Server & Client Modes
job-orchestrator provides both server and client functionality in a single binary. The mode is picked with a subcommand (server or client); everything else is configured through environment variables.
Dual-Mode Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Same Binary │
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Server Mode │ │ Client Mode │ │
│ │ │ │ │ │
│ │ - Job orchestration│ │ - Job execution │ │
│ │ - Quota management │ │ - Result packaging │ │
│ │ - Persistent DB │ │ - Persistent DB │ │
│ │ - User-facing API │ │ - Server-facing API│ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
│ job-orchestrator server job-orchestrator client │
└─────────────────────────────────────────────────────────────────┘
Server Mode
The server is the central orchestrator that:
- Receives job submissions from users/applications
- Manages job queues and enforces quotas
- Distributes jobs to available clients
- Retrieves results and serves them to users
- Handles cleanup of expired jobs
Starting the Server
PORT=5000 job-orchestrator server
Server Responsibilities
| Component | Purpose |
|---|---|
| REST API | Handle /upload and /download requests |
| Queue Manager | Enforce per-user, per-service quotas |
| Sender Task | Dispatch jobs to clients |
| Getter Task | Retrieve completed results |
| Cleaner Task | Remove expired jobs |
| SQLite DB | Persistent job tracking |
Server API Endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/upload | POST | Submit new job |
/download/:id | GET | Get results or status |
/download_partial/:id | GET | Get current job state as zip, regardless of completion |
/terminate/:id | POST | Cancel a running job |
/health | GET | Health check |
/swagger | GET | API documentation |
Client Mode
The client executes jobs on behalf of the server:
- Receives job payloads from the server
- Executes the
run.shscript - Packages results for retrieval
- Reports CPU load via
/load(the server does not use it yet)
Starting the Client
PORT=9000 job-orchestrator client
Client Responsibilities
| Component | Purpose |
|---|---|
| REST API | Handle /submit and /retrieve requests |
| Runner Task | Execute prepared payloads |
| Bash Executor | Run run.sh scripts |
| SQLite DB | Payload tracking (same DB_PATH-backed store as the server) |
Client API Endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/submit | POST | Receive job from server |
/retrieve/:id | GET | Return completed results |
/retrieve_partial/:id | GET | Return current payload state as zip, regardless of completion |
/kill/:id | POST | Terminate a running payload |
/load | GET | Report CPU usage |
/health | GET | Health check |
Communication Flow
User Server Client
│ │ │
│──POST /upload────▶│ │
│◀─── Job ID ───────│ │
│ │ │
│ │──POST /submit──────────▶│
│ │◀─── Payload ID ─────────│
│ │ │
│ │ ┌────┴────┐
│ │ │ Execute │
│ │ │ run.sh │
│ │ └────┬────┘
│ │ │
│ │──GET /retrieve/:id─────▶│
│ │◀─── results.zip ────────│
│ │ │
│──GET /download/:id▶│ │
│◀─── results.zip ──│ │
│ │ │
│──POST /terminate/:id───▶│ │
│ │──POST /kill/:id─────────▶│
│ │◀─── kill signal ────────│
Deployment Patterns
Single Machine (Development)
Both server and client on the same machine:
# Terminal 1
PORT=5000 DB_PATH=./server.sqlite DATA_PATH=./server-data \
SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit \
SERVICE_EXAMPLE_DOWNLOAD_URL=http://localhost:9000/retrieve \
SERVICE_EXAMPLE_TERMINATE_URL=http://localhost:9000/kill \
job-orchestrator server
# Terminal 2
PORT=9000 DB_PATH=./client.sqlite DATA_PATH=./client-data job-orchestrator client
Give each process its own DB_PATH and DATA_PATH. With the defaults, both would use ./db.sqlite and ./data in the current directory, and each cleaner would trip over the other’s directories. example/start_example.sh sets this up for you.
Distributed (Production)
Server on one machine, clients on compute nodes:
┌─────────────┐
│ Server │
│ (port 5000)│
└──────┬──────┘
│
┌────────────────┼────────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ Client 1 │ │ Client 2 │ │ Client 3 │
│ (compute-1) │ │ (compute-2) │ │ (compute-3) │
└─────────────┘ └─────────────┘ └─────────────┘
Multi-Service Setup
Different clients for different services:
# Server configuration
SERVICE_EXAMPLE_UPLOAD_URL: http://client-example:9000/submit
SERVICE_EXAMPLE_DOWNLOAD_URL: http://client-example:9000/retrieve
SERVICE_EXAMPLE_TERMINATE_URL: http://client-example:9000/kill
SERVICE_EXAMPLE_RUNS_PER_USER: 5
SERVICE_EXAMPLE_MAX_RUNS: 10
SERVICE_SIMULATE_UPLOAD_URL: http://client-simulate:9001/submit
SERVICE_SIMULATE_DOWNLOAD_URL: http://client-simulate:9001/retrieve
SERVICE_SIMULATE_TERMINATE_URL: http://client-simulate:9001/kill
SERVICE_SIMULATE_RUNS_PER_USER: 5
SERVICE_SIMULATE_MAX_RUNS: 10
Database Differences
Server Database (Persistent)
- Uses SQLite file on disk
- Survives restarts
- Stores complete job history
- Location configured via
DB_PATH
Client Database
- Uses the same
DB_PATH-configured SQLite backend as the server: file-backed by default, survives restarts - Only becomes in-memory if you explicitly set
DB_PATH=:memory:for the client process - Keeps a record of every payload it has received; cleaned payloads stay in the table with status
Cleaned
When to Scale
Add More Clients When:
- Job queue is consistently backing up
- Execution time is the bottleneck
- You have available compute resources
Scale Server When:
- Upload/download becomes slow
- Many concurrent users
- Database queries become slow