Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Server & Client Modes

job-orchestrator provides both server and client functionality in a single binary. The mode is picked with a subcommand (server or client); everything else is configured through environment variables.

Dual-Mode Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Same Binary                               │
│                                                                  │
│   ┌─────────────────────┐       ┌─────────────────────┐         │
│   │    Server Mode      │       │    Client Mode      │         │
│   │                     │       │                     │         │
│   │  - Job orchestration│       │  - Job execution    │         │
│   │  - Quota management │       │  - Result packaging │         │
│   │  - Persistent DB    │       │  - Persistent DB    │         │
│   │  - User-facing API  │       │  - Server-facing API│         │
│   └─────────────────────┘       └─────────────────────┘         │
│                                                                  │
│              job-orchestrator server    job-orchestrator client  │
└─────────────────────────────────────────────────────────────────┘

Server Mode

The server is the central orchestrator that:

  • Receives job submissions from users/applications
  • Manages job queues and enforces quotas
  • Distributes jobs to available clients
  • Retrieves results and serves them to users
  • Handles cleanup of expired jobs

Starting the Server

PORT=5000 job-orchestrator server

Server Responsibilities

ComponentPurpose
REST APIHandle /upload and /download requests
Queue ManagerEnforce per-user, per-service quotas
Sender TaskDispatch jobs to clients
Getter TaskRetrieve completed results
Cleaner TaskRemove expired jobs
SQLite DBPersistent job tracking

Server API Endpoints

EndpointMethodPurpose
/uploadPOSTSubmit new job
/download/:idGETGet results or status
/download_partial/:idGETGet current job state as zip, regardless of completion
/terminate/:idPOSTCancel a running job
/healthGETHealth check
/swaggerGETAPI documentation

Client Mode

The client executes jobs on behalf of the server:

  • Receives job payloads from the server
  • Executes the run.sh script
  • Packages results for retrieval
  • Reports CPU load via /load (the server does not use it yet)

Starting the Client

PORT=9000 job-orchestrator client

Client Responsibilities

ComponentPurpose
REST APIHandle /submit and /retrieve requests
Runner TaskExecute prepared payloads
Bash ExecutorRun run.sh scripts
SQLite DBPayload tracking (same DB_PATH-backed store as the server)

Client API Endpoints

EndpointMethodPurpose
/submitPOSTReceive job from server
/retrieve/:idGETReturn completed results
/retrieve_partial/:idGETReturn current payload state as zip, regardless of completion
/kill/:idPOSTTerminate a running payload
/loadGETReport CPU usage
/healthGETHealth check

Communication Flow

User                Server                    Client
  │                   │                         │
  │──POST /upload────▶│                         │
  │◀─── Job ID ───────│                         │
  │                   │                         │
  │                   │──POST /submit──────────▶│
  │                   │◀─── Payload ID ─────────│
  │                   │                         │
  │                   │                    ┌────┴────┐
  │                   │                    │ Execute │
  │                   │                    │ run.sh  │
  │                   │                    └────┬────┘
  │                   │                         │
  │                   │──GET /retrieve/:id─────▶│
  │                   │◀─── results.zip ────────│
  │                   │                         │
  │──GET /download/:id▶│                         │
  │◀─── results.zip ──│                         │
  │                   │                         │
  │──POST /terminate/:id───▶│                    │
  │                   │──POST /kill/:id─────────▶│
  │                   │◀─── kill signal ────────│

Deployment Patterns

Single Machine (Development)

Both server and client on the same machine:

# Terminal 1
PORT=5000 DB_PATH=./server.sqlite DATA_PATH=./server-data \
  SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit \
  SERVICE_EXAMPLE_DOWNLOAD_URL=http://localhost:9000/retrieve \
  SERVICE_EXAMPLE_TERMINATE_URL=http://localhost:9000/kill \
  job-orchestrator server

# Terminal 2
PORT=9000 DB_PATH=./client.sqlite DATA_PATH=./client-data job-orchestrator client

Give each process its own DB_PATH and DATA_PATH. With the defaults, both would use ./db.sqlite and ./data in the current directory, and each cleaner would trip over the other’s directories. example/start_example.sh sets this up for you.

Distributed (Production)

Server on one machine, clients on compute nodes:

                    ┌─────────────┐
                    │   Server    │
                    │  (port 5000)│
                    └──────┬──────┘
                           │
          ┌────────────────┼────────────────┐
          │                │                │
   ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐
   │  Client 1   │  │  Client 2   │  │  Client 3   │
   │ (compute-1) │  │ (compute-2) │  │ (compute-3) │
   └─────────────┘  └─────────────┘  └─────────────┘

Multi-Service Setup

Different clients for different services:

# Server configuration
SERVICE_EXAMPLE_UPLOAD_URL: http://client-example:9000/submit
SERVICE_EXAMPLE_DOWNLOAD_URL: http://client-example:9000/retrieve
SERVICE_EXAMPLE_TERMINATE_URL: http://client-example:9000/kill
SERVICE_EXAMPLE_RUNS_PER_USER: 5
SERVICE_EXAMPLE_MAX_RUNS: 10

SERVICE_SIMULATE_UPLOAD_URL: http://client-simulate:9001/submit
SERVICE_SIMULATE_DOWNLOAD_URL: http://client-simulate:9001/retrieve
SERVICE_SIMULATE_TERMINATE_URL: http://client-simulate:9001/kill
SERVICE_SIMULATE_RUNS_PER_USER: 5
SERVICE_SIMULATE_MAX_RUNS: 10

Database Differences

Server Database (Persistent)

  • Uses SQLite file on disk
  • Survives restarts
  • Stores complete job history
  • Location configured via DB_PATH

Client Database

  • Uses the same DB_PATH-configured SQLite backend as the server: file-backed by default, survives restarts
  • Only becomes in-memory if you explicitly set DB_PATH=:memory: for the client process
  • Keeps a record of every payload it has received; cleaned payloads stay in the table with status Cleaned

When to Scale

Add More Clients When:

  • Job queue is consistently backing up
  • Execution time is the bottleneck
  • You have available compute resources

Scale Server When:

  • Upload/download becomes slow
  • Many concurrent users
  • Database queries become slow

See Also