Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Client Configuration

The client is configured through environment variables and runs as a job executor.

Environment Variables

VariableDefaultDescription
PORT5000HTTP port the client listens on
DB_PATH./db.sqlitePath to the client’s SQLite payload database
DATA_PATH./dataWorking directory where job files are stored
MAX_AGE864000Seconds before the client’s cleaner deletes a payload directory (10 days)

The client shares the exact same config loader as the server, so these variables have the same names and same defaults in both roles. SERVICE_* variables are ignored by the client. There is no client-specific default for PORT: if you run a server and a client on the same host, pick a different port explicitly (9000 is a common convention, shown below) to avoid a clash.

Example Configuration

Basic Setup

export PORT=9000
export DB_PATH=/var/lib/job-orchestrator-client/db.sqlite
export DATA_PATH=/var/lib/job-orchestrator-client/data
job-orchestrator client

Docker Compose

services:
  client:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
      DATA_PATH: /opt/data
      DB_PATH: /opt/data/db.sqlite
    volumes:
      - client-data:/opt/data

volumes:
  client-data:

Note there’s no ports: mapping here — in the shipped docker-compose.yml the client is only reachable over the internal Docker network from the server, not from the host. See Client Security below.

How the Client Works

Database

The client uses the same DB_PATH-backed SQLite database mechanism as the server — by default it’s a real file on disk, not in-memory, so payload records survive a client restart.

If you actually want an ephemeral, in-memory database (e.g. for a disposable/test client), set DB_PATH=:memory: explicitly. That’s an opt-in choice, not the default behavior. Be aware that payload IDs then restart at 1 after every restart, while the server still holds the old IDs as dest_id, and old payload directories with the same numbers may still be on disk. Only use it for throwaway clients.

The client’s database keeps a record of every payload it received (cleaned ones stay with status Cleaned). The server is still the source of truth for job history.

Working Directory

The client stores job files under DATA_PATH. Each payload gets a subdirectory named after its numeric payload ID:

/opt/data/
├── 1/
│   ├── run.sh
│   ├── input.pdb
│   ├── output.txt         (created by run.sh)
│   ├── .orchestrator.exit (written by the trap)
│   └── output.zip         (created on the first completed /retrieve)
├── 2/
│   └── ...

Uploaded files are stored by base name only, so any directory structure is flattened.

Gotcha: unlike the server, the client does not create DATA_PATH at startup. It is created when the first payload arrives. Until then the client’s cleaner logs could not read directory every 60 seconds, which is harmless.

Execution Environment

When the Runner task executes a job:

  1. Validates run.sh (size, UTF-8, required trap, blocked patterns)
  2. Runs bash run.sh with the payload directory as working directory
  3. Stores the process ID; the exit code is read later from .orchestrator.exit
  4. All files in the directory are included in results

The Updater task then watches Running payloads:

  • If a payload has the killed flag set, it marks it Killed
  • If .orchestrator.exit exists, it marks it Completed (exit code 0) or Failed
  • If the process is still alive, it leaves it Running
  • If the process is gone and never wrote .orchestrator.exit, it marks it Lost and logs a warning (see Lost Processes)

Job Termination

The client supports on-demand job termination via the /kill/:id endpoint:

  • Receives termination requests from the server
  • Sends SIGTERM to the running process (only the bash process; child processes it started may keep running unless the script forwards the signal)
  • Sets the payload’s killed flag; the Updater changes the status to Killed if the payload is Running
  • The server’s /terminate/:id endpoint triggers this via HTTP

Resource Reporting

The client exposes a /load endpoint that reports CPU usage:

curl http://localhost:9000/load

Returns a float representing CPU usage percentage. This can be used by the server for load-aware scheduling (planned feature).

Multiple Clients

You can run multiple clients for:

  • Scaling: Handle more concurrent jobs
  • Isolation: Different services on different machines
  • Redundancy: Failover capability

Same Service, Multiple Clients

Not supported. Each service maps to exactly one client URL, and the server only accepts one value per variable. Do not put a load balancer in front of several clients either: payload IDs are local to each client, so a /retrieve/{id} sent to a different client would hit the wrong payload or a 404.

To spread load today, split the work into several services, each with its own client.

Different Services

Run specialized clients for different workloads:

# Client for general jobs
PORT=9000 job-orchestrator client

# Client for heavy computation (different machine)
PORT=9001 job-orchestrator client

Server configuration:

SERVICE_LIGHT_UPLOAD_URL=http://client-1:9000/submit
SERVICE_LIGHT_DOWNLOAD_URL=http://client-1:9000/retrieve
SERVICE_LIGHT_TERMINATE_URL=http://client-1:9000/kill
SERVICE_HEAVY_UPLOAD_URL=http://client-2:9001/submit
SERVICE_HEAVY_DOWNLOAD_URL=http://client-2:9001/retrieve
SERVICE_HEAVY_TERMINATE_URL=http://client-2:9001/kill

Client Security

Network Access

The client should only be accessible by the orchestrator server:

  • Use internal networks / VPCs
  • Firewall rules to restrict access
  • Never expose client ports to the internet

Execution Sandbox

The client executes arbitrary run.sh scripts. Consider:

  • Running in containers with resource limits
  • Using separate user accounts with minimal permissions
  • Mounting only necessary directories
  • Network isolation if jobs don’t need internet

Docker Resource Limits

services:
  client:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 8G
        reservations:
          cpus: '1'
          memory: 1G

Monitoring

The examples below assume the client is reachable directly (e.g. running it outside Docker, or with a ports: mapping added). With the shipped docker-compose.yml, the client has no host-exposed port — reach it with docker compose exec client curl localhost:9000/health instead, or from the server container over the internal network.

Health Check

curl http://localhost:9000/health

Load Check

curl http://localhost:9000/load

Container Logs

docker logs -f client-container

Troubleshooting

Client Not Receiving Jobs

  1. Verify server can reach client URL
  2. Check firewall rules
  3. Verify service configuration on server

Jobs Stuck in Prepared

  1. Check the client logs for errors
  2. Check for permission issues in the working directory

High Memory Usage

The client holds each incoming upload fully in memory while handling /submit, and builds /retrieve_partial archives in memory. Large inputs or partial downloads of large directories cause memory spikes. The database is file-backed and is not the cause. If memory is high:

  1. Check the size of the payloads being sent and of the partial downloads being requested
  2. Check for stuck payloads
  3. Restarting the client frees memory, but jobs running at that moment become lost (see Lost Processes)

See Also