Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

job-orchestrator is an asynchronous job orchestration system for managing and distributing computational workloads across heterogeneous computing resources with intelligent quota-based load balancing.

It serves as a middleware layer that connects web applications to computing resources: users submit files and a run.sh script via a REST API; the server queues and routes jobs to client nodes that execute them; results are returned as a ZIP archive.

It is domain-agnostic: any workload that can be expressed as a run.sh script plus input files can be orchestrated, whether that is a scientific simulation, a data-processing pipeline, or a media conversion task.

The project was originally developed as the job backend for a production web portal that exposes research software as web services.

Key Features

  • Asynchronous Job Management: Built with Rust and Tokio for high-performance async operations
  • Quota-Based Load Balancing: Per-user, per-service quotas prevent resource exhaustion
  • Dual-Mode Architecture: Runs as server (job orchestration) or client (job execution)
  • RESTful API: Simple HTTP interface for job submission, retrieval, and cancellation
  • Automatic Cleanup: Configurable retention policies for completed jobs
  • Job Termination: Cancel running jobs via API endpoints

Use Cases

job-orchestrator is designed for scenarios requiring:

  • Scientific Computing Workflows: Distribute simulations and analyses across clusters
  • Multi-Tenant Systems: Fair resource allocation with per-user quotas
  • Heterogeneous Computing: Route jobs to appropriate backends (local, HPC, cloud)
  • Web Platforms with Heavy Backends: Decouple the frontend from compute infrastructure
  • Batch Processing: Handle high-throughput job submissions with automatic queuing

Get Started

Project Status

Current State: Production-ready with server/client architecture

Planned Features:

  • DIRAC Interware integration
  • SLURM direct integration
  • Enhanced monitoring and metrics
  • Job priority queues
  • Advanced scheduling policies

Getting Help

License

MIT License - see LICENSE for details.


Parts of this documentation were automatically generated with the assistance of AI tools and human reviewed for accuracy.

Installation

There are several ways to install job-orchestrator depending on your needs.

From crates.io

The easiest way to install job-orchestrator is via Cargo:

cargo install job-orchestrator

From Source

Clone the repository and build:

git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
cargo build --release

The binary will be available at target/release/job-orchestrator.

Using Docker

Pull the pre-built image:

docker pull codeberg.org/rvhonorato/job-orchestrator:latest

Or build locally:

docker build -t job-orchestrator .
# example client (job-orchestrator + gdock), built on top of the local image
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/

Prerequisites

For Building from Source

  • Rust: 1.88 or later (edition 2024, uses let chains)
  • C compiler: SQLite is compiled from source and linked statically (via sqlx’s bundled SQLite), so no system SQLite library is needed

On Debian/Ubuntu:

apt-get install build-essential

For Running

  • Linux or macOS: client mode runs jobs with bash and manages them with the kill command, so both must be on PATH. Windows is not supported for client mode
  • Filesystem access: Write permissions for database and job storage directories

Verifying Installation

After installation, verify it works:

job-orchestrator --help

You should see the server and client subcommands listed. There is no --version flag.

Next Steps

Quick Start

The fastest way to get job-orchestrator running is with Docker Compose.

Running with Docker Compose

git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
docker compose up --build

This starts:

  • Orchestrator server on port 5000 (exposed to host)
  • Example client on an internal network (not exposed to host)

Verify It’s Running

Check the server is responding:

curl http://localhost:5000/health

You should receive a health status response.

Access the API Documentation

Open your browser and navigate to:

http://localhost:5000/swagger

This provides interactive API documentation where you can explore and test all endpoints.

What’s Next?

Now that you have job-orchestrator running, proceed to Your First Job to learn how to submit and retrieve jobs.

Stopping the Services

To stop the services:

docker compose down

To stop and remove all data (volumes):

docker compose down -v

Your First Job

This guide walks you through submitting and retrieving your first job.

Prerequisites

Make sure you have job-orchestrator running. See Quick Start if you haven’t set it up yet.

Understanding Jobs

A job in job-orchestrator consists of:

  1. Files: One or more files to be processed
  2. A run.sh script: The entry point that gets executed
  3. User ID: Identifies who submitted the job (for quota tracking)
  4. Service: Which service/backend should process this job

Creating a Simple Job

Create a simple run.sh script:

cat > run.sh << 'EOF'
#!/bin/bash
# Required: Capture exit code for the orchestrator client
trap 'echo $? > .orchestrator.exit' EXIT

echo "Hello from job-orchestrator!" > output.txt
echo "Processing complete at $(date)" >> output.txt
EOF
chmod +x run.sh

Submitting the Job

Submit the job using curl:

curl -X POST http://localhost:5000/upload \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example" | jq

You’ll receive a response like:

{
  "id": 1,
  "status": "Queued",
  "message": "Job successfully uploaded"
}

Note the id field - you’ll need this to check status and download results.

Checking Job Status

Check the job status via GET request:

curl http://localhost:5000/download/1

If the job is not yet completed, you’ll get a JSON response:

{
  "id": 1,
  "status": "Submitted",
  "message": ""
}

The status field will be one of: Queued, Processing, Submitted, Prepared, Running, Completed, Failed, Invalid, Cleaned, Unknown, Locked, Killed, or Lost. See Job States for descriptions of each.

Downloading Results

Once the status is Completed, the same endpoint returns the ZIP file:

curl -o results.zip http://localhost:5000/download/1

Extract and view:

unzip results.zip
cat output.txt

You should see:

Hello from job-orchestrator!
Processing complete at <timestamp>

The archive contains the whole job directory: your input files, everything run.sh wrote, and the .orchestrator.exit file.

Downloading Partial Results

/download/{id} only returns a ZIP for Completed jobs. For any other status, including Failed, it returns the JSON status. To get the files of a failed, stuck, or still-running job, use /download_partial/{id}:

# Get current state of job even if incomplete
curl -o partial_results.zip http://localhost:5000/download_partial/1

Canceling a Job

If you need to cancel a running job, use the terminate endpoint:

# Cancel job with ID 1
curl -X POST http://localhost:5000/terminate/1

Response on success:

{
  "id": 1,
  "status": "Unknown",
  "message": "job terminated"
}

The status field in the terminate response is not meaningful. Use GET /download/{id} to confirm the job’s actual status after termination.

Notes:

  • The server sends a termination request to the client, which sends SIGTERM to the process
  • Terminated jobs will have status Killed
  • Queued jobs cannot be cancelled: they have not reached a client yet, so the request fails with 500, the job stays Queued, and it will still run later
  • Terminating a job that already finished still “succeeds” and changes its status to Killed. After that, /download/{id} returns JSON instead of the ZIP, so the results are only reachable through /download_partial/{id} (until the client cleans them up)

Security

Script Execution

The client executes user-submitted run.sh scripts with full privileges. While a script validator rejects obviously dangerous patterns (destructive commands, network exfiltration tools, reverse shells, privilege escalation), it is NOT a sandbox and can be bypassed by determined actors.

Important: Always run the client inside a container with:

  • Resource limits (CPU, memory, PIDs)
  • Read-only root filesystem
  • Drop all capabilities (--cap-drop=ALL)
  • Internal network only (no external exposure)
  • Non-root user

See Production Deployment for details.

Path Traversal Protection

Uploaded filenames are reduced to their base name, so ../../etc/passwd is stored as passwd inside the job directory. When results are zipped, any entry that resolves outside the job directory (such as a symlink to /etc) makes the archive fail instead of being included. See Path Traversal Protection.

Important Notes

The run.sh Script

  • Must be named exactly run.sh
  • Is run as bash run.sh inside the job directory, so it does not need the executable bit or a shebang
  • Exit code 0 indicates success
  • Non-zero exit code indicates failure
  • All output files in the working directory are included in results

Important: The script MUST include the following trap at the beginning. The client checks for it before running the script and marks the job Invalid if it is missing:

#!/bin/bash
trap 'echo $? > .orchestrator.exit' EXIT
# ... rest of your script

This ensures that when your run.sh exits, it writes the exit code to .orchestrator.exit, which the client’s Updater task reads to determine if the job completed successfully or failed.

File Size Limits

The server accepts request bodies up to 400MB by default (all files in one upload combined). Change it with the MAX_UPLOAD_SIZE environment variable, in bytes. Larger uploads are rejected with 413.

User ID Parameter

Jobs can be submitted with a user_id for quota management:

curl -X POST http://localhost:5000/upload \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example"

user_id is required and must be an integer; the server returns 422 if it is missing or not a number, if service is unknown, or if no run.sh was uploaded. service must match a configured service name exactly, in lowercase (example, not EXAMPLE).

Job Retention

Job directories are deleted once they are older than MAX_AGE (default: 10 days), and the job is marked Cleaned. Age is measured from the directory’s last modification time, not from when the job finished, and the rule applies to jobs in any status. Make sure to download your results before they expire.

Next Steps

Partial Downloads and Debugging

This guide demonstrates how to use the /download_partial and /retrieve_partial endpoints to debug stuck or incomplete jobs.

Use Cases

Use partial download endpoints when:

  • A job appears stuck in Submitted or Running status
  • A job ended as Failed and you want its output (/download only returns ZIPs for Completed jobs)
  • You need to inspect the current working directory
  • Debugging why a job isn’t progressing
  • Verifying intermediate outputs of long-running jobs

Server-Side: /download_partial/

Example: Checking Incomplete Job

# Download current state of job ID 1
curl -o partial_results.zip http://localhost:5000/download_partial/1

# Extract and inspect contents
unzip -l partial_results.zip
unzip partial_results.zip

# View specific files
cat intermediate_output.txt

Client-Side: /retrieve_partial/

This endpoint is called internally by the server’s /download_partial endpoint. It provides the same functionality at the client level.

The server does not have a separate setting for this URL. It takes the service’s DOWNLOAD_URL and replaces its last path segment with retrieve_partial (http://client:9000/retrieve becomes http://client:9000/retrieve_partial). If you put a proxy in front of the client, make sure that path is routed too.

In the shipped Docker Compose setup the client has no ports: mapping — it’s only reachable from the server over the internal network (e.g. http://client:9000), not from the host. The example below assumes you’re running the client directly (outside Docker Compose) or calling it from another container on the same network.

Example (Client Developer)

# Get partial results directly from client
curl -o partial_results.zip http://localhost:9000/retrieve_partial/1

Comparison: Full vs Partial Download

EndpointReturnsUse Case
/download/{id}Final results onlyNormal completion
/download_partial/{id}Current state alwaysDebugging, monitoring

Combined Example Script

#!/bin/bash

SERVER_URL="http://localhost:5000"

# Submit job and parse the returned ID
echo "Submitting job..."
RESPONSE=$(curl -s -X POST "${SERVER_URL}/upload" \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example")
echo "$RESPONSE"
JOB_ID=$(echo "$RESPONSE" | jq -r '.id')

if [ -z "$JOB_ID" ] || [ "$JOB_ID" = "null" ]; then
  echo "Failed to get job ID from response"
  exit 1
fi

echo "Job ID: $JOB_ID"

# Poll until complete, saving partial snapshots for inspection
echo "Monitoring job..."
for i in $(seq 1 10); do
  sleep 10
  CT=$(curl -s -o response.tmp -w '%{content_type}' "${SERVER_URL}/download/${JOB_ID}")
  if [[ "$CT" == application/zip* ]]; then
    mv response.tmp final_results.zip
    echo "Job completed, results saved to final_results.zip"
    exit 0
  fi
  STATUS=$(jq -r '.status' response.tmp)
  case "$STATUS" in
    Failed|Invalid|Killed|Lost|Cleaned)
      echo "Job ended with status $STATUS"
      curl -s -o final_partial.zip "${SERVER_URL}/download_partial/${JOB_ID}"
      exit 1
      ;;
  esac
  echo "Check $i: status=$STATUS, saving partial snapshot..."
  curl -s -o "partial_${i}.zip" "${SERVER_URL}/download_partial/${JOB_ID}"
  unzip -l "partial_${i}.zip" 2>/dev/null | head -20
done

echo "Job did not complete within the polling window"
exit 1

Limitations

  • Queued jobs have not reached a client yet, so /download_partial returns 404 (“Job payload not found on client”)
  • Until run.sh writes something, the ZIP only contains the input files
  • After the client’s cleaner has removed the payload directory, the request fails
  • Partial downloads of running jobs may be incomplete or inconsistent
  • Use partial downloads for debugging, not for production result retrieval

Security Notes

  • Building the ZIP fails if any entry in the payload directory resolves outside it (for example, a symlink created by run.sh that points to /etc), so such files cannot be exfiltrated this way
  • See Path Traversal Protection for details

Architecture Overview

job-orchestrator uses a distributed architecture with a central server coordinating job execution across multiple client nodes.

High-Level Architecture

flowchart TB
 subgraph Tasks["Background Tasks"]
        Sender["Sender<br>500ms"]
        Getter["Getter<br>500ms"]
        Cleaner["Cleaner<br>60s"]
  end
 subgraph Server["Orchestrator Server"]
        API["REST API<br>upload/download/download_partial/terminate"]
        DB[("SQLite<br>Persistent")]
        FS[/"Filesystem<br>Job Storage"/]
        Tasks
        Queue["Queue Manager<br>Quota Enforcement"]
  end
 subgraph Client["Client Service"]
        ClientAPI["REST API<br>submit/retrieve/retrieve_partial/kill/load"]
        ClientDB[("SQLite<br>Persistent")]
        ClientFS[/"Working Dir"/]
        Runner["Runner Task<br>500ms"]
        Updater["Updater Task<br>500ms"]
        ClientCleaner["Cleaner Task<br>60s"]
        Executor["Bash Executor<br>run.sh"]
  end
    User(["User/Web App"]) -- POST /upload --> API
    User -- GET /download/:id --> API
    User -- GET /download_partial/:id --> API
    API -- GET /retrieve_partial/:id --> ClientAPI
    API --> DB & FS
    DB --> Queue
    Queue --> Sender
    Sender -- POST /submit --> ClientAPI
    Getter -- GET /retrieve/:id --> ClientAPI
    Getter --> FS
    Cleaner --> DB & FS
    ClientAPI --> ClientDB
    ClientDB --> Runner
    Runner --> Executor
    Executor --> ClientFS
    ClientDB --> Updater
    Updater --> ClientDB
    ClientDB --> ClientCleaner
    ClientCleaner --> ClientFS

Components

Orchestrator Server

The central server manages:

  • REST API: Handles job uploads and result downloads from users
  • SQLite Database: Persistent storage for job metadata and status
  • Filesystem Storage: Stores uploaded files and downloaded results
  • Queue Manager: Enforces per-user quotas and manages job distribution
  • Background Tasks: Automated processes for job distribution, result retrieval, and cleanup

Client Service

Each client node handles:

  • REST API: Receives jobs from server, returns results
  • SQLite Database: Payload tracking. Configured with the same DB_PATH variable as the server (file-backed by default); give each process its own file
  • Working Directory: Temporary storage for job execution
  • Runner Task: Picks up new payloads and starts them
  • Updater Task: Watches running payloads for their exit code, a kill request, or a lost process
  • Bash Executor: Runs the run.sh script for each job

Background Tasks

Server Tasks

TaskIntervalPurpose
Sender500msPicks up queued jobs, enforces quotas, dispatches to clients
Getter500msSyncs job status from clients and downloads results when complete
Cleaner60sDeletes job directories older than MAX_AGE and marks them Cleaned (records are kept)

Client Tasks

TaskIntervalPurpose
Runner500msValidates and starts prepared payloads
Updater500msMonitors running processes, handles exit codes, killed payloads and lost processes
Cleaner60sDeletes payload directories older than MAX_AGE, whatever their status

Data Flow

  1. User submits files via POST /upload
  2. Server stores files and creates job record (status: Queued)
  3. Sender task picks up job, checks quotas, sends it to the client configured for its service
  4. Client receives job, stores as payload (status: Prepared)
  5. Runner task starts run.sh; the Updater sets Completed or Failed from its exit code
  6. Getter task retrieves results, stores locally
  7. User downloads results via GET /download/:id
  8. Cleaner task removes job after retention period

Termination Flow

  1. User sends termination request via POST /terminate/:id
  2. Server locks job (status: Locked) and forwards to client
  3. Client receives request via POST /kill/:id, sends SIGTERM to the process and sets the payload’s killed flag
  4. Server sets the job to Killed as soon as the client answers with success (or restores the previous status if it fails)
  5. Server returns termination confirmation to user
  6. Client Updater later marks the payload Killed

Job Lifecycle

Understanding the job lifecycle is essential for working with job-orchestrator effectively.

Lifecycle Sequence

sequenceDiagram
    participant User
    participant Server
    participant Client
    participant Executor

    User->>Server: POST /upload (files, user_id, service)
    Server->>Server: Store job (status: Queued)
    Server-->>User: Job ID

    Note over Server: Sender task (500ms interval)
    Server->>Server: Update status: Processing
    Server->>Client: POST /submit (job files)
    Client->>Client: Store payload (status: Prepared)
    Client-->>Server: Payload ID
    Server->>Server: Update status: Submitted

    Note over Client: Runner task (500ms interval)
    Client->>Client: Update status: Running
    Client->>Executor: Execute run.sh (spawns process)

    Note over Client: Updater task (500ms interval)
    Executor-->>Client: .orchestrator.exit written
    Client->>Client: Update status: Completed

    Note over Server: Getter task (500ms interval)
    Server->>Client: GET /retrieve/:id
    Client-->>Server: JSON status (copied to job) or, once Completed, ZIP results
    Server->>Server: Store output.zip, status: Completed

    User->>Server: GET /download/:id
    Server-->>User: results.zip

    Note over Server: Cleaner task (60s interval)
    Server->>Server: Delete job dirs older than MAX_AGE, status: Cleaned

    Note over User,Server: Job Termination Flow
    User->>Server: POST /terminate/:id
    Server->>Server: Update status: Locked
    Server->>Client: POST /kill/:id
    Client->>Executor: Send SIGTERM
    Client->>Client: Set killed flag
    Client-->>Server: 200 OK
    Server->>Server: Update status: Killed
    Server-->>User: Termination confirmed
    Note over Client: Updater marks the payload Killed on its next pass

Job States

stateDiagram-v2
    [*] --> Queued: submitted
    Queued --> Processing: sender picks up
    Processing --> Submitted: sent to client
    Processing --> Failed: dispatch error

    Submitted --> Prepared: client received job
    Prepared --> Running: execution started
    Running --> Completed: exit 0
    Running --> Failed: non-zero exit
    Running --> Invalid: bad script
    Running --> Lost: process gone, no exit file
    Submitted --> Killed: terminated early
    Prepared --> Killed: terminated early
    Running --> Killed: terminated
    Completed --> Killed: terminated after finishing

    Completed --> Cleaned: MAX_AGE
    Failed --> Cleaned: MAX_AGE
    Invalid --> Cleaned: MAX_AGE
    Killed --> Cleaned: MAX_AGE
    Lost --> Cleaned: MAX_AGE
    Cleaned --> [*]

The server copies the client’s payload status into the job on every Getter pass, which is why client-side states (Prepared, Running, Invalid) also show up on the server.

Locked and Unknown are not shown above; see the table below. Cleaned can happen from any state, including Queued and Running, because the cleaner only looks at the age of the job directory.

State Descriptions

StateDescription
QueuedJob received and waiting for dispatch
ProcessingServer is sending job to a client
SubmittedJob successfully sent to client, awaiting execution
RunningClient is actively executing the job
CompletedJob finished successfully, results available
PreparedClient stored the payload and is about to run it (copied from the client)
FailedTerminal. run.sh exited non-zero, the client could not start it, or dispatch to the client failed
InvalidTerminal. Client rejected the script (run.sh missing, unsafe pattern, missing trap, too large, not UTF-8)
UnknownFallback when a status string cannot be parsed, and the placeholder in error responses. Not a retry state
LockedSet while a termination request is in flight. Restored or replaced by Killed when the request returns
KilledTerminal. Job was terminated via POST /terminate/{id}
LostTerminal. The process disappeared without writing .orchestrator.exit (client restart or crash, or bash itself killed with SIGKILL). See Lost Processes
CleanedTerminal. Job directory deleted after MAX_AGE. The database record is kept

Lifecycle Stages

1. Submission

User uploads files via POST /upload with:

  • One or more files (including run.sh)
  • user_id - identifies the submitting user
  • service - which backend should process this job

The server:

  1. Validates the request
  2. Creates a unique directory for the job
  3. Stores all uploaded files (by base name only, so subdirectories are flattened)
  4. Creates a database record with status Queued
  5. Returns the job ID to the user

2. Queuing & Quota Check

The Sender background task (runs every 500ms) loads its next batch of jobs via a round-robin fair-share scheduler:

  1. Groups all Queued jobs by service, then by user_id within each service
  2. For each service, computes available service-wide slots: MAX_RUNS (default 10) minus jobs already Processing/Submitted/Running for that service
  3. For each user in that service, computes available per-user slots: RUNS_PER_USER (default 5) minus that user’s own in-flight job count
  4. Round-robins across users with available slots — takes one queued job per user per pass — until the service-wide slot budget is exhausted or no user has both a queued job and a free slot
  5. Jobs picked this way are marked Processing; everything else remains Queued for the next cycle

Quota is enforced at two levels — per-user (RUNS_PER_USER) and per-service (MAX_RUNS) — and the round-robin means no single user can starve others of a service’s shared capacity.

3. Distribution

For jobs in Processing status:

  1. Server streams every file in the job directory as one multipart request
  2. Sends it to the service’s UPLOAD_URL (the client’s POST /submit). Each service has exactly one client URL; there is no load-based client selection
  3. On success: updates status to Submitted, stores client’s payload ID as dest_id
  4. On failure (client unreachable or non-2xx response): updates status to Failed. There is no retry

4. Execution

On the client side:

  1. Runner task (every 500ms) finds payloads in Prepared status and marks them Running
  2. Validates run.sh (see Script Validation). A rejected script makes the payload Invalid
  3. Spawns bash run.sh in the payload directory and stores its PID. If the process cannot be started, the payload becomes Failed
  4. Updater task (every 500ms) checks Running payloads. When .orchestrator.exit appears it sets Completed (exit code 0) or Failed (anything else). If the killed flag is set it sets Killed. If the process is gone and never wrote the exit file it sets Lost

5. Retrieval

The Getter background task (runs every 500ms):

  1. Finds jobs in Submitted, Prepared, or Running status (at most 10 requests in flight)
  2. Requests the payload from the client via GET /retrieve/:id
  3. If the client answers with JSON, copies the payload status into the job (Running, Failed, Invalid, …)
  4. If the client answers with a ZIP, saves it as output.zip in the job directory and sets Completed

Once a job reaches a status outside that list, the Getter stops polling it.

6. Download

User can now:

  1. Check status via GET /download/:id (returns JSON with job state when not completed)
  2. Download results via GET /download/:id (returns ZIP when completed)
  3. Results are returned as a ZIP archive

7. Cleanup

The Cleaner background task (runs every 60s):

  1. Finds job directories whose last modification time is older than MAX_AGE, whatever the job status
  2. Deletes the directory
  3. Updates the job status to Cleaned. The database record is never deleted

The client runs the same cleaner over its own payload directories.

Error Handling

Client Unreachable

If the server cannot reach a client during distribution:

  • Job status changes to Failed
  • Job will not be retried automatically
  • User can resubmit if needed

Execution Failure

If run.sh exits with non-zero code:

  • Payload status changes to Failed
  • The server copies the Failed status but does not fetch any output
  • GET /download/{id} keeps returning JSON; use GET /download_partial/{id} to get the files

Retrieval Failure

If the server cannot retrieve the payload (client down, 404, network error):

  • The job status is left unchanged and the error is logged
  • The Getter retries every 500ms, indefinitely. There is no timeout

Lost Processes

If the client is restarted or crashes while a job runs (for example a container restart), the process is gone but no .orchestrator.exit file was written:

  • The client’s Updater sees that the PID is gone and that there is no exit file, marks the payload Lost and logs a warning (payload <id> lost: pid <pid> is gone and no exit file at ...)
  • The server copies Lost into the job on its next Getter pass, which frees its RUNS_PER_USER and MAX_RUNS slot
  • Lost is terminal: resubmit the job. /download_partial/{id} still returns whatever the job wrote before it died

The trap writes .orchestrator.exit before the process ends, so a job that exits normally always leaves the file behind. The Updater checks for the file again after seeing the PID gone, so a job that finishes between two checks is still marked Completed or Failed, not Lost.

A server older than the client cannot parse Lost: it logs a retrieval error and keeps the job Running. Upgrade the server before, or together with, the clients.

Timing Considerations

EventTypical Duration
Upload to QueuedImmediate
Queued to ProcessingUp to 500ms (+ quota wait)
Processing to SubmittedDepends on file size and network
Submitted to CompletedDepends on job execution time
Completed to CleanedMAX_AGE after the job directory was last modified (default: 10 days)

Server & Client Modes

job-orchestrator provides both server and client functionality in a single binary. The mode is picked with a subcommand (server or client); everything else is configured through environment variables.

Dual-Mode Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Same Binary                               │
│                                                                  │
│   ┌─────────────────────┐       ┌─────────────────────┐         │
│   │    Server Mode      │       │    Client Mode      │         │
│   │                     │       │                     │         │
│   │  - Job orchestration│       │  - Job execution    │         │
│   │  - Quota management │       │  - Result packaging │         │
│   │  - Persistent DB    │       │  - Persistent DB    │         │
│   │  - User-facing API  │       │  - Server-facing API│         │
│   └─────────────────────┘       └─────────────────────┘         │
│                                                                  │
│              job-orchestrator server    job-orchestrator client  │
└─────────────────────────────────────────────────────────────────┘

Server Mode

The server is the central orchestrator that:

  • Receives job submissions from users/applications
  • Manages job queues and enforces quotas
  • Distributes jobs to available clients
  • Retrieves results and serves them to users
  • Handles cleanup of expired jobs

Starting the Server

PORT=5000 job-orchestrator server

Server Responsibilities

ComponentPurpose
REST APIHandle /upload and /download requests
Queue ManagerEnforce per-user, per-service quotas
Sender TaskDispatch jobs to clients
Getter TaskRetrieve completed results
Cleaner TaskRemove expired jobs
SQLite DBPersistent job tracking

Server API Endpoints

EndpointMethodPurpose
/uploadPOSTSubmit new job
/download/:idGETGet results or status
/download_partial/:idGETGet current job state as zip, regardless of completion
/terminate/:idPOSTCancel a running job
/healthGETHealth check
/swaggerGETAPI documentation

Client Mode

The client executes jobs on behalf of the server:

  • Receives job payloads from the server
  • Executes the run.sh script
  • Packages results for retrieval
  • Reports CPU load via /load (the server does not use it yet)

Starting the Client

PORT=9000 job-orchestrator client

Client Responsibilities

ComponentPurpose
REST APIHandle /submit and /retrieve requests
Runner TaskExecute prepared payloads
Bash ExecutorRun run.sh scripts
SQLite DBPayload tracking (same DB_PATH-backed store as the server)

Client API Endpoints

EndpointMethodPurpose
/submitPOSTReceive job from server
/retrieve/:idGETReturn completed results
/retrieve_partial/:idGETReturn current payload state as zip, regardless of completion
/kill/:idPOSTTerminate a running payload
/loadGETReport CPU usage
/healthGETHealth check

Communication Flow

User                Server                    Client
  │                   │                         │
  │──POST /upload────▶│                         │
  │◀─── Job ID ───────│                         │
  │                   │                         │
  │                   │──POST /submit──────────▶│
  │                   │◀─── Payload ID ─────────│
  │                   │                         │
  │                   │                    ┌────┴────┐
  │                   │                    │ Execute │
  │                   │                    │ run.sh  │
  │                   │                    └────┬────┘
  │                   │                         │
  │                   │──GET /retrieve/:id─────▶│
  │                   │◀─── results.zip ────────│
  │                   │                         │
  │──GET /download/:id▶│                         │
  │◀─── results.zip ──│                         │
  │                   │                         │
  │──POST /terminate/:id───▶│                    │
  │                   │──POST /kill/:id─────────▶│
  │                   │◀─── kill signal ────────│

Deployment Patterns

Single Machine (Development)

Both server and client on the same machine:

# Terminal 1
PORT=5000 DB_PATH=./server.sqlite DATA_PATH=./server-data \
  SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit \
  SERVICE_EXAMPLE_DOWNLOAD_URL=http://localhost:9000/retrieve \
  SERVICE_EXAMPLE_TERMINATE_URL=http://localhost:9000/kill \
  job-orchestrator server

# Terminal 2
PORT=9000 DB_PATH=./client.sqlite DATA_PATH=./client-data job-orchestrator client

Give each process its own DB_PATH and DATA_PATH. With the defaults, both would use ./db.sqlite and ./data in the current directory, and each cleaner would trip over the other’s directories. example/start_example.sh sets this up for you.

Distributed (Production)

Server on one machine, clients on compute nodes:

                    ┌─────────────┐
                    │   Server    │
                    │  (port 5000)│
                    └──────┬──────┘
                           │
          ┌────────────────┼────────────────┐
          │                │                │
   ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐
   │  Client 1   │  │  Client 2   │  │  Client 3   │
   │ (compute-1) │  │ (compute-2) │  │ (compute-3) │
   └─────────────┘  └─────────────┘  └─────────────┘

Multi-Service Setup

Different clients for different services:

# Server configuration
SERVICE_EXAMPLE_UPLOAD_URL: http://client-example:9000/submit
SERVICE_EXAMPLE_DOWNLOAD_URL: http://client-example:9000/retrieve
SERVICE_EXAMPLE_TERMINATE_URL: http://client-example:9000/kill
SERVICE_EXAMPLE_RUNS_PER_USER: 5
SERVICE_EXAMPLE_MAX_RUNS: 10

SERVICE_SIMULATE_UPLOAD_URL: http://client-simulate:9001/submit
SERVICE_SIMULATE_DOWNLOAD_URL: http://client-simulate:9001/retrieve
SERVICE_SIMULATE_TERMINATE_URL: http://client-simulate:9001/kill
SERVICE_SIMULATE_RUNS_PER_USER: 5
SERVICE_SIMULATE_MAX_RUNS: 10

Database Differences

Server Database (Persistent)

  • Uses SQLite file on disk
  • Survives restarts
  • Stores complete job history
  • Location configured via DB_PATH

Client Database

  • Uses the same DB_PATH-configured SQLite backend as the server: file-backed by default, survives restarts
  • Only becomes in-memory if you explicitly set DB_PATH=:memory: for the client process
  • Keeps a record of every payload it has received; cleaned payloads stay in the table with status Cleaned

When to Scale

Add More Clients When:

  • Job queue is consistently backing up
  • Execution time is the bottleneck
  • You have available compute resources

Scale Server When:

  • Upload/download becomes slow
  • Many concurrent users
  • Database queries become slow

See Also

Server Configuration

The orchestrator server is configured primarily through environment variables.

Environment Variables

Core Settings

VariableDefaultDescription
PORT5000HTTP port the server listens on
DB_PATH./db.sqlitePath to SQLite database file
DATA_PATH./dataDirectory for job file storage
MAX_AGE864000Job retention time in seconds (default: 10 days)
MAX_UPLOAD_SIZE419430400Maximum /upload request body in bytes (default: 400MB). Larger uploads get 413

Service Configuration

For each service you want to support, configure these variables:

Variable PatternDescription
SERVICE_<NAME>_UPLOAD_URLClient endpoint for submitting jobs
SERVICE_<NAME>_DOWNLOAD_URLClient endpoint for retrieving results
SERVICE_<NAME>_TERMINATE_URLClient endpoint for terminating jobs
SERVICE_<NAME>_RUNS_PER_USERMaximum concurrent jobs per user (default: 5)
SERVICE_<NAME>_MAX_RUNSMaximum concurrent jobs for the service across all users (default: 10)

Note: <NAME> is converted to lowercase, and that lowercase name is what clients must send in the service field (which is case-sensitive). SERVICE_EXAMPLE_* defines the service example; uploading with service=EXAMPLE returns 422 Invalid service.

Gotcha: <NAME> cannot contain underscores. The loader takes the text between the first and second _ as the name, so SERVICE_MY_TOOL_UPLOAD_URL defines a service my with an unknown setting TOOL_UPLOAD_URL, which is silently ignored.

Gotcha: PORT, MAX_AGE, MAX_UPLOAD_SIZE, RUNS_PER_USER, and MAX_RUNS are parsed as numbers with no fallback on a bad value — if you set one to something non-numeric, the server panics on startup instead of falling back to its default. Make sure these are always valid integers.

Example Configuration

Minimal Setup

export PORT=5000
export DB_PATH=/var/lib/job-orchestrator/db.sqlite
export DATA_PATH=/var/lib/job-orchestrator/data
export SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit
export SERVICE_EXAMPLE_DOWNLOAD_URL=http://localhost:9000/retrieve
export SERVICE_EXAMPLE_TERMINATE_URL=http://localhost:9000/kill

Production Setup

# Core settings
export PORT=5000
export DB_PATH=/opt/orchestrator/db.sqlite
export DATA_PATH=/opt/orchestrator/data
export MAX_AGE=864000  # 10 days

# Example service (general purpose)
export SERVICE_EXAMPLE_UPLOAD_URL=http://compute-1:9000/submit
export SERVICE_EXAMPLE_DOWNLOAD_URL=http://compute-1:9000/retrieve
export SERVICE_EXAMPLE_TERMINATE_URL=http://compute-1:9000/kill
export SERVICE_EXAMPLE_RUNS_PER_USER=10

# Simulation service (specialized, long-running)
export SERVICE_SIMULATE_UPLOAD_URL=http://sim-cluster:9001/submit
export SERVICE_SIMULATE_DOWNLOAD_URL=http://sim-cluster:9001/retrieve
export SERVICE_SIMULATE_TERMINATE_URL=http://sim-cluster:9001/kill
export SERVICE_SIMULATE_RUNS_PER_USER=3

Docker Compose

services:
  server:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "server"]
    ports:
      - "5000:5000"
    environment:
      PORT: 5000
      DB_PATH: /opt/data/db.sqlite
      DATA_PATH: /opt/data
      MAX_AGE: 864000
      SERVICE_EXAMPLE_UPLOAD_URL: http://client:9000/submit
      SERVICE_EXAMPLE_DOWNLOAD_URL: http://client:9000/retrieve
      SERVICE_EXAMPLE_TERMINATE_URL: http://client:9000/kill
      SERVICE_EXAMPLE_RUNS_PER_USER: 5
      SERVICE_EXAMPLE_MAX_RUNS: 10
    volumes:
      - server-data:/opt/data

volumes:
  server-data:

Configuration Details

PORT

The HTTP port for the REST API. Users will connect to this port to submit jobs and download results.

PORT=5000

DB_PATH

Path to the SQLite database file. The directory must exist and be writable.

DB_PATH=/var/lib/job-orchestrator/db.sqlite

The database is created automatically on first run. It stores:

  • Job metadata (ID, user, service, status)
  • Job locations and timestamps
  • Client payload references

DATA_PATH

Directory where job files are stored. Each job gets a unique subdirectory.

DATA_PATH=/var/lib/job-orchestrator/data

Structure:

/var/lib/job-orchestrator/data/
├── a1b2c3d4-e5f6-7890-abcd-ef1234567890/
│   ├── run.sh
│   ├── input.pdb
│   └── output.zip  (after completion)
├── b2c3d4e5-f6a7-8901-bcde-f12345678901/
│   └── ...

MAX_AGE

How long to keep job directories before cleanup, in seconds. Age is measured from the directory’s last modification time and the rule applies to every job, whatever its status. A job that is still running (or queued) when its directory passes MAX_AGE is cleaned too, so set it well above your longest expected job.

ValueDuration
36001 hour
8640024 hours
17280048 hours
6048001 week
86400010 days (default)
MAX_AGE=864000

Jobs older than this are removed by the Cleaner task.

Service URLs

Each service needs upload, download, and terminate URLs pointing to a client:

SERVICE_MYSERVICE_UPLOAD_URL=http://client-host:9000/submit
SERVICE_MYSERVICE_DOWNLOAD_URL=http://client-host:9000/retrieve
SERVICE_MYSERVICE_TERMINATE_URL=http://client-host:9000/kill
  • UPLOAD_URL: Where to POST job files
  • DOWNLOAD_URL: Where to GET results (/{id} is appended automatically). The partial download URL is derived from it by replacing the last path segment with retrieve_partial
  • TERMINATE_URL: Where to POST termination requests (/{id} is appended automatically)

Gotcha: if you leave UPLOAD_URL, DOWNLOAD_URL, or TERMINATE_URL unset for a service, the server does not fail at startup — the field is silently stored as an empty string, and requests routed to that service will fail at request time instead. Double-check all three are set for every service you configure.

RUNS_PER_USER

Controls how many jobs a single user can have running simultaneously for a service:

SERVICE_EXAMPLE_RUNS_PER_USER=5
  • Jobs exceeding the quota remain in Queued status
  • They’re automatically dispatched when slots become available
  • Set higher for quick jobs, lower for resource-intensive jobs

Fixed Settings

These are not configurable:

  • Task intervals: Sender and Getter every 500ms, Cleaner every 60s
  • Log level: always INFO. RUST_LOG is ignored. Every HTTP request is logged with its headers, so keep secrets out of headers or filter logs
  • Outgoing HTTP timeouts: none. A client that accepts the connection but never answers can stall the request (and a termination stays Locked meanwhile)

File Permissions

Ensure the server process has:

  • Read/Write access to DB_PATH parent directory
  • Read/Write access to DATA_PATH directory
  • Network access to all configured client URLs

Validating Configuration

Start the server and check logs:

job-orchestrator server

You should see:

  • WARN lines for every core variable left at its default
  • A Config { services: {...}, ... } line listing the parsed services. Check that every service appears with all three URLs set
  • Using database: sqlite://...
  • listening on 0.0.0.0:<port>

Test with a health check:

curl http://localhost:5000/health

See Also

Client Configuration

The client is configured through environment variables and runs as a job executor.

Environment Variables

VariableDefaultDescription
PORT5000HTTP port the client listens on
DB_PATH./db.sqlitePath to the client’s SQLite payload database
DATA_PATH./dataWorking directory where job files are stored
MAX_AGE864000Seconds before the client’s cleaner deletes a payload directory (10 days)

The client shares the exact same config loader as the server, so these variables have the same names and same defaults in both roles. SERVICE_* variables are ignored by the client. There is no client-specific default for PORT: if you run a server and a client on the same host, pick a different port explicitly (9000 is a common convention, shown below) to avoid a clash.

Example Configuration

Basic Setup

export PORT=9000
export DB_PATH=/var/lib/job-orchestrator-client/db.sqlite
export DATA_PATH=/var/lib/job-orchestrator-client/data
job-orchestrator client

Docker Compose

services:
  client:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
      DATA_PATH: /opt/data
      DB_PATH: /opt/data/db.sqlite
    volumes:
      - client-data:/opt/data

volumes:
  client-data:

Note there’s no ports: mapping here — in the shipped docker-compose.yml the client is only reachable over the internal Docker network from the server, not from the host. See Client Security below.

How the Client Works

Database

The client uses the same DB_PATH-backed SQLite database mechanism as the server — by default it’s a real file on disk, not in-memory, so payload records survive a client restart.

If you actually want an ephemeral, in-memory database (e.g. for a disposable/test client), set DB_PATH=:memory: explicitly. That’s an opt-in choice, not the default behavior. Be aware that payload IDs then restart at 1 after every restart, while the server still holds the old IDs as dest_id, and old payload directories with the same numbers may still be on disk. Only use it for throwaway clients.

The client’s database keeps a record of every payload it received (cleaned ones stay with status Cleaned). The server is still the source of truth for job history.

Working Directory

The client stores job files under DATA_PATH. Each payload gets a subdirectory named after its numeric payload ID:

/opt/data/
├── 1/
│   ├── run.sh
│   ├── input.pdb
│   ├── output.txt         (created by run.sh)
│   ├── .orchestrator.exit (written by the trap)
│   └── output.zip         (created on the first completed /retrieve)
├── 2/
│   └── ...

Uploaded files are stored by base name only, so any directory structure is flattened.

Gotcha: unlike the server, the client does not create DATA_PATH at startup. It is created when the first payload arrives. Until then the client’s cleaner logs could not read directory every 60 seconds, which is harmless.

Execution Environment

When the Runner task executes a job:

  1. Validates run.sh (size, UTF-8, required trap, blocked patterns)
  2. Runs bash run.sh with the payload directory as working directory
  3. Stores the process ID; the exit code is read later from .orchestrator.exit
  4. All files in the directory are included in results

The Updater task then watches Running payloads:

  • If a payload has the killed flag set, it marks it Killed
  • If .orchestrator.exit exists, it marks it Completed (exit code 0) or Failed
  • If the process is still alive, it leaves it Running
  • If the process is gone and never wrote .orchestrator.exit, it marks it Lost and logs a warning (see Lost Processes)

Job Termination

The client supports on-demand job termination via the /kill/:id endpoint:

  • Receives termination requests from the server
  • Sends SIGTERM to the running process (only the bash process; child processes it started may keep running unless the script forwards the signal)
  • Sets the payload’s killed flag; the Updater changes the status to Killed if the payload is Running
  • The server’s /terminate/:id endpoint triggers this via HTTP

Resource Reporting

The client exposes a /load endpoint that reports CPU usage:

curl http://localhost:9000/load

Returns a float representing CPU usage percentage. This can be used by the server for load-aware scheduling (planned feature).

Multiple Clients

You can run multiple clients for:

  • Scaling: Handle more concurrent jobs
  • Isolation: Different services on different machines
  • Redundancy: Failover capability

Same Service, Multiple Clients

Not supported. Each service maps to exactly one client URL, and the server only accepts one value per variable. Do not put a load balancer in front of several clients either: payload IDs are local to each client, so a /retrieve/{id} sent to a different client would hit the wrong payload or a 404.

To spread load today, split the work into several services, each with its own client.

Different Services

Run specialized clients for different workloads:

# Client for general jobs
PORT=9000 job-orchestrator client

# Client for heavy computation (different machine)
PORT=9001 job-orchestrator client

Server configuration:

SERVICE_LIGHT_UPLOAD_URL=http://client-1:9000/submit
SERVICE_LIGHT_DOWNLOAD_URL=http://client-1:9000/retrieve
SERVICE_LIGHT_TERMINATE_URL=http://client-1:9000/kill
SERVICE_HEAVY_UPLOAD_URL=http://client-2:9001/submit
SERVICE_HEAVY_DOWNLOAD_URL=http://client-2:9001/retrieve
SERVICE_HEAVY_TERMINATE_URL=http://client-2:9001/kill

Client Security

Network Access

The client should only be accessible by the orchestrator server:

  • Use internal networks / VPCs
  • Firewall rules to restrict access
  • Never expose client ports to the internet

Execution Sandbox

The client executes arbitrary run.sh scripts. Consider:

  • Running in containers with resource limits
  • Using separate user accounts with minimal permissions
  • Mounting only necessary directories
  • Network isolation if jobs don’t need internet

Docker Resource Limits

services:
  client:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    deploy:
      resources:
        limits:
          cpus: '4'
          memory: 8G
        reservations:
          cpus: '1'
          memory: 1G

Monitoring

The examples below assume the client is reachable directly (e.g. running it outside Docker, or with a ports: mapping added). With the shipped docker-compose.yml, the client has no host-exposed port — reach it with docker compose exec client curl localhost:9000/health instead, or from the server container over the internal network.

Health Check

curl http://localhost:9000/health

Load Check

curl http://localhost:9000/load

Container Logs

docker logs -f client-container

Troubleshooting

Client Not Receiving Jobs

  1. Verify server can reach client URL
  2. Check firewall rules
  3. Verify service configuration on server

Jobs Stuck in Prepared

  1. Check the client logs for errors
  2. Check for permission issues in the working directory

High Memory Usage

The client holds each incoming upload fully in memory while handling /submit, and builds /retrieve_partial archives in memory. Large inputs or partial downloads of large directories cause memory spikes. The database is file-backed and is not the cause. If memory is high:

  1. Check the size of the payloads being sent and of the partial downloads being requested
  2. Check for stuck payloads
  3. Restarting the client frees memory, but jobs running at that moment become lost (see Lost Processes)

See Also

Quota System

The quota system ensures fair resource allocation by limiting concurrent jobs per user and per service.

Two Independent Limits

The sender enforces two limits before dispatching a queued job:

VariableScopeDefaultDescription
SERVICE_<NAME>_RUNS_PER_USERPer user, per service5Max jobs a single user can have active at once
SERVICE_<NAME>_MAX_RUNSPer service (all users)10Max jobs the service can have active at once

Both limits must have available slots for a job to be dispatched. A job that would violate either limit remains in Queued status until a slot opens.

Jobs in Processing, Submitted, or Running status all count toward both limits. A job whose process disappears (for example after a client restart) is marked Lost and releases its slot (see Lost Processes).

How It Works

SERVICE_EXAMPLE_RUNS_PER_USER=3
SERVICE_EXAMPLE_MAX_RUNS=5

User 1 submits 4 jobs → 3 dispatched (per-user limit reached), 1 queued
User 2 submits 4 jobs → 2 dispatched (MAX_RUNS=5 reached), 2 queued

When one of User 1's jobs completes, one service slot frees up.
Both users now have a queued job and a free per-user slot, so either
User 1's or User 2's job is dispatched next. Which user goes first in a
sender pass is not deterministic.

Quota Scope

Quotas are enforced per user, per service (RUNS_PER_USER) and per service (MAX_RUNS):

UserServiceRUNS_PER_USERMAX_RUNSResult
user_1example510Up to 5 concurrent
user_2example510Up to 5 concurrent (independent)
user_1simulate24Up to 2 concurrent

Users don’t compete with each other for per-user slots. They do share the MAX_RUNS pool for the service.

Scheduling: Round-Robin Between Users

When multiple users have queued jobs for the same service, the sender distributes slots in round-robin order across users. This prevents a single user with many queued jobs from consuming all available slots ahead of others.

The rotation restarts on every sender pass (every 500ms) and the order of users within a pass is arbitrary. Within one user, jobs are in practice taken oldest first (there is no explicit ordering or priority).

Quota States

┌──────────┐   Both limits OK   ┌────────────┐
│  Queued  │ ─────────────────▶ │ Processing │
└──────────┘                    └────────────┘
     │                                │
     │ Either limit exceeded          ▼
     │                         ┌────────────┐
     └──────────────────────── │ Submitted  │
       (waits for a free slot) │ (running)  │
                               └────────────┘

Choosing Values

Factors to Consider

  1. Client capacity: MAX_RUNS should not exceed what your client hardware can handle concurrently
  2. Job duration: Longer jobs need lower per-user quotas
  3. Resource usage: CPU/memory intensive jobs need lower MAX_RUNS
  4. User base: More users may need lower per-user quotas

Guidelines

Job TypeSuggested RUNS_PER_USERSuggested MAX_RUNS
Quick jobs (< 1 min)10-2020-50
Medium jobs (1-10 min)5-1010-20
Long jobs (10+ min)2-54-10
Resource-intensive1-32-4

Example Scenarios

Scientific Computing Platform

# Quick validation jobs
SERVICE_VALIDATE_RUNS_PER_USER=10
SERVICE_VALIDATE_MAX_RUNS=50

# Standard analysis
SERVICE_ANALYZE_RUNS_PER_USER=5
SERVICE_ANALYZE_MAX_RUNS=20

# Heavy simulation
SERVICE_SIMULATE_RUNS_PER_USER=2
SERVICE_SIMULATE_MAX_RUNS=4

Educational Platform

# Student exercises
SERVICE_EXERCISE_RUNS_PER_USER=3
SERVICE_EXERCISE_MAX_RUNS=30

# Final projects
SERVICE_PROJECT_RUNS_PER_USER=5
SERVICE_PROJECT_MAX_RUNS=20

Testing Quotas

Submit multiple jobs to observe throttling:

Start the server with low limits (the variables must be set in the server’s environment, for example in docker-compose.yml, not in the shell you submit from):

SERVICE_EXAMPLE_RUNS_PER_USER=2
SERVICE_EXAMPLE_MAX_RUNS=3

Use a run.sh that takes a while (for example sleep 60 after the trap), then submit 5 jobs and check them:

for i in {1..5}; do
  curl -s -X POST http://localhost:5000/upload \
    -F "file=@run.sh" \
    -F "user_id=1" \
    -F "service=example" | jq -r '.id'
done

sleep 2
for id in 1 2 3 4 5; do
  curl -s http://localhost:5000/download/$id | jq -r '"\(.id) \(.status)"'
done

Expected behaviour (the upload response always says Queued, so check /download):

  • Jobs 1-2: dispatched (Submitted/Running); the per-user limit of 2 is now reached
  • Jobs 3-5: stay Queued. MAX_RUNS=3 would allow one more, but not for this user
  • Submitting as user_id=2 at this point would dispatch one job and fill MAX_RUNS

Fair Scheduling

The quota system provides basic fairness:

  • Per-user isolation: One user can’t starve others
  • Per-service cap: Prevents overloading a single client
  • Round-robin dispatch: Slots are shared evenly across users
  • Automatic queuing: No jobs are rejected, just delayed

Limitations

Current limitations (improvements planned):

  • No priority queues (FIFO within quota constraints)
  • No time-based quotas (e.g., jobs per hour)
  • No burst allowances

See Also

Server API Endpoints

The orchestrator server exposes a REST API for job submission and retrieval.

Base URL

http://localhost:5000

Interactive Documentation

Swagger UI is available at:

http://localhost:5000/swagger

Endpoints

POST /upload

Submit a new job for processing.

Request

  • Content-Type: multipart/form-data
  • Max size: 400MB by default, set with MAX_UPLOAD_SIZE (see Server Configuration)
FieldTypeRequiredDescription
filefileYesOne or more files (repeat for multiple). Any field name works; every part with a filename is saved
user_idintegerYesUser identifier for quota tracking
servicestringYesService name, lowercase (must be configured on server)

Example

curl -X POST http://localhost:5000/upload \
  -F "file=@run.sh" \
  -F "file=@input.pdb" \
  -F "user_id=1" \
  -F "service=example"

Response

HTTP/1.1 202 Accepted
Location: /download/1
{
  "id": 1,
  "status": "Queued",
  "message": "Job successfully uploaded"
}

The job is queued, not finished. Poll the URL in the Location header (GET /download/{id}) to follow it.

Status Codes

CodeDescription
202Job accepted and queued. The Location header points to /download/{id}
400The multipart body could not be parsed (for example, no boundary or a truncated body)
413The request body exceeds the upload limit (MAX_UPLOAD_SIZE)
415The Content-Type header is missing or not multipart/form-data
422The request parsed, but a field is invalid: user_id missing or not a number, unknown service, or no run.sh uploaded
500The server failed to store the job (disk or database error). The request can be retried

Every error response has a JSON body with a message explaining the problem.

Notes

  • The service must match a configured service on the server exactly (service names are lowercase)
  • Files are stored by base name only: directory parts are stripped, and two files with the same name overwrite each other
  • Other text fields are accepted and ignored
  • The entry point script must be named exactly run.sh (hardcoded, case-sensitive). An upload without it is rejected with 422. Its contents are not checked at upload time: an unsafe or broken script results in Invalid status when the client tries to execute it
  • dest_id is populated after the job is dispatched to a client

GET /download/

Get job status or download completed results.

Parameters

ParameterTypeDescription
idintegerJob ID from upload response

Example

# Check status (returns JSON when not completed)
curl http://localhost:5000/download/1

# Download results (returns ZIP when completed)
curl -o results.zip http://localhost:5000/download/1

Response

When the job is not yet completed, returns JSON:

{
  "id": 1,
  "status": "Running",
  "message": ""
}

When the job is completed, returns:

  • Content-Type: application/zip
  • Body: ZIP archive of all output files

Status Codes

CodeDescription
200JSON job status or ZIP file (check Content-Type)
404Job not found
500Server error

Notes

  • The same endpoint returns JSON or ZIP depending on job state. Only Completed jobs return the ZIP
  • Terminal states other than Completed (Failed, Invalid, Killed, Lost, Cleaned) keep returning JSON with 200. Use /download_partial/{id} to get the files of a failed job
  • All files present in the job directory after run.sh finishes are included

Usage Pattern

while true; do
  ct=$(curl -s -o response.tmp -w '%{content_type}' http://localhost:5000/download/1)
  if [[ "$ct" == application/zip* ]]; then
    mv response.tmp results.zip
    break
  fi
  status=$(jq -r '.status' response.tmp)
  case "$status" in
    Failed|Invalid|Killed|Lost|Cleaned) echo "Job ended: $status"; break ;;
  esac
  echo "Status: $status"
  sleep 5
done

GET /download_partial/

Retrieve current job state regardless of completion status.

Use this endpoint to debug stuck or incomplete jobs by downloading the current workspace state as a ZIP file.

Parameters

ParameterTypeDescription
idintegerJob ID from upload response

Example

curl -o partial_results.zip http://localhost:5000/download_partial/1

Response

  • Content-Type: application/zip
  • Body: ZIP archive containing current job directory state

Status Codes

CodeDescription
200ZIP file with current job state
400Job’s service is not configured on the server
404Job not found, job not dispatched yet (Queued), or the client no longer has the payload
500Server error, or the client could not build the archive (for example after cleanup)

Notes

  • The ZIP comes from the client’s working directory, not from the server’s copy
  • The client URL is derived from the service’s DOWNLOAD_URL by replacing its last path segment with retrieve_partial

POST /terminate/

Cancel a running job.

Parameters

ParameterTypeDescription
idintegerJob ID to terminate

Example

curl -X POST http://localhost:5000/terminate/1

Response

On success:

{
  "id": 1,
  "status": "Unknown",
  "message": "job terminated"
}

On failure:

{
  "id": 1,
  "status": "Unknown",
  "message": "could not terminate job"
}

The status field in the response reflects the in-memory default and is not meaningful here. Query GET /download/{id} to confirm the job’s actual status after termination.

Status Codes

CodeDescription
200Job terminated successfully
404Job not found
500Server error or termination failed

Notes

  • Termination is not gated on the job’s current status: the server locks the job, forwards the kill to the client regardless of what state it’s in, and rolls the status back to what it was before if the kill fails
  • Terminated jobs will have status Killed
  • The server sends a termination signal to the client, which then kills the process
  • Queued jobs cannot be terminated: there is no client payload yet, the client answers 404, the server returns 500 and the job stays Queued (it will still run later)
  • Terminating a job that already finished reports success and changes its status to Killed. From then on /download/{id} returns JSON instead of the ZIP, even though output.zip is still on the server

GET /health

Health check endpoint.

Example

curl http://localhost:5000/health

Response

On success:

{
  "status": "ok",
  "database": "ok"
}

On failure (database unreachable), the response is a bare 503 with no body.

Status Codes

CodeDescription
200Server is healthy
503Database is unreachable

GET /

Ping endpoint for basic connectivity check.

Example

curl http://localhost:5000/

Response

{"message": "pong"}

GET /swagger

Interactive API documentation.

Example

Open in browser:

http://localhost:5000/swagger

Provides:

  • Interactive API explorer
  • Request/response schemas
  • Try-it-out functionality

Error Responses

Error responses from /upload, /download, /download_partial and /terminate follow this format (/health returns a bare 503 instead):

{
  "id": 0,
  "status": "Unknown",
  "message": "Description of the error"
}

Rate Limiting

The server does not implement rate limiting directly. Use a reverse proxy (nginx, traefik) for rate limiting in production.

Authentication

The server does not implement authentication directly. The user_id field is trusted as provided. Implement authentication at the reverse proxy layer or in your application.

See Also

Client API Endpoints

The client exposes endpoints for the orchestrator server to submit jobs and retrieve results.

Note: These endpoints are typically only accessed by the orchestrator server, not by end users.

Base URL

http://localhost:9000

Endpoints

POST /submit

Receive a job payload from the orchestrator server.

Request

  • Content-Type: multipart/form-data
FieldTypeRequiredDescription
filefileYesOne or more job files. Any field name works; parts without a filename are ignored

Example

curl -X POST http://localhost:9000/submit \
  -F "file=@run.sh" \
  -F "file=@input.pdb"

Response

{
  "id": 1,
  "status": "Prepared",
  "loc": "/opt/data/1",
  "pid": 0,
  "killed": false
}

Status Codes

CodeDescription
200Payload received successfully
400Malformed multipart request
500Server error

Notes

  • The client stores files (by base name) in DATA_PATH/<id> and creates a payload record
  • The whole request is held in memory while it is processed, and there is no body size limit on the client
  • Status starts as Prepared, waiting for the Runner task
  • The id is returned to the server and stored as dest_id

GET /retrieve_partial/

Retrieve current payload state regardless of completion status.

Use this endpoint to debug stuck or incomplete payloads by downloading the current workspace state as a ZIP file. This endpoint is typically called by the orchestrator server, not end users.

Parameters

ParameterTypeDescription
idintegerPayload ID from submit response

Example

# Download partial results for debugging
curl -o partial_results.zip http://localhost:9000/retrieve_partial/1

Response

  • Content-Type: application/zip
  • Body: ZIP archive containing current payload directory state

Status Codes

CodeDescription
200ZIP file with current payload state
404Payload not found
500Server error

Notes

  • The /retrieve_partial endpoint on the client is used internally by the server’s /download_partial endpoint
  • Path traversal protection is applied to all ZIP operations

GET /retrieve/

Retrieve results of a completed payload.

Parameters

ParameterTypeDescription
idintegerPayload ID from submit response

Example

curl -o results.zip http://localhost:9000/retrieve/1

Response

When the payload is not yet completed, returns a JSON body:

{
  "id": 1,
  "status": "Running",
  "loc": "/opt/data/1",
  "pid": 12345,
  "killed": false
}

When the payload is completed, returns:

  • Content-Type: application/zip
  • Body: ZIP archive of all files in the payload directory

Status Codes

CodeDescription
200JSON payload status or ZIP file (check Content-Type)
404Payload not found
500Server error

Notes

  • The ZIP includes all files in the working directory after run.sh execution, including .orchestrator.exit
  • Original input files are included unless deleted by run.sh
  • The archive is written to output.zip in the payload directory on the first request and reused afterwards

POST /kill/

Terminate a running payload.

Parameters

ParameterTypeDescription
idintegerPayload ID to terminate

Example

curl -X POST http://localhost:9000/kill/1

Response

On success (200 OK):

  • Empty body

On failure:

  • 404 if payload not found (with an empty payload JSON body)
  • 500 if kill operation failed

Status Codes

CodeDescription
200Payload terminated successfully
404Payload not found
500Kill operation failed

Notes

  • Sends SIGTERM to the process PID stored for the payload; silently succeeds if no PID is set yet or the process is already gone
  • Sets the payload’s killed flag. The Updater changes the status to Killed only if the payload is Running; a payload that already finished keeps its status
  • This endpoint is called by the server’s /terminate/{id} endpoint

GET /load

Report current CPU usage.

Example

curl http://localhost:9000/load

Response

45.2

Returns a float representing CPU usage percentage (0-100). The value is sampled when the request arrives, so each call takes a fraction of a second.

Use Cases

  • Load-aware job distribution (planned feature)
  • Monitoring client health
  • Capacity planning

GET /health

Health check endpoint.

Example

curl http://localhost:9000/health

Response

{
  "status": "ok",
  "database": "ok"
}

GET /

Ping endpoint for basic connectivity check.

Example

curl http://localhost:9000/

Response

{"message": "pong"}

Payload States

Payloads on the client go through these states:

StateDescription
PreparedReceived from server, waiting for execution
RunningCurrently executing run.sh
CompletedExecution finished successfully
FailedExecution failed (non-zero exit code, or the process could not be started)
Invalidrun.sh missing, unsafe, or failed validation
KilledTerminated via /kill/{id}
LostProcess disappeared without writing .orchestrator.exit
CleanedPayload directory removed by the cleaner task

Security Considerations

The client API should never be exposed to the public internet:

  • No authentication is implemented
  • Arbitrary code execution via run.sh
  • Internal service communication only

Recommendations:

  • Use internal networks / VPCs
  • Firewall rules: allow only orchestrator server IP
  • Docker networks with no external exposure

See Also

Docker Deployment

Docker is the recommended way to deploy job-orchestrator in production.

Quick Start

git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
docker compose up --build

This starts:

  • Server on port 5000 (exposed to the host)
  • Example client on port 9000 (internal network only, not exposed to the host)

Docker Images

Official Image

docker pull codeberg.org/rvhonorato/job-orchestrator:latest

The image has no entrypoint: the binary lives at /job-orchestrator and is not on PATH, so always pass the full command (/job-orchestrator server or /job-orchestrator client).

The published image contains the binary plus bash and sqlite, without any application. It works as a server as-is. To use it as a client, build your own image on top of it with the tools your run.sh needs. example/Dockerfile shows how, adding the gdock demo application.

Build Locally

# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator .

# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/

Docker Compose

Basic Setup

services:
  server:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "server"]
    ports:
      - "5000:5000"
    environment:
      PORT: 5000
      DB_PATH: /opt/data/db.sqlite
      DATA_PATH: /opt/data
      MAX_AGE: 864000
      MAX_UPLOAD_SIZE: 419430400
      SERVICE_EXAMPLE_UPLOAD_URL: http://client:9000/submit
      SERVICE_EXAMPLE_DOWNLOAD_URL: http://client:9000/retrieve
      SERVICE_EXAMPLE_TERMINATE_URL: http://client:9000/kill
      SERVICE_EXAMPLE_RUNS_PER_USER: 5
      SERVICE_EXAMPLE_MAX_RUNS: 10
    volumes:
      - server-data:/opt/data

  client:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
      DATA_PATH: /opt/data
      DB_PATH: /opt/data/db.sqlite
    volumes:
      - client-data:/opt/data
    depends_on:
      - server

volumes:
  server-data:
  client-data:

Production Setup

See Production Deployment - Container Hardening for details on each security option.

Multiple Clients

Scaling Horizontally

A service is bound to exactly one client URL, so adding more clients for the same service does not spread its jobs. Extra clients only receive work if they are configured as separate services, as shown below. Do not place a load balancer in front of several clients: payload IDs are local to each client, so status and result requests must reach the client that got the job.

Multiple Services

services:
  server:
    environment:
      # Light jobs
      SERVICE_LIGHT_UPLOAD_URL: http://client-light:9000/submit
      SERVICE_LIGHT_DOWNLOAD_URL: http://client-light:9000/retrieve
      SERVICE_LIGHT_TERMINATE_URL: http://client-light:9000/kill
      SERVICE_LIGHT_RUNS_PER_USER: 10

      # Heavy jobs
      SERVICE_HEAVY_UPLOAD_URL: http://client-heavy:9000/submit
      SERVICE_HEAVY_DOWNLOAD_URL: http://client-heavy:9000/retrieve
      SERVICE_HEAVY_TERMINATE_URL: http://client-heavy:9000/kill
      SERVICE_HEAVY_RUNS_PER_USER: 2

  client-light:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

  client-heavy:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
    deploy:
      resources:
        limits:
          cpus: '8'
          memory: 16G

Volume Management

Persistent Storage

Always use named volumes for production:

volumes:
  server-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/job-orchestrator/server

  client-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/job-orchestrator/client

Backup Strategy

# Stop services (optional, for consistent backup)
docker compose stop

# Backup server and client data (the client holds results of running jobs)
tar -czf backup-$(date +%Y%m%d).tar.gz /data/job-orchestrator/server /data/job-orchestrator/client

# Resume services
docker compose start

Networking

Internal Network

Keep client internal:

services:
  server:
    ports:
      - "5000:5000"  # Exposed to host
    networks:
      - internal
      - external

  client:
    # No ports exposed to host
    networks:
      - internal

networks:
  internal:
    internal: true
  external:

With Reverse Proxy

services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
    depends_on:
      - server

  server:
    # No ports exposed, accessed via nginx
    networks:
      - internal

Logging

View Logs

# All services
docker compose logs -f

# Server only
docker compose logs -f server

# Last 100 lines
docker compose logs --tail 100 server

Log Rotation

services:
  server:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Commands

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart

# Rebuild and start
docker compose up --build -d

# View status
docker compose ps

# Shell into container
docker compose exec server /bin/sh

See Also

Kubernetes Deployment

The following instructions use Minikube for local testing. See kubernetes/ in the repository for the manifest files.

Quick Start (Minikube)

# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator-server .

# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator-server -t job-orchestrator-client example/

# Load into Minikube
minikube image load job-orchestrator-server
minikube image load job-orchestrator-client

# Apply manifests
minikube kubectl -- apply -f kubernetes/

# Get the external IP
minikube service job-orchestrator-server --url

Usage

# Set the base URL
export URL=$(minikube service job-orchestrator-server --url)

# Submit a job
curl -X POST $URL/upload \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example"

# Check status
curl $URL/download/1

Security Hardening

The manifests in kubernetes/ already apply the following hardening measures:

MeasureImplemented
Run as non-root (runAsUser: 1000, runAsNonRoot: true)Both server and client
Read-only root filesystemBoth server and client
Drop all capabilitiesBoth server and client
Seccomp profile (RuntimeDefault)Both server and client
CPU and memory resource limitsBoth server and client
NetworkPolicy restricting client ingress to the server and egress to the server and DNSClient

NetworkPolicy enforcement: NetworkPolicies only take effect if the cluster’s network plugin enforces them. Minikube’s default setup does not; start it with a plugin that does, for example minikube start --cni=calico.

PID limits: Kubernetes has no per-container PID limit in resources.limits, and LimitRange cannot set one either. Configure it on the node with the kubelet’s podPidsLimit setting (--pod-max-pids), which applies to every pod on that node.

Scaling: keep replicas: 1 for both deployments. The server uses SQLite on a ReadWriteOnce volume, and a second client replica behind the same Service would break result retrieval (payload IDs are local to each client).

See Also

Production Deployment

This guide covers best practices for deploying job-orchestrator in production environments.

Architecture Recommendations

Minimum Setup

┌─────────────┐     ┌─────────────┐
│   Server    │────▶│   Client    │
│ (1 instance)│     │ (1 instance)│
└─────────────┘     └─────────────┘
                    ┌──────────────┐
                    │ Load Balancer│
                    │   (nginx)    │
                    └──────┬───────┘
                           │
                    ┌──────▼───────┐
                    │    Server    │
                    │  (1 instance)│
                    └──────┬───────┘
                           │
          ┌────────────────┼────────────────┐
          │                │                │
   ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐
   │  Client 1   │  │  Client 2   │  │  Client 3   │
   └─────────────┘  └─────────────┘  └─────────────┘

Security

Script Validation

The client validates run.sh before execution. A script is rejected if it:

  • Exceeds 20 MiB in size
  • Is not valid UTF-8
  • Is missing the required exit trap — the script must contain trap 'echo $? > .orchestrator.exit' EXIT
  • Matches any dangerous pattern, including:
CategoryBlocked patterns
Destructive commandsrm followed later on the same line by any / or ~ (so rm -f tmp/file is blocked too), mkfs, dd to/from devices
Sensitive file access/etc/passwd, /etc/shadow, /etc/sudoers, /proc/, /sys/, ~/.ssh/, /root/, Docker socket
Network toolscurl, wget, nc, ncat, socat, ssh, scp, sftp, telnet, rsync
Reverse shells/dev/tcp/, /dev/udp/
Privilege escalationsudo, su, chown, dangerous chmod
Container/system escapechroot, nsenter, unshare, mount, umount, docker, kubectl
Kernel manipulationsysctl, modprobe, insmod, rmmod, iptables, nftables
Obfuscated executionbase64 | bash, eval, python -c, perl -e, ruby -e
Persistencecrontab, /etc/cron, systemctl, service, at
Fork bombs:(){ ...|: }
Resource exhaustionstress, stress-ng
Crypto minersxmrig, minerd, cpuminer
Environment secrets$AWS_*, $SECRET, $TOKEN, $PASSWORD, $API_KEY

The patterns are plain regular expressions matched against the whole file, comments included, and several have no word boundary. Expect false positives: for example su\s+ also matches words ending in “su” followed by a space, rm\s+ matches inside perform ./step, and \$TOKEN matches $TOKENIZER. If a valid script is rejected as Invalid, check the client log for the reason.

This is a sanity check, not a sandbox. It can be bypassed by determined actors. Input scripts are still expected to come from trusted or semi-trusted sources. True isolation must be enforced at the deployment level using the container hardening measures below.

Path Traversal Protection

Two separate mechanisms protect the filesystem:

  • Uploads (server /upload and client /submit): every filename is reduced to its base name before saving, so ../../etc/passwd is stored as passwd inside the job directory. Uploaded archives are never extracted.
  • Result archives (/retrieve and /retrieve_partial on the client): while building the ZIP, every entry is canonicalized and must stay inside the payload directory. If anything resolves outside it, such as a symlink created by run.sh that points to /etc, building the archive fails instead of including that file.

Container Hardening

The client executes user-submitted scripts with the full privileges of the process. Apply all of the following to limit blast radius:

MeasureDocker ComposePurpose
Read-only rootfsread_only: truePrevent filesystem tampering
Drop all capabilitiescap_drop: [ALL]Remove kernel-level privileges
No new privilegessecurity_opt: [no-new-privileges:true]Block setuid/setgid escalation
CPU limitdeploy.resources.limits.cpusPrevent CPU starvation
Memory limitdeploy.resources.limits.memoryPrevent OOM on host
PIDs limitdeploy.resources.limits.pidsPrevent fork bombs
Internal networknetworks: [internal]Block outbound internet access
Writable tmpfstmpfs: [/tmp]Provide scratch space on read-only rootfs

Example (applied to the client service):

services:
  client:
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    tmpfs:
      - /tmp
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 2G
          pids: 256
    networks:
      - internal

networks:
  internal:
    internal: true

Note: The Kubernetes manifests already run as non-root (runAsUser: 1000). For Docker deployments, the Dockerfile does not yet create a dedicated non-root user — see the TODO in the Dockerfile.

Network Security

  1. Never expose clients to the internet

    • Clients execute user-submitted scripts
    • Use internal networks only
    • Block all outbound access from client containers
  2. Use a reverse proxy

    • TLS termination
    • Rate limiting
    • Request filtering
  3. Firewall rules

    # Allow only orchestrator server to reach clients
    iptables -A INPUT -p tcp --dport 9000 -s <server-ip> -j ACCEPT
    iptables -A INPUT -p tcp --dport 9000 -j DROP
    

Reverse Proxy (nginx)

# http block — limit_req_zone must be at http context, not inside server
http {
    limit_req_zone $binary_remote_addr zone=upload:10m rate=10r/s;

    upstream orchestrator {
        server 127.0.0.1:5000;
    }

    server {
        listen 443 ssl http2;
        server_name jobs.example.com;

        ssl_certificate /etc/nginx/certs/cert.pem;
        ssl_certificate_key /etc/nginx/certs/key.pem;

        location /upload {
            limit_req zone=upload burst=20 nodelay;
            client_max_body_size 400M;  # keep in sync with MAX_UPLOAD_SIZE
            proxy_pass http://orchestrator;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }

        location /download {
            proxy_pass http://orchestrator;
            proxy_set_header Host $host;
        }

        location /terminate {
            proxy_pass http://orchestrator;
            proxy_set_header Host $host;
        }

        location /health {
            proxy_pass http://orchestrator;
        }

        # Block swagger in production (optional)
        location /swagger {
            deny all;
        }
    }
}

client_max_body_size is checked by nginx before the request reaches the server. If it is lower than MAX_UPLOAD_SIZE, nginx rejects large uploads with its own 413 page instead of the server’s JSON response, so keep the two values equal.

Authentication

job-orchestrator does not implement authentication. Options:

  1. Reverse proxy authentication

    location / {
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://orchestrator;
    }
    
  2. Application-level authentication

    • Wrap the API in your application
    • Validate users before calling job-orchestrator
  3. OAuth2 Proxy

    • Use oauth2-proxy in front of the service
    • Integrates with identity providers

Resource Planning

Server Requirements

Load LevelCPUMemoryStorage
Light (< 100 jobs/day)1 core512MB10GB
Medium (100-1000 jobs/day)2 cores1GB50GB
Heavy (> 1000 jobs/day)4 cores2GB100GB+

Storage depends heavily on job file sizes and retention period.

Client Requirements

Depends entirely on your job workloads:

Job TypeCPUMemory
Text processing1 core512MB
Scientific computing4-8 cores8-16GB
ML/Deep learning8+ cores + GPU32GB+

Storage Calculation

Storage = (avg_job_size) × (jobs_per_day) × (retention_days)

Example:
- 10MB average job
- 500 jobs/day
- 10 day retention (default MAX_AGE)
= 10MB × 500 × 10 = 50GB

Monitoring

Health Checks

# Server health
curl -f http://localhost:5000/health

# Client health
curl -f http://localhost:9000/health

# Client load
curl http://localhost:9000/load

Log Aggregation

services:
  server:
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        tag: "job-orchestrator.server"

Backup & Recovery

What to Backup

  1. Server database (DB_PATH)

    • Contains job history and status
    • Critical for job tracking
  2. Server data directory (DATA_PATH)

    • Contains job files and results
    • Large, may use incremental backups
  3. Client database and data directory

    • Holds payload IDs that the server references as dest_id, plus the working files of running jobs
    • Losing them breaks status and result retrieval for jobs that were in flight

Backup Script

#!/bin/bash
BACKUP_DIR=/backups/job-orchestrator
DATE=$(date +%Y%m%d_%H%M%S)

# Backup database
sqlite3 /opt/data/db.sqlite ".backup '${BACKUP_DIR}/db_${DATE}.sqlite'"

# Backup data (incremental with rsync)
rsync -av --delete /opt/data/ ${BACKUP_DIR}/data/

# Cleanup old backups (keep 7 days)
find ${BACKUP_DIR} -name "db_*.sqlite" -mtime +7 -delete

Recovery

# Stop server
docker compose stop server

# Restore database (pick the backup file you want)
cp /backups/job-orchestrator/db_YYYYMMDD_HHMMSS.sqlite /opt/data/db.sqlite

# Restore data
rsync -av /backups/job-orchestrator/data/ /opt/data/

# Start server
docker compose start server

High Availability

Current Limitations

  • Single server architecture
  • No built-in clustering
  • SQLite doesn’t support concurrent writes

Workarounds

  1. Quick recovery

    • Automated health checks
    • Container auto-restart
    • Fast backup restoration
  2. Client restarts

    • Clients are not stateless: keep their database and data directory on persistent storage
    • Restarting a client while jobs run loses those processes, and the jobs are marked Lost and must be resubmitted (see Lost Processes)
    • Drain a client (wait for its jobs to finish) before restarting it
  3. Future improvements

    • PostgreSQL support (planned)
    • Server clustering (planned)

Maintenance

Database Maintenance

# Vacuum database (reclaim space)
sqlite3 /opt/data/db.sqlite "VACUUM;"

# Check integrity
sqlite3 /opt/data/db.sqlite "PRAGMA integrity_check;"

Log Rotation

Ensure logs don’t fill disk:

services:
  server:
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"

Updates

# Pull latest image
docker pull codeberg.org/rvhonorato/job-orchestrator:latest

# Recreate containers
docker compose up -d

Troubleshooting

See Troubleshooting Guide for common issues.

See Also

Building from Source

This guide covers building job-orchestrator from source code.

Prerequisites

Rust Toolchain

Install Rust via rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Minimum version: Rust 1.88 (edition 2024, uses let chains)

Verify installation:

rustc --version
cargo --version

System Dependencies

Only a C compiler is needed. SQLite is compiled from source and linked statically through sqlx’s bundled SQLite, and TLS uses rustls, so no system SQLite or OpenSSL libraries are required.

Debian/Ubuntu

apt-get update
apt-get install -y build-essential

Fedora/RHEL

dnf install gcc

macOS

xcode-select --install

Windows

The code compiles with the Visual Studio Build Tools, but client mode runs jobs through bash and the kill command, so it is not supported on Windows. Use WSL or a Linux container.

Clone Repository

git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator

Build Commands

Debug Build

Fast compilation, includes debug symbols:

cargo build

Binary location: target/debug/job-orchestrator

Release Build

Optimized for performance:

cargo build --release

Binary location: target/release/job-orchestrator

Check (No Build)

Verify code compiles without producing binary:

cargo check

Running

From Cargo

# Server mode
PORT=5000 cargo run -- server

# Client mode
PORT=9000 cargo run -- client

From Binary

# After release build
PORT=5000 ./target/release/job-orchestrator server

Build Options

Features

Currently no optional features. All functionality is included by default.

Target Platforms

Cross-compile for different targets:

# Add target
rustup target add x86_64-unknown-linux-musl

# Build for target
cargo build --release --target x86_64-unknown-linux-musl

Common targets:

  • x86_64-unknown-linux-gnu - Linux (glibc)
  • x86_64-unknown-linux-musl - Linux (static)
  • x86_64-apple-darwin - macOS Intel
  • aarch64-apple-darwin - macOS Apple Silicon
  • x86_64-pc-windows-msvc - Windows (builds, but client mode does not work there)

Docker Build

Using Dockerfile

# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator .

# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/

Multi-stage Build

The Dockerfile produces a minimal Alpine image with the binary, bash, and sqlite. Where the binary comes from is picked with the BINARY_SOURCE build argument:

  1. build (default): compiles job-orchestrator from source with the full Rust toolchain
  2. prebuilt: copies an already compiled static (musl) binary from dist/job-orchestrator. CI uses this: it compiles the binary in a separate job (with caching) and only packages it with buildah
cargo build --release --target x86_64-unknown-linux-musl
mkdir -p dist && cp target/x86_64-unknown-linux-musl/release/job-orchestrator dist/
docker build --build-arg BINARY_SOURCE=prebuilt -t job-orchestrator .

The image has no application in it. example/Dockerfile builds a client image on top of it with the example application (gdock) compiled from source.

See Also

Testing

This guide covers running and writing tests for job-orchestrator.

Running Tests

All Tests

cargo test

With Output

See println! output from tests:

cargo test -- --nocapture

Specific Test

# By name
cargo test test_upload

# By module
cargo test queue_dto::tests

Ignored Tests

No tests are marked #[ignore] at the moment. If you add slow tests that need extra setup, mark them with #[ignore] and run them with:

cargo test -- --ignored

Test Coverage

Using cargo-tarpaulin

Install:

cargo install cargo-tarpaulin

Generate coverage:

# HTML report
cargo tarpaulin --out Html --output-dir ./coverage

# XML report (for CI)
cargo tarpaulin --out Xml --output-dir ./coverage

View report:

open coverage/tarpaulin-report.html

Test Structure

Tests are inline #[cfg(test)] modules within the same file as the code they test:

src/
├── config/loader.rs          # Config loading and env var tests
├── controllers/server.rs     # Upload, download, terminate handler tests
├── controllers/client.rs     # Submit, retrieve, kill handler tests
├── models/status_dto.rs      # Status serialization tests
├── models/queue_dto.rs       # Quota and round-robin scheduling tests
├── services/server.rs        # Sender, getter, cleaner task tests
├── services/client.rs        # Runner, updater, cleaner task tests
└── utils/io.rs               # Script validator, file I/O tests

There is no separate tests/ directory — this is a binary crate with no public library interface.

Writing Tests

Unit Tests

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_something() {
        let result = function_under_test();
        assert_eq!(result, expected_value);
    }
}
}

Async Tests

Use tokio::test for async functions:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn test_async_function() {
        let result = async_function().await;
        assert!(result.is_ok());
    }
}
}

Mocking

Using mockito

For HTTP mocking — this is what the codebase actually uses (src/controllers/server.rs, src/services/client.rs, src/services/server.rs):

#![allow(unused)]
fn main() {
use mockito::Server;

#[tokio::test]
async fn test_http_client() {
    let mut server = Server::new_async().await;
    let mock = server
        .mock("GET", "/health")
        .with_status(200)
        .create_async()
        .await;

    // Test against server.url()

    mock.assert_async().await;
}
}

mockall is listed as a dev-dependency in Cargo.toml for trait-based mocking, but nothing in the codebase currently uses it — every existing test mocks HTTP boundaries with mockito instead.

Test Utilities

Test Fixtures

Create reusable test data:

#![allow(unused)]
fn main() {
#[cfg(test)]
mod test_utils {
    pub fn create_test_job() -> Job {
        let mut job = Job::new("/tmp/test-data");
        job.set_user_id(1);
        job.set_service("test".to_string());
        job
    }
}
}

Temporary Directories

Use tempfile for temporary test directories:

#![allow(unused)]
fn main() {
use tempfile::TempDir;

#[test]
fn test_file_operations() {
    let temp_dir = TempDir::new().unwrap();
    let file_path = temp_dir.path().join("test.txt");

    // Test file operations

    // TempDir is automatically cleaned up
}
}

CI Testing

Tests run automatically via Forgejo Actions (.forgejo/workflows/ci.yml), in a single ci job with two steps:

  • Run clippy — cargo clippy -- -D warnings
  • Run tests with coverage — cargo tarpaulin --engine llvm

There’s no separate cargo test step — tarpaulin runs the test suite while measuring coverage.

Linting

Clippy

Run Clippy for additional checks:

cargo clippy -- -D warnings

Common fixes:

  • #[allow(clippy::lint_name)] to suppress specific lints
  • Configure in clippy.toml or Cargo.toml

Formatting

Check formatting:

cargo fmt -- --check

Fix formatting:

cargo fmt

Debugging Tests

With println

#![allow(unused)]
fn main() {
#[test]
fn test_debug() {
    let value = compute_something();
    println!("Debug value: {:?}", value);  // Use --nocapture to see
    assert!(value.is_valid());
}
}

With RUST_BACKTRACE

RUST_BACKTRACE=1 cargo test

See Also

Contributing

Contributions to job-orchestrator are welcome! This guide explains how to contribute.

Ways to Contribute

  • Bug reports: Found a bug? Open an issue
  • Feature requests: Have an idea? Open an issue to discuss
  • Code contributions: Fix bugs or implement features
  • Documentation: Improve docs, fix typos, add examples
  • Testing: Add tests, report edge cases

Please read the AI Policy before opening a pull request or issue.

Getting Started

1. Fork the Repository

Click “Fork” on Codeberg, then clone your fork:

git clone https://codeberg.org/YOUR_USERNAME/job-orchestrator.git
cd job-orchestrator

2. Set Up Development Environment

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install a C compiler (Debian/Ubuntu); SQLite is bundled
apt-get install build-essential

# Build
cargo build

# Run tests
cargo test

3. Create a Branch

git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix

Development Workflow

Making Changes

  1. Write code
  2. Add tests for new functionality
  3. Run tests: cargo test
  4. Run linter: cargo clippy -- -D warnings
  5. Format code: cargo fmt

Commit Messages

Follow conventional commits:

type(scope): description

[optional body]

Types:

  • feat: New feature
  • fix: Bug fix
  • docs: Documentation
  • refactor: Code refactoring
  • test: Adding tests
  • chore: Maintenance

Examples:

feat(quota): add per-service quota limits
fix(client): handle non-zero exit codes correctly
docs(readme): update quick start instructions

Pull Request Process

  1. Push your branch:

    git push origin feature/your-feature-name
    
  2. Open a Pull Request on Codeberg

  3. In the PR description (there is no template):

    • Describe your changes
    • Link related issues
    • Note any breaking changes
  4. Wait for review

    • CI must pass
    • Maintainer will review
    • Address feedback
  5. Merge!

Code Style

Rust Style

Follow standard Rust conventions:

  • Use cargo fmt for formatting
  • Use cargo clippy for linting
  • Prefer descriptive variable names
  • Add doc comments for public APIs
#![allow(unused)]
fn main() {
/// Uploads a job to the orchestrator.
///
/// # Arguments
///
/// * `files` - Files to upload
/// * `user_id` - User submitting the job
/// * `service` - Target service name
///
/// # Returns
///
/// The created job with its ID and initial status.
pub async fn upload_job(
    files: Vec<File>,
    user_id: i32,
    service: String,
) -> Result<Job, Error> {
    // Implementation
}
}

Error Handling

  • Use Result types, not panics
  • Provide context with error messages
  • Use ? operator for propagation
#![allow(unused)]
fn main() {
// Good
let file = File::open(path)
    .map_err(|e| Error::FileOpen { path: path.clone(), source: e })?;

// Avoid
let file = File::open(path).unwrap();
}

Testing Guidelines

Write Tests For

  • New functionality
  • Bug fixes (regression tests)
  • Edge cases

Test Structure

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_function_describes_behavior() {
        // Arrange
        let input = create_test_input();

        // Act
        let result = function_under_test(input);

        // Assert
        assert_eq!(result, expected);
    }
}
}

Async Tests

#![allow(unused)]
fn main() {
#[tokio::test]
async fn test_async_operation() {
    let result = async_function().await;
    assert!(result.is_ok());
}
}

Documentation

Code Documentation

Add doc comments for:

  • Public functions
  • Public structs
  • Public modules
#![allow(unused)]
fn main() {
/// A job represents a unit of work to be processed.
pub struct Job {
    /// Unique identifier for the job.
    pub id: i32,
    /// User who submitted the job.
    pub user_id: i32,
    // ...
}
}

mdbook Documentation

Documentation is in docs/src/. To preview:

# Install mdbook
cargo install mdbook

# Serve locally
cd docs
mdbook serve --open

Reporting Issues

Bug Reports

Include:

  • job-orchestrator version
  • Operating system
  • Steps to reproduce
  • Expected vs actual behavior
  • Logs if relevant

Feature Requests

Include:

  • Use case description
  • Proposed solution (if any)
  • Alternatives considered

Code of Conduct

Be respectful and constructive. We’re all here to build something useful together.

Questions?

License

Contributions are licensed under the MIT License.

Troubleshooting

Common issues and solutions for job-orchestrator.

Server Issues

Server Won’t Start

Symptom: Server fails to start, exits immediately

Possible Causes:

  1. Port already in use

    Error: Address already in use
    

    Solution:

    # Find process using the port
    lsof -i :5000
    # Kill it or use a different port
    PORT=5001 job-orchestrator server
    
  2. Database path not writable

    Error: unable to open database file
    

    Solution:

    # Check directory exists and is writable
    mkdir -p /opt/data
    chmod 755 /opt/data
    
  3. Missing service configuration

    The server starts without services but will return 422 Invalid service on every upload. Configure at least one service:

    export SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit
    export SERVICE_EXAMPLE_DOWNLOAD_URL=http://client:9000/retrieve
    export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill
    

Jobs Stuck in Queued

Symptom: Jobs stay in Queued status indefinitely

Possible Causes:

  1. Quota exhausted

    There are two independent limits — either can hold a job in Queued:

    • RUNS_PER_USER (default: 5) — user has too many active jobs for this service
    • MAX_RUNS (default: 10) — the service has reached its total concurrent job cap

    Wait for running jobs to complete, or increase the relevant limit.

  2. Slots held by stuck jobs

    Jobs whose process died are marked Lost and release their slot on their own (see Job Status is Lost). A job that stays Running while its client is unreachable still counts toward both limits. Find them and terminate them:

    sqlite3 /opt/data/db.sqlite "SELECT id, user_id, service, status FROM jobs WHERE status IN ('processing','submitted','running')"
    curl -X POST http://localhost:5000/terminate/<id>
    
  3. Service removed from the configuration

    Jobs queued for a service that is no longer configured (for example after a restart with different SERVICE_* variables) are skipped by the sender and stay Queued forever.

An unreachable client does not keep jobs Queued: the dispatch fails and the job becomes Failed.

Jobs Stuck in Submitted

Symptom: Jobs move to Submitted but never complete

Possible Causes:

  1. Client not executing jobs

    Check client logs for errors

    docker logs client
    
  2. run.sh hanging

    Your script may be waiting for input or stuck in a loop. Use /download_partial/{id} to look at its working directory

  3. Client lost the payload

    If the client was restarted mid-job, or its data was wiped, the server keeps polling and logs errors such as There was some error while trying to retrieve job .... The job status never changes on its own; terminate it and resubmit

Upload Is Rejected

Symptom: POST /upload returns an error instead of 202 Accepted. The message field in the JSON response says what was wrong.

Possible Causes:

  1. Missing required fields or run.sh (422)

    user_id (a number), service and a file named run.sh are required:

    curl -X POST http://localhost:5000/upload \
      -F "file=@run.sh" \
      -F "user_id=1" \
      -F "service=example"
    
  2. Unknown service (422)

    Service must be configured on server, and the service field must be the lowercase name (example, not EXAMPLE):

    export SERVICE_EXAMPLE_UPLOAD_URL=...
    

    Service names cannot contain underscores; see Server Configuration.

  3. Upload too large (413)

    The default limit is 400MB for all files combined. Check file sizes, or raise MAX_UPLOAD_SIZE on the server.

  4. Wrong Content-Type (415) or unreadable body (400)

    The request must be multipart/form-data with a boundary. Let your HTTP client build it (curl -F, or FormData in a browser) instead of setting the Content-Type header by hand, which drops the boundary.

Client Issues

Client Not Receiving Jobs

Symptom: Client running but no jobs arrive

Check:

  1. Network connectivity

    # From server, can you reach client?
    curl http://client:9000/health
    
  2. Firewall rules

    # Client port must be accessible from server
    iptables -L -n | grep 9000
    
  3. Docker networking

    # Containers must be on same network
    # The shipped compose file uses the "api" network for server-client traffic
    docker network inspect job-orchestrator_api
    

Termination Fails

Symptom: Job remains Running or Submitted after termination request

Possible Causes:

  1. Job not in a terminable state

    • Termination is not gated by job status: the server attempts it for a job in any status
    • Queued jobs have no client payload yet (dest_id is still 0), so the client answers 404, the server returns 500, and the job stays Queued
    • Already completed or failed jobs: termination “succeeds” and the job becomes Killed, after which /download/{id} no longer returns the ZIP
    • Solution: Check job status first with GET /download/{id}
  2. Client unreachable

    • Server cannot reach client to send termination request
    • Verify client connectivity:
    curl http://client:9000/health
    
  3. Process already terminated

    • The process may have finished between the request and processing
    • The job is still marked Killed in that case
  4. Missing TERMINATE_URL configuration

    • Server needs SERVICE_<NAME>_TERMINATE_URL configured
    • Solution: Add to server configuration:
    export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill
    

Jobs Stuck in Locked

Symptom: Job stays in Locked status indefinitely

Possible Causes:

Locked only lasts while the server waits for the client’s answer to a termination request. It stays longer when:

  1. The client accepted the connection but does not answer

    • The server sets no HTTP timeout, so it waits as long as the connection stays open
    • Check client logs and health
  2. The server restarted during a termination

    • Nothing picks up Locked jobs afterwards, so they stay Locked permanently
    • They are not polled and do not count toward quotas
    • Fix the status in the database manually, or send another POST /terminate/{id}

Jobs Stuck in Prepared

Symptom: Payloads stay in Prepared status

Possible Causes:

  1. Runner task not running

    Check client logs, may need restart

  2. Permission issues

    Client working directory may have permission issues

A script that fails validation does not stay Prepared: it becomes Invalid (see below).

Job Status is Invalid

Symptom: Job status is Invalid — the client rejected the script before running it

Possible Causes:

  1. Missing exit trap — script must contain:

    trap 'echo $? > .orchestrator.exit' EXIT
    
  2. Script too large — maximum script size is 20 MiB

  3. Script not valid UTF-8

  4. Script contains a blocked pattern (the client log names the pattern; false positives are possible, for example rm with any relative path containing /). See Script Validation for the full list (network tools, destructive commands, privilege escalation, etc.)

Invalid jobs are cleaned up after MAX_AGE like any other terminal state.

Job Status is Lost

Symptom: Job status is Lost

The job’s process disappeared without writing .orchestrator.exit. The client log has a warning such as payload 12 lost: pid 4321 is gone and no exit file at "/opt/data/12".

Possible Causes:

  1. Client restarted or crashed while the job ran (for example a container restart). Drain clients before restarting them, see Production
  2. The bash process running run.sh was killed with SIGKILL (for example kill -9, or the OOM killer picking it). SIGKILL cannot be trapped, so the exit code is never written

Solution: Lost is terminal. Use /download_partial/{id} to look at what the job wrote, then resubmit it.

Execution Fails

Symptom: Jobs complete but with Failed status

Check:

  1. Exit code

    run.sh must exit with code 0 for success:

    #!/bin/bash
    # Your commands here
    exit 0  # Explicit success
    
  2. Script errors

    Check output files for error messages. /download/{id} does not return files for Failed jobs; use /download_partial/{id}

  3. Missing dependencies

    Your script may need tools not available in the container

Database Issues

Database Locked

Symptom: “database is locked” errors

Causes: Multiple processes accessing SQLite

Solution:

  • Ensure only one server instance runs
  • Check for zombie processes
  • Restart server

Database Corrupted

Symptom: Strange errors, missing data

Solution:

  1. Stop server

  2. Backup current database

  3. Run integrity check:

    sqlite3 db.sqlite "PRAGMA integrity_check;"
    
  4. If corrupted, restore from backup or delete and restart

Out of Disk Space

Symptom: “disk full” errors

Solution:

  1. Check disk usage:

    df -h
    
  2. Clean old jobs:

    # Reduce MAX_AGE and restart
    export MAX_AGE=3600  # 1 hour
    
  3. Manually clean data directory

Docker Issues

Container Exits Immediately

Check logs:

docker logs container_name

Common causes:

  • Missing environment variables
  • Port conflicts
  • Permission issues

Cannot Connect Between Containers

Ensure same network:

services:
  server:
    networks:
      - app-network
  client:
    networks:
      - app-network

networks:
  app-network:

Use service names, not localhost:

# Wrong
SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit

# Correct
SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit

Volume Permission Issues

Symptom: Permission denied when writing to volumes

Solution:

services:
  server:
    user: "1000:1000"  # Match host user
    volumes:
      - ./data:/opt/data

Or fix permissions:

sudo chown -R 1000:1000 ./data

Performance Issues

Slow Job Processing

Possible Causes:

  1. Slow database

    • Use SSD storage for database
    • Run VACUUM periodically
  2. Network latency

    • Place server and clients on same network
    • Check for packet loss
  3. Client overloaded

    • Add more clients
    • Reduce RUNS_PER_USER

High Memory Usage

Server:

  • /download/{id} reads the whole output.zip into memory before sending it, so large results cause spikes
  • Uploads are streamed to disk and do not

Client:

  • /submit holds the whole upload in memory, and /retrieve//retrieve_partial build or read the whole archive in memory
  • The database is file-backed and is not a significant factor

Disk Usage Growing

Check:

du -sh /opt/data/*

Solutions:

  • Reduce MAX_AGE (the cleanup interval itself is fixed at 60 seconds)
  • Archive old results externally

Getting Help

If you can’t resolve an issue:

  1. Check logs for specific error messages
  2. Search existing issues: Codeberg Issues
  3. Open new issue with:
    • Version
    • Configuration
    • Steps to reproduce
    • Logs

See Also