Introduction
job-orchestrator is an asynchronous job orchestration system for managing and distributing computational workloads across heterogeneous computing resources with intelligent quota-based load balancing.
It serves as a middleware layer that connects web applications to computing resources: users submit files and a run.sh script via a REST API; the server queues and routes jobs to client nodes that execute them; results are returned as a ZIP archive.
It is domain-agnostic: any workload that can be expressed as a run.sh script plus input files can be orchestrated, whether that is a scientific simulation, a data-processing pipeline, or a media conversion task.
The project was originally developed as the job backend for a production web portal that exposes research software as web services.
Key Features
- Asynchronous Job Management: Built with Rust and Tokio for high-performance async operations
- Quota-Based Load Balancing: Per-user, per-service quotas prevent resource exhaustion
- Dual-Mode Architecture: Runs as server (job orchestration) or client (job execution)
- RESTful API: Simple HTTP interface for job submission, retrieval, and cancellation
- Automatic Cleanup: Configurable retention policies for completed jobs
- Job Termination: Cancel running jobs via API endpoints
Use Cases
job-orchestrator is designed for scenarios requiring:
- Scientific Computing Workflows: Distribute simulations and analyses across clusters
- Multi-Tenant Systems: Fair resource allocation with per-user quotas
- Heterogeneous Computing: Route jobs to appropriate backends (local, HPC, cloud)
- Web Platforms with Heavy Backends: Decouple the frontend from compute infrastructure
- Batch Processing: Handle high-throughput job submissions with automatic queuing
Get Started
- Quick Start — up and running with Docker Compose in minutes
- Your First Job — submit and retrieve a job
- Architecture Overview — understand how the system works
Project Status
Current State: Production-ready with server/client architecture
Planned Features:
- DIRAC Interware integration
- SLURM direct integration
- Enhanced monitoring and metrics
- Job priority queues
- Advanced scheduling policies
Getting Help
- API Documentation: Available via Swagger UI at
/swaggerwhen running - Issues: Codeberg Issues
- Email: Rodrigo V. Honorato rvhonorato@protonmail.com
License
MIT License - see LICENSE for details.
Parts of this documentation were automatically generated with the assistance of AI tools and human reviewed for accuracy.
Installation
There are several ways to install job-orchestrator depending on your needs.
From crates.io
The easiest way to install job-orchestrator is via Cargo:
cargo install job-orchestrator
From Source
Clone the repository and build:
git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
cargo build --release
The binary will be available at target/release/job-orchestrator.
Using Docker
Pull the pre-built image:
docker pull codeberg.org/rvhonorato/job-orchestrator:latest
Or build locally:
docker build -t job-orchestrator .
# example client (job-orchestrator + gdock), built on top of the local image
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/
Prerequisites
For Building from Source
- Rust: 1.88 or later (edition 2024, uses
letchains) - C compiler: SQLite is compiled from source and linked statically (via
sqlx’s bundled SQLite), so no system SQLite library is needed
On Debian/Ubuntu:
apt-get install build-essential
For Running
- Linux or macOS: client mode runs jobs with
bashand manages them with thekillcommand, so both must be onPATH. Windows is not supported for client mode - Filesystem access: Write permissions for database and job storage directories
Verifying Installation
After installation, verify it works:
job-orchestrator --help
You should see the server and client subcommands listed. There is no --version flag.
Next Steps
- Quick Start - Get running with Docker Compose
- Your First Job - Submit and retrieve a job
Quick Start
The fastest way to get job-orchestrator running is with Docker Compose.
Running with Docker Compose
git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
docker compose up --build
This starts:
- Orchestrator server on port 5000 (exposed to host)
- Example client on an internal network (not exposed to host)
Verify It’s Running
Check the server is responding:
curl http://localhost:5000/health
You should receive a health status response.
Access the API Documentation
Open your browser and navigate to:
http://localhost:5000/swagger
This provides interactive API documentation where you can explore and test all endpoints.
What’s Next?
Now that you have job-orchestrator running, proceed to Your First Job to learn how to submit and retrieve jobs.
Stopping the Services
To stop the services:
docker compose down
To stop and remove all data (volumes):
docker compose down -v
Your First Job
This guide walks you through submitting and retrieving your first job.
Prerequisites
Make sure you have job-orchestrator running. See Quick Start if you haven’t set it up yet.
Understanding Jobs
A job in job-orchestrator consists of:
- Files: One or more files to be processed
- A
run.shscript: The entry point that gets executed - User ID: Identifies who submitted the job (for quota tracking)
- Service: Which service/backend should process this job
Creating a Simple Job
Create a simple run.sh script:
cat > run.sh << 'EOF'
#!/bin/bash
# Required: Capture exit code for the orchestrator client
trap 'echo $? > .orchestrator.exit' EXIT
echo "Hello from job-orchestrator!" > output.txt
echo "Processing complete at $(date)" >> output.txt
EOF
chmod +x run.sh
Submitting the Job
Submit the job using curl:
curl -X POST http://localhost:5000/upload \
-F "file=@run.sh" \
-F "user_id=1" \
-F "service=example" | jq
You’ll receive a response like:
{
"id": 1,
"status": "Queued",
"message": "Job successfully uploaded"
}
Note the id field - you’ll need this to check status and download results.
Checking Job Status
Check the job status via GET request:
curl http://localhost:5000/download/1
If the job is not yet completed, you’ll get a JSON response:
{
"id": 1,
"status": "Submitted",
"message": ""
}
The status field will be one of: Queued, Processing, Submitted, Prepared, Running, Completed, Failed, Invalid, Cleaned, Unknown, Locked, Killed, or Lost. See Job States for descriptions of each.
Downloading Results
Once the status is Completed, the same endpoint returns the ZIP file:
curl -o results.zip http://localhost:5000/download/1
Extract and view:
unzip results.zip
cat output.txt
You should see:
Hello from job-orchestrator!
Processing complete at <timestamp>
The archive contains the whole job directory: your input files, everything
run.sh wrote, and the .orchestrator.exit file.
Downloading Partial Results
/download/{id} only returns a ZIP for Completed jobs. For any other status,
including Failed, it returns the JSON status. To get the files of a failed,
stuck, or still-running job, use /download_partial/{id}:
# Get current state of job even if incomplete
curl -o partial_results.zip http://localhost:5000/download_partial/1
Canceling a Job
If you need to cancel a running job, use the terminate endpoint:
# Cancel job with ID 1
curl -X POST http://localhost:5000/terminate/1
Response on success:
{
"id": 1,
"status": "Unknown",
"message": "job terminated"
}
The
statusfield in the terminate response is not meaningful. UseGET /download/{id}to confirm the job’s actual status after termination.
Notes:
- The server sends a termination request to the client, which sends
SIGTERMto the process - Terminated jobs will have status
Killed Queuedjobs cannot be cancelled: they have not reached a client yet, so the request fails with500, the job staysQueued, and it will still run later- Terminating a job that already finished still “succeeds” and changes its status to
Killed. After that,/download/{id}returns JSON instead of the ZIP, so the results are only reachable through/download_partial/{id}(until the client cleans them up)
Security
Script Execution
The client executes user-submitted run.sh scripts with full privileges.
While a script validator rejects obviously dangerous patterns (destructive
commands, network exfiltration tools, reverse shells, privilege escalation),
it is NOT a sandbox and can be bypassed by determined actors.
Important: Always run the client inside a container with:
- Resource limits (CPU, memory, PIDs)
- Read-only root filesystem
- Drop all capabilities (
--cap-drop=ALL) - Internal network only (no external exposure)
- Non-root user
See Production Deployment for details.
Path Traversal Protection
Uploaded filenames are reduced to their base name, so ../../etc/passwd is
stored as passwd inside the job directory. When results are zipped, any entry
that resolves outside the job directory (such as a symlink to /etc) makes
the archive fail instead of being included. See
Path Traversal Protection.
Important Notes
The run.sh Script
- Must be named exactly
run.sh - Is run as
bash run.shinside the job directory, so it does not need the executable bit or a shebang - Exit code
0indicates success - Non-zero exit code indicates failure
- All output files in the working directory are included in results
Important: The script MUST include the following trap at the beginning. The client checks for it before running the script and marks the job Invalid if it is missing:
#!/bin/bash
trap 'echo $? > .orchestrator.exit' EXIT
# ... rest of your script
This ensures that when your run.sh exits, it writes the exit code to .orchestrator.exit, which the client’s Updater task reads to determine if the job completed successfully or failed.
File Size Limits
The server accepts request bodies up to 400MB by default (all files in one upload combined). Change it with the MAX_UPLOAD_SIZE environment variable, in bytes. Larger uploads are rejected with 413.
User ID Parameter
Jobs can be submitted with a user_id for quota management:
curl -X POST http://localhost:5000/upload \
-F "file=@run.sh" \
-F "user_id=1" \
-F "service=example"
user_id is required and must be an integer; the server returns 422 if it is missing or not a number, if service is unknown, or if no run.sh was uploaded. service must match a configured service name exactly, in lowercase (example, not EXAMPLE).
Job Retention
Job directories are deleted once they are older than MAX_AGE (default: 10 days), and the job is marked Cleaned. Age is measured from the directory’s last modification time, not from when the job finished, and the rule applies to jobs in any status. Make sure to download your results before they expire.
Next Steps
- Learn about the Job Lifecycle
- Configure Quotas for your users
- Set up Production Deployment
Partial Downloads and Debugging
This guide demonstrates how to use the /download_partial and /retrieve_partial endpoints to debug stuck or incomplete jobs.
Use Cases
Use partial download endpoints when:
- A job appears stuck in
SubmittedorRunningstatus - A job ended as
Failedand you want its output (/downloadonly returns ZIPs forCompletedjobs) - You need to inspect the current working directory
- Debugging why a job isn’t progressing
- Verifying intermediate outputs of long-running jobs
Server-Side: /download_partial/
Example: Checking Incomplete Job
# Download current state of job ID 1
curl -o partial_results.zip http://localhost:5000/download_partial/1
# Extract and inspect contents
unzip -l partial_results.zip
unzip partial_results.zip
# View specific files
cat intermediate_output.txt
Client-Side: /retrieve_partial/
This endpoint is called internally by the server’s /download_partial endpoint.
It provides the same functionality at the client level.
The server does not have a separate setting for this URL. It takes the
service’s DOWNLOAD_URL and replaces its last path segment with
retrieve_partial (http://client:9000/retrieve becomes
http://client:9000/retrieve_partial). If you put a proxy in front of the
client, make sure that path is routed too.
In the shipped Docker Compose setup the client has no ports: mapping — it’s
only reachable from the server over the internal network (e.g.
http://client:9000), not from the host. The example below assumes you’re
running the client directly (outside Docker Compose) or calling it from
another container on the same network.
Example (Client Developer)
# Get partial results directly from client
curl -o partial_results.zip http://localhost:9000/retrieve_partial/1
Comparison: Full vs Partial Download
| Endpoint | Returns | Use Case |
|---|---|---|
/download/{id} | Final results only | Normal completion |
/download_partial/{id} | Current state always | Debugging, monitoring |
Combined Example Script
#!/bin/bash
SERVER_URL="http://localhost:5000"
# Submit job and parse the returned ID
echo "Submitting job..."
RESPONSE=$(curl -s -X POST "${SERVER_URL}/upload" \
-F "file=@run.sh" \
-F "user_id=1" \
-F "service=example")
echo "$RESPONSE"
JOB_ID=$(echo "$RESPONSE" | jq -r '.id')
if [ -z "$JOB_ID" ] || [ "$JOB_ID" = "null" ]; then
echo "Failed to get job ID from response"
exit 1
fi
echo "Job ID: $JOB_ID"
# Poll until complete, saving partial snapshots for inspection
echo "Monitoring job..."
for i in $(seq 1 10); do
sleep 10
CT=$(curl -s -o response.tmp -w '%{content_type}' "${SERVER_URL}/download/${JOB_ID}")
if [[ "$CT" == application/zip* ]]; then
mv response.tmp final_results.zip
echo "Job completed, results saved to final_results.zip"
exit 0
fi
STATUS=$(jq -r '.status' response.tmp)
case "$STATUS" in
Failed|Invalid|Killed|Lost|Cleaned)
echo "Job ended with status $STATUS"
curl -s -o final_partial.zip "${SERVER_URL}/download_partial/${JOB_ID}"
exit 1
;;
esac
echo "Check $i: status=$STATUS, saving partial snapshot..."
curl -s -o "partial_${i}.zip" "${SERVER_URL}/download_partial/${JOB_ID}"
unzip -l "partial_${i}.zip" 2>/dev/null | head -20
done
echo "Job did not complete within the polling window"
exit 1
Limitations
Queuedjobs have not reached a client yet, so/download_partialreturns404(“Job payload not found on client”)- Until
run.shwrites something, the ZIP only contains the input files - After the client’s cleaner has removed the payload directory, the request fails
- Partial downloads of running jobs may be incomplete or inconsistent
- Use partial downloads for debugging, not for production result retrieval
Security Notes
- Building the ZIP fails if any entry in the payload directory resolves outside it (for example, a symlink created by
run.shthat points to/etc), so such files cannot be exfiltrated this way - See Path Traversal Protection for details
Architecture Overview
job-orchestrator uses a distributed architecture with a central server coordinating job execution across multiple client nodes.
High-Level Architecture
flowchart TB
subgraph Tasks["Background Tasks"]
Sender["Sender<br>500ms"]
Getter["Getter<br>500ms"]
Cleaner["Cleaner<br>60s"]
end
subgraph Server["Orchestrator Server"]
API["REST API<br>upload/download/download_partial/terminate"]
DB[("SQLite<br>Persistent")]
FS[/"Filesystem<br>Job Storage"/]
Tasks
Queue["Queue Manager<br>Quota Enforcement"]
end
subgraph Client["Client Service"]
ClientAPI["REST API<br>submit/retrieve/retrieve_partial/kill/load"]
ClientDB[("SQLite<br>Persistent")]
ClientFS[/"Working Dir"/]
Runner["Runner Task<br>500ms"]
Updater["Updater Task<br>500ms"]
ClientCleaner["Cleaner Task<br>60s"]
Executor["Bash Executor<br>run.sh"]
end
User(["User/Web App"]) -- POST /upload --> API
User -- GET /download/:id --> API
User -- GET /download_partial/:id --> API
API -- GET /retrieve_partial/:id --> ClientAPI
API --> DB & FS
DB --> Queue
Queue --> Sender
Sender -- POST /submit --> ClientAPI
Getter -- GET /retrieve/:id --> ClientAPI
Getter --> FS
Cleaner --> DB & FS
ClientAPI --> ClientDB
ClientDB --> Runner
Runner --> Executor
Executor --> ClientFS
ClientDB --> Updater
Updater --> ClientDB
ClientDB --> ClientCleaner
ClientCleaner --> ClientFS
Components
Orchestrator Server
The central server manages:
- REST API: Handles job uploads and result downloads from users
- SQLite Database: Persistent storage for job metadata and status
- Filesystem Storage: Stores uploaded files and downloaded results
- Queue Manager: Enforces per-user quotas and manages job distribution
- Background Tasks: Automated processes for job distribution, result retrieval, and cleanup
Client Service
Each client node handles:
- REST API: Receives jobs from server, returns results
- SQLite Database: Payload tracking. Configured with the same
DB_PATHvariable as the server (file-backed by default); give each process its own file - Working Directory: Temporary storage for job execution
- Runner Task: Picks up new payloads and starts them
- Updater Task: Watches running payloads for their exit code, a kill request, or a lost process
- Bash Executor: Runs the
run.shscript for each job
Background Tasks
Server Tasks
| Task | Interval | Purpose |
|---|---|---|
| Sender | 500ms | Picks up queued jobs, enforces quotas, dispatches to clients |
| Getter | 500ms | Syncs job status from clients and downloads results when complete |
| Cleaner | 60s | Deletes job directories older than MAX_AGE and marks them Cleaned (records are kept) |
Client Tasks
| Task | Interval | Purpose |
|---|---|---|
| Runner | 500ms | Validates and starts prepared payloads |
| Updater | 500ms | Monitors running processes, handles exit codes, killed payloads and lost processes |
| Cleaner | 60s | Deletes payload directories older than MAX_AGE, whatever their status |
Data Flow
- User submits files via
POST /upload - Server stores files and creates job record (status:
Queued) - Sender task picks up job, checks quotas, sends it to the client configured for its service
- Client receives job, stores as payload (status:
Prepared) - Runner task starts
run.sh; the Updater setsCompletedorFailedfrom its exit code - Getter task retrieves results, stores locally
- User downloads results via
GET /download/:id - Cleaner task removes job after retention period
Termination Flow
- User sends termination request via
POST /terminate/:id - Server locks job (status:
Locked) and forwards to client - Client receives request via
POST /kill/:id, sends SIGTERM to the process and sets the payload’s killed flag - Server sets the job to
Killedas soon as the client answers with success (or restores the previous status if it fails) - Server returns termination confirmation to user
- Client Updater later marks the payload
Killed
Job Lifecycle
Understanding the job lifecycle is essential for working with job-orchestrator effectively.
Lifecycle Sequence
sequenceDiagram
participant User
participant Server
participant Client
participant Executor
User->>Server: POST /upload (files, user_id, service)
Server->>Server: Store job (status: Queued)
Server-->>User: Job ID
Note over Server: Sender task (500ms interval)
Server->>Server: Update status: Processing
Server->>Client: POST /submit (job files)
Client->>Client: Store payload (status: Prepared)
Client-->>Server: Payload ID
Server->>Server: Update status: Submitted
Note over Client: Runner task (500ms interval)
Client->>Client: Update status: Running
Client->>Executor: Execute run.sh (spawns process)
Note over Client: Updater task (500ms interval)
Executor-->>Client: .orchestrator.exit written
Client->>Client: Update status: Completed
Note over Server: Getter task (500ms interval)
Server->>Client: GET /retrieve/:id
Client-->>Server: JSON status (copied to job) or, once Completed, ZIP results
Server->>Server: Store output.zip, status: Completed
User->>Server: GET /download/:id
Server-->>User: results.zip
Note over Server: Cleaner task (60s interval)
Server->>Server: Delete job dirs older than MAX_AGE, status: Cleaned
Note over User,Server: Job Termination Flow
User->>Server: POST /terminate/:id
Server->>Server: Update status: Locked
Server->>Client: POST /kill/:id
Client->>Executor: Send SIGTERM
Client->>Client: Set killed flag
Client-->>Server: 200 OK
Server->>Server: Update status: Killed
Server-->>User: Termination confirmed
Note over Client: Updater marks the payload Killed on its next pass
Job States
stateDiagram-v2
[*] --> Queued: submitted
Queued --> Processing: sender picks up
Processing --> Submitted: sent to client
Processing --> Failed: dispatch error
Submitted --> Prepared: client received job
Prepared --> Running: execution started
Running --> Completed: exit 0
Running --> Failed: non-zero exit
Running --> Invalid: bad script
Running --> Lost: process gone, no exit file
Submitted --> Killed: terminated early
Prepared --> Killed: terminated early
Running --> Killed: terminated
Completed --> Killed: terminated after finishing
Completed --> Cleaned: MAX_AGE
Failed --> Cleaned: MAX_AGE
Invalid --> Cleaned: MAX_AGE
Killed --> Cleaned: MAX_AGE
Lost --> Cleaned: MAX_AGE
Cleaned --> [*]
The server copies the client’s payload status into the job on every Getter pass, which is why client-side states (
Prepared,Running,Invalid) also show up on the server.
LockedandUnknownare not shown above; see the table below.Cleanedcan happen from any state, includingQueuedandRunning, because the cleaner only looks at the age of the job directory.
State Descriptions
| State | Description |
|---|---|
| Queued | Job received and waiting for dispatch |
| Processing | Server is sending job to a client |
| Submitted | Job successfully sent to client, awaiting execution |
| Running | Client is actively executing the job |
| Completed | Job finished successfully, results available |
| Prepared | Client stored the payload and is about to run it (copied from the client) |
| Failed | Terminal. run.sh exited non-zero, the client could not start it, or dispatch to the client failed |
| Invalid | Terminal. Client rejected the script (run.sh missing, unsafe pattern, missing trap, too large, not UTF-8) |
| Unknown | Fallback when a status string cannot be parsed, and the placeholder in error responses. Not a retry state |
| Locked | Set while a termination request is in flight. Restored or replaced by Killed when the request returns |
| Killed | Terminal. Job was terminated via POST /terminate/{id} |
| Lost | Terminal. The process disappeared without writing .orchestrator.exit (client restart or crash, or bash itself killed with SIGKILL). See Lost Processes |
| Cleaned | Terminal. Job directory deleted after MAX_AGE. The database record is kept |
Lifecycle Stages
1. Submission
User uploads files via POST /upload with:
- One or more files (including
run.sh) user_id- identifies the submitting userservice- which backend should process this job
The server:
- Validates the request
- Creates a unique directory for the job
- Stores all uploaded files (by base name only, so subdirectories are flattened)
- Creates a database record with status
Queued - Returns the job ID to the user
2. Queuing & Quota Check
The Sender background task (runs every 500ms) loads its next batch of jobs via a round-robin fair-share scheduler:
- Groups all
Queuedjobs byservice, then byuser_idwithin each service - For each service, computes available service-wide slots:
MAX_RUNS(default 10) minus jobs alreadyProcessing/Submitted/Runningfor that service - For each user in that service, computes available per-user slots:
RUNS_PER_USER(default 5) minus that user’s own in-flight job count - Round-robins across users with available slots — takes one queued job per user per pass — until the service-wide slot budget is exhausted or no user has both a queued job and a free slot
- Jobs picked this way are marked
Processing; everything else remainsQueuedfor the next cycle
Quota is enforced at two levels — per-user (RUNS_PER_USER) and per-service (MAX_RUNS) — and the round-robin means no single user can starve others of a service’s shared capacity.
3. Distribution
For jobs in Processing status:
- Server streams every file in the job directory as one multipart request
- Sends it to the service’s
UPLOAD_URL(the client’sPOST /submit). Each service has exactly one client URL; there is no load-based client selection - On success: updates status to
Submitted, stores client’s payload ID asdest_id - On failure (client unreachable or non-2xx response): updates status to
Failed. There is no retry
4. Execution
On the client side:
- Runner task (every 500ms) finds payloads in
Preparedstatus and marks themRunning - Validates
run.sh(see Script Validation). A rejected script makes the payloadInvalid - Spawns
bash run.shin the payload directory and stores its PID. If the process cannot be started, the payload becomesFailed - Updater task (every 500ms) checks
Runningpayloads. When.orchestrator.exitappears it setsCompleted(exit code0) orFailed(anything else). If the killed flag is set it setsKilled. If the process is gone and never wrote the exit file it setsLost
5. Retrieval
The Getter background task (runs every 500ms):
- Finds jobs in
Submitted,Prepared, orRunningstatus (at most 10 requests in flight) - Requests the payload from the client via
GET /retrieve/:id - If the client answers with JSON, copies the payload status into the job (
Running,Failed,Invalid, …) - If the client answers with a ZIP, saves it as
output.zipin the job directory and setsCompleted
Once a job reaches a status outside that list, the Getter stops polling it.
6. Download
User can now:
- Check status via
GET /download/:id(returns JSON with job state when not completed) - Download results via
GET /download/:id(returns ZIP when completed) - Results are returned as a ZIP archive
7. Cleanup
The Cleaner background task (runs every 60s):
- Finds job directories whose last modification time is older than
MAX_AGE, whatever the job status - Deletes the directory
- Updates the job status to
Cleaned. The database record is never deleted
The client runs the same cleaner over its own payload directories.
Error Handling
Client Unreachable
If the server cannot reach a client during distribution:
- Job status changes to
Failed - Job will not be retried automatically
- User can resubmit if needed
Execution Failure
If run.sh exits with non-zero code:
- Payload status changes to
Failed - The server copies the
Failedstatus but does not fetch any output GET /download/{id}keeps returning JSON; useGET /download_partial/{id}to get the files
Retrieval Failure
If the server cannot retrieve the payload (client down, 404, network error):
- The job status is left unchanged and the error is logged
- The Getter retries every 500ms, indefinitely. There is no timeout
Lost Processes
If the client is restarted or crashes while a job runs (for example a container restart), the process is gone but no .orchestrator.exit file was written:
- The client’s Updater sees that the PID is gone and that there is no exit file, marks the payload
Lostand logs a warning (payload <id> lost: pid <pid> is gone and no exit file at ...) - The server copies
Lostinto the job on its next Getter pass, which frees itsRUNS_PER_USERandMAX_RUNSslot Lostis terminal: resubmit the job./download_partial/{id}still returns whatever the job wrote before it died
The trap writes .orchestrator.exit before the process ends, so a job that exits normally always leaves the file behind. The Updater checks for the file again after seeing the PID gone, so a job that finishes between two checks is still marked Completed or Failed, not Lost.
A server older than the client cannot parse Lost: it logs a retrieval error and keeps the job Running. Upgrade the server before, or together with, the clients.
Timing Considerations
| Event | Typical Duration |
|---|---|
| Upload to Queued | Immediate |
| Queued to Processing | Up to 500ms (+ quota wait) |
| Processing to Submitted | Depends on file size and network |
| Submitted to Completed | Depends on job execution time |
| Completed to Cleaned | MAX_AGE after the job directory was last modified (default: 10 days) |
Server & Client Modes
job-orchestrator provides both server and client functionality in a single binary. The mode is picked with a subcommand (server or client); everything else is configured through environment variables.
Dual-Mode Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Same Binary │
│ │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ Server Mode │ │ Client Mode │ │
│ │ │ │ │ │
│ │ - Job orchestration│ │ - Job execution │ │
│ │ - Quota management │ │ - Result packaging │ │
│ │ - Persistent DB │ │ - Persistent DB │ │
│ │ - User-facing API │ │ - Server-facing API│ │
│ └─────────────────────┘ └─────────────────────┘ │
│ │
│ job-orchestrator server job-orchestrator client │
└─────────────────────────────────────────────────────────────────┘
Server Mode
The server is the central orchestrator that:
- Receives job submissions from users/applications
- Manages job queues and enforces quotas
- Distributes jobs to available clients
- Retrieves results and serves them to users
- Handles cleanup of expired jobs
Starting the Server
PORT=5000 job-orchestrator server
Server Responsibilities
| Component | Purpose |
|---|---|
| REST API | Handle /upload and /download requests |
| Queue Manager | Enforce per-user, per-service quotas |
| Sender Task | Dispatch jobs to clients |
| Getter Task | Retrieve completed results |
| Cleaner Task | Remove expired jobs |
| SQLite DB | Persistent job tracking |
Server API Endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/upload | POST | Submit new job |
/download/:id | GET | Get results or status |
/download_partial/:id | GET | Get current job state as zip, regardless of completion |
/terminate/:id | POST | Cancel a running job |
/health | GET | Health check |
/swagger | GET | API documentation |
Client Mode
The client executes jobs on behalf of the server:
- Receives job payloads from the server
- Executes the
run.shscript - Packages results for retrieval
- Reports CPU load via
/load(the server does not use it yet)
Starting the Client
PORT=9000 job-orchestrator client
Client Responsibilities
| Component | Purpose |
|---|---|
| REST API | Handle /submit and /retrieve requests |
| Runner Task | Execute prepared payloads |
| Bash Executor | Run run.sh scripts |
| SQLite DB | Payload tracking (same DB_PATH-backed store as the server) |
Client API Endpoints
| Endpoint | Method | Purpose |
|---|---|---|
/submit | POST | Receive job from server |
/retrieve/:id | GET | Return completed results |
/retrieve_partial/:id | GET | Return current payload state as zip, regardless of completion |
/kill/:id | POST | Terminate a running payload |
/load | GET | Report CPU usage |
/health | GET | Health check |
Communication Flow
User Server Client
│ │ │
│──POST /upload────▶│ │
│◀─── Job ID ───────│ │
│ │ │
│ │──POST /submit──────────▶│
│ │◀─── Payload ID ─────────│
│ │ │
│ │ ┌────┴────┐
│ │ │ Execute │
│ │ │ run.sh │
│ │ └────┬────┘
│ │ │
│ │──GET /retrieve/:id─────▶│
│ │◀─── results.zip ────────│
│ │ │
│──GET /download/:id▶│ │
│◀─── results.zip ──│ │
│ │ │
│──POST /terminate/:id───▶│ │
│ │──POST /kill/:id─────────▶│
│ │◀─── kill signal ────────│
Deployment Patterns
Single Machine (Development)
Both server and client on the same machine:
# Terminal 1
PORT=5000 DB_PATH=./server.sqlite DATA_PATH=./server-data \
SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit \
SERVICE_EXAMPLE_DOWNLOAD_URL=http://localhost:9000/retrieve \
SERVICE_EXAMPLE_TERMINATE_URL=http://localhost:9000/kill \
job-orchestrator server
# Terminal 2
PORT=9000 DB_PATH=./client.sqlite DATA_PATH=./client-data job-orchestrator client
Give each process its own DB_PATH and DATA_PATH. With the defaults, both would use ./db.sqlite and ./data in the current directory, and each cleaner would trip over the other’s directories. example/start_example.sh sets this up for you.
Distributed (Production)
Server on one machine, clients on compute nodes:
┌─────────────┐
│ Server │
│ (port 5000)│
└──────┬──────┘
│
┌────────────────┼────────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ Client 1 │ │ Client 2 │ │ Client 3 │
│ (compute-1) │ │ (compute-2) │ │ (compute-3) │
└─────────────┘ └─────────────┘ └─────────────┘
Multi-Service Setup
Different clients for different services:
# Server configuration
SERVICE_EXAMPLE_UPLOAD_URL: http://client-example:9000/submit
SERVICE_EXAMPLE_DOWNLOAD_URL: http://client-example:9000/retrieve
SERVICE_EXAMPLE_TERMINATE_URL: http://client-example:9000/kill
SERVICE_EXAMPLE_RUNS_PER_USER: 5
SERVICE_EXAMPLE_MAX_RUNS: 10
SERVICE_SIMULATE_UPLOAD_URL: http://client-simulate:9001/submit
SERVICE_SIMULATE_DOWNLOAD_URL: http://client-simulate:9001/retrieve
SERVICE_SIMULATE_TERMINATE_URL: http://client-simulate:9001/kill
SERVICE_SIMULATE_RUNS_PER_USER: 5
SERVICE_SIMULATE_MAX_RUNS: 10
Database Differences
Server Database (Persistent)
- Uses SQLite file on disk
- Survives restarts
- Stores complete job history
- Location configured via
DB_PATH
Client Database
- Uses the same
DB_PATH-configured SQLite backend as the server: file-backed by default, survives restarts - Only becomes in-memory if you explicitly set
DB_PATH=:memory:for the client process - Keeps a record of every payload it has received; cleaned payloads stay in the table with status
Cleaned
When to Scale
Add More Clients When:
- Job queue is consistently backing up
- Execution time is the bottleneck
- You have available compute resources
Scale Server When:
- Upload/download becomes slow
- Many concurrent users
- Database queries become slow
See Also
Server Configuration
The orchestrator server is configured primarily through environment variables.
Environment Variables
Core Settings
| Variable | Default | Description |
|---|---|---|
PORT | 5000 | HTTP port the server listens on |
DB_PATH | ./db.sqlite | Path to SQLite database file |
DATA_PATH | ./data | Directory for job file storage |
MAX_AGE | 864000 | Job retention time in seconds (default: 10 days) |
MAX_UPLOAD_SIZE | 419430400 | Maximum /upload request body in bytes (default: 400MB). Larger uploads get 413 |
Service Configuration
For each service you want to support, configure these variables:
| Variable Pattern | Description |
|---|---|
SERVICE_<NAME>_UPLOAD_URL | Client endpoint for submitting jobs |
SERVICE_<NAME>_DOWNLOAD_URL | Client endpoint for retrieving results |
SERVICE_<NAME>_TERMINATE_URL | Client endpoint for terminating jobs |
SERVICE_<NAME>_RUNS_PER_USER | Maximum concurrent jobs per user (default: 5) |
SERVICE_<NAME>_MAX_RUNS | Maximum concurrent jobs for the service across all users (default: 10) |
Note: <NAME> is converted to lowercase, and that lowercase name is what clients must send in the service field (which is case-sensitive). SERVICE_EXAMPLE_* defines the service example; uploading with service=EXAMPLE returns 422 Invalid service.
Gotcha: <NAME> cannot contain underscores. The loader takes the text between the first and second _ as the name, so SERVICE_MY_TOOL_UPLOAD_URL defines a service my with an unknown setting TOOL_UPLOAD_URL, which is silently ignored.
Gotcha: PORT, MAX_AGE, MAX_UPLOAD_SIZE, RUNS_PER_USER, and MAX_RUNS are parsed as numbers with no fallback on a bad value — if you set one to something non-numeric, the server panics on startup instead of falling back to its default. Make sure these are always valid integers.
Example Configuration
Minimal Setup
export PORT=5000
export DB_PATH=/var/lib/job-orchestrator/db.sqlite
export DATA_PATH=/var/lib/job-orchestrator/data
export SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit
export SERVICE_EXAMPLE_DOWNLOAD_URL=http://localhost:9000/retrieve
export SERVICE_EXAMPLE_TERMINATE_URL=http://localhost:9000/kill
Production Setup
# Core settings
export PORT=5000
export DB_PATH=/opt/orchestrator/db.sqlite
export DATA_PATH=/opt/orchestrator/data
export MAX_AGE=864000 # 10 days
# Example service (general purpose)
export SERVICE_EXAMPLE_UPLOAD_URL=http://compute-1:9000/submit
export SERVICE_EXAMPLE_DOWNLOAD_URL=http://compute-1:9000/retrieve
export SERVICE_EXAMPLE_TERMINATE_URL=http://compute-1:9000/kill
export SERVICE_EXAMPLE_RUNS_PER_USER=10
# Simulation service (specialized, long-running)
export SERVICE_SIMULATE_UPLOAD_URL=http://sim-cluster:9001/submit
export SERVICE_SIMULATE_DOWNLOAD_URL=http://sim-cluster:9001/retrieve
export SERVICE_SIMULATE_TERMINATE_URL=http://sim-cluster:9001/kill
export SERVICE_SIMULATE_RUNS_PER_USER=3
Docker Compose
services:
server:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "server"]
ports:
- "5000:5000"
environment:
PORT: 5000
DB_PATH: /opt/data/db.sqlite
DATA_PATH: /opt/data
MAX_AGE: 864000
SERVICE_EXAMPLE_UPLOAD_URL: http://client:9000/submit
SERVICE_EXAMPLE_DOWNLOAD_URL: http://client:9000/retrieve
SERVICE_EXAMPLE_TERMINATE_URL: http://client:9000/kill
SERVICE_EXAMPLE_RUNS_PER_USER: 5
SERVICE_EXAMPLE_MAX_RUNS: 10
volumes:
- server-data:/opt/data
volumes:
server-data:
Configuration Details
PORT
The HTTP port for the REST API. Users will connect to this port to submit jobs and download results.
PORT=5000
DB_PATH
Path to the SQLite database file. The directory must exist and be writable.
DB_PATH=/var/lib/job-orchestrator/db.sqlite
The database is created automatically on first run. It stores:
- Job metadata (ID, user, service, status)
- Job locations and timestamps
- Client payload references
DATA_PATH
Directory where job files are stored. Each job gets a unique subdirectory.
DATA_PATH=/var/lib/job-orchestrator/data
Structure:
/var/lib/job-orchestrator/data/
├── a1b2c3d4-e5f6-7890-abcd-ef1234567890/
│ ├── run.sh
│ ├── input.pdb
│ └── output.zip (after completion)
├── b2c3d4e5-f6a7-8901-bcde-f12345678901/
│ └── ...
MAX_AGE
How long to keep job directories before cleanup, in seconds. Age is measured from the directory’s last modification time and the rule applies to every job, whatever its status. A job that is still running (or queued) when its directory passes MAX_AGE is cleaned too, so set it well above your longest expected job.
| Value | Duration |
|---|---|
3600 | 1 hour |
86400 | 24 hours |
172800 | 48 hours |
604800 | 1 week |
864000 | 10 days (default) |
MAX_AGE=864000
Jobs older than this are removed by the Cleaner task.
Service URLs
Each service needs upload, download, and terminate URLs pointing to a client:
SERVICE_MYSERVICE_UPLOAD_URL=http://client-host:9000/submit
SERVICE_MYSERVICE_DOWNLOAD_URL=http://client-host:9000/retrieve
SERVICE_MYSERVICE_TERMINATE_URL=http://client-host:9000/kill
- UPLOAD_URL: Where to POST job files
- DOWNLOAD_URL: Where to GET results (
/{id}is appended automatically). The partial download URL is derived from it by replacing the last path segment withretrieve_partial - TERMINATE_URL: Where to POST termination requests (
/{id}is appended automatically)
Gotcha: if you leave UPLOAD_URL, DOWNLOAD_URL, or TERMINATE_URL unset for a service, the server does not fail at startup — the field is silently stored as an empty string, and requests routed to that service will fail at request time instead. Double-check all three are set for every service you configure.
RUNS_PER_USER
Controls how many jobs a single user can have running simultaneously for a service:
SERVICE_EXAMPLE_RUNS_PER_USER=5
- Jobs exceeding the quota remain in
Queuedstatus - They’re automatically dispatched when slots become available
- Set higher for quick jobs, lower for resource-intensive jobs
Fixed Settings
These are not configurable:
- Task intervals: Sender and Getter every 500ms, Cleaner every 60s
- Log level: always
INFO.RUST_LOGis ignored. Every HTTP request is logged with its headers, so keep secrets out of headers or filter logs - Outgoing HTTP timeouts: none. A client that accepts the connection but never answers can stall the request (and a termination stays
Lockedmeanwhile)
File Permissions
Ensure the server process has:
- Read/Write access to
DB_PATHparent directory - Read/Write access to
DATA_PATHdirectory - Network access to all configured client URLs
Validating Configuration
Start the server and check logs:
job-orchestrator server
You should see:
WARNlines for every core variable left at its default- A
Config { services: {...}, ... }line listing the parsed services. Check that every service appears with all three URLs set Using database: sqlite://...listening on 0.0.0.0:<port>
Test with a health check:
curl http://localhost:5000/health
See Also
Client Configuration
The client is configured through environment variables and runs as a job executor.
Environment Variables
| Variable | Default | Description |
|---|---|---|
PORT | 5000 | HTTP port the client listens on |
DB_PATH | ./db.sqlite | Path to the client’s SQLite payload database |
DATA_PATH | ./data | Working directory where job files are stored |
MAX_AGE | 864000 | Seconds before the client’s cleaner deletes a payload directory (10 days) |
The client shares the exact same config loader as the server, so these variables have the same names and same defaults in both roles. SERVICE_* variables are ignored by the client. There is no client-specific default for PORT: if you run a server and a client on the same host, pick a different port explicitly (9000 is a common convention, shown below) to avoid a clash.
Example Configuration
Basic Setup
export PORT=9000
export DB_PATH=/var/lib/job-orchestrator-client/db.sqlite
export DATA_PATH=/var/lib/job-orchestrator-client/data
job-orchestrator client
Docker Compose
services:
client:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "client"]
environment:
PORT: 9000
DATA_PATH: /opt/data
DB_PATH: /opt/data/db.sqlite
volumes:
- client-data:/opt/data
volumes:
client-data:
Note there’s no ports: mapping here — in the shipped docker-compose.yml the client is only reachable over the internal Docker network from the server, not from the host. See Client Security below.
How the Client Works
Database
The client uses the same DB_PATH-backed SQLite database mechanism as the server — by default it’s a real file on disk, not in-memory, so payload records survive a client restart.
If you actually want an ephemeral, in-memory database (e.g. for a disposable/test client), set DB_PATH=:memory: explicitly. That’s an opt-in choice, not the default behavior. Be aware that payload IDs then restart at 1 after every restart, while the server still holds the old IDs as dest_id, and old payload directories with the same numbers may still be on disk. Only use it for throwaway clients.
The client’s database keeps a record of every payload it received (cleaned ones stay with status Cleaned). The server is still the source of truth for job history.
Working Directory
The client stores job files under DATA_PATH. Each payload gets a subdirectory named after its numeric payload ID:
/opt/data/
├── 1/
│ ├── run.sh
│ ├── input.pdb
│ ├── output.txt (created by run.sh)
│ ├── .orchestrator.exit (written by the trap)
│ └── output.zip (created on the first completed /retrieve)
├── 2/
│ └── ...
Uploaded files are stored by base name only, so any directory structure is flattened.
Gotcha: unlike the server, the client does not create DATA_PATH at startup. It is created when the first payload arrives. Until then the client’s cleaner logs could not read directory every 60 seconds, which is harmless.
Execution Environment
When the Runner task executes a job:
- Validates
run.sh(size, UTF-8, required trap, blocked patterns) - Runs
bash run.shwith the payload directory as working directory - Stores the process ID; the exit code is read later from
.orchestrator.exit - All files in the directory are included in results
The Updater task then watches Running payloads:
- If a payload has the killed flag set, it marks it
Killed - If
.orchestrator.exitexists, it marks itCompleted(exit code0) orFailed - If the process is still alive, it leaves it
Running - If the process is gone and never wrote
.orchestrator.exit, it marks itLostand logs a warning (see Lost Processes)
Job Termination
The client supports on-demand job termination via the /kill/:id endpoint:
- Receives termination requests from the server
- Sends SIGTERM to the running process (only the
bashprocess; child processes it started may keep running unless the script forwards the signal) - Sets the payload’s killed flag; the Updater changes the status to
Killedif the payload isRunning - The server’s
/terminate/:idendpoint triggers this via HTTP
Resource Reporting
The client exposes a /load endpoint that reports CPU usage:
curl http://localhost:9000/load
Returns a float representing CPU usage percentage. This can be used by the server for load-aware scheduling (planned feature).
Multiple Clients
You can run multiple clients for:
- Scaling: Handle more concurrent jobs
- Isolation: Different services on different machines
- Redundancy: Failover capability
Same Service, Multiple Clients
Not supported. Each service maps to exactly one client URL, and the server only accepts one value per variable. Do not put a load balancer in front of several clients either: payload IDs are local to each client, so a /retrieve/{id} sent to a different client would hit the wrong payload or a 404.
To spread load today, split the work into several services, each with its own client.
Different Services
Run specialized clients for different workloads:
# Client for general jobs
PORT=9000 job-orchestrator client
# Client for heavy computation (different machine)
PORT=9001 job-orchestrator client
Server configuration:
SERVICE_LIGHT_UPLOAD_URL=http://client-1:9000/submit
SERVICE_LIGHT_DOWNLOAD_URL=http://client-1:9000/retrieve
SERVICE_LIGHT_TERMINATE_URL=http://client-1:9000/kill
SERVICE_HEAVY_UPLOAD_URL=http://client-2:9001/submit
SERVICE_HEAVY_DOWNLOAD_URL=http://client-2:9001/retrieve
SERVICE_HEAVY_TERMINATE_URL=http://client-2:9001/kill
Client Security
Network Access
The client should only be accessible by the orchestrator server:
- Use internal networks / VPCs
- Firewall rules to restrict access
- Never expose client ports to the internet
Execution Sandbox
The client executes arbitrary run.sh scripts. Consider:
- Running in containers with resource limits
- Using separate user accounts with minimal permissions
- Mounting only necessary directories
- Network isolation if jobs don’t need internet
Docker Resource Limits
services:
client:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "client"]
deploy:
resources:
limits:
cpus: '4'
memory: 8G
reservations:
cpus: '1'
memory: 1G
Monitoring
The examples below assume the client is reachable directly (e.g. running it outside Docker, or with a ports: mapping added). With the shipped docker-compose.yml, the client has no host-exposed port — reach it with docker compose exec client curl localhost:9000/health instead, or from the server container over the internal network.
Health Check
curl http://localhost:9000/health
Load Check
curl http://localhost:9000/load
Container Logs
docker logs -f client-container
Troubleshooting
Client Not Receiving Jobs
- Verify server can reach client URL
- Check firewall rules
- Verify service configuration on server
Jobs Stuck in Prepared
- Check the client logs for errors
- Check for permission issues in the working directory
High Memory Usage
The client holds each incoming upload fully in memory while handling /submit, and builds /retrieve_partial archives in memory. Large inputs or partial downloads of large directories cause memory spikes. The database is file-backed and is not the cause. If memory is high:
- Check the size of the payloads being sent and of the partial downloads being requested
- Check for stuck payloads
- Restarting the client frees memory, but jobs running at that moment become lost (see Lost Processes)
See Also
Quota System
The quota system ensures fair resource allocation by limiting concurrent jobs per user and per service.
Two Independent Limits
The sender enforces two limits before dispatching a queued job:
| Variable | Scope | Default | Description |
|---|---|---|---|
SERVICE_<NAME>_RUNS_PER_USER | Per user, per service | 5 | Max jobs a single user can have active at once |
SERVICE_<NAME>_MAX_RUNS | Per service (all users) | 10 | Max jobs the service can have active at once |
Both limits must have available slots for a job to be dispatched. A job that
would violate either limit remains in Queued status until a slot opens.
Jobs in Processing, Submitted, or Running status all count toward
both limits. A job whose process disappears (for example after a client
restart) is marked Lost and releases its slot (see
Lost Processes).
How It Works
SERVICE_EXAMPLE_RUNS_PER_USER=3
SERVICE_EXAMPLE_MAX_RUNS=5
User 1 submits 4 jobs → 3 dispatched (per-user limit reached), 1 queued
User 2 submits 4 jobs → 2 dispatched (MAX_RUNS=5 reached), 2 queued
When one of User 1's jobs completes, one service slot frees up.
Both users now have a queued job and a free per-user slot, so either
User 1's or User 2's job is dispatched next. Which user goes first in a
sender pass is not deterministic.
Quota Scope
Quotas are enforced per user, per service (RUNS_PER_USER) and per service (MAX_RUNS):
| User | Service | RUNS_PER_USER | MAX_RUNS | Result |
|---|---|---|---|---|
| user_1 | example | 5 | 10 | Up to 5 concurrent |
| user_2 | example | 5 | 10 | Up to 5 concurrent (independent) |
| user_1 | simulate | 2 | 4 | Up to 2 concurrent |
Users don’t compete with each other for per-user slots. They do share the
MAX_RUNS pool for the service.
Scheduling: Round-Robin Between Users
When multiple users have queued jobs for the same service, the sender distributes slots in round-robin order across users. This prevents a single user with many queued jobs from consuming all available slots ahead of others.
The rotation restarts on every sender pass (every 500ms) and the order of users within a pass is arbitrary. Within one user, jobs are in practice taken oldest first (there is no explicit ordering or priority).
Quota States
┌──────────┐ Both limits OK ┌────────────┐
│ Queued │ ─────────────────▶ │ Processing │
└──────────┘ └────────────┘
│ │
│ Either limit exceeded ▼
│ ┌────────────┐
└──────────────────────── │ Submitted │
(waits for a free slot) │ (running) │
└────────────┘
Choosing Values
Factors to Consider
- Client capacity:
MAX_RUNSshould not exceed what your client hardware can handle concurrently - Job duration: Longer jobs need lower per-user quotas
- Resource usage: CPU/memory intensive jobs need lower
MAX_RUNS - User base: More users may need lower per-user quotas
Guidelines
| Job Type | Suggested RUNS_PER_USER | Suggested MAX_RUNS |
|---|---|---|
| Quick jobs (< 1 min) | 10-20 | 20-50 |
| Medium jobs (1-10 min) | 5-10 | 10-20 |
| Long jobs (10+ min) | 2-5 | 4-10 |
| Resource-intensive | 1-3 | 2-4 |
Example Scenarios
Scientific Computing Platform
# Quick validation jobs
SERVICE_VALIDATE_RUNS_PER_USER=10
SERVICE_VALIDATE_MAX_RUNS=50
# Standard analysis
SERVICE_ANALYZE_RUNS_PER_USER=5
SERVICE_ANALYZE_MAX_RUNS=20
# Heavy simulation
SERVICE_SIMULATE_RUNS_PER_USER=2
SERVICE_SIMULATE_MAX_RUNS=4
Educational Platform
# Student exercises
SERVICE_EXERCISE_RUNS_PER_USER=3
SERVICE_EXERCISE_MAX_RUNS=30
# Final projects
SERVICE_PROJECT_RUNS_PER_USER=5
SERVICE_PROJECT_MAX_RUNS=20
Testing Quotas
Submit multiple jobs to observe throttling:
Start the server with low limits (the variables must be set in the server’s environment, for example in docker-compose.yml, not in the shell you submit from):
SERVICE_EXAMPLE_RUNS_PER_USER=2
SERVICE_EXAMPLE_MAX_RUNS=3
Use a run.sh that takes a while (for example sleep 60 after the trap), then submit 5 jobs and check them:
for i in {1..5}; do
curl -s -X POST http://localhost:5000/upload \
-F "file=@run.sh" \
-F "user_id=1" \
-F "service=example" | jq -r '.id'
done
sleep 2
for id in 1 2 3 4 5; do
curl -s http://localhost:5000/download/$id | jq -r '"\(.id) \(.status)"'
done
Expected behaviour (the upload response always says Queued, so check /download):
- Jobs 1-2: dispatched (
Submitted/Running); the per-user limit of 2 is now reached - Jobs 3-5: stay
Queued.MAX_RUNS=3would allow one more, but not for this user - Submitting as
user_id=2at this point would dispatch one job and fillMAX_RUNS
Fair Scheduling
The quota system provides basic fairness:
- Per-user isolation: One user can’t starve others
- Per-service cap: Prevents overloading a single client
- Round-robin dispatch: Slots are shared evenly across users
- Automatic queuing: No jobs are rejected, just delayed
Limitations
Current limitations (improvements planned):
- No priority queues (FIFO within quota constraints)
- No time-based quotas (e.g., jobs per hour)
- No burst allowances
See Also
Server API Endpoints
The orchestrator server exposes a REST API for job submission and retrieval.
Base URL
http://localhost:5000
Interactive Documentation
Swagger UI is available at:
http://localhost:5000/swagger
Endpoints
POST /upload
Submit a new job for processing.
Request
- Content-Type:
multipart/form-data - Max size: 400MB by default, set with
MAX_UPLOAD_SIZE(see Server Configuration)
| Field | Type | Required | Description |
|---|---|---|---|
file | file | Yes | One or more files (repeat for multiple). Any field name works; every part with a filename is saved |
user_id | integer | Yes | User identifier for quota tracking |
service | string | Yes | Service name, lowercase (must be configured on server) |
Example
curl -X POST http://localhost:5000/upload \
-F "file=@run.sh" \
-F "file=@input.pdb" \
-F "user_id=1" \
-F "service=example"
Response
HTTP/1.1 202 Accepted
Location: /download/1
{
"id": 1,
"status": "Queued",
"message": "Job successfully uploaded"
}
The job is queued, not finished. Poll the URL in the Location header (GET /download/{id}) to follow it.
Status Codes
| Code | Description |
|---|---|
202 | Job accepted and queued. The Location header points to /download/{id} |
400 | The multipart body could not be parsed (for example, no boundary or a truncated body) |
413 | The request body exceeds the upload limit (MAX_UPLOAD_SIZE) |
415 | The Content-Type header is missing or not multipart/form-data |
422 | The request parsed, but a field is invalid: user_id missing or not a number, unknown service, or no run.sh uploaded |
500 | The server failed to store the job (disk or database error). The request can be retried |
Every error response has a JSON body with a message explaining the problem.
Notes
- The
servicemust match a configured service on the server exactly (service names are lowercase) - Files are stored by base name only: directory parts are stripped, and two files with the same name overwrite each other
- Other text fields are accepted and ignored
- The entry point script must be named exactly
run.sh(hardcoded, case-sensitive). An upload without it is rejected with422. Its contents are not checked at upload time: an unsafe or broken script results inInvalidstatus when the client tries to execute it dest_idis populated after the job is dispatched to a client
GET /download/
Get job status or download completed results.
Parameters
| Parameter | Type | Description |
|---|---|---|
id | integer | Job ID from upload response |
Example
# Check status (returns JSON when not completed)
curl http://localhost:5000/download/1
# Download results (returns ZIP when completed)
curl -o results.zip http://localhost:5000/download/1
Response
When the job is not yet completed, returns JSON:
{
"id": 1,
"status": "Running",
"message": ""
}
When the job is completed, returns:
- Content-Type:
application/zip - Body: ZIP archive of all output files
Status Codes
| Code | Description |
|---|---|
200 | JSON job status or ZIP file (check Content-Type) |
404 | Job not found |
500 | Server error |
Notes
- The same endpoint returns JSON or ZIP depending on job state. Only
Completedjobs return the ZIP - Terminal states other than
Completed(Failed,Invalid,Killed,Lost,Cleaned) keep returning JSON with200. Use/download_partial/{id}to get the files of a failed job - All files present in the job directory after
run.shfinishes are included
Usage Pattern
while true; do
ct=$(curl -s -o response.tmp -w '%{content_type}' http://localhost:5000/download/1)
if [[ "$ct" == application/zip* ]]; then
mv response.tmp results.zip
break
fi
status=$(jq -r '.status' response.tmp)
case "$status" in
Failed|Invalid|Killed|Lost|Cleaned) echo "Job ended: $status"; break ;;
esac
echo "Status: $status"
sleep 5
done
GET /download_partial/
Retrieve current job state regardless of completion status.
Use this endpoint to debug stuck or incomplete jobs by downloading the current workspace state as a ZIP file.
Parameters
| Parameter | Type | Description |
|---|---|---|
id | integer | Job ID from upload response |
Example
curl -o partial_results.zip http://localhost:5000/download_partial/1
Response
- Content-Type:
application/zip - Body: ZIP archive containing current job directory state
Status Codes
| Code | Description |
|---|---|
200 | ZIP file with current job state |
400 | Job’s service is not configured on the server |
404 | Job not found, job not dispatched yet (Queued), or the client no longer has the payload |
500 | Server error, or the client could not build the archive (for example after cleanup) |
Notes
- The ZIP comes from the client’s working directory, not from the server’s copy
- The client URL is derived from the service’s
DOWNLOAD_URLby replacing its last path segment withretrieve_partial
POST /terminate/
Cancel a running job.
Parameters
| Parameter | Type | Description |
|---|---|---|
id | integer | Job ID to terminate |
Example
curl -X POST http://localhost:5000/terminate/1
Response
On success:
{
"id": 1,
"status": "Unknown",
"message": "job terminated"
}
On failure:
{
"id": 1,
"status": "Unknown",
"message": "could not terminate job"
}
The
statusfield in the response reflects the in-memory default and is not meaningful here. QueryGET /download/{id}to confirm the job’s actual status after termination.
Status Codes
| Code | Description |
|---|---|
200 | Job terminated successfully |
404 | Job not found |
500 | Server error or termination failed |
Notes
- Termination is not gated on the job’s current status: the server locks the job, forwards the kill to the client regardless of what state it’s in, and rolls the status back to what it was before if the kill fails
- Terminated jobs will have status
Killed - The server sends a termination signal to the client, which then kills the process
Queuedjobs cannot be terminated: there is no client payload yet, the client answers404, the server returns500and the job staysQueued(it will still run later)- Terminating a job that already finished reports success and changes its status to
Killed. From then on/download/{id}returns JSON instead of the ZIP, even thoughoutput.zipis still on the server
GET /health
Health check endpoint.
Example
curl http://localhost:5000/health
Response
On success:
{
"status": "ok",
"database": "ok"
}
On failure (database unreachable), the response is a bare 503 with no body.
Status Codes
| Code | Description |
|---|---|
200 | Server is healthy |
503 | Database is unreachable |
GET /
Ping endpoint for basic connectivity check.
Example
curl http://localhost:5000/
Response
{"message": "pong"}
GET /swagger
Interactive API documentation.
Example
Open in browser:
http://localhost:5000/swagger
Provides:
- Interactive API explorer
- Request/response schemas
- Try-it-out functionality
Error Responses
Error responses from /upload, /download, /download_partial and /terminate follow this format (/health returns a bare 503 instead):
{
"id": 0,
"status": "Unknown",
"message": "Description of the error"
}
Rate Limiting
The server does not implement rate limiting directly. Use a reverse proxy (nginx, traefik) for rate limiting in production.
Authentication
The server does not implement authentication directly. The user_id field is trusted as provided. Implement authentication at the reverse proxy layer or in your application.
See Also
Client API Endpoints
The client exposes endpoints for the orchestrator server to submit jobs and retrieve results.
Note: These endpoints are typically only accessed by the orchestrator server, not by end users.
Base URL
http://localhost:9000
Endpoints
POST /submit
Receive a job payload from the orchestrator server.
Request
- Content-Type:
multipart/form-data
| Field | Type | Required | Description |
|---|---|---|---|
file | file | Yes | One or more job files. Any field name works; parts without a filename are ignored |
Example
curl -X POST http://localhost:9000/submit \
-F "file=@run.sh" \
-F "file=@input.pdb"
Response
{
"id": 1,
"status": "Prepared",
"loc": "/opt/data/1",
"pid": 0,
"killed": false
}
Status Codes
| Code | Description |
|---|---|
200 | Payload received successfully |
400 | Malformed multipart request |
500 | Server error |
Notes
- The client stores files (by base name) in
DATA_PATH/<id>and creates a payload record - The whole request is held in memory while it is processed, and there is no body size limit on the client
- Status starts as
Prepared, waiting for the Runner task - The
idis returned to the server and stored asdest_id
GET /retrieve_partial/
Retrieve current payload state regardless of completion status.
Use this endpoint to debug stuck or incomplete payloads by downloading the current workspace state as a ZIP file. This endpoint is typically called by the orchestrator server, not end users.
Parameters
| Parameter | Type | Description |
|---|---|---|
id | integer | Payload ID from submit response |
Example
# Download partial results for debugging
curl -o partial_results.zip http://localhost:9000/retrieve_partial/1
Response
- Content-Type:
application/zip - Body: ZIP archive containing current payload directory state
Status Codes
| Code | Description |
|---|---|
200 | ZIP file with current payload state |
404 | Payload not found |
500 | Server error |
Notes
- The
/retrieve_partialendpoint on the client is used internally by the server’s/download_partialendpoint - Path traversal protection is applied to all ZIP operations
GET /retrieve/
Retrieve results of a completed payload.
Parameters
| Parameter | Type | Description |
|---|---|---|
id | integer | Payload ID from submit response |
Example
curl -o results.zip http://localhost:9000/retrieve/1
Response
When the payload is not yet completed, returns a JSON body:
{
"id": 1,
"status": "Running",
"loc": "/opt/data/1",
"pid": 12345,
"killed": false
}
When the payload is completed, returns:
- Content-Type:
application/zip - Body: ZIP archive of all files in the payload directory
Status Codes
| Code | Description |
|---|---|
200 | JSON payload status or ZIP file (check Content-Type) |
404 | Payload not found |
500 | Server error |
Notes
- The ZIP includes all files in the working directory after
run.shexecution, including.orchestrator.exit - Original input files are included unless deleted by
run.sh - The archive is written to
output.zipin the payload directory on the first request and reused afterwards
POST /kill/
Terminate a running payload.
Parameters
| Parameter | Type | Description |
|---|---|---|
id | integer | Payload ID to terminate |
Example
curl -X POST http://localhost:9000/kill/1
Response
On success (200 OK):
- Empty body
On failure:
404if payload not found (with an empty payload JSON body)500if kill operation failed
Status Codes
| Code | Description |
|---|---|
200 | Payload terminated successfully |
404 | Payload not found |
500 | Kill operation failed |
Notes
- Sends SIGTERM to the process PID stored for the payload; silently succeeds if no PID is set yet or the process is already gone
- Sets the payload’s
killedflag. The Updater changes the status toKilledonly if the payload isRunning; a payload that already finished keeps its status - This endpoint is called by the server’s
/terminate/{id}endpoint
GET /load
Report current CPU usage.
Example
curl http://localhost:9000/load
Response
45.2
Returns a float representing CPU usage percentage (0-100). The value is sampled when the request arrives, so each call takes a fraction of a second.
Use Cases
- Load-aware job distribution (planned feature)
- Monitoring client health
- Capacity planning
GET /health
Health check endpoint.
Example
curl http://localhost:9000/health
Response
{
"status": "ok",
"database": "ok"
}
GET /
Ping endpoint for basic connectivity check.
Example
curl http://localhost:9000/
Response
{"message": "pong"}
Payload States
Payloads on the client go through these states:
| State | Description |
|---|---|
Prepared | Received from server, waiting for execution |
Running | Currently executing run.sh |
Completed | Execution finished successfully |
Failed | Execution failed (non-zero exit code, or the process could not be started) |
Invalid | run.sh missing, unsafe, or failed validation |
Killed | Terminated via /kill/{id} |
Lost | Process disappeared without writing .orchestrator.exit |
Cleaned | Payload directory removed by the cleaner task |
Security Considerations
The client API should never be exposed to the public internet:
- No authentication is implemented
- Arbitrary code execution via
run.sh - Internal service communication only
Recommendations:
- Use internal networks / VPCs
- Firewall rules: allow only orchestrator server IP
- Docker networks with no external exposure
See Also
Docker Deployment
Docker is the recommended way to deploy job-orchestrator in production.
Quick Start
git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
docker compose up --build
This starts:
- Server on port 5000 (exposed to the host)
- Example client on port 9000 (internal network only, not exposed to the host)
Docker Images
Official Image
docker pull codeberg.org/rvhonorato/job-orchestrator:latest
The image has no entrypoint: the binary lives at /job-orchestrator and is not on PATH, so always pass the full command (/job-orchestrator server or /job-orchestrator client).
The published image contains the binary plus bash and sqlite, without any application. It works as a server as-is. To use it as a client, build your own image on top of it with the tools your run.sh needs. example/Dockerfile shows how, adding the gdock demo application.
Build Locally
# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator .
# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/
Docker Compose
Basic Setup
services:
server:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "server"]
ports:
- "5000:5000"
environment:
PORT: 5000
DB_PATH: /opt/data/db.sqlite
DATA_PATH: /opt/data
MAX_AGE: 864000
MAX_UPLOAD_SIZE: 419430400
SERVICE_EXAMPLE_UPLOAD_URL: http://client:9000/submit
SERVICE_EXAMPLE_DOWNLOAD_URL: http://client:9000/retrieve
SERVICE_EXAMPLE_TERMINATE_URL: http://client:9000/kill
SERVICE_EXAMPLE_RUNS_PER_USER: 5
SERVICE_EXAMPLE_MAX_RUNS: 10
volumes:
- server-data:/opt/data
client:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "client"]
environment:
PORT: 9000
DATA_PATH: /opt/data
DB_PATH: /opt/data/db.sqlite
volumes:
- client-data:/opt/data
depends_on:
- server
volumes:
server-data:
client-data:
Production Setup
See Production Deployment - Container Hardening for details on each security option.
Multiple Clients
Scaling Horizontally
A service is bound to exactly one client URL, so adding more clients for the same service does not spread its jobs. Extra clients only receive work if they are configured as separate services, as shown below. Do not place a load balancer in front of several clients: payload IDs are local to each client, so status and result requests must reach the client that got the job.
Multiple Services
services:
server:
environment:
# Light jobs
SERVICE_LIGHT_UPLOAD_URL: http://client-light:9000/submit
SERVICE_LIGHT_DOWNLOAD_URL: http://client-light:9000/retrieve
SERVICE_LIGHT_TERMINATE_URL: http://client-light:9000/kill
SERVICE_LIGHT_RUNS_PER_USER: 10
# Heavy jobs
SERVICE_HEAVY_UPLOAD_URL: http://client-heavy:9000/submit
SERVICE_HEAVY_DOWNLOAD_URL: http://client-heavy:9000/retrieve
SERVICE_HEAVY_TERMINATE_URL: http://client-heavy:9000/kill
SERVICE_HEAVY_RUNS_PER_USER: 2
client-light:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "client"]
environment:
PORT: 9000
deploy:
resources:
limits:
cpus: '2'
memory: 2G
client-heavy:
image: codeberg.org/rvhonorato/job-orchestrator:latest
command: ["/job-orchestrator", "client"]
environment:
PORT: 9000
deploy:
resources:
limits:
cpus: '8'
memory: 16G
Volume Management
Persistent Storage
Always use named volumes for production:
volumes:
server-data:
driver: local
driver_opts:
type: none
o: bind
device: /data/job-orchestrator/server
client-data:
driver: local
driver_opts:
type: none
o: bind
device: /data/job-orchestrator/client
Backup Strategy
# Stop services (optional, for consistent backup)
docker compose stop
# Backup server and client data (the client holds results of running jobs)
tar -czf backup-$(date +%Y%m%d).tar.gz /data/job-orchestrator/server /data/job-orchestrator/client
# Resume services
docker compose start
Networking
Internal Network
Keep client internal:
services:
server:
ports:
- "5000:5000" # Exposed to host
networks:
- internal
- external
client:
# No ports exposed to host
networks:
- internal
networks:
internal:
internal: true
external:
With Reverse Proxy
services:
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./certs:/etc/nginx/certs:ro
depends_on:
- server
server:
# No ports exposed, accessed via nginx
networks:
- internal
Logging
View Logs
# All services
docker compose logs -f
# Server only
docker compose logs -f server
# Last 100 lines
docker compose logs --tail 100 server
Log Rotation
services:
server:
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
Commands
# Start
docker compose up -d
# Stop
docker compose down
# Restart
docker compose restart
# Rebuild and start
docker compose up --build -d
# View status
docker compose ps
# Shell into container
docker compose exec server /bin/sh
See Also
Kubernetes Deployment
The following instructions use Minikube for local testing.
See kubernetes/ in the repository for the manifest files.
Quick Start (Minikube)
# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator-server .
# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator-server -t job-orchestrator-client example/
# Load into Minikube
minikube image load job-orchestrator-server
minikube image load job-orchestrator-client
# Apply manifests
minikube kubectl -- apply -f kubernetes/
# Get the external IP
minikube service job-orchestrator-server --url
Usage
# Set the base URL
export URL=$(minikube service job-orchestrator-server --url)
# Submit a job
curl -X POST $URL/upload \
-F "file=@run.sh" \
-F "user_id=1" \
-F "service=example"
# Check status
curl $URL/download/1
Security Hardening
The manifests in kubernetes/ already apply the following hardening measures:
| Measure | Implemented |
|---|---|
Run as non-root (runAsUser: 1000, runAsNonRoot: true) | Both server and client |
| Read-only root filesystem | Both server and client |
| Drop all capabilities | Both server and client |
Seccomp profile (RuntimeDefault) | Both server and client |
| CPU and memory resource limits | Both server and client |
| NetworkPolicy restricting client ingress to the server and egress to the server and DNS | Client |
NetworkPolicy enforcement: NetworkPolicies only take effect if the cluster’s network plugin enforces them. Minikube’s default setup does not; start it with a plugin that does, for example minikube start --cni=calico.
PID limits: Kubernetes has no per-container PID limit in resources.limits, and LimitRange cannot set one either. Configure it on the node with the kubelet’s podPidsLimit setting (--pod-max-pids), which applies to every pod on that node.
Scaling: keep replicas: 1 for both deployments. The server uses SQLite on a ReadWriteOnce volume, and a second client replica behind the same Service would break result retrieval (payload IDs are local to each client).
See Also
Production Deployment
This guide covers best practices for deploying job-orchestrator in production environments.
Architecture Recommendations
Minimum Setup
┌─────────────┐ ┌─────────────┐
│ Server │────▶│ Client │
│ (1 instance)│ │ (1 instance)│
└─────────────┘ └─────────────┘
Recommended Setup
┌──────────────┐
│ Load Balancer│
│ (nginx) │
└──────┬───────┘
│
┌──────▼───────┐
│ Server │
│ (1 instance)│
└──────┬───────┘
│
┌────────────────┼────────────────┐
│ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ Client 1 │ │ Client 2 │ │ Client 3 │
└─────────────┘ └─────────────┘ └─────────────┘
Security
Script Validation
The client validates run.sh before execution. A script is rejected if it:
- Exceeds 20 MiB in size
- Is not valid UTF-8
- Is missing the required exit trap — the script must contain
trap 'echo $? > .orchestrator.exit' EXIT - Matches any dangerous pattern, including:
| Category | Blocked patterns |
|---|---|
| Destructive commands | rm followed later on the same line by any / or ~ (so rm -f tmp/file is blocked too), mkfs, dd to/from devices |
| Sensitive file access | /etc/passwd, /etc/shadow, /etc/sudoers, /proc/, /sys/, ~/.ssh/, /root/, Docker socket |
| Network tools | curl, wget, nc, ncat, socat, ssh, scp, sftp, telnet, rsync |
| Reverse shells | /dev/tcp/, /dev/udp/ |
| Privilege escalation | sudo, su, chown, dangerous chmod |
| Container/system escape | chroot, nsenter, unshare, mount, umount, docker, kubectl |
| Kernel manipulation | sysctl, modprobe, insmod, rmmod, iptables, nftables |
| Obfuscated execution | base64 | bash, eval, python -c, perl -e, ruby -e |
| Persistence | crontab, /etc/cron, systemctl, service, at |
| Fork bombs | :(){ ...|: } |
| Resource exhaustion | stress, stress-ng |
| Crypto miners | xmrig, minerd, cpuminer |
| Environment secrets | $AWS_*, $SECRET, $TOKEN, $PASSWORD, $API_KEY |
The patterns are plain regular expressions matched against the whole file, comments included, and several have no word boundary. Expect false positives: for example su\s+ also matches words ending in “su” followed by a space, rm\s+ matches inside perform ./step, and \$TOKEN matches $TOKENIZER. If a valid script is rejected as Invalid, check the client log for the reason.
This is a sanity check, not a sandbox. It can be bypassed by determined actors. Input scripts are still expected to come from trusted or semi-trusted sources. True isolation must be enforced at the deployment level using the container hardening measures below.
Path Traversal Protection
Two separate mechanisms protect the filesystem:
- Uploads (server
/uploadand client/submit): every filename is reduced to its base name before saving, so../../etc/passwdis stored aspasswdinside the job directory. Uploaded archives are never extracted. - Result archives (
/retrieveand/retrieve_partialon the client): while building the ZIP, every entry is canonicalized and must stay inside the payload directory. If anything resolves outside it, such as a symlink created byrun.shthat points to/etc, building the archive fails instead of including that file.
Container Hardening
The client executes user-submitted scripts with the full privileges of the process. Apply all of the following to limit blast radius:
| Measure | Docker Compose | Purpose |
|---|---|---|
| Read-only rootfs | read_only: true | Prevent filesystem tampering |
| Drop all capabilities | cap_drop: [ALL] | Remove kernel-level privileges |
| No new privileges | security_opt: [no-new-privileges:true] | Block setuid/setgid escalation |
| CPU limit | deploy.resources.limits.cpus | Prevent CPU starvation |
| Memory limit | deploy.resources.limits.memory | Prevent OOM on host |
| PIDs limit | deploy.resources.limits.pids | Prevent fork bombs |
| Internal network | networks: [internal] | Block outbound internet access |
| Writable tmpfs | tmpfs: [/tmp] | Provide scratch space on read-only rootfs |
Example (applied to the client service):
services:
client:
read_only: true
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
tmpfs:
- /tmp
deploy:
resources:
limits:
cpus: "2"
memory: 2G
pids: 256
networks:
- internal
networks:
internal:
internal: true
Note: The Kubernetes manifests already run as non-root (runAsUser: 1000).
For Docker deployments, the Dockerfile does not yet create a dedicated non-root
user — see the TODO in the Dockerfile.
Network Security
-
Never expose clients to the internet
- Clients execute user-submitted scripts
- Use internal networks only
- Block all outbound access from client containers
-
Use a reverse proxy
- TLS termination
- Rate limiting
- Request filtering
-
Firewall rules
# Allow only orchestrator server to reach clients iptables -A INPUT -p tcp --dport 9000 -s <server-ip> -j ACCEPT iptables -A INPUT -p tcp --dport 9000 -j DROP
Reverse Proxy (nginx)
# http block — limit_req_zone must be at http context, not inside server
http {
limit_req_zone $binary_remote_addr zone=upload:10m rate=10r/s;
upstream orchestrator {
server 127.0.0.1:5000;
}
server {
listen 443 ssl http2;
server_name jobs.example.com;
ssl_certificate /etc/nginx/certs/cert.pem;
ssl_certificate_key /etc/nginx/certs/key.pem;
location /upload {
limit_req zone=upload burst=20 nodelay;
client_max_body_size 400M; # keep in sync with MAX_UPLOAD_SIZE
proxy_pass http://orchestrator;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /download {
proxy_pass http://orchestrator;
proxy_set_header Host $host;
}
location /terminate {
proxy_pass http://orchestrator;
proxy_set_header Host $host;
}
location /health {
proxy_pass http://orchestrator;
}
# Block swagger in production (optional)
location /swagger {
deny all;
}
}
}
client_max_body_size is checked by nginx before the request reaches the server. If it is lower than MAX_UPLOAD_SIZE, nginx rejects large uploads with its own 413 page instead of the server’s JSON response, so keep the two values equal.
Authentication
job-orchestrator does not implement authentication. Options:
-
Reverse proxy authentication
location / { auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; proxy_pass http://orchestrator; } -
Application-level authentication
- Wrap the API in your application
- Validate users before calling job-orchestrator
-
OAuth2 Proxy
- Use oauth2-proxy in front of the service
- Integrates with identity providers
Resource Planning
Server Requirements
| Load Level | CPU | Memory | Storage |
|---|---|---|---|
| Light (< 100 jobs/day) | 1 core | 512MB | 10GB |
| Medium (100-1000 jobs/day) | 2 cores | 1GB | 50GB |
| Heavy (> 1000 jobs/day) | 4 cores | 2GB | 100GB+ |
Storage depends heavily on job file sizes and retention period.
Client Requirements
Depends entirely on your job workloads:
| Job Type | CPU | Memory |
|---|---|---|
| Text processing | 1 core | 512MB |
| Scientific computing | 4-8 cores | 8-16GB |
| ML/Deep learning | 8+ cores + GPU | 32GB+ |
Storage Calculation
Storage = (avg_job_size) × (jobs_per_day) × (retention_days)
Example:
- 10MB average job
- 500 jobs/day
- 10 day retention (default MAX_AGE)
= 10MB × 500 × 10 = 50GB
Monitoring
Health Checks
# Server health
curl -f http://localhost:5000/health
# Client health
curl -f http://localhost:9000/health
# Client load
curl http://localhost:9000/load
Log Aggregation
services:
server:
logging:
driver: "fluentd"
options:
fluentd-address: "localhost:24224"
tag: "job-orchestrator.server"
Backup & Recovery
What to Backup
-
Server database (
DB_PATH)- Contains job history and status
- Critical for job tracking
-
Server data directory (
DATA_PATH)- Contains job files and results
- Large, may use incremental backups
-
Client database and data directory
- Holds payload IDs that the server references as
dest_id, plus the working files of running jobs - Losing them breaks status and result retrieval for jobs that were in flight
- Holds payload IDs that the server references as
Backup Script
#!/bin/bash
BACKUP_DIR=/backups/job-orchestrator
DATE=$(date +%Y%m%d_%H%M%S)
# Backup database
sqlite3 /opt/data/db.sqlite ".backup '${BACKUP_DIR}/db_${DATE}.sqlite'"
# Backup data (incremental with rsync)
rsync -av --delete /opt/data/ ${BACKUP_DIR}/data/
# Cleanup old backups (keep 7 days)
find ${BACKUP_DIR} -name "db_*.sqlite" -mtime +7 -delete
Recovery
# Stop server
docker compose stop server
# Restore database (pick the backup file you want)
cp /backups/job-orchestrator/db_YYYYMMDD_HHMMSS.sqlite /opt/data/db.sqlite
# Restore data
rsync -av /backups/job-orchestrator/data/ /opt/data/
# Start server
docker compose start server
High Availability
Current Limitations
- Single server architecture
- No built-in clustering
- SQLite doesn’t support concurrent writes
Workarounds
-
Quick recovery
- Automated health checks
- Container auto-restart
- Fast backup restoration
-
Client restarts
- Clients are not stateless: keep their database and data directory on persistent storage
- Restarting a client while jobs run loses those processes, and the jobs are marked
Lostand must be resubmitted (see Lost Processes) - Drain a client (wait for its jobs to finish) before restarting it
-
Future improvements
- PostgreSQL support (planned)
- Server clustering (planned)
Maintenance
Database Maintenance
# Vacuum database (reclaim space)
sqlite3 /opt/data/db.sqlite "VACUUM;"
# Check integrity
sqlite3 /opt/data/db.sqlite "PRAGMA integrity_check;"
Log Rotation
Ensure logs don’t fill disk:
services:
server:
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "5"
Updates
# Pull latest image
docker pull codeberg.org/rvhonorato/job-orchestrator:latest
# Recreate containers
docker compose up -d
Troubleshooting
See Troubleshooting Guide for common issues.
See Also
Building from Source
This guide covers building job-orchestrator from source code.
Prerequisites
Rust Toolchain
Install Rust via rustup:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Minimum version: Rust 1.88 (edition 2024, uses let chains)
Verify installation:
rustc --version
cargo --version
System Dependencies
Only a C compiler is needed. SQLite is compiled from source and linked statically through sqlx’s bundled SQLite, and TLS uses rustls, so no system SQLite or OpenSSL libraries are required.
Debian/Ubuntu
apt-get update
apt-get install -y build-essential
Fedora/RHEL
dnf install gcc
macOS
xcode-select --install
Windows
The code compiles with the Visual Studio Build Tools, but client mode runs jobs through bash and the kill command, so it is not supported on Windows. Use WSL or a Linux container.
Clone Repository
git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
Build Commands
Debug Build
Fast compilation, includes debug symbols:
cargo build
Binary location: target/debug/job-orchestrator
Release Build
Optimized for performance:
cargo build --release
Binary location: target/release/job-orchestrator
Check (No Build)
Verify code compiles without producing binary:
cargo check
Running
From Cargo
# Server mode
PORT=5000 cargo run -- server
# Client mode
PORT=9000 cargo run -- client
From Binary
# After release build
PORT=5000 ./target/release/job-orchestrator server
Build Options
Features
Currently no optional features. All functionality is included by default.
Target Platforms
Cross-compile for different targets:
# Add target
rustup target add x86_64-unknown-linux-musl
# Build for target
cargo build --release --target x86_64-unknown-linux-musl
Common targets:
x86_64-unknown-linux-gnu- Linux (glibc)x86_64-unknown-linux-musl- Linux (static)x86_64-apple-darwin- macOS Intelaarch64-apple-darwin- macOS Apple Siliconx86_64-pc-windows-msvc- Windows (builds, but client mode does not work there)
Docker Build
Using Dockerfile
# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator .
# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/
Multi-stage Build
The Dockerfile produces a minimal Alpine image with the binary, bash, and sqlite. Where the binary comes from is picked with the BINARY_SOURCE build argument:
- build (default): compiles
job-orchestratorfrom source with the full Rust toolchain - prebuilt: copies an already compiled static (musl) binary from
dist/job-orchestrator. CI uses this: it compiles the binary in a separate job (with caching) and only packages it with buildah
cargo build --release --target x86_64-unknown-linux-musl
mkdir -p dist && cp target/x86_64-unknown-linux-musl/release/job-orchestrator dist/
docker build --build-arg BINARY_SOURCE=prebuilt -t job-orchestrator .
The image has no application in it. example/Dockerfile builds a client image on top of it with the example application (gdock) compiled from source.
See Also
Testing
This guide covers running and writing tests for job-orchestrator.
Running Tests
All Tests
cargo test
With Output
See println! output from tests:
cargo test -- --nocapture
Specific Test
# By name
cargo test test_upload
# By module
cargo test queue_dto::tests
Ignored Tests
No tests are marked #[ignore] at the moment. If you add slow tests that need extra setup, mark them with #[ignore] and run them with:
cargo test -- --ignored
Test Coverage
Using cargo-tarpaulin
Install:
cargo install cargo-tarpaulin
Generate coverage:
# HTML report
cargo tarpaulin --out Html --output-dir ./coverage
# XML report (for CI)
cargo tarpaulin --out Xml --output-dir ./coverage
View report:
open coverage/tarpaulin-report.html
Test Structure
Tests are inline #[cfg(test)] modules within the same file as the code they test:
src/
├── config/loader.rs # Config loading and env var tests
├── controllers/server.rs # Upload, download, terminate handler tests
├── controllers/client.rs # Submit, retrieve, kill handler tests
├── models/status_dto.rs # Status serialization tests
├── models/queue_dto.rs # Quota and round-robin scheduling tests
├── services/server.rs # Sender, getter, cleaner task tests
├── services/client.rs # Runner, updater, cleaner task tests
└── utils/io.rs # Script validator, file I/O tests
There is no separate tests/ directory — this is a binary crate with no public library interface.
Writing Tests
Unit Tests
#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_something() {
let result = function_under_test();
assert_eq!(result, expected_value);
}
}
}
Async Tests
Use tokio::test for async functions:
#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_async_function() {
let result = async_function().await;
assert!(result.is_ok());
}
}
}
Mocking
Using mockito
For HTTP mocking — this is what the codebase actually uses (src/controllers/server.rs, src/services/client.rs, src/services/server.rs):
#![allow(unused)]
fn main() {
use mockito::Server;
#[tokio::test]
async fn test_http_client() {
let mut server = Server::new_async().await;
let mock = server
.mock("GET", "/health")
.with_status(200)
.create_async()
.await;
// Test against server.url()
mock.assert_async().await;
}
}
mockallis listed as a dev-dependency inCargo.tomlfor trait-based mocking, but nothing in the codebase currently uses it — every existing test mocks HTTP boundaries withmockitoinstead.
Test Utilities
Test Fixtures
Create reusable test data:
#![allow(unused)]
fn main() {
#[cfg(test)]
mod test_utils {
pub fn create_test_job() -> Job {
let mut job = Job::new("/tmp/test-data");
job.set_user_id(1);
job.set_service("test".to_string());
job
}
}
}
Temporary Directories
Use tempfile for temporary test directories:
#![allow(unused)]
fn main() {
use tempfile::TempDir;
#[test]
fn test_file_operations() {
let temp_dir = TempDir::new().unwrap();
let file_path = temp_dir.path().join("test.txt");
// Test file operations
// TempDir is automatically cleaned up
}
}
CI Testing
Tests run automatically via Forgejo Actions (.forgejo/workflows/ci.yml), in a
single ci job with two steps:
- Run clippy —
cargo clippy -- -D warnings - Run tests with coverage —
cargo tarpaulin --engine llvm
There’s no separate cargo test step — tarpaulin runs the test suite while
measuring coverage.
Linting
Clippy
Run Clippy for additional checks:
cargo clippy -- -D warnings
Common fixes:
#[allow(clippy::lint_name)]to suppress specific lints- Configure in
clippy.tomlorCargo.toml
Formatting
Check formatting:
cargo fmt -- --check
Fix formatting:
cargo fmt
Debugging Tests
With println
#![allow(unused)]
fn main() {
#[test]
fn test_debug() {
let value = compute_something();
println!("Debug value: {:?}", value); // Use --nocapture to see
assert!(value.is_valid());
}
}
With RUST_BACKTRACE
RUST_BACKTRACE=1 cargo test
See Also
Contributing
Contributions to job-orchestrator are welcome! This guide explains how to contribute.
Ways to Contribute
- Bug reports: Found a bug? Open an issue
- Feature requests: Have an idea? Open an issue to discuss
- Code contributions: Fix bugs or implement features
- Documentation: Improve docs, fix typos, add examples
- Testing: Add tests, report edge cases
Please read the AI Policy before opening a pull request or issue.
Getting Started
1. Fork the Repository
Click “Fork” on Codeberg, then clone your fork:
git clone https://codeberg.org/YOUR_USERNAME/job-orchestrator.git
cd job-orchestrator
2. Set Up Development Environment
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install a C compiler (Debian/Ubuntu); SQLite is bundled
apt-get install build-essential
# Build
cargo build
# Run tests
cargo test
3. Create a Branch
git checkout -b feature/your-feature-name
# or
git checkout -b fix/your-bug-fix
Development Workflow
Making Changes
- Write code
- Add tests for new functionality
- Run tests:
cargo test - Run linter:
cargo clippy -- -D warnings - Format code:
cargo fmt
Commit Messages
Follow conventional commits:
type(scope): description
[optional body]
Types:
feat: New featurefix: Bug fixdocs: Documentationrefactor: Code refactoringtest: Adding testschore: Maintenance
Examples:
feat(quota): add per-service quota limits
fix(client): handle non-zero exit codes correctly
docs(readme): update quick start instructions
Pull Request Process
-
Push your branch:
git push origin feature/your-feature-name -
Open a Pull Request on Codeberg
-
In the PR description (there is no template):
- Describe your changes
- Link related issues
- Note any breaking changes
-
Wait for review
- CI must pass
- Maintainer will review
- Address feedback
-
Merge!
Code Style
Rust Style
Follow standard Rust conventions:
- Use
cargo fmtfor formatting - Use
cargo clippyfor linting - Prefer descriptive variable names
- Add doc comments for public APIs
#![allow(unused)]
fn main() {
/// Uploads a job to the orchestrator.
///
/// # Arguments
///
/// * `files` - Files to upload
/// * `user_id` - User submitting the job
/// * `service` - Target service name
///
/// # Returns
///
/// The created job with its ID and initial status.
pub async fn upload_job(
files: Vec<File>,
user_id: i32,
service: String,
) -> Result<Job, Error> {
// Implementation
}
}
Error Handling
- Use
Resulttypes, not panics - Provide context with error messages
- Use
?operator for propagation
#![allow(unused)]
fn main() {
// Good
let file = File::open(path)
.map_err(|e| Error::FileOpen { path: path.clone(), source: e })?;
// Avoid
let file = File::open(path).unwrap();
}
Testing Guidelines
Write Tests For
- New functionality
- Bug fixes (regression tests)
- Edge cases
Test Structure
#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_function_describes_behavior() {
// Arrange
let input = create_test_input();
// Act
let result = function_under_test(input);
// Assert
assert_eq!(result, expected);
}
}
}
Async Tests
#![allow(unused)]
fn main() {
#[tokio::test]
async fn test_async_operation() {
let result = async_function().await;
assert!(result.is_ok());
}
}
Documentation
Code Documentation
Add doc comments for:
- Public functions
- Public structs
- Public modules
#![allow(unused)]
fn main() {
/// A job represents a unit of work to be processed.
pub struct Job {
/// Unique identifier for the job.
pub id: i32,
/// User who submitted the job.
pub user_id: i32,
// ...
}
}
mdbook Documentation
Documentation is in docs/src/. To preview:
# Install mdbook
cargo install mdbook
# Serve locally
cd docs
mdbook serve --open
Reporting Issues
Bug Reports
Include:
- job-orchestrator version
- Operating system
- Steps to reproduce
- Expected vs actual behavior
- Logs if relevant
Feature Requests
Include:
- Use case description
- Proposed solution (if any)
- Alternatives considered
Code of Conduct
Be respectful and constructive. We’re all here to build something useful together.
Questions?
- Open an issue for questions
- Email: Rodrigo V. Honorato rvhonorato@protonmail.com
License
Contributions are licensed under the MIT License.
Troubleshooting
Common issues and solutions for job-orchestrator.
Server Issues
Server Won’t Start
Symptom: Server fails to start, exits immediately
Possible Causes:
-
Port already in use
Error: Address already in useSolution:
# Find process using the port lsof -i :5000 # Kill it or use a different port PORT=5001 job-orchestrator server -
Database path not writable
Error: unable to open database fileSolution:
# Check directory exists and is writable mkdir -p /opt/data chmod 755 /opt/data -
Missing service configuration
The server starts without services but will return
422 Invalid serviceon every upload. Configure at least one service:export SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit export SERVICE_EXAMPLE_DOWNLOAD_URL=http://client:9000/retrieve export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill
Jobs Stuck in Queued
Symptom: Jobs stay in Queued status indefinitely
Possible Causes:
-
Quota exhausted
There are two independent limits — either can hold a job in
Queued:RUNS_PER_USER(default: 5) — user has too many active jobs for this serviceMAX_RUNS(default: 10) — the service has reached its total concurrent job cap
Wait for running jobs to complete, or increase the relevant limit.
-
Slots held by stuck jobs
Jobs whose process died are marked
Lostand release their slot on their own (see Job Status is Lost). A job that staysRunningwhile its client is unreachable still counts toward both limits. Find them and terminate them:sqlite3 /opt/data/db.sqlite "SELECT id, user_id, service, status FROM jobs WHERE status IN ('processing','submitted','running')" curl -X POST http://localhost:5000/terminate/<id> -
Service removed from the configuration
Jobs queued for a service that is no longer configured (for example after a restart with different
SERVICE_*variables) are skipped by the sender and stayQueuedforever.
An unreachable client does not keep jobs Queued: the dispatch fails and the job becomes Failed.
Jobs Stuck in Submitted
Symptom: Jobs move to Submitted but never complete
Possible Causes:
-
Client not executing jobs
Check client logs for errors
docker logs client -
run.shhangingYour script may be waiting for input or stuck in a loop. Use
/download_partial/{id}to look at its working directory -
Client lost the payload
If the client was restarted mid-job, or its data was wiped, the server keeps polling and logs errors such as
There was some error while trying to retrieve job .... The job status never changes on its own; terminate it and resubmit
Upload Is Rejected
Symptom: POST /upload returns an error instead of 202 Accepted. The message field in the JSON response says what was wrong.
Possible Causes:
-
Missing required fields or
run.sh(422)user_id(a number),serviceand a file namedrun.share required:curl -X POST http://localhost:5000/upload \ -F "file=@run.sh" \ -F "user_id=1" \ -F "service=example" -
Unknown service (
422)Service must be configured on server, and the
servicefield must be the lowercase name (example, notEXAMPLE):export SERVICE_EXAMPLE_UPLOAD_URL=...Service names cannot contain underscores; see Server Configuration.
-
Upload too large (
413)The default limit is 400MB for all files combined. Check file sizes, or raise
MAX_UPLOAD_SIZEon the server. -
Wrong
Content-Type(415) or unreadable body (400)The request must be
multipart/form-datawith a boundary. Let your HTTP client build it (curl -F, orFormDatain a browser) instead of setting theContent-Typeheader by hand, which drops the boundary.
Client Issues
Client Not Receiving Jobs
Symptom: Client running but no jobs arrive
Check:
-
Network connectivity
# From server, can you reach client? curl http://client:9000/health -
Firewall rules
# Client port must be accessible from server iptables -L -n | grep 9000 -
Docker networking
# Containers must be on same network # The shipped compose file uses the "api" network for server-client traffic docker network inspect job-orchestrator_api
Termination Fails
Symptom: Job remains Running or Submitted after termination request
Possible Causes:
-
Job not in a terminable state
- Termination is not gated by job status: the server attempts it for a job in any status
Queuedjobs have no client payload yet (dest_idis still 0), so the client answers404, the server returns500, and the job staysQueued- Already completed or failed jobs: termination “succeeds” and the job becomes
Killed, after which/download/{id}no longer returns the ZIP - Solution: Check job status first with
GET /download/{id}
-
Client unreachable
- Server cannot reach client to send termination request
- Verify client connectivity:
curl http://client:9000/health -
Process already terminated
- The process may have finished between the request and processing
- The job is still marked
Killedin that case
-
Missing TERMINATE_URL configuration
- Server needs
SERVICE_<NAME>_TERMINATE_URLconfigured - Solution: Add to server configuration:
export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill - Server needs
Jobs Stuck in Locked
Symptom: Job stays in Locked status indefinitely
Possible Causes:
Locked only lasts while the server waits for the client’s answer to a termination request. It stays longer when:
-
The client accepted the connection but does not answer
- The server sets no HTTP timeout, so it waits as long as the connection stays open
- Check client logs and health
-
The server restarted during a termination
- Nothing picks up
Lockedjobs afterwards, so they stayLockedpermanently - They are not polled and do not count toward quotas
- Fix the status in the database manually, or send another
POST /terminate/{id}
- Nothing picks up
Jobs Stuck in Prepared
Symptom: Payloads stay in Prepared status
Possible Causes:
-
Runner task not running
Check client logs, may need restart
-
Permission issues
Client working directory may have permission issues
A script that fails validation does not stay Prepared: it becomes Invalid (see below).
Job Status is Invalid
Symptom: Job status is Invalid — the client rejected the script before running it
Possible Causes:
-
Missing exit trap — script must contain:
trap 'echo $? > .orchestrator.exit' EXIT -
Script too large — maximum script size is 20 MiB
-
Script not valid UTF-8
-
Script contains a blocked pattern (the client log names the pattern; false positives are possible, for example
rmwith any relative path containing/). See Script Validation for the full list (network tools, destructive commands, privilege escalation, etc.)
Invalid jobs are cleaned up after MAX_AGE like any other terminal state.
Job Status is Lost
Symptom: Job status is Lost
The job’s process disappeared without writing .orchestrator.exit. The client log has a warning such as payload 12 lost: pid 4321 is gone and no exit file at "/opt/data/12".
Possible Causes:
- Client restarted or crashed while the job ran (for example a container restart). Drain clients before restarting them, see Production
- The
bashprocess runningrun.shwas killed with SIGKILL (for examplekill -9, or the OOM killer picking it). SIGKILL cannot be trapped, so the exit code is never written
Solution: Lost is terminal. Use /download_partial/{id} to look at what the job wrote, then resubmit it.
Execution Fails
Symptom: Jobs complete but with Failed status
Check:
-
Exit code
run.shmust exit with code 0 for success:#!/bin/bash # Your commands here exit 0 # Explicit success -
Script errors
Check output files for error messages.
/download/{id}does not return files forFailedjobs; use/download_partial/{id} -
Missing dependencies
Your script may need tools not available in the container
Database Issues
Database Locked
Symptom: “database is locked” errors
Causes: Multiple processes accessing SQLite
Solution:
- Ensure only one server instance runs
- Check for zombie processes
- Restart server
Database Corrupted
Symptom: Strange errors, missing data
Solution:
-
Stop server
-
Backup current database
-
Run integrity check:
sqlite3 db.sqlite "PRAGMA integrity_check;" -
If corrupted, restore from backup or delete and restart
Out of Disk Space
Symptom: “disk full” errors
Solution:
-
Check disk usage:
df -h -
Clean old jobs:
# Reduce MAX_AGE and restart export MAX_AGE=3600 # 1 hour -
Manually clean data directory
Docker Issues
Container Exits Immediately
Check logs:
docker logs container_name
Common causes:
- Missing environment variables
- Port conflicts
- Permission issues
Cannot Connect Between Containers
Ensure same network:
services:
server:
networks:
- app-network
client:
networks:
- app-network
networks:
app-network:
Use service names, not localhost:
# Wrong
SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit
# Correct
SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit
Volume Permission Issues
Symptom: Permission denied when writing to volumes
Solution:
services:
server:
user: "1000:1000" # Match host user
volumes:
- ./data:/opt/data
Or fix permissions:
sudo chown -R 1000:1000 ./data
Performance Issues
Slow Job Processing
Possible Causes:
-
Slow database
- Use SSD storage for database
- Run VACUUM periodically
-
Network latency
- Place server and clients on same network
- Check for packet loss
-
Client overloaded
- Add more clients
- Reduce RUNS_PER_USER
High Memory Usage
Server:
/download/{id}reads the wholeoutput.zipinto memory before sending it, so large results cause spikes- Uploads are streamed to disk and do not
Client:
/submitholds the whole upload in memory, and/retrieve//retrieve_partialbuild or read the whole archive in memory- The database is file-backed and is not a significant factor
Disk Usage Growing
Check:
du -sh /opt/data/*
Solutions:
- Reduce
MAX_AGE(the cleanup interval itself is fixed at 60 seconds) - Archive old results externally
Getting Help
If you can’t resolve an issue:
- Check logs for specific error messages
- Search existing issues: Codeberg Issues
- Open new issue with:
- Version
- Configuration
- Steps to reproduce
- Logs