Architecture Overview
job-orchestrator uses a distributed architecture with a central server coordinating job execution across multiple client nodes.
High-Level Architecture
flowchart TB
subgraph Tasks["Background Tasks"]
Sender["Sender<br>500ms"]
Getter["Getter<br>500ms"]
Cleaner["Cleaner<br>60s"]
end
subgraph Server["Orchestrator Server"]
API["REST API<br>upload/download/download_partial/terminate"]
DB[("SQLite<br>Persistent")]
FS[/"Filesystem<br>Job Storage"/]
Tasks
Queue["Queue Manager<br>Quota Enforcement"]
end
subgraph Client["Client Service"]
ClientAPI["REST API<br>submit/retrieve/retrieve_partial/kill/load"]
ClientDB[("SQLite<br>Persistent")]
ClientFS[/"Working Dir"/]
Runner["Runner Task<br>500ms"]
Updater["Updater Task<br>500ms"]
ClientCleaner["Cleaner Task<br>60s"]
Executor["Bash Executor<br>run.sh"]
end
User(["User/Web App"]) -- POST /upload --> API
User -- GET /download/:id --> API
User -- GET /download_partial/:id --> API
API -- GET /retrieve_partial/:id --> ClientAPI
API --> DB & FS
DB --> Queue
Queue --> Sender
Sender -- POST /submit --> ClientAPI
Getter -- GET /retrieve/:id --> ClientAPI
Getter --> FS
Cleaner --> DB & FS
ClientAPI --> ClientDB
ClientDB --> Runner
Runner --> Executor
Executor --> ClientFS
ClientDB --> Updater
Updater --> ClientDB
ClientDB --> ClientCleaner
ClientCleaner --> ClientFS
Components
Orchestrator Server
The central server manages:
- REST API: Handles job uploads and result downloads from users
- SQLite Database: Persistent storage for job metadata and status
- Filesystem Storage: Stores uploaded files and downloaded results
- Queue Manager: Enforces per-user quotas and manages job distribution
- Background Tasks: Automated processes for job distribution, result retrieval, and cleanup
Client Service
Each client node handles:
- REST API: Receives jobs from server, returns results
- SQLite Database: Payload tracking. Configured with the same
DB_PATHvariable as the server (file-backed by default); give each process its own file - Working Directory: Temporary storage for job execution
- Runner Task: Picks up new payloads and starts them
- Updater Task: Watches running payloads for their exit code, a kill request, or a lost process
- Bash Executor: Runs the
run.shscript for each job
Background Tasks
Server Tasks
| Task | Interval | Purpose |
|---|---|---|
| Sender | 500ms | Picks up queued jobs, enforces quotas, dispatches to clients |
| Getter | 500ms | Syncs job status from clients and downloads results when complete |
| Cleaner | 60s | Deletes job directories older than MAX_AGE and marks them Cleaned (records are kept) |
Client Tasks
| Task | Interval | Purpose |
|---|---|---|
| Runner | 500ms | Validates and starts prepared payloads |
| Updater | 500ms | Monitors running processes, handles exit codes, killed payloads and lost processes |
| Cleaner | 60s | Deletes payload directories older than MAX_AGE, whatever their status |
Data Flow
- User submits files via
POST /upload - Server stores files and creates job record (status:
Queued) - Sender task picks up job, checks quotas, sends it to the client configured for its service
- Client receives job, stores as payload (status:
Prepared) - Runner task starts
run.sh; the Updater setsCompletedorFailedfrom its exit code - Getter task retrieves results, stores locally
- User downloads results via
GET /download/:id - Cleaner task removes job after retention period
Termination Flow
- User sends termination request via
POST /terminate/:id - Server locks job (status:
Locked) and forwards to client - Client receives request via
POST /kill/:id, sends SIGTERM to the process and sets the payload’s killed flag - Server sets the job to
Killedas soon as the client answers with success (or restores the previous status if it fails) - Server returns termination confirmation to user
- Client Updater later marks the payload
Killed