Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Architecture Overview

job-orchestrator uses a distributed architecture with a central server coordinating job execution across multiple client nodes.

High-Level Architecture

flowchart TB
 subgraph Tasks["Background Tasks"]
        Sender["Sender<br>500ms"]
        Getter["Getter<br>500ms"]
        Cleaner["Cleaner<br>60s"]
  end
 subgraph Server["Orchestrator Server"]
        API["REST API<br>upload/download/download_partial/terminate"]
        DB[("SQLite<br>Persistent")]
        FS[/"Filesystem<br>Job Storage"/]
        Tasks
        Queue["Queue Manager<br>Quota Enforcement"]
  end
 subgraph Client["Client Service"]
        ClientAPI["REST API<br>submit/retrieve/retrieve_partial/kill/load"]
        ClientDB[("SQLite<br>Persistent")]
        ClientFS[/"Working Dir"/]
        Runner["Runner Task<br>500ms"]
        Updater["Updater Task<br>500ms"]
        ClientCleaner["Cleaner Task<br>60s"]
        Executor["Bash Executor<br>run.sh"]
  end
    User(["User/Web App"]) -- POST /upload --> API
    User -- GET /download/:id --> API
    User -- GET /download_partial/:id --> API
    API -- GET /retrieve_partial/:id --> ClientAPI
    API --> DB & FS
    DB --> Queue
    Queue --> Sender
    Sender -- POST /submit --> ClientAPI
    Getter -- GET /retrieve/:id --> ClientAPI
    Getter --> FS
    Cleaner --> DB & FS
    ClientAPI --> ClientDB
    ClientDB --> Runner
    Runner --> Executor
    Executor --> ClientFS
    ClientDB --> Updater
    Updater --> ClientDB
    ClientDB --> ClientCleaner
    ClientCleaner --> ClientFS

Components

Orchestrator Server

The central server manages:

  • REST API: Handles job uploads and result downloads from users
  • SQLite Database: Persistent storage for job metadata and status
  • Filesystem Storage: Stores uploaded files and downloaded results
  • Queue Manager: Enforces per-user quotas and manages job distribution
  • Background Tasks: Automated processes for job distribution, result retrieval, and cleanup

Client Service

Each client node handles:

  • REST API: Receives jobs from server, returns results
  • SQLite Database: Payload tracking. Configured with the same DB_PATH variable as the server (file-backed by default); give each process its own file
  • Working Directory: Temporary storage for job execution
  • Runner Task: Picks up new payloads and starts them
  • Updater Task: Watches running payloads for their exit code, a kill request, or a lost process
  • Bash Executor: Runs the run.sh script for each job

Background Tasks

Server Tasks

TaskIntervalPurpose
Sender500msPicks up queued jobs, enforces quotas, dispatches to clients
Getter500msSyncs job status from clients and downloads results when complete
Cleaner60sDeletes job directories older than MAX_AGE and marks them Cleaned (records are kept)

Client Tasks

TaskIntervalPurpose
Runner500msValidates and starts prepared payloads
Updater500msMonitors running processes, handles exit codes, killed payloads and lost processes
Cleaner60sDeletes payload directories older than MAX_AGE, whatever their status

Data Flow

  1. User submits files via POST /upload
  2. Server stores files and creates job record (status: Queued)
  3. Sender task picks up job, checks quotas, sends it to the client configured for its service
  4. Client receives job, stores as payload (status: Prepared)
  5. Runner task starts run.sh; the Updater sets Completed or Failed from its exit code
  6. Getter task retrieves results, stores locally
  7. User downloads results via GET /download/:id
  8. Cleaner task removes job after retention period

Termination Flow

  1. User sends termination request via POST /terminate/:id
  2. Server locks job (status: Locked) and forwards to client
  3. Client receives request via POST /kill/:id, sends SIGTERM to the process and sets the payload’s killed flag
  4. Server sets the job to Killed as soon as the client answers with success (or restores the previous status if it fails)
  5. Server returns termination confirmation to user
  6. Client Updater later marks the payload Killed