Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Job Lifecycle

Understanding the job lifecycle is essential for working with job-orchestrator effectively.

Lifecycle Sequence

sequenceDiagram
    participant User
    participant Server
    participant Client
    participant Executor

    User->>Server: POST /upload (files, user_id, service)
    Server->>Server: Store job (status: Queued)
    Server-->>User: Job ID

    Note over Server: Sender task (500ms interval)
    Server->>Server: Update status: Processing
    Server->>Client: POST /submit (job files)
    Client->>Client: Store payload (status: Prepared)
    Client-->>Server: Payload ID
    Server->>Server: Update status: Submitted

    Note over Client: Runner task (500ms interval)
    Client->>Client: Update status: Running
    Client->>Executor: Execute run.sh (spawns process)

    Note over Client: Updater task (500ms interval)
    Executor-->>Client: .orchestrator.exit written
    Client->>Client: Update status: Completed

    Note over Server: Getter task (500ms interval)
    Server->>Client: GET /retrieve/:id
    Client-->>Server: JSON status (copied to job) or, once Completed, ZIP results
    Server->>Server: Store output.zip, status: Completed

    User->>Server: GET /download/:id
    Server-->>User: results.zip

    Note over Server: Cleaner task (60s interval)
    Server->>Server: Delete job dirs older than MAX_AGE, status: Cleaned

    Note over User,Server: Job Termination Flow
    User->>Server: POST /terminate/:id
    Server->>Server: Update status: Locked
    Server->>Client: POST /kill/:id
    Client->>Executor: Send SIGTERM
    Client->>Client: Set killed flag
    Client-->>Server: 200 OK
    Server->>Server: Update status: Killed
    Server-->>User: Termination confirmed
    Note over Client: Updater marks the payload Killed on its next pass

Job States

stateDiagram-v2
    [*] --> Queued: submitted
    Queued --> Processing: sender picks up
    Processing --> Submitted: sent to client
    Processing --> Failed: dispatch error

    Submitted --> Prepared: client received job
    Prepared --> Running: execution started
    Running --> Completed: exit 0
    Running --> Failed: non-zero exit
    Running --> Invalid: bad script
    Running --> Lost: process gone, no exit file
    Submitted --> Killed: terminated early
    Prepared --> Killed: terminated early
    Running --> Killed: terminated
    Completed --> Killed: terminated after finishing

    Completed --> Cleaned: MAX_AGE
    Failed --> Cleaned: MAX_AGE
    Invalid --> Cleaned: MAX_AGE
    Killed --> Cleaned: MAX_AGE
    Lost --> Cleaned: MAX_AGE
    Cleaned --> [*]

The server copies the client’s payload status into the job on every Getter pass, which is why client-side states (Prepared, Running, Invalid) also show up on the server.

Locked and Unknown are not shown above; see the table below. Cleaned can happen from any state, including Queued and Running, because the cleaner only looks at the age of the job directory.

State Descriptions

StateDescription
QueuedJob received and waiting for dispatch
ProcessingServer is sending job to a client
SubmittedJob successfully sent to client, awaiting execution
RunningClient is actively executing the job
CompletedJob finished successfully, results available
PreparedClient stored the payload and is about to run it (copied from the client)
FailedTerminal. run.sh exited non-zero, the client could not start it, or dispatch to the client failed
InvalidTerminal. Client rejected the script (run.sh missing, unsafe pattern, missing trap, too large, not UTF-8)
UnknownFallback when a status string cannot be parsed, and the placeholder in error responses. Not a retry state
LockedSet while a termination request is in flight. Restored or replaced by Killed when the request returns
KilledTerminal. Job was terminated via POST /terminate/{id}
LostTerminal. The process disappeared without writing .orchestrator.exit (client restart or crash, or bash itself killed with SIGKILL). See Lost Processes
CleanedTerminal. Job directory deleted after MAX_AGE. The database record is kept

Lifecycle Stages

1. Submission

User uploads files via POST /upload with:

  • One or more files (including run.sh)
  • user_id - identifies the submitting user
  • service - which backend should process this job

The server:

  1. Validates the request
  2. Creates a unique directory for the job
  3. Stores all uploaded files (by base name only, so subdirectories are flattened)
  4. Creates a database record with status Queued
  5. Returns the job ID to the user

2. Queuing & Quota Check

The Sender background task (runs every 500ms) loads its next batch of jobs via a round-robin fair-share scheduler:

  1. Groups all Queued jobs by service, then by user_id within each service
  2. For each service, computes available service-wide slots: MAX_RUNS (default 10) minus jobs already Processing/Submitted/Running for that service
  3. For each user in that service, computes available per-user slots: RUNS_PER_USER (default 5) minus that user’s own in-flight job count
  4. Round-robins across users with available slots — takes one queued job per user per pass — until the service-wide slot budget is exhausted or no user has both a queued job and a free slot
  5. Jobs picked this way are marked Processing; everything else remains Queued for the next cycle

Quota is enforced at two levels — per-user (RUNS_PER_USER) and per-service (MAX_RUNS) — and the round-robin means no single user can starve others of a service’s shared capacity.

3. Distribution

For jobs in Processing status:

  1. Server streams every file in the job directory as one multipart request
  2. Sends it to the service’s UPLOAD_URL (the client’s POST /submit). Each service has exactly one client URL; there is no load-based client selection
  3. On success: updates status to Submitted, stores client’s payload ID as dest_id
  4. On failure (client unreachable or non-2xx response): updates status to Failed. There is no retry

4. Execution

On the client side:

  1. Runner task (every 500ms) finds payloads in Prepared status and marks them Running
  2. Validates run.sh (see Script Validation). A rejected script makes the payload Invalid
  3. Spawns bash run.sh in the payload directory and stores its PID. If the process cannot be started, the payload becomes Failed
  4. Updater task (every 500ms) checks Running payloads. When .orchestrator.exit appears it sets Completed (exit code 0) or Failed (anything else). If the killed flag is set it sets Killed. If the process is gone and never wrote the exit file it sets Lost

5. Retrieval

The Getter background task (runs every 500ms):

  1. Finds jobs in Submitted, Prepared, or Running status (at most 10 requests in flight)
  2. Requests the payload from the client via GET /retrieve/:id
  3. If the client answers with JSON, copies the payload status into the job (Running, Failed, Invalid, …)
  4. If the client answers with a ZIP, saves it as output.zip in the job directory and sets Completed

Once a job reaches a status outside that list, the Getter stops polling it.

6. Download

User can now:

  1. Check status via GET /download/:id (returns JSON with job state when not completed)
  2. Download results via GET /download/:id (returns ZIP when completed)
  3. Results are returned as a ZIP archive

7. Cleanup

The Cleaner background task (runs every 60s):

  1. Finds job directories whose last modification time is older than MAX_AGE, whatever the job status
  2. Deletes the directory
  3. Updates the job status to Cleaned. The database record is never deleted

The client runs the same cleaner over its own payload directories.

Error Handling

Client Unreachable

If the server cannot reach a client during distribution:

  • Job status changes to Failed
  • Job will not be retried automatically
  • User can resubmit if needed

Execution Failure

If run.sh exits with non-zero code:

  • Payload status changes to Failed
  • The server copies the Failed status but does not fetch any output
  • GET /download/{id} keeps returning JSON; use GET /download_partial/{id} to get the files

Retrieval Failure

If the server cannot retrieve the payload (client down, 404, network error):

  • The job status is left unchanged and the error is logged
  • The Getter retries every 500ms, indefinitely. There is no timeout

Lost Processes

If the client is restarted or crashes while a job runs (for example a container restart), the process is gone but no .orchestrator.exit file was written:

  • The client’s Updater sees that the PID is gone and that there is no exit file, marks the payload Lost and logs a warning (payload <id> lost: pid <pid> is gone and no exit file at ...)
  • The server copies Lost into the job on its next Getter pass, which frees its RUNS_PER_USER and MAX_RUNS slot
  • Lost is terminal: resubmit the job. /download_partial/{id} still returns whatever the job wrote before it died

The trap writes .orchestrator.exit before the process ends, so a job that exits normally always leaves the file behind. The Updater checks for the file again after seeing the PID gone, so a job that finishes between two checks is still marked Completed or Failed, not Lost.

A server older than the client cannot parse Lost: it logs a retrieval error and keeps the job Running. Upgrade the server before, or together with, the clients.

Timing Considerations

EventTypical Duration
Upload to QueuedImmediate
Queued to ProcessingUp to 500ms (+ quota wait)
Processing to SubmittedDepends on file size and network
Submitted to CompletedDepends on job execution time
Completed to CleanedMAX_AGE after the job directory was last modified (default: 10 days)