Job Lifecycle
Understanding the job lifecycle is essential for working with job-orchestrator effectively.
Lifecycle Sequence
sequenceDiagram
participant User
participant Server
participant Client
participant Executor
User->>Server: POST /upload (files, user_id, service)
Server->>Server: Store job (status: Queued)
Server-->>User: Job ID
Note over Server: Sender task (500ms interval)
Server->>Server: Update status: Processing
Server->>Client: POST /submit (job files)
Client->>Client: Store payload (status: Prepared)
Client-->>Server: Payload ID
Server->>Server: Update status: Submitted
Note over Client: Runner task (500ms interval)
Client->>Client: Update status: Running
Client->>Executor: Execute run.sh (spawns process)
Note over Client: Updater task (500ms interval)
Executor-->>Client: .orchestrator.exit written
Client->>Client: Update status: Completed
Note over Server: Getter task (500ms interval)
Server->>Client: GET /retrieve/:id
Client-->>Server: JSON status (copied to job) or, once Completed, ZIP results
Server->>Server: Store output.zip, status: Completed
User->>Server: GET /download/:id
Server-->>User: results.zip
Note over Server: Cleaner task (60s interval)
Server->>Server: Delete job dirs older than MAX_AGE, status: Cleaned
Note over User,Server: Job Termination Flow
User->>Server: POST /terminate/:id
Server->>Server: Update status: Locked
Server->>Client: POST /kill/:id
Client->>Executor: Send SIGTERM
Client->>Client: Set killed flag
Client-->>Server: 200 OK
Server->>Server: Update status: Killed
Server-->>User: Termination confirmed
Note over Client: Updater marks the payload Killed on its next pass
Job States
stateDiagram-v2
[*] --> Queued: submitted
Queued --> Processing: sender picks up
Processing --> Submitted: sent to client
Processing --> Failed: dispatch error
Submitted --> Prepared: client received job
Prepared --> Running: execution started
Running --> Completed: exit 0
Running --> Failed: non-zero exit
Running --> Invalid: bad script
Running --> Lost: process gone, no exit file
Submitted --> Killed: terminated early
Prepared --> Killed: terminated early
Running --> Killed: terminated
Completed --> Killed: terminated after finishing
Completed --> Cleaned: MAX_AGE
Failed --> Cleaned: MAX_AGE
Invalid --> Cleaned: MAX_AGE
Killed --> Cleaned: MAX_AGE
Lost --> Cleaned: MAX_AGE
Cleaned --> [*]
The server copies the client’s payload status into the job on every Getter pass, which is why client-side states (
Prepared,Running,Invalid) also show up on the server.
LockedandUnknownare not shown above; see the table below.Cleanedcan happen from any state, includingQueuedandRunning, because the cleaner only looks at the age of the job directory.
State Descriptions
| State | Description |
|---|---|
| Queued | Job received and waiting for dispatch |
| Processing | Server is sending job to a client |
| Submitted | Job successfully sent to client, awaiting execution |
| Running | Client is actively executing the job |
| Completed | Job finished successfully, results available |
| Prepared | Client stored the payload and is about to run it (copied from the client) |
| Failed | Terminal. run.sh exited non-zero, the client could not start it, or dispatch to the client failed |
| Invalid | Terminal. Client rejected the script (run.sh missing, unsafe pattern, missing trap, too large, not UTF-8) |
| Unknown | Fallback when a status string cannot be parsed, and the placeholder in error responses. Not a retry state |
| Locked | Set while a termination request is in flight. Restored or replaced by Killed when the request returns |
| Killed | Terminal. Job was terminated via POST /terminate/{id} |
| Lost | Terminal. The process disappeared without writing .orchestrator.exit (client restart or crash, or bash itself killed with SIGKILL). See Lost Processes |
| Cleaned | Terminal. Job directory deleted after MAX_AGE. The database record is kept |
Lifecycle Stages
1. Submission
User uploads files via POST /upload with:
- One or more files (including
run.sh) user_id- identifies the submitting userservice- which backend should process this job
The server:
- Validates the request
- Creates a unique directory for the job
- Stores all uploaded files (by base name only, so subdirectories are flattened)
- Creates a database record with status
Queued - Returns the job ID to the user
2. Queuing & Quota Check
The Sender background task (runs every 500ms) loads its next batch of jobs via a round-robin fair-share scheduler:
- Groups all
Queuedjobs byservice, then byuser_idwithin each service - For each service, computes available service-wide slots:
MAX_RUNS(default 10) minus jobs alreadyProcessing/Submitted/Runningfor that service - For each user in that service, computes available per-user slots:
RUNS_PER_USER(default 5) minus that user’s own in-flight job count - Round-robins across users with available slots — takes one queued job per user per pass — until the service-wide slot budget is exhausted or no user has both a queued job and a free slot
- Jobs picked this way are marked
Processing; everything else remainsQueuedfor the next cycle
Quota is enforced at two levels — per-user (RUNS_PER_USER) and per-service (MAX_RUNS) — and the round-robin means no single user can starve others of a service’s shared capacity.
3. Distribution
For jobs in Processing status:
- Server streams every file in the job directory as one multipart request
- Sends it to the service’s
UPLOAD_URL(the client’sPOST /submit). Each service has exactly one client URL; there is no load-based client selection - On success: updates status to
Submitted, stores client’s payload ID asdest_id - On failure (client unreachable or non-2xx response): updates status to
Failed. There is no retry
4. Execution
On the client side:
- Runner task (every 500ms) finds payloads in
Preparedstatus and marks themRunning - Validates
run.sh(see Script Validation). A rejected script makes the payloadInvalid - Spawns
bash run.shin the payload directory and stores its PID. If the process cannot be started, the payload becomesFailed - Updater task (every 500ms) checks
Runningpayloads. When.orchestrator.exitappears it setsCompleted(exit code0) orFailed(anything else). If the killed flag is set it setsKilled. If the process is gone and never wrote the exit file it setsLost
5. Retrieval
The Getter background task (runs every 500ms):
- Finds jobs in
Submitted,Prepared, orRunningstatus (at most 10 requests in flight) - Requests the payload from the client via
GET /retrieve/:id - If the client answers with JSON, copies the payload status into the job (
Running,Failed,Invalid, …) - If the client answers with a ZIP, saves it as
output.zipin the job directory and setsCompleted
Once a job reaches a status outside that list, the Getter stops polling it.
6. Download
User can now:
- Check status via
GET /download/:id(returns JSON with job state when not completed) - Download results via
GET /download/:id(returns ZIP when completed) - Results are returned as a ZIP archive
7. Cleanup
The Cleaner background task (runs every 60s):
- Finds job directories whose last modification time is older than
MAX_AGE, whatever the job status - Deletes the directory
- Updates the job status to
Cleaned. The database record is never deleted
The client runs the same cleaner over its own payload directories.
Error Handling
Client Unreachable
If the server cannot reach a client during distribution:
- Job status changes to
Failed - Job will not be retried automatically
- User can resubmit if needed
Execution Failure
If run.sh exits with non-zero code:
- Payload status changes to
Failed - The server copies the
Failedstatus but does not fetch any output GET /download/{id}keeps returning JSON; useGET /download_partial/{id}to get the files
Retrieval Failure
If the server cannot retrieve the payload (client down, 404, network error):
- The job status is left unchanged and the error is logged
- The Getter retries every 500ms, indefinitely. There is no timeout
Lost Processes
If the client is restarted or crashes while a job runs (for example a container restart), the process is gone but no .orchestrator.exit file was written:
- The client’s Updater sees that the PID is gone and that there is no exit file, marks the payload
Lostand logs a warning (payload <id> lost: pid <pid> is gone and no exit file at ...) - The server copies
Lostinto the job on its next Getter pass, which frees itsRUNS_PER_USERandMAX_RUNSslot Lostis terminal: resubmit the job./download_partial/{id}still returns whatever the job wrote before it died
The trap writes .orchestrator.exit before the process ends, so a job that exits normally always leaves the file behind. The Updater checks for the file again after seeing the PID gone, so a job that finishes between two checks is still marked Completed or Failed, not Lost.
A server older than the client cannot parse Lost: it logs a retrieval error and keeps the job Running. Upgrade the server before, or together with, the clients.
Timing Considerations
| Event | Typical Duration |
|---|---|
| Upload to Queued | Immediate |
| Queued to Processing | Up to 500ms (+ quota wait) |
| Processing to Submitted | Depends on file size and network |
| Submitted to Completed | Depends on job execution time |
| Completed to Cleaned | MAX_AGE after the job directory was last modified (default: 10 days) |