Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Your First Job

This guide walks you through submitting and retrieving your first job.

Prerequisites

Make sure you have job-orchestrator running. See Quick Start if you haven’t set it up yet.

Understanding Jobs

A job in job-orchestrator consists of:

  1. Files: One or more files to be processed
  2. A run.sh script: The entry point that gets executed
  3. User ID: Identifies who submitted the job (for quota tracking)
  4. Service: Which service/backend should process this job

Creating a Simple Job

Create a simple run.sh script:

cat > run.sh << 'EOF'
#!/bin/bash
# Required: Capture exit code for the orchestrator client
trap 'echo $? > .orchestrator.exit' EXIT

echo "Hello from job-orchestrator!" > output.txt
echo "Processing complete at $(date)" >> output.txt
EOF
chmod +x run.sh

Submitting the Job

Submit the job using curl:

curl -X POST http://localhost:5000/upload \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example" | jq

You’ll receive a response like:

{
  "id": 1,
  "status": "Queued",
  "message": "Job successfully uploaded"
}

Note the id field - you’ll need this to check status and download results.

Checking Job Status

Check the job status via GET request:

curl http://localhost:5000/download/1

If the job is not yet completed, you’ll get a JSON response:

{
  "id": 1,
  "status": "Submitted",
  "message": ""
}

The status field will be one of: Queued, Processing, Submitted, Prepared, Running, Completed, Failed, Invalid, Cleaned, Unknown, Locked, Killed, or Lost. See Job States for descriptions of each.

Downloading Results

Once the status is Completed, the same endpoint returns the ZIP file:

curl -o results.zip http://localhost:5000/download/1

Extract and view:

unzip results.zip
cat output.txt

You should see:

Hello from job-orchestrator!
Processing complete at <timestamp>

The archive contains the whole job directory: your input files, everything run.sh wrote, and the .orchestrator.exit file.

Downloading Partial Results

/download/{id} only returns a ZIP for Completed jobs. For any other status, including Failed, it returns the JSON status. To get the files of a failed, stuck, or still-running job, use /download_partial/{id}:

# Get current state of job even if incomplete
curl -o partial_results.zip http://localhost:5000/download_partial/1

Canceling a Job

If you need to cancel a running job, use the terminate endpoint:

# Cancel job with ID 1
curl -X POST http://localhost:5000/terminate/1

Response on success:

{
  "id": 1,
  "status": "Unknown",
  "message": "job terminated"
}

The status field in the terminate response is not meaningful. Use GET /download/{id} to confirm the job’s actual status after termination.

Notes:

  • The server sends a termination request to the client, which sends SIGTERM to the process
  • Terminated jobs will have status Killed
  • Queued jobs cannot be cancelled: they have not reached a client yet, so the request fails with 500, the job stays Queued, and it will still run later
  • Terminating a job that already finished still “succeeds” and changes its status to Killed. After that, /download/{id} returns JSON instead of the ZIP, so the results are only reachable through /download_partial/{id} (until the client cleans them up)

Security

Script Execution

The client executes user-submitted run.sh scripts with full privileges. While a script validator rejects obviously dangerous patterns (destructive commands, network exfiltration tools, reverse shells, privilege escalation), it is NOT a sandbox and can be bypassed by determined actors.

Important: Always run the client inside a container with:

  • Resource limits (CPU, memory, PIDs)
  • Read-only root filesystem
  • Drop all capabilities (--cap-drop=ALL)
  • Internal network only (no external exposure)
  • Non-root user

See Production Deployment for details.

Path Traversal Protection

Uploaded filenames are reduced to their base name, so ../../etc/passwd is stored as passwd inside the job directory. When results are zipped, any entry that resolves outside the job directory (such as a symlink to /etc) makes the archive fail instead of being included. See Path Traversal Protection.

Important Notes

The run.sh Script

  • Must be named exactly run.sh
  • Is run as bash run.sh inside the job directory, so it does not need the executable bit or a shebang
  • Exit code 0 indicates success
  • Non-zero exit code indicates failure
  • All output files in the working directory are included in results

Important: The script MUST include the following trap at the beginning. The client checks for it before running the script and marks the job Invalid if it is missing:

#!/bin/bash
trap 'echo $? > .orchestrator.exit' EXIT
# ... rest of your script

This ensures that when your run.sh exits, it writes the exit code to .orchestrator.exit, which the client’s Updater task reads to determine if the job completed successfully or failed.

File Size Limits

The server accepts request bodies up to 400MB by default (all files in one upload combined). Change it with the MAX_UPLOAD_SIZE environment variable, in bytes. Larger uploads are rejected with 413.

User ID Parameter

Jobs can be submitted with a user_id for quota management:

curl -X POST http://localhost:5000/upload \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example"

user_id is required and must be an integer; the server returns 422 if it is missing or not a number, if service is unknown, or if no run.sh was uploaded. service must match a configured service name exactly, in lowercase (example, not EXAMPLE).

Job Retention

Job directories are deleted once they are older than MAX_AGE (default: 10 days), and the job is marked Cleaned. Age is measured from the directory’s last modification time, not from when the job finished, and the rule applies to jobs in any status. Make sure to download your results before they expire.

Next Steps