Troubleshooting
Common issues and solutions for job-orchestrator.
Server Issues
Server Won’t Start
Symptom: Server fails to start, exits immediately
Possible Causes:
-
Port already in use
Error: Address already in useSolution:
# Find process using the port lsof -i :5000 # Kill it or use a different port PORT=5001 job-orchestrator server -
Database path not writable
Error: unable to open database fileSolution:
# Check directory exists and is writable mkdir -p /opt/data chmod 755 /opt/data -
Missing service configuration
The server starts without services but will return
422 Invalid serviceon every upload. Configure at least one service:export SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit export SERVICE_EXAMPLE_DOWNLOAD_URL=http://client:9000/retrieve export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill
Jobs Stuck in Queued
Symptom: Jobs stay in Queued status indefinitely
Possible Causes:
-
Quota exhausted
There are two independent limits — either can hold a job in
Queued:RUNS_PER_USER(default: 5) — user has too many active jobs for this serviceMAX_RUNS(default: 10) — the service has reached its total concurrent job cap
Wait for running jobs to complete, or increase the relevant limit.
-
Slots held by stuck jobs
Jobs whose process died are marked
Lostand release their slot on their own (see Job Status is Lost). A job that staysRunningwhile its client is unreachable still counts toward both limits. Find them and terminate them:sqlite3 /opt/data/db.sqlite "SELECT id, user_id, service, status FROM jobs WHERE status IN ('processing','submitted','running')" curl -X POST http://localhost:5000/terminate/<id> -
Service removed from the configuration
Jobs queued for a service that is no longer configured (for example after a restart with different
SERVICE_*variables) are skipped by the sender and stayQueuedforever.
An unreachable client does not keep jobs Queued: the dispatch fails and the job becomes Failed.
Jobs Stuck in Submitted
Symptom: Jobs move to Submitted but never complete
Possible Causes:
-
Client not executing jobs
Check client logs for errors
docker logs client -
run.shhangingYour script may be waiting for input or stuck in a loop. Use
/download_partial/{id}to look at its working directory -
Client lost the payload
If the client was restarted mid-job, or its data was wiped, the server keeps polling and logs errors such as
There was some error while trying to retrieve job .... The job status never changes on its own; terminate it and resubmit
Upload Is Rejected
Symptom: POST /upload returns an error instead of 202 Accepted. The message field in the JSON response says what was wrong.
Possible Causes:
-
Missing required fields or
run.sh(422)user_id(a number),serviceand a file namedrun.share required:curl -X POST http://localhost:5000/upload \ -F "file=@run.sh" \ -F "user_id=1" \ -F "service=example" -
Unknown service (
422)Service must be configured on server, and the
servicefield must be the lowercase name (example, notEXAMPLE):export SERVICE_EXAMPLE_UPLOAD_URL=...Service names cannot contain underscores; see Server Configuration.
-
Upload too large (
413)The default limit is 400MB for all files combined. Check file sizes, or raise
MAX_UPLOAD_SIZEon the server. -
Wrong
Content-Type(415) or unreadable body (400)The request must be
multipart/form-datawith a boundary. Let your HTTP client build it (curl -F, orFormDatain a browser) instead of setting theContent-Typeheader by hand, which drops the boundary.
Client Issues
Client Not Receiving Jobs
Symptom: Client running but no jobs arrive
Check:
-
Network connectivity
# From server, can you reach client? curl http://client:9000/health -
Firewall rules
# Client port must be accessible from server iptables -L -n | grep 9000 -
Docker networking
# Containers must be on same network # The shipped compose file uses the "api" network for server-client traffic docker network inspect job-orchestrator_api
Termination Fails
Symptom: Job remains Running or Submitted after termination request
Possible Causes:
-
Job not in a terminable state
- Termination is not gated by job status: the server attempts it for a job in any status
Queuedjobs have no client payload yet (dest_idis still 0), so the client answers404, the server returns500, and the job staysQueued- Already completed or failed jobs: termination “succeeds” and the job becomes
Killed, after which/download/{id}no longer returns the ZIP - Solution: Check job status first with
GET /download/{id}
-
Client unreachable
- Server cannot reach client to send termination request
- Verify client connectivity:
curl http://client:9000/health -
Process already terminated
- The process may have finished between the request and processing
- The job is still marked
Killedin that case
-
Missing TERMINATE_URL configuration
- Server needs
SERVICE_<NAME>_TERMINATE_URLconfigured - Solution: Add to server configuration:
export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill - Server needs
Jobs Stuck in Locked
Symptom: Job stays in Locked status indefinitely
Possible Causes:
Locked only lasts while the server waits for the client’s answer to a termination request. It stays longer when:
-
The client accepted the connection but does not answer
- The server sets no HTTP timeout, so it waits as long as the connection stays open
- Check client logs and health
-
The server restarted during a termination
- Nothing picks up
Lockedjobs afterwards, so they stayLockedpermanently - They are not polled and do not count toward quotas
- Fix the status in the database manually, or send another
POST /terminate/{id}
- Nothing picks up
Jobs Stuck in Prepared
Symptom: Payloads stay in Prepared status
Possible Causes:
-
Runner task not running
Check client logs, may need restart
-
Permission issues
Client working directory may have permission issues
A script that fails validation does not stay Prepared: it becomes Invalid (see below).
Job Status is Invalid
Symptom: Job status is Invalid — the client rejected the script before running it
Possible Causes:
-
Missing exit trap — script must contain:
trap 'echo $? > .orchestrator.exit' EXIT -
Script too large — maximum script size is 20 MiB
-
Script not valid UTF-8
-
Script contains a blocked pattern (the client log names the pattern; false positives are possible, for example
rmwith any relative path containing/). See Script Validation for the full list (network tools, destructive commands, privilege escalation, etc.)
Invalid jobs are cleaned up after MAX_AGE like any other terminal state.
Job Status is Lost
Symptom: Job status is Lost
The job’s process disappeared without writing .orchestrator.exit. The client log has a warning such as payload 12 lost: pid 4321 is gone and no exit file at "/opt/data/12".
Possible Causes:
- Client restarted or crashed while the job ran (for example a container restart). Drain clients before restarting them, see Production
- The
bashprocess runningrun.shwas killed with SIGKILL (for examplekill -9, or the OOM killer picking it). SIGKILL cannot be trapped, so the exit code is never written
Solution: Lost is terminal. Use /download_partial/{id} to look at what the job wrote, then resubmit it.
Execution Fails
Symptom: Jobs complete but with Failed status
Check:
-
Exit code
run.shmust exit with code 0 for success:#!/bin/bash # Your commands here exit 0 # Explicit success -
Script errors
Check output files for error messages.
/download/{id}does not return files forFailedjobs; use/download_partial/{id} -
Missing dependencies
Your script may need tools not available in the container
Database Issues
Database Locked
Symptom: “database is locked” errors
Causes: Multiple processes accessing SQLite
Solution:
- Ensure only one server instance runs
- Check for zombie processes
- Restart server
Database Corrupted
Symptom: Strange errors, missing data
Solution:
-
Stop server
-
Backup current database
-
Run integrity check:
sqlite3 db.sqlite "PRAGMA integrity_check;" -
If corrupted, restore from backup or delete and restart
Out of Disk Space
Symptom: “disk full” errors
Solution:
-
Check disk usage:
df -h -
Clean old jobs:
# Reduce MAX_AGE and restart export MAX_AGE=3600 # 1 hour -
Manually clean data directory
Docker Issues
Container Exits Immediately
Check logs:
docker logs container_name
Common causes:
- Missing environment variables
- Port conflicts
- Permission issues
Cannot Connect Between Containers
Ensure same network:
services:
server:
networks:
- app-network
client:
networks:
- app-network
networks:
app-network:
Use service names, not localhost:
# Wrong
SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit
# Correct
SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit
Volume Permission Issues
Symptom: Permission denied when writing to volumes
Solution:
services:
server:
user: "1000:1000" # Match host user
volumes:
- ./data:/opt/data
Or fix permissions:
sudo chown -R 1000:1000 ./data
Performance Issues
Slow Job Processing
Possible Causes:
-
Slow database
- Use SSD storage for database
- Run VACUUM periodically
-
Network latency
- Place server and clients on same network
- Check for packet loss
-
Client overloaded
- Add more clients
- Reduce RUNS_PER_USER
High Memory Usage
Server:
/download/{id}reads the wholeoutput.zipinto memory before sending it, so large results cause spikes- Uploads are streamed to disk and do not
Client:
/submitholds the whole upload in memory, and/retrieve//retrieve_partialbuild or read the whole archive in memory- The database is file-backed and is not a significant factor
Disk Usage Growing
Check:
du -sh /opt/data/*
Solutions:
- Reduce
MAX_AGE(the cleanup interval itself is fixed at 60 seconds) - Archive old results externally
Getting Help
If you can’t resolve an issue:
- Check logs for specific error messages
- Search existing issues: Codeberg Issues
- Open new issue with:
- Version
- Configuration
- Steps to reproduce
- Logs