Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Troubleshooting

Common issues and solutions for job-orchestrator.

Server Issues

Server Won’t Start

Symptom: Server fails to start, exits immediately

Possible Causes:

  1. Port already in use

    Error: Address already in use
    

    Solution:

    # Find process using the port
    lsof -i :5000
    # Kill it or use a different port
    PORT=5001 job-orchestrator server
    
  2. Database path not writable

    Error: unable to open database file
    

    Solution:

    # Check directory exists and is writable
    mkdir -p /opt/data
    chmod 755 /opt/data
    
  3. Missing service configuration

    The server starts without services but will return 422 Invalid service on every upload. Configure at least one service:

    export SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit
    export SERVICE_EXAMPLE_DOWNLOAD_URL=http://client:9000/retrieve
    export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill
    

Jobs Stuck in Queued

Symptom: Jobs stay in Queued status indefinitely

Possible Causes:

  1. Quota exhausted

    There are two independent limits — either can hold a job in Queued:

    • RUNS_PER_USER (default: 5) — user has too many active jobs for this service
    • MAX_RUNS (default: 10) — the service has reached its total concurrent job cap

    Wait for running jobs to complete, or increase the relevant limit.

  2. Slots held by stuck jobs

    Jobs whose process died are marked Lost and release their slot on their own (see Job Status is Lost). A job that stays Running while its client is unreachable still counts toward both limits. Find them and terminate them:

    sqlite3 /opt/data/db.sqlite "SELECT id, user_id, service, status FROM jobs WHERE status IN ('processing','submitted','running')"
    curl -X POST http://localhost:5000/terminate/<id>
    
  3. Service removed from the configuration

    Jobs queued for a service that is no longer configured (for example after a restart with different SERVICE_* variables) are skipped by the sender and stay Queued forever.

An unreachable client does not keep jobs Queued: the dispatch fails and the job becomes Failed.

Jobs Stuck in Submitted

Symptom: Jobs move to Submitted but never complete

Possible Causes:

  1. Client not executing jobs

    Check client logs for errors

    docker logs client
    
  2. run.sh hanging

    Your script may be waiting for input or stuck in a loop. Use /download_partial/{id} to look at its working directory

  3. Client lost the payload

    If the client was restarted mid-job, or its data was wiped, the server keeps polling and logs errors such as There was some error while trying to retrieve job .... The job status never changes on its own; terminate it and resubmit

Upload Is Rejected

Symptom: POST /upload returns an error instead of 202 Accepted. The message field in the JSON response says what was wrong.

Possible Causes:

  1. Missing required fields or run.sh (422)

    user_id (a number), service and a file named run.sh are required:

    curl -X POST http://localhost:5000/upload \
      -F "file=@run.sh" \
      -F "user_id=1" \
      -F "service=example"
    
  2. Unknown service (422)

    Service must be configured on server, and the service field must be the lowercase name (example, not EXAMPLE):

    export SERVICE_EXAMPLE_UPLOAD_URL=...
    

    Service names cannot contain underscores; see Server Configuration.

  3. Upload too large (413)

    The default limit is 400MB for all files combined. Check file sizes, or raise MAX_UPLOAD_SIZE on the server.

  4. Wrong Content-Type (415) or unreadable body (400)

    The request must be multipart/form-data with a boundary. Let your HTTP client build it (curl -F, or FormData in a browser) instead of setting the Content-Type header by hand, which drops the boundary.

Client Issues

Client Not Receiving Jobs

Symptom: Client running but no jobs arrive

Check:

  1. Network connectivity

    # From server, can you reach client?
    curl http://client:9000/health
    
  2. Firewall rules

    # Client port must be accessible from server
    iptables -L -n | grep 9000
    
  3. Docker networking

    # Containers must be on same network
    # The shipped compose file uses the "api" network for server-client traffic
    docker network inspect job-orchestrator_api
    

Termination Fails

Symptom: Job remains Running or Submitted after termination request

Possible Causes:

  1. Job not in a terminable state

    • Termination is not gated by job status: the server attempts it for a job in any status
    • Queued jobs have no client payload yet (dest_id is still 0), so the client answers 404, the server returns 500, and the job stays Queued
    • Already completed or failed jobs: termination “succeeds” and the job becomes Killed, after which /download/{id} no longer returns the ZIP
    • Solution: Check job status first with GET /download/{id}
  2. Client unreachable

    • Server cannot reach client to send termination request
    • Verify client connectivity:
    curl http://client:9000/health
    
  3. Process already terminated

    • The process may have finished between the request and processing
    • The job is still marked Killed in that case
  4. Missing TERMINATE_URL configuration

    • Server needs SERVICE_<NAME>_TERMINATE_URL configured
    • Solution: Add to server configuration:
    export SERVICE_EXAMPLE_TERMINATE_URL=http://client:9000/kill
    

Jobs Stuck in Locked

Symptom: Job stays in Locked status indefinitely

Possible Causes:

Locked only lasts while the server waits for the client’s answer to a termination request. It stays longer when:

  1. The client accepted the connection but does not answer

    • The server sets no HTTP timeout, so it waits as long as the connection stays open
    • Check client logs and health
  2. The server restarted during a termination

    • Nothing picks up Locked jobs afterwards, so they stay Locked permanently
    • They are not polled and do not count toward quotas
    • Fix the status in the database manually, or send another POST /terminate/{id}

Jobs Stuck in Prepared

Symptom: Payloads stay in Prepared status

Possible Causes:

  1. Runner task not running

    Check client logs, may need restart

  2. Permission issues

    Client working directory may have permission issues

A script that fails validation does not stay Prepared: it becomes Invalid (see below).

Job Status is Invalid

Symptom: Job status is Invalid — the client rejected the script before running it

Possible Causes:

  1. Missing exit trap — script must contain:

    trap 'echo $? > .orchestrator.exit' EXIT
    
  2. Script too large — maximum script size is 20 MiB

  3. Script not valid UTF-8

  4. Script contains a blocked pattern (the client log names the pattern; false positives are possible, for example rm with any relative path containing /). See Script Validation for the full list (network tools, destructive commands, privilege escalation, etc.)

Invalid jobs are cleaned up after MAX_AGE like any other terminal state.

Job Status is Lost

Symptom: Job status is Lost

The job’s process disappeared without writing .orchestrator.exit. The client log has a warning such as payload 12 lost: pid 4321 is gone and no exit file at "/opt/data/12".

Possible Causes:

  1. Client restarted or crashed while the job ran (for example a container restart). Drain clients before restarting them, see Production
  2. The bash process running run.sh was killed with SIGKILL (for example kill -9, or the OOM killer picking it). SIGKILL cannot be trapped, so the exit code is never written

Solution: Lost is terminal. Use /download_partial/{id} to look at what the job wrote, then resubmit it.

Execution Fails

Symptom: Jobs complete but with Failed status

Check:

  1. Exit code

    run.sh must exit with code 0 for success:

    #!/bin/bash
    # Your commands here
    exit 0  # Explicit success
    
  2. Script errors

    Check output files for error messages. /download/{id} does not return files for Failed jobs; use /download_partial/{id}

  3. Missing dependencies

    Your script may need tools not available in the container

Database Issues

Database Locked

Symptom: “database is locked” errors

Causes: Multiple processes accessing SQLite

Solution:

  • Ensure only one server instance runs
  • Check for zombie processes
  • Restart server

Database Corrupted

Symptom: Strange errors, missing data

Solution:

  1. Stop server

  2. Backup current database

  3. Run integrity check:

    sqlite3 db.sqlite "PRAGMA integrity_check;"
    
  4. If corrupted, restore from backup or delete and restart

Out of Disk Space

Symptom: “disk full” errors

Solution:

  1. Check disk usage:

    df -h
    
  2. Clean old jobs:

    # Reduce MAX_AGE and restart
    export MAX_AGE=3600  # 1 hour
    
  3. Manually clean data directory

Docker Issues

Container Exits Immediately

Check logs:

docker logs container_name

Common causes:

  • Missing environment variables
  • Port conflicts
  • Permission issues

Cannot Connect Between Containers

Ensure same network:

services:
  server:
    networks:
      - app-network
  client:
    networks:
      - app-network

networks:
  app-network:

Use service names, not localhost:

# Wrong
SERVICE_EXAMPLE_UPLOAD_URL=http://localhost:9000/submit

# Correct
SERVICE_EXAMPLE_UPLOAD_URL=http://client:9000/submit

Volume Permission Issues

Symptom: Permission denied when writing to volumes

Solution:

services:
  server:
    user: "1000:1000"  # Match host user
    volumes:
      - ./data:/opt/data

Or fix permissions:

sudo chown -R 1000:1000 ./data

Performance Issues

Slow Job Processing

Possible Causes:

  1. Slow database

    • Use SSD storage for database
    • Run VACUUM periodically
  2. Network latency

    • Place server and clients on same network
    • Check for packet loss
  3. Client overloaded

    • Add more clients
    • Reduce RUNS_PER_USER

High Memory Usage

Server:

  • /download/{id} reads the whole output.zip into memory before sending it, so large results cause spikes
  • Uploads are streamed to disk and do not

Client:

  • /submit holds the whole upload in memory, and /retrieve//retrieve_partial build or read the whole archive in memory
  • The database is file-backed and is not a significant factor

Disk Usage Growing

Check:

du -sh /opt/data/*

Solutions:

  • Reduce MAX_AGE (the cleanup interval itself is fixed at 60 seconds)
  • Archive old results externally

Getting Help

If you can’t resolve an issue:

  1. Check logs for specific error messages
  2. Search existing issues: Codeberg Issues
  3. Open new issue with:
    • Version
    • Configuration
    • Steps to reproduce
    • Logs

See Also