Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Docker Deployment

Docker is the recommended way to deploy job-orchestrator in production.

Quick Start

git clone https://codeberg.org/rvhonorato/job-orchestrator.git
cd job-orchestrator
docker compose up --build

This starts:

  • Server on port 5000 (exposed to the host)
  • Example client on port 9000 (internal network only, not exposed to the host)

Docker Images

Official Image

docker pull codeberg.org/rvhonorato/job-orchestrator:latest

The image has no entrypoint: the binary lives at /job-orchestrator and is not on PATH, so always pass the full command (/job-orchestrator server or /job-orchestrator client).

The published image contains the binary plus bash and sqlite, without any application. It works as a server as-is. To use it as a client, build your own image on top of it with the tools your run.sh needs. example/Dockerfile shows how, adding the gdock demo application.

Build Locally

# Build the job-orchestrator image (server, and base for clients)
docker build -t job-orchestrator .

# Build the example client image (adds gdock on top of the local image)
docker build --build-arg BASE_IMAGE=job-orchestrator -t job-orchestrator-client example/

Docker Compose

Basic Setup

services:
  server:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "server"]
    ports:
      - "5000:5000"
    environment:
      PORT: 5000
      DB_PATH: /opt/data/db.sqlite
      DATA_PATH: /opt/data
      MAX_AGE: 864000
      MAX_UPLOAD_SIZE: 419430400
      SERVICE_EXAMPLE_UPLOAD_URL: http://client:9000/submit
      SERVICE_EXAMPLE_DOWNLOAD_URL: http://client:9000/retrieve
      SERVICE_EXAMPLE_TERMINATE_URL: http://client:9000/kill
      SERVICE_EXAMPLE_RUNS_PER_USER: 5
      SERVICE_EXAMPLE_MAX_RUNS: 10
    volumes:
      - server-data:/opt/data

  client:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
      DATA_PATH: /opt/data
      DB_PATH: /opt/data/db.sqlite
    volumes:
      - client-data:/opt/data
    depends_on:
      - server

volumes:
  server-data:
  client-data:

Production Setup

See Production Deployment - Container Hardening for details on each security option.

Multiple Clients

Scaling Horizontally

A service is bound to exactly one client URL, so adding more clients for the same service does not spread its jobs. Extra clients only receive work if they are configured as separate services, as shown below. Do not place a load balancer in front of several clients: payload IDs are local to each client, so status and result requests must reach the client that got the job.

Multiple Services

services:
  server:
    environment:
      # Light jobs
      SERVICE_LIGHT_UPLOAD_URL: http://client-light:9000/submit
      SERVICE_LIGHT_DOWNLOAD_URL: http://client-light:9000/retrieve
      SERVICE_LIGHT_TERMINATE_URL: http://client-light:9000/kill
      SERVICE_LIGHT_RUNS_PER_USER: 10

      # Heavy jobs
      SERVICE_HEAVY_UPLOAD_URL: http://client-heavy:9000/submit
      SERVICE_HEAVY_DOWNLOAD_URL: http://client-heavy:9000/retrieve
      SERVICE_HEAVY_TERMINATE_URL: http://client-heavy:9000/kill
      SERVICE_HEAVY_RUNS_PER_USER: 2

  client-light:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G

  client-heavy:
    image: codeberg.org/rvhonorato/job-orchestrator:latest
    command: ["/job-orchestrator", "client"]
    environment:
      PORT: 9000
    deploy:
      resources:
        limits:
          cpus: '8'
          memory: 16G

Volume Management

Persistent Storage

Always use named volumes for production:

volumes:
  server-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/job-orchestrator/server

  client-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/job-orchestrator/client

Backup Strategy

# Stop services (optional, for consistent backup)
docker compose stop

# Backup server and client data (the client holds results of running jobs)
tar -czf backup-$(date +%Y%m%d).tar.gz /data/job-orchestrator/server /data/job-orchestrator/client

# Resume services
docker compose start

Networking

Internal Network

Keep client internal:

services:
  server:
    ports:
      - "5000:5000"  # Exposed to host
    networks:
      - internal
      - external

  client:
    # No ports exposed to host
    networks:
      - internal

networks:
  internal:
    internal: true
  external:

With Reverse Proxy

services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
    depends_on:
      - server

  server:
    # No ports exposed, accessed via nginx
    networks:
      - internal

Logging

View Logs

# All services
docker compose logs -f

# Server only
docker compose logs -f server

# Last 100 lines
docker compose logs --tail 100 server

Log Rotation

services:
  server:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Commands

# Start
docker compose up -d

# Stop
docker compose down

# Restart
docker compose restart

# Rebuild and start
docker compose up --build -d

# View status
docker compose ps

# Shell into container
docker compose exec server /bin/sh

See Also