Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

job-orchestrator is an asynchronous job orchestration system for managing and distributing computational workloads across heterogeneous computing resources with intelligent quota-based load balancing.

It serves as a middleware layer that connects web applications to computing resources: users submit files and a run.sh script via a REST API; the server queues and routes jobs to client nodes that execute them; results are returned as a ZIP archive.

It is domain-agnostic: any workload that can be expressed as a run.sh script plus input files can be orchestrated, whether that is a scientific simulation, a data-processing pipeline, or a media conversion task.

The project was originally developed as the job backend for a production web portal that exposes research software as web services.

Key Features

  • Asynchronous Job Management: Built with Rust and Tokio for high-performance async operations
  • Quota-Based Load Balancing: Per-user, per-service quotas prevent resource exhaustion
  • Dual-Mode Architecture: Runs as server (job orchestration) or client (job execution)
  • RESTful API: Simple HTTP interface for job submission, retrieval, and cancellation
  • Automatic Cleanup: Configurable retention policies for completed jobs
  • Job Termination: Cancel running jobs via API endpoints

Use Cases

job-orchestrator is designed for scenarios requiring:

  • Scientific Computing Workflows: Distribute simulations and analyses across clusters
  • Multi-Tenant Systems: Fair resource allocation with per-user quotas
  • Heterogeneous Computing: Route jobs to appropriate backends (local, HPC, cloud)
  • Web Platforms with Heavy Backends: Decouple the frontend from compute infrastructure
  • Batch Processing: Handle high-throughput job submissions with automatic queuing

Get Started

Project Status

Current State: Production-ready with server/client architecture

Planned Features:

  • DIRAC Interware integration
  • SLURM direct integration
  • Enhanced monitoring and metrics
  • Job priority queues
  • Advanced scheduling policies

Getting Help

License

MIT License - see LICENSE for details.


Parts of this documentation were automatically generated with the assistance of AI tools and human reviewed for accuracy.