Introduction
job-orchestrator is an asynchronous job orchestration system for managing and distributing computational workloads across heterogeneous computing resources with intelligent quota-based load balancing.
It serves as a middleware layer that connects web applications to computing resources: users submit files and a run.sh script via a REST API; the server queues and routes jobs to client nodes that execute them; results are returned as a ZIP archive.
It is domain-agnostic: any workload that can be expressed as a run.sh script plus input files can be orchestrated, whether that is a scientific simulation, a data-processing pipeline, or a media conversion task.
The project was originally developed as the job backend for a production web portal that exposes research software as web services.
Key Features
- Asynchronous Job Management: Built with Rust and Tokio for high-performance async operations
- Quota-Based Load Balancing: Per-user, per-service quotas prevent resource exhaustion
- Dual-Mode Architecture: Runs as server (job orchestration) or client (job execution)
- RESTful API: Simple HTTP interface for job submission, retrieval, and cancellation
- Automatic Cleanup: Configurable retention policies for completed jobs
- Job Termination: Cancel running jobs via API endpoints
Use Cases
job-orchestrator is designed for scenarios requiring:
- Scientific Computing Workflows: Distribute simulations and analyses across clusters
- Multi-Tenant Systems: Fair resource allocation with per-user quotas
- Heterogeneous Computing: Route jobs to appropriate backends (local, HPC, cloud)
- Web Platforms with Heavy Backends: Decouple the frontend from compute infrastructure
- Batch Processing: Handle high-throughput job submissions with automatic queuing
Get Started
- Quick Start — up and running with Docker Compose in minutes
- Your First Job — submit and retrieve a job
- Architecture Overview — understand how the system works
Project Status
Current State: Production-ready with server/client architecture
Planned Features:
- DIRAC Interware integration
- SLURM direct integration
- Enhanced monitoring and metrics
- Job priority queues
- Advanced scheduling policies
Getting Help
- API Documentation: Available via Swagger UI at
/swaggerwhen running - Issues: Codeberg Issues
- Email: Rodrigo V. Honorato rvhonorato@protonmail.com
License
MIT License - see LICENSE for details.
Parts of this documentation were automatically generated with the assistance of AI tools and human reviewed for accuracy.