Quota System
The quota system ensures fair resource allocation by limiting concurrent jobs per user and per service.
Two Independent Limits
The sender enforces two limits before dispatching a queued job:
| Variable | Scope | Default | Description |
|---|---|---|---|
SERVICE_<NAME>_RUNS_PER_USER | Per user, per service | 5 | Max jobs a single user can have active at once |
SERVICE_<NAME>_MAX_RUNS | Per service (all users) | 10 | Max jobs the service can have active at once |
Both limits must have available slots for a job to be dispatched. A job that
would violate either limit remains in Queued status until a slot opens.
Jobs in Processing, Submitted, or Running status all count toward
both limits. A job whose process disappears (for example after a client
restart) is marked Lost and releases its slot (see
Lost Processes).
How It Works
SERVICE_EXAMPLE_RUNS_PER_USER=3
SERVICE_EXAMPLE_MAX_RUNS=5
User 1 submits 4 jobs → 3 dispatched (per-user limit reached), 1 queued
User 2 submits 4 jobs → 2 dispatched (MAX_RUNS=5 reached), 2 queued
When one of User 1's jobs completes, one service slot frees up.
Both users now have a queued job and a free per-user slot, so either
User 1's or User 2's job is dispatched next. Which user goes first in a
sender pass is not deterministic.
Quota Scope
Quotas are enforced per user, per service (RUNS_PER_USER) and per service (MAX_RUNS):
| User | Service | RUNS_PER_USER | MAX_RUNS | Result |
|---|---|---|---|---|
| user_1 | example | 5 | 10 | Up to 5 concurrent |
| user_2 | example | 5 | 10 | Up to 5 concurrent (independent) |
| user_1 | simulate | 2 | 4 | Up to 2 concurrent |
Users don’t compete with each other for per-user slots. They do share the
MAX_RUNS pool for the service.
Scheduling: Round-Robin Between Users
When multiple users have queued jobs for the same service, the sender distributes slots in round-robin order across users. This prevents a single user with many queued jobs from consuming all available slots ahead of others.
The rotation restarts on every sender pass (every 500ms) and the order of users within a pass is arbitrary. Within one user, jobs are in practice taken oldest first (there is no explicit ordering or priority).
Quota States
┌──────────┐ Both limits OK ┌────────────┐
│ Queued │ ─────────────────▶ │ Processing │
└──────────┘ └────────────┘
│ │
│ Either limit exceeded ▼
│ ┌────────────┐
└──────────────────────── │ Submitted │
(waits for a free slot) │ (running) │
└────────────┘
Choosing Values
Factors to Consider
- Client capacity:
MAX_RUNSshould not exceed what your client hardware can handle concurrently - Job duration: Longer jobs need lower per-user quotas
- Resource usage: CPU/memory intensive jobs need lower
MAX_RUNS - User base: More users may need lower per-user quotas
Guidelines
| Job Type | Suggested RUNS_PER_USER | Suggested MAX_RUNS |
|---|---|---|
| Quick jobs (< 1 min) | 10-20 | 20-50 |
| Medium jobs (1-10 min) | 5-10 | 10-20 |
| Long jobs (10+ min) | 2-5 | 4-10 |
| Resource-intensive | 1-3 | 2-4 |
Example Scenarios
Scientific Computing Platform
# Quick validation jobs
SERVICE_VALIDATE_RUNS_PER_USER=10
SERVICE_VALIDATE_MAX_RUNS=50
# Standard analysis
SERVICE_ANALYZE_RUNS_PER_USER=5
SERVICE_ANALYZE_MAX_RUNS=20
# Heavy simulation
SERVICE_SIMULATE_RUNS_PER_USER=2
SERVICE_SIMULATE_MAX_RUNS=4
Educational Platform
# Student exercises
SERVICE_EXERCISE_RUNS_PER_USER=3
SERVICE_EXERCISE_MAX_RUNS=30
# Final projects
SERVICE_PROJECT_RUNS_PER_USER=5
SERVICE_PROJECT_MAX_RUNS=20
Testing Quotas
Submit multiple jobs to observe throttling:
Start the server with low limits (the variables must be set in the server’s environment, for example in docker-compose.yml, not in the shell you submit from):
SERVICE_EXAMPLE_RUNS_PER_USER=2
SERVICE_EXAMPLE_MAX_RUNS=3
Use a run.sh that takes a while (for example sleep 60 after the trap), then submit 5 jobs and check them:
for i in {1..5}; do
curl -s -X POST http://localhost:5000/upload \
-F "file=@run.sh" \
-F "user_id=1" \
-F "service=example" | jq -r '.id'
done
sleep 2
for id in 1 2 3 4 5; do
curl -s http://localhost:5000/download/$id | jq -r '"\(.id) \(.status)"'
done
Expected behaviour (the upload response always says Queued, so check /download):
- Jobs 1-2: dispatched (
Submitted/Running); the per-user limit of 2 is now reached - Jobs 3-5: stay
Queued.MAX_RUNS=3would allow one more, but not for this user - Submitting as
user_id=2at this point would dispatch one job and fillMAX_RUNS
Fair Scheduling
The quota system provides basic fairness:
- Per-user isolation: One user can’t starve others
- Per-service cap: Prevents overloading a single client
- Round-robin dispatch: Slots are shared evenly across users
- Automatic queuing: No jobs are rejected, just delayed
Limitations
Current limitations (improvements planned):
- No priority queues (FIFO within quota constraints)
- No time-based quotas (e.g., jobs per hour)
- No burst allowances