Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Partial Downloads and Debugging

This guide demonstrates how to use the /download_partial and /retrieve_partial endpoints to debug stuck or incomplete jobs.

Use Cases

Use partial download endpoints when:

  • A job appears stuck in Submitted or Running status
  • A job ended as Failed and you want its output (/download only returns ZIPs for Completed jobs)
  • You need to inspect the current working directory
  • Debugging why a job isn’t progressing
  • Verifying intermediate outputs of long-running jobs

Server-Side: /download_partial/

Example: Checking Incomplete Job

# Download current state of job ID 1
curl -o partial_results.zip http://localhost:5000/download_partial/1

# Extract and inspect contents
unzip -l partial_results.zip
unzip partial_results.zip

# View specific files
cat intermediate_output.txt

Client-Side: /retrieve_partial/

This endpoint is called internally by the server’s /download_partial endpoint. It provides the same functionality at the client level.

The server does not have a separate setting for this URL. It takes the service’s DOWNLOAD_URL and replaces its last path segment with retrieve_partial (http://client:9000/retrieve becomes http://client:9000/retrieve_partial). If you put a proxy in front of the client, make sure that path is routed too.

In the shipped Docker Compose setup the client has no ports: mapping — it’s only reachable from the server over the internal network (e.g. http://client:9000), not from the host. The example below assumes you’re running the client directly (outside Docker Compose) or calling it from another container on the same network.

Example (Client Developer)

# Get partial results directly from client
curl -o partial_results.zip http://localhost:9000/retrieve_partial/1

Comparison: Full vs Partial Download

EndpointReturnsUse Case
/download/{id}Final results onlyNormal completion
/download_partial/{id}Current state alwaysDebugging, monitoring

Combined Example Script

#!/bin/bash

SERVER_URL="http://localhost:5000"

# Submit job and parse the returned ID
echo "Submitting job..."
RESPONSE=$(curl -s -X POST "${SERVER_URL}/upload" \
  -F "file=@run.sh" \
  -F "user_id=1" \
  -F "service=example")
echo "$RESPONSE"
JOB_ID=$(echo "$RESPONSE" | jq -r '.id')

if [ -z "$JOB_ID" ] || [ "$JOB_ID" = "null" ]; then
  echo "Failed to get job ID from response"
  exit 1
fi

echo "Job ID: $JOB_ID"

# Poll until complete, saving partial snapshots for inspection
echo "Monitoring job..."
for i in $(seq 1 10); do
  sleep 10
  CT=$(curl -s -o response.tmp -w '%{content_type}' "${SERVER_URL}/download/${JOB_ID}")
  if [[ "$CT" == application/zip* ]]; then
    mv response.tmp final_results.zip
    echo "Job completed, results saved to final_results.zip"
    exit 0
  fi
  STATUS=$(jq -r '.status' response.tmp)
  case "$STATUS" in
    Failed|Invalid|Killed|Lost|Cleaned)
      echo "Job ended with status $STATUS"
      curl -s -o final_partial.zip "${SERVER_URL}/download_partial/${JOB_ID}"
      exit 1
      ;;
  esac
  echo "Check $i: status=$STATUS, saving partial snapshot..."
  curl -s -o "partial_${i}.zip" "${SERVER_URL}/download_partial/${JOB_ID}"
  unzip -l "partial_${i}.zip" 2>/dev/null | head -20
done

echo "Job did not complete within the polling window"
exit 1

Limitations

  • Queued jobs have not reached a client yet, so /download_partial returns 404 (“Job payload not found on client”)
  • Until run.sh writes something, the ZIP only contains the input files
  • After the client’s cleaner has removed the payload directory, the request fails
  • Partial downloads of running jobs may be incomplete or inconsistent
  • Use partial downloads for debugging, not for production result retrieval

Security Notes

  • Building the ZIP fails if any entry in the payload directory resolves outside it (for example, a symlink created by run.sh that points to /etc), so such files cannot be exfiltrated this way
  • See Path Traversal Protection for details