Harbor datapoint

Job flow

Set SANDBOX_GATEWAY_URL once (local default http://localhost:8780):

export SANDBOX_GATEWAY_URL="${SANDBOX_GATEWAY_URL:-http://localhost:8780}"

Enqueue (v2)

Send x-correlation-id on enqueue so the same id appears in logs, worker traces, and on the Langfuse trace (metadata correlation_id). See Observability.

curl -sS -H "X-Api-Key: dev-local-key" \
  -H "x-correlation-id: my-eval-run-42" \
  -H "Content-Type: application/json" \
  -d '{
    "task_slug": "my-task",
    "metadata": {"task_archive_url": "https://..."},
    "agents": [{
      "name": "a",
      "harbor_agent": "oracle",
      "harbor_extensions": {"agent_env": {"FOO": "bar"}}
    }]
  }' \
  "$SANDBOX_GATEWAY_URL/sandbox/harbor/v2/jobs/execute-tasks"

Returns 202 with {"job_id": "...", "status": "queued"}. Poll GET /sandbox/harbor/v2/jobs/{job_id}.

task_archive_url goes inside metadata. Sent at the top level it is ignored and the request fails with metadata.task_archive_url is required.

The task archive contract

You host the archive; the platform fetches it. There is no upload endpoint and no task registry to push to, so the zip has to live somewhere the worker can reach over HTTP for the duration of the run.

A URL that breaks one of the first four rules is refused with 400 at request time on both routes, rather than being accepted and failing minutes later. The size bounds cannot be known without fetching, so they surface as the run's error.

Generating a presigned URL for a private GCS object:

gsutil signurl -d 1h service-account.json gs://my-bucket/tasks/my-task.zip

The size limits are deployment-tunable (SANDBOX_HARBOR_MAX_ARCHIVE_BYTES, SANDBOX_HARBOR_MAX_EXTRACTED_BYTES, SANDBOX_HARBOR_MAX_ARCHIVE_ENTRIES) but never absent: a worker's disk is shared with every other job on its node, so an unbounded archive is one consumer's zip against everyone else's runs.

Two routes, two contracts

run/datapoint blocks until the run finishes and returns the whole response rather than a job_id; it also sets Server-Timing: total;dur=…. Earlier revisions of this page showed that route while describing the 202-and-poll contract, which belongs to execute-tasks.

Both routes draw on the same per-project concurrency quota, so the sync route is not a way around a full queue: over the limit it answers 429 with Retry-After. The async route accepts the job and the worker retries it as capacity frees, which is why a queued job can sit in queued for a while under load rather than failing.

Underneath, both routes put the work on the same queue and the same worker runs it — the sync one waits for the verdict on your behalf. That matters in one case you can observe: if the run outlasts the synchronous wait, you get 504 with the job id in the response headers, and the run keeps going. Poll GET /jobs/{job_id} or follow its event stream; do not resubmit, or you will pay for the same task twice. If your tasks routinely take that long, use execute-tasks and skip the waiting connection entirely.

Using oracle as the harbor_agent, as above, requires the task tree to carry solution/solve.sh — the oracle applies that solution rather than solving the task, which is what makes it the way to check a task and its verifier are sound.

Verify task (authoring)

When you only need to answer "does tests/test.sh pass for this workspace state?" — without an LLM agent or solution/solve.sh — use verify-task. Results always set purpose: task_authoring and valid_for_scoring: false; they are not benchmark scores.

Workspace seeding order:

1. Materialize the task archive (same metadata.task_archive_url contract as above).
2. Copy visible_seed/* into /workspace when present in the task tree.
3. Optionally overlay a workspace_fixture_url zip (fixture wins on path conflicts).
4. Run tests/test.sh in the task's environment/Dockerfile image.

curl -sS -H "X-Api-Key: dev-local-key" \
  -H "Content-Type: application/json" \
  -d '{
    "task_slug": "my-task",
    "metadata": {"task_archive_url": "https://storage.example/tasks/my-task.zip"},
    "workspace_fixture_url": "https://storage.example/fixtures/my-workspace.zip"
  }' \
  "$SANDBOX_GATEWAY_URL/sandbox/harbor/v2/jobs/verify-task"

Poll GET /sandbox/harbor/v2/jobs/{job_id} for result.verifier.exit_code, reward, and artifact_download_urls. For end-to-end task soundness (solution + verifier), keep using datapoint with harbor_agent: oracle.

Langfuse groups LLM spans under session_id = job_id when tracing is enabled.

Reading the result: status is not a verdict

A completed job carries two different judgements, and conflating them will eventually make you score a run that never happened.

status answers "did the orchestration finish?". outcome answers "should you believe the rewards?". Gate scoring on outcome.valid_for_scoring, not on status.

{
  "status": "failed",
  "outcome": {
    "status": "errored",
    "valid_for_scoring": false,
    "reasons": [
      "codex: 1 trial(s) raised (AgentTimeoutError); an incomplete rollout cannot be scored",
      "codex: the verifier report contains zero tests; exiting 0 without collecting a test is not evidence of a pass"
    ],
    "trials": { "scored": 1, "errored": 1, "successful": 0 },
    "exceptions": ["AgentTimeoutError"],
    "verifier": { "evidence": "empty", "tests": 0 }
  }
}

A run is marked invalid when a trial raised, when nothing was scored, or when the verifier did not prove that tests ran — either because its report contained zero tests (evidence: "empty") or because it produced no report at all (evidence: "none"). A verifier that exits 0 without collecting a test has demonstrated nothing, and reading it as a pass fabricates a result.

evidence: "none" requires a word of explanation, because it is the common case for a task whose verifier is a shell script writing reward.txt. Such a verifier may well have checked the work properly — but nothing in what it produced says so, and from the outside it is indistinguishable from one that writes 1.0 without looking. So the platform withholds the claim rather than granting it. This is a change: absence of a report used to satisfy valid_for_scoring, and that let the exact defect described above through by a shorter route.

Withholding the claim is not a job failure. status stays succeeded and the rewards are returned untouched; only valid_for_scoring says false, with the reason naming the override. If you trust the verifier and want to score on its reward alone, say so explicitly:

{ "task_slug": "...", "agents": [...], "require_verifier_evidence": false }

A deployment can opt out for every request with SANDBOX_HARBOR_REQUIRE_VERIFIER_EVIDENCE=false, and an explicit value on the request always wins. Calibration pipelines should leave the default alone — a fabricated pass does the most damage there. An empty report is refused even with the override, because a report accounting for zero tests contradicts itself; only absence is a matter of trust.

The best answer, where you control the task, is to make the verifier emit a CTRF report at /logs/verifier/ctrf.json. Then the reward carries a named check behind it and no trust setting is needed.

One thing this deliberately does not do: rewards are never rewritten. agent_results[].result stays exactly as Harbor recorded it, including a reward we are refusing to vouch for, so an audit can see what happened.

status becomes failed when no trial produced a real result. A partially errored job stays succeeded — some trials did work — with the detail in outcome.

Pass@k roster and selective retry

When pass_at_k > 1, each attempt appears in result.outcome.trials_detail:

"trials_detail": [
  {
    "trial_name": "extracted__AqpFqaL",
    "status": "completed",
    "reward": 0.0,
    "exception": null
  },
  {
    "trial_name": "extracted__XnyGhDD",
    "status": "errored",
    "reward": null,
    "exception": "TerminusNoProgressError",
    "exception_message": "pane unchanged for 6 episodes with a stuck-shell marker…"
  }
]

Do not treat pass@k as editable row seats. Refill the budget of fair attempts:

curl -sS -X POST \
  -H "X-Api-Key: $SANDBOX_API_KEY" \
  -H "Idempotency-Key: retry-$JOB_ID-$(date +%s)" \
  "$SANDBOX_GATEWAY_URL/sandbox/harbor/v2/jobs/$JOB_ID/retry-errored-trials"

202 body:

{
  "job_id": "<child>",
  "retry_of": "<parent>",
  "pass_at_k": 3,
  "retried_exceptions": ["TerminusNoProgressError"],
  "status": "queued"
}

• pass_at_k on the child equals how many parent trials errored (1 if only one failed; 3 if three failed) — not “retry trial #6 only”.
• Parent archive stays immutable. Combine parent + child for an overall pass@k client-side.
• v1 supports single-agent datapoints only. Jobs enqueued before request snapshots existed return 400 — resubmit via execute-tasks for those.
• Side effects: notifications / deliver / writeback are cleared on the child by default so a refill does not re-fire webhooks. Pass ?include_side_effects=true to keep them.
• Idempotency: send Idempotency-Key to get the same child job_id on a double-click.

Unset harbor_extensions.n_concurrent defaults to min(pass_at_k, 16) so pass@8 fans out with -n 8. Terminus busy-hangs at a heredoc/> EOF prompt fail that trial early (TerminusNoProgressError) instead of holding siblings until agent.timeout_sec. Operator knobs (Harbor worker): SANDBOX_TERMINUS_NO_PROGRESS (default on), SANDBOX_TERMINUS_STUCK_EPISODES (default 6).

SSE events

curl -N -H "X-Api-Key: dev-local-key" \
  "$SANDBOX_GATEWAY_URL/sandbox/harbor/v2/jobs/{job_id}/events"

Artifacts

• GET .../jobs/{job_id}/artifacts — inventory
• GET .../jobs/{job_id}/artifacts/archive — presigned archive URL

See Harbor v2 API reference for full schemas.