Harbor

Run a benchmark task and collect its results. Use these paths for new integrations.

Every request needs a credential — see Authentication.

Method	Path	Summary
`GET`	`/sandbox/harbor/api-versions`	List Harbor API versions
`GET`	`/sandbox/harbor/v2/capabilities`	Harbor API v2 capabilities
`GET`	`/sandbox/harbor/v2/jobs/{job_id}`	Get job status (v2)
`GET`	`/sandbox/harbor/v2/jobs/{job_id}/artifacts`	List job artifacts (v2)
`GET`	`/sandbox/harbor/v2/jobs/{job_id}/artifacts/archive`	Download job archive (v2)
`GET`	`/sandbox/harbor/v2/jobs/{job_id}/artifacts/test-stdout/{relative_path}`	Download verifier test stdout (v2)
`POST`	`/sandbox/harbor/v2/jobs/{job_id}/cancel`	Cancel job (v2)
`GET`	`/sandbox/harbor/v2/jobs/{job_id}/events`	SSE job events (v2)
`POST`	`/sandbox/harbor/v2/jobs/{job_id}/retry-errored-trials`	Retry only errored pass@k trials (v2)
`POST`	`/sandbox/harbor/v2/jobs/{job_id}/retry-provider-blocked`	Retry full pass@k after provider-blocked failure (v2)
`POST`	`/sandbox/harbor/v2/jobs/execute-tasks`	Enqueue Harbor datapoint job (v2)
`POST`	`/sandbox/harbor/v2/jobs/verify-task`	Enqueue Harbor verify-task job (v2)
`POST`	`/sandbox/harbor/v2/run/datapoint`	Run Harbor datapoint (v2)
`POST`	`/sandbox/harbor/v2/verify-task`	Run Harbor verifier only (v2)

Common headers

Accepted on every endpoint on this page.

Name	In	Type	Required	Description
`x-correlation-id`	header	string	no	End-to-end correlation id for this eval run. When set, the gateway and downstream services honor it verbatim in logs, Cloud Trace span attributes, async job payloads, and Langfuse trace.id. When omitted, the gateway generates a ULID and echoes it on the response.

Endpoints

List Harbor API versions

GET /sandbox/harbor/api-versions

Responses

Status	Description	Body
`200`	Successful Response	object

Harbor API v2 capabilities

GET /sandbox/harbor/v2/capabilities

Responses

Status	Description	Body
`200`	Successful Response	object

Get job status (v2)

GET /sandbox/harbor/v2/jobs/{job_id}

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Job status	object
`404`	No job with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError

Example response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running",
  "timings": {
    "enqueued_at": 1753420800.12,
    "started_at": 1753420802.44,
    "queue_wait_ms": 2320
  },
  "created_at": 1753420800.12,
  "updated_at": 1753420802.44
}

List job artifacts (v2)

GET /sandbox/harbor/v2/jobs/{job_id}/artifacts

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Successful Response	object
`422`	Validation Error	HTTPValidationError

Download job archive (v2)

GET /sandbox/harbor/v2/jobs/{job_id}/artifacts/archive

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`presigned`	query	boolean	no
`as`	query	string \| null	no	Set to "json" to receive {"url": ...} instead of a 307 redirect. Requires presigned=true.
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Successful Response	any
`404`	No artifact with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError

Download verifier test stdout (v2)

GET /sandbox/harbor/v2/jobs/{job_id}/artifacts/test-stdout/{relative_path}

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`relative_path`	path	string	yes
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Successful Response	any
`404`	No artifact with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError

Cancel job (v2)

POST /sandbox/harbor/v2/jobs/{job_id}/cancel

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Successful Response	object
`404`	No job with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError

SSE job events (v2)

GET /sandbox/harbor/v2/jobs/{job_id}/events

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`Last-Event-ID`	header	string \| null	no
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`200`	Server-sent events stream	any
`404`	No job with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError

Retry only errored pass@k trials (v2)

POST /sandbox/harbor/v2/jobs/{job_id}/retry-errored-trials

Batch-refill incomplete pass@k attempts for a terminal datapoint job. Counts result.outcome.trials_detail rows with status=errored and enqueues a new job with pass_at_k=N_errored plus metadata.retry_of / retry_kind=errored_trials. Completed attempts (including reward 0.0) are not retried. Parent archive is immutable; combine scores client-side. By default notifications/deliver/writeback are not copied (pass include_side_effects=true to keep them). Optional Idempotency-Key returns the same child job on repeat. v1: single-agent datapoints only. Requires a durable request snapshot from enqueue. Returns 400 when nothing errored or the snapshot/contract is missing; 409 when the parent is still running.

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`include_side_effects`	query	boolean	no	When true, copy the parent's notifications/deliver/writeback onto the child. Default false so a refill does not re-fire webhooks.
`Idempotency-Key`	header	string \| null	no
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`202`	Follow-up job accepted	object
`400`	Nothing to retry, multi-agent, or missing request snapshot	—
`409`	Parent job is not terminal	—
`422`	Validation Error	HTTPValidationError

Example response:

{
  "job_id": "child-uuid",
  "retry_of": "parent-uuid",
  "pass_at_k": 3,
  "retried_exceptions": [
    "TerminusNoProgressError"
  ],
  "status": "queued"
}

Retry full pass@k after provider-blocked failure (v2)

POST /sandbox/harbor/v2/jobs/{job_id}/retry-provider-blocked

Enqueue a new job with the parent's original pass_at_k when result.outcome.failure_class=provider (quota/auth/rate-limit). Unlike retry-errored-trials, this re-runs the full k budget with a fresh KeyHive lease. Sets metadata.retry_of / retry_kind=provider_blocked. Optional Idempotency-Key returns the same child job on repeat. v1: single-agent datapoints only.

Parameters

Name	In	Type	Required	Description
`job_id`	path	string	yes
`include_side_effects`	query	boolean	no
`Idempotency-Key`	header	string \| null	no
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Responses

Status	Description	Body
`202`	Follow-up job accepted	object
`400`	Parent is not provider-blocked or snapshot missing	—
`409`	Parent job is not terminal	—
`422`	Validation Error	HTTPValidationError

Enqueue Harbor datapoint job (v2)

POST /sandbox/harbor/v2/jobs/execute-tasks

Parameters

Name	In	Type	Required	Description
`Idempotency-Key`	header	string \| null	no
`X-Sandbox-Job-Id`	header	string \| null	no
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Request body

application/json, required — ExecuteTasksRequest

Field	Type	Required	Description
`task_slug`	string	yes
`agents`	DatapointAgentBody[]	yes	One or more agents to run against the same materialized task (max 8).
`metadata`	object	no
`artifacts`	string[] \| null	no
`sandbox`	string \| null	no
`notifications`	object[] \| null	no
`writeback`	HarborTaskWritebackSpec \| null	no
`deliver`	DeliverSpec \| null	no
`require_verifier_evidence`	boolean \| null	no	Whether outcome.valid_for_scoring requires the verifier to have produced a machine-readable test report. Defaults to true: a verifier that writes a reward without collecting any test proves nothing, and with no report there is no way to tell the difference. Set false when the task uses a shell verifier you trust and you want to score on its reward alone — the rewards themselves are returned either way, and the job still succeeds. Deployment-wide override: SANDBOX_HARBOR_REQUIRE_VERIFIER_EVIDENCE=false.

Responses

Status	Description	Body
`202`	Successful Response	object
`422`	Validation Error	HTTPValidationError

Enqueue Harbor verify-task job (v2)

POST /sandbox/harbor/v2/jobs/verify-task

Parameters

Name	In	Type	Required	Description
`Idempotency-Key`	header	string \| null	no
`X-Sandbox-Job-Id`	header	string \| null	no
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Request body

application/json, required — VerifyTaskRequest

Field	Type	Required
`task_slug`	string	yes
`metadata`	object	no
`workspace_fixture_url`	string \| null	no
`sandbox`	string \| null	no

Responses

Status	Description	Body
`202`	Successful Response	object
`422`	Validation Error	HTTPValidationError

Run Harbor datapoint (v2)

POST /sandbox/harbor/v2/run/datapoint

Parameters

Name	In	Type	Required	Description
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Request body

application/json, required — RunDatapointRequest

Field	Type	Required	Description
`task_slug`	string	yes
`agents`	DatapointAgentBody[]	yes	One or more agents to run against the same materialized task (max 8).
`metadata`	object	no
`artifacts`	string[] \| null	no
`sandbox`	string \| null	no
`notifications`	object[] \| null	no
`writeback`	HarborTaskWritebackSpec \| null	no
`deliver`	DeliverSpec \| null	no
`require_verifier_evidence`	boolean \| null	no	Whether outcome.valid_for_scoring requires the verifier to have produced a machine-readable test report. Defaults to true: a verifier that writes a reward without collecting any test proves nothing, and with no report there is no way to tell the difference. Set false when the task uses a shell verifier you trust and you want to score on its reward alone — the rewards themselves are returned either way, and the job still succeeds. Deployment-wide override: SANDBOX_HARBOR_REQUIRE_VERIFIER_EVIDENCE=false.

Example:

{
  "task_slug": "example-task",
  "metadata": {
    "task_archive_url": "https://storage.example/tasks/example-task.zip"
  },
  "agents": [
    {
      "name": "oracle",
      "harbor_agent": "oracle",
      "harbor_extensions": {
        "agent_env": {
          "SANDBOX_DOCS": "1"
        }
      }
    }
  ]
}

Responses

Status	Description	Body
`200`	Datapoint completed	object
`404`	No job with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError
`429`	The project is already running its maximum number of concurrent jobs. Nothing was enqueued, so retrying after `Retry-After` is safe and cannot duplicate a run. Use `POST /jobs/execute-tasks` instead if you would rather the platform queue the work than refuse it: the async route accepts the job with a `202` and lets it wait for capacity.	—
`504`	The run did not finish inside the synchronous wait. It has not been cancelled — it is still running, and the job id is in the response headers. Poll `GET /jobs/{job_id}`, or follow `GET /jobs/{job_id}/events`, rather than resubmitting. Use `POST /jobs/execute-tasks` instead if your tasks routinely run this long.	—

Example response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "succeeded",
  "results": [
    {
      "agent": "oracle",
      "reward": 1
    }
  ]
}

Run Harbor verifier only (v2)

POST /sandbox/harbor/v2/verify-task

Parameters

Name	In	Type	Required	Description
`X-Sandbox-Project-Id`	header	string \| null	no	Project to bill and rate-limit this request against. Validated against the credential: a project-scoped token may only name its own project, and gets 403 otherwise. Defaults to the token's project when omitted.
`X-Sandbox-Tenant-Id`	header	string \| null	no	Tenant for this request. Must match the credential's tenant when the credential carries one.

Request body

application/json, required — VerifyTaskRequest

Field	Type	Required
`task_slug`	string	yes
`metadata`	object	no
`workspace_fixture_url`	string \| null	no
`sandbox`	string \| null	no

Example:

{
  "task_slug": "my-task",
  "metadata": {
    "task_archive_url": "https://storage.example/tasks/my-task.zip"
  }
}

Responses

Status	Description	Body
`200`	Datapoint completed	object
`404`	No job with this id in this tenancy. Job records expire, so this is also the answer for a job that has aged out.	—
`422`	Validation Error	HTTPValidationError
`429`	The project is already running its maximum number of concurrent jobs. Nothing was enqueued, so retrying after `Retry-After` is safe and cannot duplicate a run. Use `POST /jobs/execute-tasks` instead if you would rather the platform queue the work than refuse it: the async route accepts the job with a `202` and lets it wait for capacity.	—
`504`	The run did not finish inside the synchronous wait. It has not been cancelled — it is still running, and the job id is in the response headers. Poll `GET /jobs/{job_id}`, or follow `GET /jobs/{job_id}/events`, rather than resubmitting. Use `POST /jobs/execute-tasks` instead if your tasks routinely run this long.	—

Example response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "succeeded",
  "results": [
    {
      "agent": "oracle",
      "reward": 1
    }
  ]
}

Schemas

The object shapes referenced above.

DatapointAgentBody

Field	Type	Required	Description
`name`	string	yes	Logical agent label in the datapoint result
`harbor_agent`	string	yes	Harbor agent id (catalog or factory)
`model`	string \| null	no	LLM model override
`pass_at_k`	integer	no	Default: 1.
`params`	object	no	Per-request agent kwargs forwarded to Harbor as `--ak key=value`. Merged with `harbor_extensions.agent_kwargs` (extensions win on conflict).
`harbor_extensions`	object \| null	no	Harbor passthrough: agent_env (--ae), agent_kwargs (--ak), sandbox (-e), n_concurrent, custom_agent_import. When harbor_agent is opencode: opencode_agent sets default_agent in opencode.json (plan \| build); opencode_config deep-merges an allowlisted subset of opencode.json (agent, permission, tools, model, small_model, subagent_depth, instructions, command, mcp) — e.g. a custom agent.review subagent.

DeliverSpec

v2 customer delivery — maps to staged_zip writeback internally.

Field	Type	Required	Description
`agent_run_zip`	PresignedDeliverable \| null	no	Upload agent outputs as a single zip after the run.

ExecuteTasksRequest

Field	Type	Required	Description
`task_slug`	string	yes
`agents`	DatapointAgentBody[]	yes	One or more agents to run against the same materialized task (max 8).
`metadata`	object	no
`artifacts`	string[] \| null	no
`sandbox`	string \| null	no
`notifications`	object[] \| null	no
`writeback`	HarborTaskWritebackSpec \| null	no
`deliver`	DeliverSpec \| null	no
`require_verifier_evidence`	boolean \| null	no	Whether outcome.valid_for_scoring requires the verifier to have produced a machine-readable test report. Defaults to true: a verifier that writes a reward without collecting any test proves nothing, and with no report there is no way to tell the difference. Set false when the task uses a shell verifier you trust and you want to score on its reward alone — the rewards themselves are returned either way, and the job still succeeds. Deployment-wide override: SANDBOX_HARBOR_REQUIRE_VERIFIER_EVIDENCE=false.

HTTPValidationError

Field	Type	Required	Description
`detail`	ValidationError[]	no

HarborTaskWritebackSpec

Field	Type	Required	Description
`enabled`	boolean	no	Default: true.
`destination`	WritebackDestination	yes
`mode`	"in_place_overwrite" \| "versioned_suffix" \| "staged_zip"	no	Default: "in_place_overwrite".
`include_paths`	string[] \| null	no
`include_run_outputs`	boolean	no	Default: true.
`include_judge_outputs`	boolean	no	Default: false.
`deliverable_files`	string[] \| null	no
`deliverable_destination_prefix`	string	no	Default: "outputs/{label}/".
`missing_deliverable_is_failure`	boolean	no	Default: true.
`include_telemetry`	boolean	no	Default: false.
`deliverables_only`	boolean	no	Default: false.

PresignedDeliverable

One presigned HTTP upload target.

Field	Type	Required	Description
`url`	string	yes	Presigned PUT (or POST) URL.
`method`	"PUT" \| "POST"	no	Default: "PUT".
`headers`	object	no
`key`	string	no	Logical artifact name in writeback outcomes. Default: "agent_run.zip".

PresignedPut

One presigned write target, keyed by relative path within the destination.

Field	Type	Required	Description
`key`	string	yes	Relative path under the destination root. The writeback step matches files to URLs by this key.
`url`	string	yes
`method`	"PUT" \| "POST"	no	Default: "PUT".
`headers`	object	no

RunDatapointRequest

Field	Type	Required	Description
`task_slug`	string	yes
`agents`	DatapointAgentBody[]	yes	One or more agents to run against the same materialized task (max 8).
`metadata`	object	no
`artifacts`	string[] \| null	no
`sandbox`	string \| null	no
`notifications`	object[] \| null	no
`writeback`	HarborTaskWritebackSpec \| null	no
`deliver`	DeliverSpec \| null	no
`require_verifier_evidence`	boolean \| null	no	Whether outcome.valid_for_scoring requires the verifier to have produced a machine-readable test report. Defaults to true: a verifier that writes a reward without collecting any test proves nothing, and with no report there is no way to tell the difference. Set false when the task uses a shell verifier you trust and you want to score on its reward alone — the rewards themselves are returned either way, and the job still succeeds. Deployment-wide override: SANDBOX_HARBOR_REQUIRE_VERIFIER_EVIDENCE=false.

ValidationError

Field	Type	Required
`loc`	string \| integer[]	yes
`msg`	string	yes
`type`	string	yes
`input`	any	no
`ctx`	object	no

VerifyTaskRequest

Field	Type	Required
`task_slug`	string	yes
`metadata`	object	no
`workspace_fixture_url`	string \| null	no
`sandbox`	string \| null	no

WritebackDestination

Field	Type	Required	Description
`url`	string \| null	no	Destination root URL (`s3://`, `gs://`, or presigned PUT for `staged_zip`). Optional when `presigned_writes` fully specifies targets.
`presigned_writes`	PresignedPut[] \| null	no
`storage_connector_id`	integer \| null	no

Machine-readable spec: OpenAPI YAML.

Harbor

On this page