Observability and correlation

Every Sandbox API request can carry a correlation id so your own system, the platform's workers, and LLM observability all share one key.

Following a run while it runs

A run is asynchronous and can take minutes. Two routes let you watch one rather than guess:

Route	Use
`GET /sandbox/agent/v1/runs/{job_id}`	Point-in-time status, timings and result
`GET /sandbox/agent/v1/runs/{job_id}/events`	Server-sent events, as they happen

Harbor datapoint jobs expose the same pair under /sandbox/harbor/v2/jobs/{job_id} and /sandbox/harbor/v2/jobs/{job_id}/events. You do not have to remember which: the 202 from an enqueue names both.

{
  "job_id": "agent-run-550e8400",
  "status": "queued",
  "status_url": "/sandbox/agent/v1/runs/agent-run-550e8400",
  "events_url": "/sandbox/agent/v1/runs/agent-run-550e8400/events"
}

The event stream

curl -N -H "X-Api-Key: $SANDBOX_API_KEY" \
  "$SANDBOX_GATEWAY_URL/sandbox/agent/v1/runs/$JOB_ID/events"

A meta frame arrives immediately, so an empty stream means a transport problem rather than a quiet job. Then one phase frame per transition, then done:

id: 0
event: meta
data: {"job_id":"agent-run-550e8400","delivery":"sse"}

id: 2
event: phase
data: {"event":"phase","phase":"materialize","state":"started"}

id: 4
event: phase
data: {"event":"phase","phase":"agent","state":"started","model":"anthropic/claude-sonnet-4-5"}

id: 5
event: phase
data: {"event":"phase","phase":"agent","state":"finished","duration_ms":112873}

event: done
data: {"job_id":"agent-run-550e8400","status":"succeeded","timings":{"run_ms":112873}}

Phase	What is happening
`admission`	Waiting for a concurrency slot against your project quota
`credentials`	A model provider key was leased. `model_key_leased: false` means the run will proceed without one, which is the usual reason an agent fails for no visible reason
`materialize`	Fetching and expanding the task archive, or synthesizing a workspace from your instruction
`agent`	The agent is working. On a datapoint run, one per agent, with `index` and `of`
`artifacts`	Publishing the job directory. `after_failure: true` means the run failed and this is the workspace being saved for you

Each phase reports started then finished with a duration_ms, or failed with an error_type. A stream that ends on a failed phase tells you which stage died. The exception message is deliberately not published, because it routinely contains a presigned URL or a provider error echoing a key — get the detail from the error field on the status route.

Reconnecting. Every frame carries a monotonic id. Send the last one you saw as Last-Event-ID and the stream resumes after it instead of replaying the run from the beginning.

These live phases are not the same thing as the harbor.phase.* spans in Cloud Trace. Those are reconstructed from the trial's result.json after the run finishes, with exact sub-phase timings — better for latency analysis, useless for watching a run in flight.

Request header

Header	Required	Behavior
`x-correlation-id`	No	If you send a non-empty value, Sandbox uses it verbatim everywhere. If omitted, the gateway generates a ULID and echoes it on the response.

export SANDBOX_GATEWAY_URL="${SANDBOX_GATEWAY_URL:-http://localhost:8780}"

curl -sS \
  -H "X-Api-Key: dev-local-key" \
  -H "x-correlation-id: prism-eval-run-42" \
  -H "Content-Type: application/json" \
  -d '{"task_slug":"my-task","metadata":{"task_archive_url":"https://..."},"agents":[{"name":"a","harbor_agent":"oracle"}]}' \
  "$SANDBOX_GATEWAY_URL/sandbox/harbor/v2/jobs/execute-tasks"

Check the response headers — you should see the same x-correlation-id value.

ID mapping

Id	Who sets it	Where it appears
correlation_id	Your client (preferred) or gateway ULID	Logs (`correlation_id` field), OTEL span attribute, Langfuse trace metadata `correlation_id`
job_id	Harbor (`202` response, poll URL)	Langfuse `session_id` (groups all LLM calls in one run)
otel trace_id	Automatic HTTP instrumentation	Cloud Trace; also Langfuse metadata `otel_trace_id` when enabled

Pick one stable correlation id per logical run — whatever your own system already calls it — and send it on the initial submission.

Async harbor jobs

For POST /sandbox/harbor/v2/jobs/execute-tasks (and Agent Runtime POST /sandbox/agent/v1/runs):

Gateway/harbor API reads x-correlation-id from the HTTP request.
The id is stored on the Pub/Sub job payload (correlation_id).
The worker restores it for logs and passes it into harbor subprocess env (SANDBOX_CORRELATION_ID, HARBOR_TRACE_ID).

You can filter Pub/Sub messages in GCP by attribute correlation_id.

Langfuse (LLM eval plane)

When the platform has Langfuse keys configured (LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, hosted at https://langfuse.turing.com):

LiteLLM generations from harbor/agent runs are traced in Langfuse.
Find your run by searching metadata correlation_id for the id you sent. Langfuse trace ids are 32-character hex OTEL ids, so your id cannot be the trace id itself; it is recorded on the trace instead, and the trace id is derived from it deterministically (Langfuse.create_trace_id(seed=<your id>)), so the same correlation id always resolves to the same trace.
session_id = harbor job_id, which groups every LLM call in one run.

This is separate from Cloud Trace (HTTP/SLO plane). Both can share the same correlation id for cross-linking.

Local make up works without Langfuse keys — tracing bootstrap is a no-op.

Python client

SandboxClient sends x-correlation-id when you pass correlation_id= (or legacy request_id=):

await client.submit_execute_tasks(
    body,
    correlation_id="prism-eval-run-42",
)

See Python SandboxClient.

OpenAPI

The platform spec documents x-correlation-id on every /sandbox/* operation:

OpenAPI YAML — parameter CorrelationId