Skip to content

feat(sandbox): instrument file transfers - #4934

Merged
ananthsub merged 2 commits into
mainfrom
ananthsub/sandbox-transfer-telemetry
Nov 9, 2026
Merged

ananthsub merged 2 commits into
mainfrom
ananthsub/sandbox-transfer-telemetry

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changed

AsyncSandbox.upload() and download() called the provider directly, with no span and no metric. The seed files in SandboxSpec.files are uploaded after start() records gym.sandbox.startup_duration_ms, so they were not counted anywhere either. A slow or failing file transfer was invisible in traces and dashboards.

Every transfer now goes through one helper in nemo_gym/sandbox/api.py: upload(), download(), and the seed-file loop in start(), which now calls upload(). The helper uses the same gate as the other sandbox call sites: it runs while metrics are exported or the sandbox span group is enabled. It:

  • opens a gym.sandbox.upload or gym.sandbox.download span while the sandbox span group is enabled. The span carries nemo.gym.sandbox.provider, nemo.gym.sandbox.id, the rollout id, and the sandbox request attributes the stop span carries, so it joins to gym.sandbox.request. Once the transfer succeeds, it also records nemo.gym.sandbox.transfer.size_bytes. A failed transfer marks its span as an error.
  • records gym.sandbox.transfer_duration_ms (histogram, ms) by nemo.gym.sandbox.provider, nemo.gym.sandbox.transfer.direction (upload or download), and nemo.gym.sandbox.transfer.outcome (ok, error, or cancelled). Its bucket boundaries run from 10 ms to ten minutes, because a small file copied into a local container takes tens of milliseconds and the lifecycle boundaries start at 250 ms.
  • adds the transferred file's size to gym.sandbox.transfer_bytes_total (counter, bytes) by provider and direction, for successful transfers of regular files only.

Reading the local file size costs a metadata round trip on a shared filesystem such as Lustre. The size is read with a single stat, and only when a recording span or a metrics exporter will use it. Reading it can never fail a transfer: when the file cannot be read or is not a regular file, the size is left out.

Paths are not recorded, for the same reason exec spans omit the command: a path can name task content.

The metrics catalog gains a "Sandbox Metrics" section that also documents the four existing sandbox metrics, which were missing from it. That section explains which failures count as error and which as cancelled. The span-groups table now lists the transfer spans, the existing gym.sandbox.stop and gym.sandbox.request spans, and the resource spans.

Sidecar downloads are not instrumented, matching sidecar exec, which has no telemetry today.

Why

Harness agents increasingly move files into sandboxes during a session, for example prompt images for multimodal tasks (#4917, #4922, #4923). Without telemetry, a slow transfer looks like slow agent startup and a failed one shows up only as an error string. The names follow the existing sandbox conventions: _ms durations, a _total counter, and nemo.gym. attributes.

No issue: this is a small, self-contained observability addition. It does not depend on #4917; the two touch adjacent lines in the seed-file loop of start(), and whichever lands second keeps both changes.

Compatibility

No behavior change for callers. With metrics not exported and the sandbox span group disabled, transfers run exactly as before, record nothing, and read no file sizes.

Validation

  • Run in an environment with the telemetry and sandbox extras (nemo-lens installed), so the gated code executes and the lens-only tests run.
  • pytest -n 16 tests/unit_tests/telemetry tests/unit_tests/sandbox: everything passes except a few Docker terminal tests that fail only under parallel load; they pass when that file runs on its own and are untouched by this change. The remaining skips are for enroot, OpenShell, live OpenSandbox, and user namespaces, which this machine does not provide.
  • The metric tests check that:
    • a seed file, an explicit upload and a download each record a duration in milliseconds with the right direction and outcome;
    • with only a metrics exporter on (the default preset), durations and the byte counter are recorded;
    • nothing is recorded with both metrics and the span group off;
    • a failed upload records an error duration and no bytes, and a cancelled download records cancelled;
    • a missing file, a stat that raises, and a directory each still yield a successful transfer with no bytes;
    • the file is stat'ed exactly once per successful transfer when a recording span or an exporter uses the size, never when neither does, and never after a failed transfer;
    • the instruments declare milliseconds and bytes as their units.
  • The span tests check the transfer span attributes, including the sandbox request id, that a path never appears in them, and that a failed upload marks its span as an error.
  • The new tests fail without the change.
  • pre-commit run --files <changed files> passed.

Rollout evidence

N/A: this adds spans and metrics around existing sandbox calls and changes no environment or agent behavior.

@ananthsub ananthsub added feature New capabilities, enhancements, or enablement work area:sandbox Sandbox providers, isolation, and sandbox runtime contracts labels Oct 22, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 22, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 22, 2026 17:50
@ananthsub
ananthsub requested review from hemildesai and iguehring-nvidia and removed request for a team October 22, 2026 17:50
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

AsyncSandbox.upload() and download() called the provider with no span and no metric,
and the seed files in SandboxSpec.files are uploaded after the startup duration is recorded.
A slow or failing file transfer was therefore invisible in traces and dashboards.

Every transfer, including seed files, now goes through one helper that, while the sandbox
span group is enabled, opens a gym.sandbox.upload or gym.sandbox.download span with the
provider, sandbox id and file size, and records gym.sandbox.transfer_duration_ms by
direction and outcome and gym.sandbox.transfer_bytes_total by direction.
Paths are not recorded, because a path can name task content.
Reading the local file size never fails a transfer.

The metrics catalog gains a sandbox section covering the existing sandbox metrics and the
new ones, and the span-groups table lists the transfer and stop spans.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Since #4612, every sandbox metric is recorded whenever metrics are
exported, whatever the span groups. Transfers were still gated on the
`sandbox` span group, so the default preset with metrics on reported
startup and exec but no transfers. They now use the same gate.

Reading the local file size costs a metadata round trip on a shared
filesystem such as Lustre, and it ran twice after every transfer even
when nothing used the result. The size is now read with one stat, and
only when a recording span or a metrics exporter takes it.

Also:
- Transfer spans carry the sandbox request attributes, like the stop
  span, so they join to `gym.sandbox.request`.
- Seed files go through `upload()`, and direction and outcome are
  typed as literals.
- The duration histogram gains 2 and 10 minute boundaries.
- The metrics and span-groups pages describe the new gate, sidecar
  downloads, and which failures count as errors or cancellations.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub force-pushed the ananthsub/sandbox-transfer-telemetry branch from 0cc9555 to 520701d Compare November 9, 2026 19:21

This branch was previously deployed

1 inactive deployment
public — 520701d8 Deployed Nov 9, 2026 by copy-pr-bot[bot] via release / finalize / notify #11136
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sandbox Sandbox providers, isolation, and sandbox runtime contracts feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants