Repository navigation
feat(sandbox): instrument file transfers - #4934
Merged
Merged
Conversation
ananthsub
requested review from
hemildesai and
iguehring-nvidia
and removed request for
a team
October 22, 2026 17:50
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4934-520701d8eb28.docs.buildwithfern.com/nemo/gym |
ananthsub
force-pushed
the
ananthsub/sandbox-transfer-telemetry
branch
from
November 9, 2026 03:28
cc96ce4 to
0cc9555
Compare
AsyncSandbox.upload() and download() called the provider with no span and no metric, and the seed files in SandboxSpec.files are uploaded after the startup duration is recorded. A slow or failing file transfer was therefore invisible in traces and dashboards. Every transfer, including seed files, now goes through one helper that, while the sandbox span group is enabled, opens a gym.sandbox.upload or gym.sandbox.download span with the provider, sandbox id and file size, and records gym.sandbox.transfer_duration_ms by direction and outcome and gym.sandbox.transfer_bytes_total by direction. Paths are not recorded, because a path can name task content. Reading the local file size never fails a transfer. The metrics catalog gains a sandbox section covering the existing sandbox metrics and the new ones, and the span-groups table lists the transfer and stop spans. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Since #4612, every sandbox metric is recorded whenever metrics are exported, whatever the span groups. Transfers were still gated on the `sandbox` span group, so the default preset with metrics on reported startup and exec but no transfers. They now use the same gate. Reading the local file size costs a metadata round trip on a shared filesystem such as Lustre, and it ran twice after every transfer even when nothing used the result. The size is now read with one stat, and only when a recording span or a metrics exporter takes it. Also: - Transfer spans carry the sandbox request attributes, like the stop span, so they join to `gym.sandbox.request`. - Seed files go through `upload()`, and direction and outcome are typed as literals. - The duration histogram gains 2 and 10 minute boundaries. - The metrics and span-groups pages describe the new gate, sidecar downloads, and which failures count as errors or cancellations. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ananthsub/sandbox-transfer-telemetry
branch
from
November 9, 2026 19:21
0cc9555 to
520701d
Compare
hemildesai
approved these changes
Nov 9, 2026
This was referenced Nov 9, 2026
7 tasks done
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
AsyncSandbox.upload()anddownload()called the provider directly, with no span and no metric. The seed files inSandboxSpec.filesare uploaded afterstart()recordsgym.sandbox.startup_duration_ms, so they were not counted anywhere either. A slow or failing file transfer was invisible in traces and dashboards.Every transfer now goes through one helper in
nemo_gym/sandbox/api.py:upload(),download(), and the seed-file loop instart(), which now callsupload(). The helper uses the same gate as the other sandbox call sites: it runs while metrics are exported or thesandboxspan group is enabled. It:gym.sandbox.uploadorgym.sandbox.downloadspan while thesandboxspan group is enabled. The span carriesnemo.gym.sandbox.provider,nemo.gym.sandbox.id, the rollout id, and the sandbox request attributes the stop span carries, so it joins togym.sandbox.request. Once the transfer succeeds, it also recordsnemo.gym.sandbox.transfer.size_bytes. A failed transfer marks its span as an error.gym.sandbox.transfer_duration_ms(histogram, ms) bynemo.gym.sandbox.provider,nemo.gym.sandbox.transfer.direction(uploadordownload), andnemo.gym.sandbox.transfer.outcome(ok,error, orcancelled). Its bucket boundaries run from 10 ms to ten minutes, because a small file copied into a local container takes tens of milliseconds and the lifecycle boundaries start at 250 ms.gym.sandbox.transfer_bytes_total(counter, bytes) by provider and direction, for successful transfers of regular files only.Reading the local file size costs a metadata round trip on a shared filesystem such as Lustre. The size is read with a single stat, and only when a recording span or a metrics exporter will use it. Reading it can never fail a transfer: when the file cannot be read or is not a regular file, the size is left out.
Paths are not recorded, for the same reason exec spans omit the command: a path can name task content.
The metrics catalog gains a "Sandbox Metrics" section that also documents the four existing sandbox metrics, which were missing from it. That section explains which failures count as
errorand which ascancelled. The span-groups table now lists the transfer spans, the existinggym.sandbox.stopandgym.sandbox.requestspans, and the resource spans.Sidecar downloads are not instrumented, matching sidecar
exec, which has no telemetry today.Why
Harness agents increasingly move files into sandboxes during a session, for example prompt images for multimodal tasks (#4917, #4922, #4923). Without telemetry, a slow transfer looks like slow agent startup and a failed one shows up only as an error string. The names follow the existing sandbox conventions:
_msdurations, a_totalcounter, andnemo.gym.attributes.No issue: this is a small, self-contained observability addition. It does not depend on #4917; the two touch adjacent lines in the seed-file loop of
start(), and whichever lands second keeps both changes.Compatibility
No behavior change for callers. With metrics not exported and the
sandboxspan group disabled, transfers run exactly as before, record nothing, and read no file sizes.Validation
telemetryandsandboxextras (nemo-lens installed), so the gated code executes and the lens-only tests run.pytest -n 16 tests/unit_tests/telemetry tests/unit_tests/sandbox: everything passes except a few Docker terminal tests that fail only under parallel load; they pass when that file runs on its own and are untouched by this change. The remaining skips are for enroot, OpenShell, live OpenSandbox, and user namespaces, which this machine does not provide.errorduration and no bytes, and a cancelled download recordscancelled;pre-commit run --files <changed files>passed.Rollout evidence
N/A: this adds spans and metrics around existing sandbox calls and changes no environment or agent behavior.