Repository navigation
fix(telemetry): record metrics whenever metrics export, independent of span groups - #4612
Merged
Merged
Conversation
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-4612-da4267da894b.docs.buildwithfern.com/nemo/gym |
ananthsub
force-pushed
the
ananthsub/metrics-independent-of-span-groups
branch
from
October 14, 2026 19:44
ef9d667 to
3e63eb5
Compare
5 of 6 tasks
This was referenced Oct 22, 2026
7 tasks done
iguehring-nvidia
previously approved these changes
Nov 7, 2026
…f span groups Span groups decide which spans exist, but several metrics were recorded only when a span group was on. The `default` preset leaves the `verify` and `sandbox` groups off, so a run with metrics on reported no `gym.verify.*` and no sandbox metrics at all. Each site now records its metrics whenever a metrics exporter is active, and its span whenever its span group is on: - `gym.verify.*` and `gym.rollout.duration_ms` in `telemetry/endpoints.py`. - `gym.rollout.completed_total` and `gym.rollout.rewarded_total` in the rollout driver. - The sandbox start and exec durations, the active-sandbox count, and both providers' create-retry counters. Sites that do both take their instrumented path through `is_span_group_or_metrics_active(group)` in `telemetry/setup.py`; `managed_span` still checks the span group itself. This also prepares Gym for nemo-lens enabling no span groups while traces are off. Without it, these metrics would stop in every metrics-only run. The rollout-health counters already followed this rule, and `span-groups.mdx` now states it for all metrics. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ananthsub/metrics-independent-of-span-groups
branch
from
November 9, 2026 14:31
3e63eb5 to
bfbb637
Compare
…e rollout counter The rollout counter now follows the metrics exporter, so the stage-routing test turns the exporter on to observe it. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
iguehring-nvidia
approved these changes
Nov 9, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42650569113 |
ananthsub
added a commit
that referenced
this pull request
Nov 9, 2026
Since #4612, every sandbox metric is recorded whenever metrics are exported, whatever the span groups. Transfers were still gated on the `sandbox` span group, so the default preset with metrics on reported startup and exec but no transfers. They now use the same gate. Reading the local file size costs a metadata round trip on a shared filesystem such as Lustre, and it ran twice after every transfer even when nothing used the result. The size is now read with one stat, and only when a recording span or a metrics exporter takes it. Also: - Transfer spans carry the sandbox request attributes, like the stop span, so they join to `gym.sandbox.request`. - Seed files go through `upload()`, and direction and outcome are typed as literals. - The duration histogram gains 2 and 10 minute boundaries. - The metrics and span-groups pages describe the new gate, sidecar downloads, and which failures count as errors or cancellations. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
added a commit
that referenced
this pull request
Nov 9, 2026
* feat(sandbox): instrument file transfers AsyncSandbox.upload() and download() called the provider with no span and no metric, and the seed files in SandboxSpec.files are uploaded after the startup duration is recorded. A slow or failing file transfer was therefore invisible in traces and dashboards. Every transfer, including seed files, now goes through one helper that, while the sandbox span group is enabled, opens a gym.sandbox.upload or gym.sandbox.download span with the provider, sandbox id and file size, and records gym.sandbox.transfer_duration_ms by direction and outcome and gym.sandbox.transfer_bytes_total by direction. Paths are not recorded, because a path can name task content. Reading the local file size never fails a transfer. The metrics catalog gains a sandbox section covering the existing sandbox metrics and the new ones, and the span-groups table lists the transfer and stop spans. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> * fix(sandbox): follow the metrics exporter and stat transfers once Since #4612, every sandbox metric is recorded whenever metrics are exported, whatever the span groups. Transfers were still gated on the `sandbox` span group, so the default preset with metrics on reported startup and exec but no transfers. They now use the same gate. Reading the local file size costs a metadata round trip on a shared filesystem such as Lustre, and it ran twice after every transfer even when nothing used the result. The size is now read with one stat, and only when a recording span or a metrics exporter takes it. Also: - Transfer spans carry the sandbox request attributes, like the stop span, so they join to `gym.sandbox.request`. - Seed files go through `upload()`, and direction and outcome are typed as literals. - The duration histogram gains 2 and 10 minute boundaries. - The metrics and span-groups pages describe the new gate, sidecar downloads, and which failures count as errors or cancellations. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> --------- Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Span groups decide which spans exist, but several Gym metrics were recorded only when a span group was on.
The
defaultpreset leaves theverifyandsandboxgroups off, so a default run with metrics on reported nogym.verify.*and no sandbox metrics at all.Each site now records its metrics whenever a metrics exporter is active, and its span whenever its span group is on:
gym.verify.duration_ms,gym.verify.success_ratetraced_verify_endpointintelemetry/endpoints.pyverifygroupgym.rollout.duration_mstraced_rollout_endpointintelemetry/endpoints.pyrolloutgroupgym.rollout.completed_total,gym.rollout.rewarded_total_record_rollout_outcomein the rollout driverrolloutgroupgym.sandbox.startup_duration_ms,gym.sandbox.exec_duration_ms,gym.sandbox.activeAsyncSandbox.start,exec,stopsandboxgroupgym.sandbox.create_retry_totalsandboxgroupWhere a site has both, it takes its instrumented path when either is on, through
is_span_group_or_metrics_active(group)intelemetry/setup.py;managed_spanstill checks the span group itself, so spans followspan_groupsexactly as before.The rollout-health counters already followed this rule, and
span-groups.mdxnow states it for every metric.Why
Two reasons.
span_groups, which also widens trace volume.gym.verify.*,gym.rollout.*, and the sandbox metrics in every metrics-only Gym run. After it, Gym's metrics do not depend on span groups, so the nemo-lens fix can land and a later pin bump makes metrics-only runs cheaper with no metric lost.Issue
Follow-up to #4481, which skipped Gym's span-only work while traces are off and found the coupling. No separate issue.
NVIDIA-NeMo/Lens#87 makes nemo-lens enable no span group while traces are disabled. This PR should merge before Gym bumps its nemo-lens pin past that change.
Compatibility
gym.verify.*,gym.rollout.duration_ms,gym.rollout.completed_total,gym.rollout.rewarded_total, and the sandbox metrics regardless ofspan_groups. Under the default preset,gym.verify.*and the sandbox metrics are new series.is_span_group_enabledorGymSpanGroup, so both names leave its entry in the import-surface fixture. They were imported helpers, not part of the provider's interface.Validation
312de5eb4d,pytest tests/unit_tests/telemetry tests/unit_tests/sandbox tests/unit_tests/rollouts tests/unit_tests/test_moved_paths.py -n 16with thetelemetryextra installed: the only failures are intest_sandbox_failures.py,test_opensandbox_compose.py,test_opensandbox_http_transport.py,test_docker_pty.py, andtest_sandbox_stop_retry.py, and the same tests fail on the base commit in this environment.sandboxgroup off; the rollout-outcome counter following the exporter and not the span group; create-retry counts following the exporter. Existing "records nothing" tests now pin both the group and the exporter off. The reward counter that main added under the same gate is covered the same way. Restoring the old gating, or dropping the exporter branch from the helper, makes the new tests fail.gym.verify.*; this PR exportsgym.verify.duration_msandgym.verify.success_rate. No rollout failed in either run.pre-commit runon the changed files.Rollout evidence
N/A for model rollouts. The change decides when telemetry is recorded and does not change agent, model, or verifier decisions. The simulator runs above exercise it through real rollouts with a mock inference engine.