Skip to content

fix(telemetry): record metrics whenever metrics export, independent of span groups - #4612

Merged
ananthsub merged 2 commits into
mainfrom
ananthsub/metrics-independent-of-span-groups
Nov 9, 2026
Merged

ananthsub merged 2 commits into
mainfrom
ananthsub/metrics-independent-of-span-groups

Conversation

@ananthsub

@ananthsub ananthsub commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Span groups decide which spans exist, but several Gym metrics were recorded only when a span group was on.
The default preset leaves the verify and sandbox groups off, so a default run with metrics on reported no gym.verify.* and no sandbox metrics at all.

Each site now records its metrics whenever a metrics exporter is active, and its span whenever its span group is on:

Metrics Site Previously gated on
gym.verify.duration_ms, gym.verify.success_rate traced_verify_endpoint in telemetry/endpoints.py verify group
gym.rollout.duration_ms traced_rollout_endpoint in telemetry/endpoints.py rollout group
gym.rollout.completed_total, gym.rollout.rewarded_total _record_rollout_outcome in the rollout driver rollout group
gym.sandbox.startup_duration_ms, gym.sandbox.exec_duration_ms, gym.sandbox.active AsyncSandbox.start, exec, stop sandbox group
gym.sandbox.create_retry_total Daytona and OpenSandbox create-retry callbacks sandbox group

Where a site has both, it takes its instrumented path when either is on, through is_span_group_or_metrics_active(group) in telemetry/setup.py; managed_span still checks the span group itself, so spans follow span_groups exactly as before.
The rollout-health counters already followed this rule, and span-groups.mdx now states it for every metric.

Why

Two reasons.

  • A default run should report the metrics that are documented for it. Today an operator who turns metrics on sees no verification or sandbox metrics unless they also widen span_groups, which also widens trace volume.
  • nemo-lens enables span groups even when traces are disabled, so every gated span site pays for spans no exporter receives. Fixing that in nemo-lens means enabling no group while traces are off. Before this PR, that fix would have silently stopped gym.verify.*, gym.rollout.*, and the sandbox metrics in every metrics-only Gym run. After it, Gym's metrics do not depend on span groups, so the nemo-lens fix can land and a later pin bump makes metrics-only runs cheaper with no metric lost.

Issue

Follow-up to #4481, which skipped Gym's span-only work while traces are off and found the coupling. No separate issue.

NVIDIA-NeMo/Lens#87 makes nemo-lens enable no span group while traces are disabled. This PR should merge before Gym bumps its nemo-lens pin past that change.

Compatibility

  • Runs with metrics on now report gym.verify.*, gym.rollout.duration_ms, gym.rollout.completed_total, gym.rollout.rewarded_total, and the sandbox metrics regardless of span_groups. Under the default preset, gym.verify.* and the sandbox metrics are new series.
  • Spans are unchanged.
  • With telemetry off, each site adds one global read.
  • The Daytona provider no longer imports is_span_group_enabled or GymSpanGroup, so both names leave its entry in the import-surface fixture. They were imported helpers, not part of the provider's interface.

Validation

  • After rebasing on 312de5eb4d, pytest tests/unit_tests/telemetry tests/unit_tests/sandbox tests/unit_tests/rollouts tests/unit_tests/test_moved_paths.py -n 16 with the telemetry extra installed: the only failures are in test_sandbox_failures.py, test_opensandbox_compose.py, test_opensandbox_http_transport.py, test_docker_pty.py, and test_sandbox_stop_retry.py, and the same tests fail on the base commit in this environment.
  • New tests: verify and rollout metrics recorded with every span group off and an exporter active; sandbox start and exec durations recorded with the sandbox group off; the rollout-outcome counter following the exporter and not the span group; create-retry counts following the exporter. Existing "records nothing" tests now pin both the group and the exporter off. The reward counter that main added under the same gate is covered the same way. Restoring the old gating, or dropping the exporter branch from the helper, makes the new tests fail.
  • The same suite against a nemo-lens build that enables no span group while traces are off: same results, so this PR is what keeps Gym's metrics intact under that change.
  • The CPU capacity simulator from Add a high-concurrency Gym dispatch and resources capacity simulator #3156, production topology with the single-agent-turn environment server, 256 rollouts each, metrics on, traces off, default span groups. The base commit exported no gym.verify.*; this PR exports gym.verify.duration_ms and gym.verify.success_rate. No rollout failed in either run.
  • The GDPVal stage-routing test, which counts rollout outcomes, now turns on the metrics exporter instead of the span group; it passes locally after failing in the server suite.
  • pre-commit run on the changed files.

Rollout evidence

N/A for model rollouts. The change decides when telemetry is recorded and does not change agent, model, or verifier decisions. The simulator runs above exercise it through real rollouts with a mock inference engine.

@ananthsub ananthsub added bug Something isn't working area:core Shared APIs, servers, telemetry, health, and registries needs-review PR is ready for code review and waiting on a reviewer labels Oct 14, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 14, 2026 19:41
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

@ananthsub
ananthsub force-pushed the ananthsub/metrics-independent-of-span-groups branch from ef9d667 to 3e63eb5 Compare October 14, 2026 19:44
@yaoyu-33 yaoyu-33 added the complexity:medium Single-domain change with interacting parts or a moderate review surface label Oct 14, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Oct 15, 2026
Comment thread nemo_gym/sandbox/api.py Resolved
…f span groups

Span groups decide which spans exist, but several metrics were recorded only when a span
group was on. The `default` preset leaves the `verify` and `sandbox` groups off, so a run with
metrics on reported no `gym.verify.*` and no sandbox metrics at all.

Each site now records its metrics whenever a metrics exporter is active, and its span whenever
its span group is on:

- `gym.verify.*` and `gym.rollout.duration_ms` in `telemetry/endpoints.py`.
- `gym.rollout.completed_total` and `gym.rollout.rewarded_total` in the rollout driver.
- The sandbox start and exec durations, the active-sandbox count, and both providers'
  create-retry counters.

Sites that do both take their instrumented path through
`is_span_group_or_metrics_active(group)` in `telemetry/setup.py`; `managed_span` still checks the
span group itself.

This also prepares Gym for nemo-lens enabling no span groups while traces are off. Without it,
these metrics would stop in every metrics-only run. The rollout-health counters already followed
this rule, and `span-groups.mdx` now states it for all metrics.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
…e rollout counter

The rollout counter now follows the metrics exporter, so the stage-routing test turns the
exporter on to observe it.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub added this pull request to the merge queue Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42650569113

Merged via the queue into main with commit c442d05 Nov 9, 2026
70 checks passed
@ananthsub
ananthsub deleted the ananthsub/metrics-independent-of-span-groups branch November 9, 2026 16:17
ananthsub added a commit that referenced this pull request Nov 9, 2026
Since #4612, every sandbox metric is recorded whenever metrics are
exported, whatever the span groups. Transfers were still gated on the
`sandbox` span group, so the default preset with metrics on reported
startup and exec but no transfers. They now use the same gate.

Reading the local file size costs a metadata round trip on a shared
filesystem such as Lustre, and it ran twice after every transfer even
when nothing used the result. The size is now read with one stat, and
only when a recording span or a metrics exporter takes it.

Also:
- Transfer spans carry the sandbox request attributes, like the stop
  span, so they join to `gym.sandbox.request`.
- Seed files go through `upload()`, and direction and outcome are
  typed as literals.
- The duration histogram gains 2 and 10 minute boundaries.
- The metrics and span-groups pages describe the new gate, sidecar
  downloads, and which failures count as errors or cancellations.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub added a commit that referenced this pull request Nov 9, 2026
* feat(sandbox): instrument file transfers

AsyncSandbox.upload() and download() called the provider with no span and no metric,
and the seed files in SandboxSpec.files are uploaded after the startup duration is recorded.
A slow or failing file transfer was therefore invisible in traces and dashboards.

Every transfer, including seed files, now goes through one helper that, while the sandbox
span group is enabled, opens a gym.sandbox.upload or gym.sandbox.download span with the
provider, sandbox id and file size, and records gym.sandbox.transfer_duration_ms by
direction and outcome and gym.sandbox.transfer_bytes_total by direction.
Paths are not recorded, because a path can name task content.
Reading the local file size never fails a transfer.

The metrics catalog gains a sandbox section covering the existing sandbox metrics and the
new ones, and the span-groups table lists the transfer and stop spans.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

* fix(sandbox): follow the metrics exporter and stat transfers once

Since #4612, every sandbox metric is recorded whenever metrics are
exported, whatever the span groups. Transfers were still gated on the
`sandbox` span group, so the default preset with metrics on reported
startup and exec but no transfers. They now use the same gate.

Reading the local file size costs a metadata round trip on a shared
filesystem such as Lustre, and it ran twice after every transfer even
when nothing used the result. The size is now read with one stat, and
only when a recording span or a metrics exporter takes it.

Also:
- Transfer spans carry the sandbox request attributes, like the stop
  span, so they join to `gym.sandbox.request`.
- Seed files go through `upload()`, and direction and outcome are
  typed as literals.
- The duration histogram gains 2 and 10 minute boundaries.
- The metrics and span-groups pages describe the new gate, sidecar
  downloads, and which failures count as errors or cancellations.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

---------

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — da4267da Deployed Nov 9, 2026 by copy-pr-bot[bot] via release / finalize / notify #11039
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working complexity:medium Single-domain change with interacting parts or a moderate review surface needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants