Skip to content

feat(telemetry): export the reward of each scored rollout - #4609

Merged
iguehring-nvidia merged 1 commit into
mainfrom
iguehring/rollout-reward
Oct 15, 2026
Merged

iguehring-nvidia merged 1 commit into
mainfrom
iguehring/rollout-reward

Conversation

@iguehring-nvidia

@iguehring-nvidia iguehring-nvidia commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What

A new rollout metric, recorded by the collection driver next to gym.rollout.completed_total for every scored attempt that carries a numeric reward:

Metric Type Attributes
gym.rollout.rewarded_total Counter nemo.gym.rollout.dispatch.name, nemo.gym.rollout.masked, nemo.gym.rollout.reward.range
  • nemo.gym.rollout.reward.range is negative, zero, partial (strictly between 0 and 1), full (exactly 1) or above_one.
  • nemo.gym.rollout.masked is the result's mask_sample, so masked rewards can be left out the way the aggregate metrics leave them out.
  • A result without a numeric reward (missing, None, bool, non-finite) is unscored, not a zero, and is not counted. Failed and omitted attempts have no reward.

Why

Telemetry already shows how many rollouts finished and why some failed, but not whether the scored ones earn anything. A run whose rewards are all zero, drop after a configuration change, or are mostly masked is only visible once /aggregate_metrics runs at the end. This counter makes the reward distribution visible while the run is going. Benchmark scores (pass@k, per-task averages) still come from /aggregate_metrics; the docs say the live view can differ from them.

Why ranges, not a histogram: rewards are environment-defined floats and can be negative penalties (blackjack returns -1, generative_reward_model defaults its parse-failure penalty to -100, wordle subtracts penalties). OpenTelemetry histograms reject negative values. A counter with a bounded range attribute works for every environment and keeps exactly 0 and exactly 1 apart from partial credit.

No linked issue: a small, self-contained addition to the existing rollout telemetry.

Changes

  • nemo_gym/telemetry/gym_metrics.py: instrument, record_rollout_reward(), module docs; _record_counter takes a unit.
  • nemo_gym/rollout_collection.py: _record_rollout_outcome() counts the reward for scored attempts. Same gate as the outcome counter (rollout span group).
  • fern/versions/latest/pages/observability/metrics.mdx: table row, a "What the Reward Metrics Show" section, and the intro now says that per-rollout rewards are exported while benchmark scores stay with AggregateMetricsMixin.
  • tests/unit_tests/telemetry/test_rollout_reward.py: every range including penalties, scored-only recording, masked split, non-numeric rewards, span group off.

Validation

  • tests/unit_tests/telemetry/ and tests/unit_tests/test_rollout_collection.py on the rebased head: 674 passed, 3 failed. The 3 failures (test_connection_pool.py::test_multi_destination_waits_are_attributed_to_the_binding_limit, two in test_memory.py) fail identically on main on macOS.
  • pre-commit run --all-files: pass. DCO sign-off on the commit.
  • Rollout evidence: SciCode, 16 tasks, 1 repeat, Nemotron 3.5 Nano on vLLM, gym eval submit with telemetry on, job-side Gym at this branch before the rebase onto main (1bb0245f; the change itself is identical in the rebased head). Completed in 8.5 min.
    • rollouts.jsonl: 16 scored rollouts, all reward 0, none masked (mean/problem_accuracy 0.0, mean/subtask_accuracy 0.051); failures file empty.
    • Exported gym.rollout.rewarded_total (job directory otel/metrics.jsonl and the metrics backend agree): {dispatch.name: scicode_benchmark_agent, masked: false, reward.range: zero} = 16, equal to gym.rollout.completed_total{outcome: scored} = 16.
  • Second run with rewards: IFBench, 64 tasks, 1 repeat, same setup. Completed in 7.4 min.
    • rollouts.jsonl: 64 scored, 27 with reward 1 and 37 with reward 0, none masked; mean/reward 0.4219 (= 27/64); failures file empty.
    • Exported gym.rollout.rewarded_total (job directory and metrics backend agree): full = 27, zero = 37, matching gym.rollout.completed_total{outcome: scored} = 64.
  • No run produced partial, negative, above_one or masked rewards (IFBench grades by fraction, but these prompts scored all-or-nothing); those are covered by the unit tests.

Compatibility: additive. No change to existing metrics, outputs or scores.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Oct 14, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@iguehring-nvidia iguehring-nvidia self-assigned this Oct 14, 2026
@iguehring-nvidia

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no reliability concerns. Reviewed all four changed files and traced reward recording through outcome routing, mask handling, and the telemetry counter helper. The change preserves existing outcome counting while excluding failed, omitted, and non-numeric rewards from the new counter.

Static review only; tests were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex

Record gym.rollout.rewarded_total next to gym.rollout.completed_total
for every scored attempt with a numeric reward, by environment, by
mask_sample and by reward range (negative, zero, partial, full,
above_one). Rewards are environment-defined floats and can be negative
penalties, which OpenTelemetry histograms reject, so the distribution is
a counter with a bounded range attribute.

Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
@iguehring-nvidia
iguehring-nvidia force-pushed the iguehring/rollout-reward branch from 1bb0245 to a84d46a Compare October 14, 2026 20:51
@iguehring-nvidia
iguehring-nvidia marked this pull request as ready for review October 14, 2026 20:52
@iguehring-nvidia iguehring-nvidia added feature New capabilities, enhancements, or enablement work area:core Shared APIs, servers, telemetry, health, and registries needs-review PR is ready for code review and waiting on a reviewer labels Oct 14, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 14, 2026 20:53
@github-actions

Copy link
Copy Markdown
Contributor

@yaoyu-33 yaoyu-33 added the complexity:low Localized change in one scope with a small, straightforward review surface label Oct 14, 2026
@iguehring-nvidia
iguehring-nvidia added this pull request to the merge queue Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38698734721

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38699489619

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 15, 2026
@iguehring-nvidia
iguehring-nvidia added this pull request to the merge queue Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38711534570

Merged via the queue into main with commit 4330be2 Oct 15, 2026
52 of 57 checks passed

This branch was previously deployed

1 inactive deployment
public — a84d46a4 Deployed Oct 14, 2026 by copy-pr-bot[bot] via release / finalize / notify #5382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants