Repository navigation
feat(telemetry): export the reward of each scored rollout - #4609
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no reliability concerns. Reviewed all four changed files and traced reward recording through outcome routing, mask handling, and the telemetry counter helper. The change preserves existing outcome counting while excluding failed, omitted, and non-numeric rewards from the new counter.
Static review only; tests were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex
Record gym.rollout.rewarded_total next to gym.rollout.completed_total for every scored attempt with a numeric reward, by environment, by mask_sample and by reward range (negative, zero, partial, full, above_one). Rewards are environment-defined floats and can be negative penalties, which OpenTelemetry histograms reject, so the distribution is a counter with a bounded range attribute. Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
1bb0245 to
a84d46a
Compare
|
🌿 Preview your docs: https://nvidia-preview-pr-4609-a84d46a4ec11.docs.buildwithfern.com/nemo/gym |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38698734721 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38699489619 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38711534570 |
What
A new rollout metric, recorded by the collection driver next to
gym.rollout.completed_totalfor everyscoredattempt that carries a numeric reward:gym.rollout.rewarded_totalnemo.gym.rollout.dispatch.name,nemo.gym.rollout.masked,nemo.gym.rollout.reward.rangenemo.gym.rollout.reward.rangeisnegative,zero,partial(strictly between 0 and 1),full(exactly 1) orabove_one.nemo.gym.rollout.maskedis the result'smask_sample, so masked rewards can be left out the way the aggregate metrics leave them out.None, bool, non-finite) is unscored, not a zero, and is not counted. Failed and omitted attempts have no reward.Why
Telemetry already shows how many rollouts finished and why some failed, but not whether the scored ones earn anything. A run whose rewards are all zero, drop after a configuration change, or are mostly masked is only visible once
/aggregate_metricsruns at the end. This counter makes the reward distribution visible while the run is going. Benchmark scores (pass@k, per-task averages) still come from/aggregate_metrics; the docs say the live view can differ from them.Why ranges, not a histogram: rewards are environment-defined floats and can be negative penalties (
blackjackreturns -1,generative_reward_modeldefaults its parse-failure penalty to -100,wordlesubtracts penalties). OpenTelemetry histograms reject negative values. A counter with a bounded range attribute works for every environment and keeps exactly 0 and exactly 1 apart from partial credit.No linked issue: a small, self-contained addition to the existing rollout telemetry.
Changes
nemo_gym/telemetry/gym_metrics.py: instrument,record_rollout_reward(), module docs;_record_countertakes a unit.nemo_gym/rollout_collection.py:_record_rollout_outcome()counts the reward forscoredattempts. Same gate as the outcome counter (rollout span group).fern/versions/latest/pages/observability/metrics.mdx: table row, a "What the Reward Metrics Show" section, and the intro now says that per-rollout rewards are exported while benchmark scores stay withAggregateMetricsMixin.tests/unit_tests/telemetry/test_rollout_reward.py: every range including penalties, scored-only recording, masked split, non-numeric rewards, span group off.Validation
tests/unit_tests/telemetry/andtests/unit_tests/test_rollout_collection.pyon the rebased head: 674 passed, 3 failed. The 3 failures (test_connection_pool.py::test_multi_destination_waits_are_attributed_to_the_binding_limit, two intest_memory.py) fail identically onmainon macOS.pre-commit run --all-files: pass. DCO sign-off on the commit.gym eval submitwith telemetry on, job-side Gym at this branch before the rebase ontomain(1bb0245f; the change itself is identical in the rebased head). Completed in 8.5 min.rollouts.jsonl: 16 scored rollouts, all reward 0, none masked (mean/problem_accuracy0.0,mean/subtask_accuracy0.051); failures file empty.gym.rollout.rewarded_total(job directoryotel/metrics.jsonland the metrics backend agree):{dispatch.name: scicode_benchmark_agent, masked: false, reward.range: zero} = 16, equal togym.rollout.completed_total{outcome: scored} = 16.rollouts.jsonl: 64 scored, 27 with reward 1 and 37 with reward 0, none masked;mean/reward0.4219 (= 27/64); failures file empty.gym.rollout.rewarded_total(job directory and metrics backend agree):full = 27,zero = 37, matchinggym.rollout.completed_total{outcome: scored} = 64.partial,negative,above_oneor masked rewards (IFBench grades by fraction, but these prompts scored all-or-nothing); those are covered by the unit tests.Compatibility: additive. No change to existing metrics, outputs or scores.
🤖 Generated with Claude Code