Repository navigation
feat(eval): count each rollout of the in-process API in Gym's rollout metrics - #5801
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
41c8e68 to
37f4125
Compare
3f94ccf to
b4e2adc
Compare
abc4275 to
ee6acc1
Compare
|
/review |
|
🌿 Preview your docs: https://nvidia-preview-pr-5801-34de5b65bee1.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP — one non-blocking documentation NOTE; no correctness or lifecycle defects found.
I reviewed all three changed files and traced dispatch resolution, failure normalization, metric recording, and the existing telemetry tests. This was a static review; I didn't run tests or a live exporter.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Inline comments (escaped v1):
-
nemo_gym/telemetry/gym_metrics.pyline 73: NOTE — Suggest documenting the counter's new producer.With telemetry active, a completed TaskRunner.run_task now increments this counter without saving an outcome. This definition still says only
run_from_configrecords it and makes persistence a condition for every increment. The user-facing metric definition likewise describes every increment as a persisted CLI attempt. Readers can therefore mistake Python API counts for durable saved outcomes.Please qualify the existing persistence/routing descriptions as collection-driver behavior, remove the exclusivity claim here, and add this distinction to both definitions. Suggested paragraph:
TaskRunner.run_task also records each returned rollout outcome, without a recovery-store write. For this API, scored means a reward is present and failed means none is present; a reported failure kind labels either outcome. rewarded_total counts only finite rewards, including masked rewards.Preserve the CLI distinctions: cached results don't increment the counter, saved omissions retain
omitted, and its historicalscoredlabel includes supported unscored completions. Static acceptance: both definitions identify the two producers and distinguish their persistence and outcome rules; neither implies that aTaskRunnerincrement proves an outcome was saved. No runtime change is needed.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
AdamRajfer
left a comment
There was a problem hiding this comment.
Failure handling mostly lines up: a retry of one run_task is a single count, a typed /run failure is counted failed with its kind, and the standard path is untouched (the diff only adds code, docstrings and docs, and the existing in-process tests pass here: 72 passed). Two things I'd fix before this lands, plus a smaller label mismatch.
-
A judge fault is counted as a scored zero here, and as a failed rollout on the standard path.
judge_failsafereturnsreward: 0.0,mask_sample: True,failure_kind: judge_failedand_ng_failure_class: judge_failed(nemo_gym/verification/judge.py:350-363).gym eval runroutes that row to the failures sidecar and recordsfailed(rollout_collection.py:4427,4511-4540)._outcome(in_process.py:380-401) only readsfailure_kind, so_record_outcomesees a reward and recordsscored, plus arewarded_totalsample in thezerorange, masked. The new testtest_a_scored_rollout_that_carries_a_kind_is_counted_under_itpins exactly this. Any dashboard or health gate that treatsscoredas "the model was measured" will see eval defects as measured zeros from in-process callers, and not fromgym eval run. Suggested fix: treat a result that carries_ng_failure_class(or a judge/env failure kind that the loop would drop) asfailedand skiprewarded_total, i.e. decide the outcome with the same predicate the loop uses instead of "reward is not None". Keepscoredfor masked rewards that have no failure class. -
Labels differ from the loop on cases that are not faults. On the standard path a scored row never carries
failure_class/failure_kind(rollout_collection.py:2437), so agroup by failure_kindpanel only contains non-scored rows. Here scored rollouts with a kind carry them too (in_process.py:366-371). Also, for a typed failure with no kind, the loop setsfailure_classto the routing class and leavesfailure_kindempty ("without inventing a canonical failure kind",rollout_outcomes.py:53-58), but_failure_of_recordputssidecar_failure_classintoTaskFailure.kind, so the new code emitsfailure_kind=agent_request_failed(the test's third case shows it).failure_typeisexception_typeon both paths, fine. Suggested fix: no failure labels onscored, and leavefailure_kindunset when the record'sfailure.failure_kindis None. -
A completion without a reward and without a failure is
failedwith no class (in_process.py:368-370). The loop counts a supported unscored completion asscored, and a bare no-persist reply asomitted. Either map those the same way or say in the docs that these show up asfailedwith empty labels, because a failed-with-no-class series will otherwise look like an unclassified fault.
Smaller: _dispatch_name(resolved_row) runs after the rollout finished (in_process.py:353) and raises KeyError if a row has neither agent_ref nor an environment server key. That would lose a completed outcome over a label, so I'd fall back to a placeholder rather than raise. I did not exercise that with a real collated row.
Not verified: no live exporter or real servers, and I didn't compare emitted attributes against a running gym eval run. The comparison is from the code on both sides.
AdamRajfer
left a comment
There was a problem hiding this comment.
A second pass, this time from the side of a caller that embeds TaskRunner in its own long-lived worker: one runner per benchmark and model, many concurrent run_task calls, and its own OpenTelemetry setup. The recording itself works there. It never raises, a cancelled run_task records nothing, and each caller retry counts as one more attempt, which matches the counter's definition. The in-process tests pass at 45d326f (72 passed). Two things matter for that kind of caller.
-
The documented opt-in breaks a host that already has an SDK
MeterProvider(fern/versions/latest/pages/api-reference/index.mdx:148-151). If such a process follows "callinit_telemetryfirst", OTel refuses the second global provider ("Overriding of current MeterProvider is not allowed") and Gym's meter resolves to the host's provider, so the counters do arrive. But Gym still marks itself as the owner (_OWNS_PROVIDERS = True), andshutdown_telemetry()then shuts down the host's provider. I reproduced it with a hostMeterProviderand anInMemoryMetricReader,NEMO_GYM_OTEL_ENABLED=1,init_telemetry()andshutdown_telemetry(): the host provider ends up shut down. The doc also doesn't say thatinit_telemetrydoes nothing unlessNEMO_GYM_OTEL_ENABLEDis set. The root cause is ininit_telemetry, which only recognises nemo-lens as an existing owner, so treating any installed SDK meter provider the same way (reuse it, don't own it) would fix it. If that belongs in a separate PR, I'd at least spell out the env var and the shutdown caveat in this doc. -
The counts can't be attributed to a run when one process holds several runners (
nemo_gym/environment/in_process.py:353,357-373). The only per-point attributes are the outcome, the dispatch name and the failure labels, and resource attributes are process-wide. A worker with two models open on the same benchmark gets one merged series, so a jump infailedcan't be traced to the run that caused it.gym eval rundoesn't hit this because it is one run per process. Letting the caller pass a few extra attributes when it builds theTaskRunner(something likemetric_attributes={...}), added to both counters, would close it.
Smaller, and related to the judge-failure point in my earlier review: an embedder that treats a masked or kind-carrying outcome as not measured will count differently from scored here. Whichever predicate you settle on, stating it next to the RolloutOutcome fields would let callers line their own numbers up with Gym's.
Not checked: a real collector, and Gym servers spawned by the host that inherit the env var.
|
@AdamRajfer thank you for both passes. The changes are in First review
Second review
Checked against real Gym servers (IFEval, stub model): a run with failing prompts gave 15 scored and 5 failed, and a run with the environment server killed gave 21 scored and 39 failed (kind |
AdamRajfer
left a comment
There was a problem hiding this comment.
Thanks, 527973c covers the first review: a failure now decides the outcome, so a judge fault is failed on both paths, scored rows carry no failure labels, and unscored plus the stated predicate make the in-process rule clear. "kind, else class" for the routing fallback works for me. Approving with the two open points as follow-ups: the init_telemetry fix for a host that already has a MeterProvider (until then the env var and shutdown caveat aren't in the docs), and per-runner metric attributes once a caller needs them.
…metrics run_task records gym.rollout.completed_total (scored when the rollout has a reward, failed when it has none, labelled with the failure's kind) and gym.rollout.rewarded_total, so a caller of the in-process API does not call the metric recorders itself. Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
…ir reward A rollout with a failure is counted failed and labeled with its kind only. Without a failure it is scored when it has a reward and unscored when it has none. A failed rollout adds nothing to rewarded_total. The failure_class label is no longer emitted from the in-process API, and a row that names neither an agent nor an environment server is counted under unknown instead of raising after the rollout finished. Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
527973c to
34de5b6
Compare
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42841921077 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42844994879 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42848006586 |
Counts each rollout of
TaskRunner.run_taskin Gym's rollout metrics, so a caller of the in-process API does not call the recorders itself. Follows #5595, which is merged.What changed
run_taskrecords for its rollout:gym.rollout.completed_total:failedwhen the outcome has a failure, labeled with the failure's kind (failure_kind) and exception type;scoredwhen the outcome has a reward, andunscoredwhen it has none.gym.rollout.rewarded_totalfor a scored rollout with a finite reward. A masked rollout keeps its reward, so it is counted and labeled masked, as in the rollout loop.The recorders do nothing without a meter. Duration comes from the agent server. A row that names neither an agent nor an environment server is counted under
unknowninstead of raising after the rollout finished.Decisions
failedwhatever its reward is, as in the rollout loop (a judge fault returns reward 0.0 and a failure, and isfailedon both paths). The returnedRolloutOutcomeis unchanged: the caller still gets the reward and the failure and decides what to do with them.failure_kind, taken fromTaskFailure.kind. The in-process API does not emitfailure_class, and a scored rollout carries no failure label.unscored. The rollout loop folds such completions intoscored; this PR does not change the loop. The metric docs say so.failure_kind; a routing class alone does not make it one.Validation
mainon this machine (test_benchmarks.py::...test_every_repo_benchmark_appears_in_listing, connection-pool and memory telemetry tests, and sandbox transfer metrics errors). New tests cover a judge fault, an unscored completion, no failure label on any scored row, no reward sample for a failed rollout, and theunknowndispatch name.agent_request_failed). Not run: the full suite, a live exporter beyond a local collector.