Skip to content

feat(eval): count each rollout of the in-process API in Gym's rollout metrics - #5801

Merged
iguehring-nvidia merged 2 commits into
mainfrom
iguehring/in-process-telemetry
Nov 10, 2026
Merged

iguehring-nvidia merged 2 commits into
mainfrom
iguehring/in-process-telemetry

Conversation

@iguehring-nvidia

@iguehring-nvidia iguehring-nvidia commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Counts each rollout of TaskRunner.run_task in Gym's rollout metrics, so a caller of the in-process API does not call the recorders itself. Follows #5595, which is merged.

What changed

run_task records for its rollout:

  • gym.rollout.completed_total:
    • failed when the outcome has a failure, labeled with the failure's kind (failure_kind) and exception type;
    • otherwise scored when the outcome has a reward, and unscored when it has none.
  • gym.rollout.rewarded_total for a scored rollout with a finite reward. A masked rollout keeps its reward, so it is counted and labeled masked, as in the rollout loop.

The recorders do nothing without a meter. Duration comes from the agent server. A row that names neither an agent nor an environment server is counted under unknown instead of raising after the rollout finished.

Decisions

  • The outcome follows the failure. A rollout with a failure is failed whatever its reward is, as in the rollout loop (a judge fault returns reward 0.0 and a failure, and is failed on both paths). The returned RolloutOutcome is unchanged: the caller still gets the reward and the failure and decides what to do with them.
  • One failure label, failure_kind, taken from TaskFailure.kind. The in-process API does not emit failure_class, and a scored rollout carries no failure label.
  • A rollout with neither a reward nor a failure (a completion without a score) is unscored. The rollout loop folds such completions into scored; this PR does not change the loop. The metric docs say so.
  • A result is a failure when it carries failure_kind; a routing class alone does not make it one.
  • Not gated on span groups, as fix(telemetry): record metrics whenever metrics export, independent of span groups #4612 proposes for the loop.
  • The counter definition in the code and in the metrics page now names both producers and states the rule next to the in-process API.

Validation

  • Unit tests: the in-process, public API and docs-link tests pass (208). A wider run of the environment, rollout recovery, decorator and telemetry tests passes except tests that also fail on main on this machine (test_benchmarks.py::...test_every_repo_benchmark_appears_in_listing, connection-pool and memory telemetry tests, and sandbox transfer metrics errors). New tests cover a judge fault, an unscored completion, no failure label on any scored row, no reward sample for a failed rollout, and the unknown dispatch name.
  • Run against real Gym servers (IFEval, stub model): the counters matched the trial outcomes, including a run with failing prompts (15 scored, 5 failed) and a run with the environment server killed (21 scored, 39 failed with kind agent_request_failed). Not run: the full suite, a live exporter beyond a local collector.

@copy-pr-bot

copy-pr-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@iguehring-nvidia iguehring-nvidia self-assigned this Nov 7, 2026
@iguehring-nvidia
iguehring-nvidia force-pushed the iguehring/in-process-telemetry branch 2 times, most recently from 41c8e68 to 37f4125 Compare November 7, 2026 21:17
@iguehring-nvidia
iguehring-nvidia force-pushed the iguehring/in-process-outcome branch from 3f94ccf to b4e2adc Compare November 7, 2026 21:46
@iguehring-nvidia
iguehring-nvidia force-pushed the iguehring/in-process-telemetry branch from abc4275 to ee6acc1 Compare November 7, 2026 21:46
@iguehring-nvidia
iguehring-nvidia marked this pull request as ready for review November 7, 2026 22:58
@iguehring-nvidia

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP — one non-blocking documentation NOTE; no correctness or lifecycle defects found.

I reviewed all three changed files and traced dispatch resolution, failure normalization, metric recording, and the existing telemetry tests. This was a static review; I didn't run tests or a live exporter.

Note: formal approval is not available for this request, so this review is posted as a comment only.


Inline comments (escaped v1):

  • nemo_gym/telemetry/gym_metrics.py line 73: NOTE — Suggest documenting the counter's new producer.

    With telemetry active, a completed TaskRunner.run_task now increments this counter without saving an outcome. This definition still says only run_from_config records it and makes persistence a condition for every increment. The user-facing metric definition likewise describes every increment as a persisted CLI attempt. Readers can therefore mistake Python API counts for durable saved outcomes.

    Please qualify the existing persistence/routing descriptions as collection-driver behavior, remove the exclusivity claim here, and add this distinction to both definitions. Suggested paragraph:

    TaskRunner.run_task also records each returned rollout outcome, without a recovery-store write. For this API, scored means a reward is present and failed means none is present; a reported failure kind labels either outcome. rewarded_total counts only finite rewards, including masked rewards.
    

    Preserve the CLI distinctions: cached results don't increment the counter, saved omissions retain omitted, and its historical scored label includes supported unscored completions. Static acceptance: both definitions identify the two producers and distinguish their persistence and outcome rules; neither implies that a TaskRunner increment proves an outcome was saved. No runtime change is needed.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added feature New capabilities, enhancements, or enablement work area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters needs-review PR is ready for code review and waiting on a reviewer labels Nov 8, 2026

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failure handling mostly lines up: a retry of one run_task is a single count, a typed /run failure is counted failed with its kind, and the standard path is untouched (the diff only adds code, docstrings and docs, and the existing in-process tests pass here: 72 passed). Two things I'd fix before this lands, plus a smaller label mismatch.

  1. A judge fault is counted as a scored zero here, and as a failed rollout on the standard path. judge_failsafe returns reward: 0.0, mask_sample: True, failure_kind: judge_failed and _ng_failure_class: judge_failed (nemo_gym/verification/judge.py:350-363). gym eval run routes that row to the failures sidecar and records failed (rollout_collection.py:4427, 4511-4540). _outcome (in_process.py:380-401) only reads failure_kind, so _record_outcome sees a reward and records scored, plus a rewarded_total sample in the zero range, masked. The new test test_a_scored_rollout_that_carries_a_kind_is_counted_under_it pins exactly this. Any dashboard or health gate that treats scored as "the model was measured" will see eval defects as measured zeros from in-process callers, and not from gym eval run. Suggested fix: treat a result that carries _ng_failure_class (or a judge/env failure kind that the loop would drop) as failed and skip rewarded_total, i.e. decide the outcome with the same predicate the loop uses instead of "reward is not None". Keep scored for masked rewards that have no failure class.

  2. Labels differ from the loop on cases that are not faults. On the standard path a scored row never carries failure_class/failure_kind (rollout_collection.py:2437), so a group by failure_kind panel only contains non-scored rows. Here scored rollouts with a kind carry them too (in_process.py:366-371). Also, for a typed failure with no kind, the loop sets failure_class to the routing class and leaves failure_kind empty ("without inventing a canonical failure kind", rollout_outcomes.py:53-58), but _failure_of_record puts sidecar_failure_class into TaskFailure.kind, so the new code emits failure_kind=agent_request_failed (the test's third case shows it). failure_type is exception_type on both paths, fine. Suggested fix: no failure labels on scored, and leave failure_kind unset when the record's failure.failure_kind is None.

  3. A completion without a reward and without a failure is failed with no class (in_process.py:368-370). The loop counts a supported unscored completion as scored, and a bare no-persist reply as omitted. Either map those the same way or say in the docs that these show up as failed with empty labels, because a failed-with-no-class series will otherwise look like an unclassified fault.

Smaller: _dispatch_name(resolved_row) runs after the rollout finished (in_process.py:353) and raises KeyError if a row has neither agent_ref nor an environment server key. That would lose a completed outcome over a label, so I'd fall back to a placeholder rather than raise. I did not exercise that with a real collated row.

Not verified: no live exporter or real servers, and I didn't compare emitted attributes against a running gym eval run. The comparison is from the code on both sides.

Comment thread nemo_gym/environment/in_process.py Resolved
Comment thread nemo_gym/environment/in_process.py Resolved

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A second pass, this time from the side of a caller that embeds TaskRunner in its own long-lived worker: one runner per benchmark and model, many concurrent run_task calls, and its own OpenTelemetry setup. The recording itself works there. It never raises, a cancelled run_task records nothing, and each caller retry counts as one more attempt, which matches the counter's definition. The in-process tests pass at 45d326f (72 passed). Two things matter for that kind of caller.

  1. The documented opt-in breaks a host that already has an SDK MeterProvider (fern/versions/latest/pages/api-reference/index.mdx:148-151). If such a process follows "call init_telemetry first", OTel refuses the second global provider ("Overriding of current MeterProvider is not allowed") and Gym's meter resolves to the host's provider, so the counters do arrive. But Gym still marks itself as the owner (_OWNS_PROVIDERS = True), and shutdown_telemetry() then shuts down the host's provider. I reproduced it with a host MeterProvider and an InMemoryMetricReader, NEMO_GYM_OTEL_ENABLED=1, init_telemetry() and shutdown_telemetry(): the host provider ends up shut down. The doc also doesn't say that init_telemetry does nothing unless NEMO_GYM_OTEL_ENABLED is set. The root cause is in init_telemetry, which only recognises nemo-lens as an existing owner, so treating any installed SDK meter provider the same way (reuse it, don't own it) would fix it. If that belongs in a separate PR, I'd at least spell out the env var and the shutdown caveat in this doc.

  2. The counts can't be attributed to a run when one process holds several runners (nemo_gym/environment/in_process.py:353, 357-373). The only per-point attributes are the outcome, the dispatch name and the failure labels, and resource attributes are process-wide. A worker with two models open on the same benchmark gets one merged series, so a jump in failed can't be traced to the run that caused it. gym eval run doesn't hit this because it is one run per process. Letting the caller pass a few extra attributes when it builds the TaskRunner (something like metric_attributes={...}), added to both counters, would close it.

Smaller, and related to the judge-failure point in my earlier review: an embedder that treats a masked or kind-carrying outcome as not measured will count differently from scored here. Whichever predicate you settle on, stating it next to the RolloutOutcome fields would let callers line their own numbers up with Gym's.

Not checked: a real collector, and Gym servers spawned by the host that inherit the env var.

@iguehring-nvidia

Copy link
Copy Markdown
Contributor Author

@AdamRajfer thank you for both passes. The changes are in 527973c59, and the PR description states the new rule.

First review

  1. Judge fault counted as scored. Fixed. The outcome follows the failure: a rollout with a failure is failed whatever its reward is, so the judge fault (reward 0.0, masked, judge_failed) is failed on both paths. A failed rollout adds nothing to rewarded_total. The returned RolloutOutcome is unchanged: the caller still gets the reward and the failure. test_a_scored_rollout_that_carries_a_kind_is_counted_under_it is replaced by tests for the judge fault and for "no scored row carries a failure label".
  2. Labels.
    • No failure labels on scored: done. A scored rollout has no failure, so it has no kind to label it with.
    • Not done the way you suggested. The in-process API exposes one failure value, the kind (TaskFailure.kind), and not the routing class, so I stopped emitting failure_class from it and emit only failure_kind. For a record with no canonical kind, that value is the routing fallback, where the loop leaves failure_kind empty and sets failure_class. Reading "kind, else class" gives the same value on both paths. I made that fill on my side of the dashboards; the loop is unchanged here.
  3. No reward and no failure. Counted as a new outcome, unscored, not failed and not scored. The loop folds these completions into scored; the metric page now says so. I did not change the loop here and am happy to align it in a follow-up.
  4. _dispatch_name after the rollout. A row that names neither an agent nor an environment server is counted under unknown instead of raising. Test added.

Second review

  1. Host with its own MeterProvider. You are right, and the root cause is in init_telemetry. It is not in this PR; I will send it separately (treat any installed SDK meter provider as an existing owner: reuse it, do not shut it down). I did not add the env var or shutdown caveat to the docs of this PR, so that stays open until that PR.
  2. Several runners in one process. Not done. A metric_attributes argument on TaskRunner is a good fit; I would rather add it when a caller needs it.
  3. State the predicate. Done. The API page and the metric page now give the rule next to the in-process API, say that a result is a failure when it carries failure_kind (a routing class alone does not make it one), and say that a caller may apply another rule to the same outcome.

Checked against real Gym servers (IFEval, stub model): a run with failing prompts gave 15 scored and 5 failed, and a run with the environment server killed gave 21 scored and 39 failed (kind agent_request_failed), matching the trial outcomes.

AdamRajfer
AdamRajfer previously approved these changes Nov 10, 2026

@AdamRajfer AdamRajfer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, 527973c covers the first review: a failure now decides the outcome, so a judge fault is failed on both paths, scored rows carry no failure labels, and unscored plus the stated predicate make the in-process rule clear. "kind, else class" for the routing fallback works for me. Approving with the two open points as follow-ups: the init_telemetry fix for a host that already has a MeterProvider (until then the env var and shutdown caveat aren't in the docs), and per-runner metric attributes once a caller needs them.

…metrics

run_task records gym.rollout.completed_total (scored when the rollout has a reward, failed when it has none, labelled with the failure's kind) and gym.rollout.rewarded_total, so a caller of the in-process API does not call the metric recorders itself.

Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
…ir reward

A rollout with a failure is counted failed and labeled with its kind only. Without a failure it is scored when it has a reward and unscored when it has none. A failed rollout adds nothing to rewarded_total. The failure_class label is no longer emitted from the in-process API, and a row that names neither an agent nor an environment server is counted under unknown instead of raising after the rollout finished.

Signed-off-by: Ingo Guehring <iguehring@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42841921077

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 10, 2026
@iguehring-nvidia
iguehring-nvidia added this pull request to the merge queue Nov 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42844994879

@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42848006586

Merged via the queue into main with commit c582734 Nov 10, 2026
108 of 111 checks passed
@iguehring-nvidia
iguehring-nvidia deleted the iguehring/in-process-telemetry branch November 10, 2026 16:09

This branch was previously deployed

1 inactive deployment
public — 34de5b65 Deployed Nov 10, 2026 by copy-pr-bot[bot] via release / finalize / notify #11433
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants