Skip to content

feat(eval): report timed-out rollouts in the run health summary - #4841

Merged
AdamRajfer merged 27 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:eval/time-limit-detection
Oct 27, 2026
Merged

AdamRajfer merged 27 commits into
NVIDIA-NeMo:mainfrom
AdamRajfer:eval/time-limit-detection

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Timeout diagnostics now live under quality_summary.json::run.stats.timeouts, so gym eval health-check can distinguish recorded model, agent and judge/sandbox timeout evidence without changing scores or health verdicts. The live timing table also projects materialized episode replies the same way as saved health reporting; the raw reply returned to collection is unchanged.

Coverage is deliberately partial and counts final outcomes. A failed attempt followed by a successful rollout is not counted as a final timeout. Retaining every failure observation, including recovered attempts, is follow-up work; CLI diagnostic counts do not need to match that broader history. This PR does not expand instrumentation or change retry policy.

#4996 and #4921 are now merged. This branch incorporates main at 4945282322ebbef8a77c90a4b937d20d412618ec, preserving shard-aware rechecking and context-overflow behavior alongside timeout diagnostics. The joint merged-shard regression and documentation update are included.

Validation at ed7e5db77180c295622199c6aa12de6eddae5bd0:

  • 878 focused collection, health, timeout and timing tests passed; changed-file pre-commit hooks passed.
  • The joint regression checks that two source shards sharing a task/repeat identity retain both expected rollouts and their timeout accounting through a default merged-directory health recheck.
  • Existing positive episode-deadline and nested-timeout regressions remain covered. No broader instrumentation or recovered-attempt accounting is introduced.

Current-head hosted CI and required human review must complete before merging.

A rollout that runs out of time scores as a failure of the evaluated
model, whatever made it slow. Until now a run did not say how many
rollouts that was or how long they ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer AdamRajfer self-assigned this Oct 21, 2026
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/ok to test 623d7d7

@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work labels Oct 21, 2026
@AdamRajfer
AdamRajfer marked this pull request as ready for review October 21, 2026 15:00
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 21, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — two RISK findings: standalone aggregation loses capture-only timeout evidence, and a failed diagnostic rewrite can corrupt previously written aggregate metrics.

Inspected all four changed files and traced collection, aggregation, capture read-back, failure routing, and exporter contracts. Review complete. Static source review only; tests were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/rollout_collection.py Resolved
Comment thread nemo_gym/rollout_collection.py Resolved
Review finding 4339034971 on NVIDIA-NeMo#4841. The section writer rewrote the aggregate metrics file in place, so a write that failed partway (a full disk) left a truncated document behind, and the next read of the metrics failed the command. The new document is now written to a temporary file in the same directory and replaces the metrics file only once it is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 21, 2026 19:01
AdamRajfer added a commit to AdamRajfer/Gym that referenced this pull request Oct 21, 2026
Review finding 4339034965 (follow-up to NVIDIA-NeMo#4841). A failure that returned no agent evidence is written to the sidecar without its model-call captures; collection's time-limit section read them back from its capture store and counted a timed-out policy call, but gym eval aggregate has no capture store, so it counted the same run's timeouts as zero. Collection now reads the store once for such a row and, when a captured policy call timed out, records _ng_failure_policy_call_timed_out on the sidecar row (kept out of the metrics like the other _ng_failure_ fields). The time-limit section reads that fact as it reads the captures. The row itself still carries no captures, and scores are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@giuliolovisotto
giuliolovisotto self-requested a review October 22, 2026 10:24

@giuliolovisotto giuliolovisotto left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Adam, thanks for having a look at this.

I have two high level comments:

  • I think time limit diagnostics should be part of health-check, and not aggregate, because it is related to health. They could go for example in quality_summary.json::run.stats. Individual model call errors already go in there for example.
  • as of today, nothing enforces that any of the event labels this module looks for (agent_timeout, transport_timeout, timed_out ....) are actually produced by components that can time out.

I think it would be more productive to have a short call, I have done already some work and investigation on a shared Timeout Event in this RFC.

Brings in the run health counts (NVIDIA-NeMo#4788) and the exporter changes (NVIDIA-NeMo#4852, NVIDIA-NeMo#4873), which the time-limit diagnostics now build on. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/ok to test b4709b6

@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — three RISK findings affect timeout accounting and attribution; two remain unresolved from review 5637038210.

Completed the broad RESET review of all 18 changed files, including collection, capture readback, trajectory projection, CLI forwarding, aggregation, parsers, and exporters.

Verified the fixes for falsy malformed timeout containers and missing-call-ID tool timing in OpenCode and sandboxed OpenCode. Those concrete prior findings are closed.

Review complete. Static source analysis only; tests and suggested changes were not executed. Verify suggested code before adoption.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/rollout_health.py Resolved
Comment thread nemo_gym/health/timeouts.py Resolved
AdamRajfer and others added 2 commits October 24, 2026 08:44
…ts message

Both OpenCode parsers skip a tool part whose owning message cannot be
read, has no role, or belongs to no session. The tool was known but
left out, yet the tool list read without it counted as the whole tool
history. Those parts now also report tool_timing_unavailable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Three timeout-section tests ran an agent's own transcript parser
(Codex, OpenClaw) from tests/unit_tests. Importing those agent modules
there counted their code in the core unit job's coverage, whose tests
for them run in the server suites. The tests now live in each agent's
test_observability.py, with the same assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/ok to test 87b123c

@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — two RISK findings remain unresolved: merged-run rechecks lose timeout accounting, and configured policy-server aliases are misattributed.

Completed the broad RESET review of all 19 changed files and traced collection, capture readback, trajectory projection, CLI forwarding, aggregation, and export paths.

The latest prior OpenCode finding is fixed: both the local parser and sandboxed parser now mark tools omitted because their owner cannot be established. Projection preserves those gaps, and the timeout reader leaves tool and untracked time unknown.

Review complete. Static source analysis only; tests were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/rollout_health.py Resolved
Comment thread nemo_gym/health/timeouts.py Resolved
AdamRajfer and others added 2 commits October 26, 2026 00:08
…timed out

Terminus 2 scores a rollout whose run budget (the task's agent timeout) expires: the episode is verified, and the stop is
recorded only on the response it returns, as `response.metadata.terminus2_outcome == "timeout"`. No timeout signal
listed that field, so the timeouts section counted such rollouts as not timed out.

Add it as a legacy timeout signal, cited from the Terminus 2 agent like the other producers. Tests: a synthetic Terminus
row that ended on its budget counts under that signal, and a completed one does not; the parametrized checks cover the
new entry's match and its citation.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Backport the R2E parser and regression tests from NVIDIA-NeMo#5026
(upstream commit 1857c77).
Strip SGR sequences before interpreting statuses and test IDs. Cover plain
and colored output, including missing and nonpassing required tests.

The CI setup changes from that PR are not included.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — two RISK findings remain unresolved: merged-run rechecks lose timeout accounting, and configured policy-server aliases are misattributed.

Completed the broad RESET review of all 22 changed files and traced affected collection, capture readback, trajectory projection, CLI, aggregation, scoring, and export paths.

The earlier OpenCode unknown-owner failure is fixed in both the local parser and sandboxed parser; it is not repeated here.

Static source analysis only; tests were not executed. Two delegated report payloads were unavailable, so their assigned lifecycle and agent-parser areas were inspected directly by the parent reviewer.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread nemo_gym/rollout_health.py Resolved
Comment thread nemo_gym/health/timeouts.py Resolved
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Record the configured policy server on landed and failed rows so offline capture joins keep policy calls separate from judges and simulators. Preserve legacy fallback and captured identities.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

/review reset=true

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — merged-run rechecks still lose failure accounting, and the live timing table misses timeout evidence inside episode responses.

Completed the broad RESET review of all 19 changed files and their affected collection, capture-readback, projection, CLI, aggregation, and export paths.

The prior configured-policy-alias finding is fixed in the inspected source. The collector-to-offline regression checks the real failure sidecar, policy/judge shares 0.4/0.1, one policy call, and 10 output tokens/s. The existing OpenCode omission guards also remain present in both parsers.

Static source analysis only; tests were not executed. Two delegated report payloads were unavailable, so their assigned timeout-calculation and parser/documentation areas were inspected directly.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

if failures_path_for(path).is_file()
for row in _jsonl_rows(failures_path_for(path))
)
summary["run"]["stats"]["timeouts"] = summarize_timeouts(timings, attempts).model_dump(mode="json")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — retain source-shard provenance for default merged-run rechecks.

Failure / blast radius: Review 5650496115/comment 4376149369 remains unresolved. Aggregate two shards into a third directory: A contains one successful row; B has an empty rollout file and one failure-sidecar row with _ng_failure_class="timeout_exceeded". Both inventories owe task/repeat 0/0. Aggregation checks the original sources, reporting rollouts=2, timed_out=1. However, merging persists only landed rows, and a default recheck selects only the merged file. This assignment overwrites the section with rollouts=1, timed_out=0. Scores are unaffected, but unchanged artifacts lose timeout/failure accounting. The acknowledged dependency is not incorporated in this head.

Correction: Persist ordered original-input provenance in RolloutAggregationHelper.run_from_config; resolve it in health_check_run_dir for default rechecks and pass those paths to run_health_checks, retaining report output and persisted limits in the merged directory. Explicit rollout_file selection must bypass provenance. Preserve each source's inventory, sidecar, and source index through _read_failures: concatenating sidecars by task/repeat would let A's success suppress B's failure. Update the documented limitation. The invariant is identical accounting between aggregation and a default recheck of unchanged inputs. Manifest storage and relocation semantics require an implementation choice, so no safe drop-in patch is established.

Preservation / acceptance — proposed, not executed: The two-shard setup must retain rollouts=2, timed_out=1, by_evidence.legacy=1, and run.artifacts.failed.total=1 before and after rechecking, including reused 0/0 identities. Within one source, retain suppression of a failure followed by a landed row and selection of only the last failed attempt. With provenance present, explicitly select an alternate file containing one success and require only it to contribute (rollouts=1, timed_out=0). Retain the single-source agreement and score assertions, including mean/reward == 0.5.

def record_result(self, result: Mapping[str, Any], seconds: float) -> None:
self._durations.append(seconds)
if result.get("timed_out") or result.get(NG_FAILURE_CLASS_KEY) in ("timeout_exceeded", AGENT_TIMEOUT):
if classify_timeout(result):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — classify the projected episode record in both timeout reports.

Failure / blast radius: One classifier still receives different inputs. _post_subroutine calls record_result on the raw response, while collection subsequently projects materialized episode replies. A materialized task returning {episode_id: {...}, task_id: {...}, failure: {failure_reason: "Episode timed out", terminal: false}} has no recognized top-level signal in that raw envelope. The timing table therefore reports 0.0% (0) hitting the wall-clock limit. _episode_record converts the same reply to _ng_failure_message="Episode timed out", so saved health counts one timeout. This is the normal environment episode-deadline response. Signals inside an episode's successful result object have the same mismatch. Scores are unaffected, but the live deployment diagnostic can falsely report no timeouts.

Correction: Replace the record_result block in _post_subroutine at lines 4549–4550 with a timing-only projection, leaving _CompletedRollout.result raw:

if run_timing is not None and isinstance(result, Mapping):
    timing_result = (
        _episode_record(result)
        if _materialized_taskset(row) is not None and _is_episode_response(result)
        else result
    )
    run_timing.record_result(timing_result, time.monotonic() - started)

The invariant is that table and saved-health classification see the same canonical fields. The projection constructs a separate record. Retain the request-shape guard: a legacy reply merely echoing episode-like identities must not be projected. This suggestion was traced through projection and dispatch contracts, not executed; verify it before adoption.

Preservation / acceptance — proposed, not executed: Dispatch one materialized row (task_id={"taskset":"t","task_id":"x"}, task_input={}) through a configured environment with a fake /run returning the deadline-failure envelope above. With timing enabled, require hit wall-clock limit to contain 100.0% (1) and saved health to report timed_out=1. Repeat with an episode result containing agent_timed_out=True, and with a valid typed timeout event inside result; both reports must agree. A completed episode without evidence must remain 0.0% (0). Preserve flat Codex/BrowseComp classification, the legacy identity-echo case, and run_examples returning raw replies unchanged.

@giuliolovisotto giuliolovisotto left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Adam—this addresses my earlier concern: timeout diagnostics now live under run.stats.timeouts without changing scores or health verdicts.

Two remaining requests:

  1. Stack this on #4996 and validate them together, so re-running health checks on merged outputs preserves timeout counts from source-shard failures.
  2. Document the current scope and intended follow-up: partial producer coverage and final-outcome counting are fine for this PR. Longer term, saved observability should retain all NgFailureEvents, including failed attempts followed by successful retries. The CLI can have a narrower reporting scope; its counts don’t need to match.

No need to expand instrumentation or implement recovered-attempt accounting in this PR.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer

Copy link
Copy Markdown
Contributor Author

Thanks Giulio. I added the partial-coverage/final-outcome scope and the all-attempts observability follow-up. I also fixed the live timing table to classify episode timeout replies consistently with saved health, without changing the returned result.

The published fix passes 791 focused tests. I independently tested it together with #4996: 1,213 tests pass, including the two-shard case preserving two expected rollouts and one timeout after aggregation and recheck; explicit single-file checking remains narrow.

Both branches are in my fork, so I cannot target #4996 directly in the upstream PR. I will merge #4996 first once its required approval is in, then bring main into this branch and add the prepared joint regression. #4996 is still review-required; this new head also needs CI/review. No instrumentation expansion or recovered-attempt accounting is included.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Preserve omitted-rollout selection in the shared failure reader so timeout and health summaries use the same final outcomes. Retain both sets of collection regressions and cover interrupted versus never-started retries.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40475678899

This branch was previously deployed

1 inactive deployment
public — dec7c376 Deployed Oct 27, 2026 by copy-pr-bot[bot] via release / finalize / notify #7538
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants