Repository navigation
feat(eval): report timed-out rollouts in the run health summary - #4841
Conversation
A rollout that runs out of time scores as a failure of the evaluated model, whatever made it slow. Until now a run did not say how many rollouts that was or how long they ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/ok to test 623d7d7 |
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-4841-dec7c3767926.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — two RISK findings: standalone aggregation loses capture-only timeout evidence, and a failed diagnostic rewrite can corrupt previously written aggregate metrics.
Inspected all four changed files and traced collection, aggregation, capture read-back, failure routing, and exporter contracts. Review complete. Static source review only; tests were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Review finding 4339034971 on NVIDIA-NeMo#4841. The section writer rewrote the aggregate metrics file in place, so a write that failed partway (a full disk) left a truncated document behind, and the next read of the metrics failed the command. The new document is now written to a temporary file in the same directory and replaces the metrics file only once it is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Review finding 4339034965 (follow-up to NVIDIA-NeMo#4841). A failure that returned no agent evidence is written to the sidecar without its model-call captures; collection's time-limit section read them back from its capture store and counted a timed-out policy call, but gym eval aggregate has no capture store, so it counted the same run's timeouts as zero. Collection now reads the store once for such a row and, when a captured policy call timed out, records _ng_failure_policy_call_timed_out on the sidecar row (kept out of the metrics like the other _ng_failure_ fields). The time-limit section reads that fact as it reads the captures. The row itself still carries no captures, and scores are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
giuliolovisotto
left a comment
There was a problem hiding this comment.
Hi Adam, thanks for having a look at this.
I have two high level comments:
- I think time limit diagnostics should be part of
health-check, and notaggregate, because it is related to health. They could go for example inquality_summary.json::run.stats. Individual model call errors already go in there for example. - as of today, nothing enforces that any of the event labels this module looks for (
agent_timeout, transport_timeout, timed_out ....) are actually produced by components that can time out.
I think it would be more productive to have a short call, I have done already some work and investigation on a shared Timeout Event in this RFC.
Brings in the run health counts (NVIDIA-NeMo#4788) and the exporter changes (NVIDIA-NeMo#4852, NVIDIA-NeMo#4873), which the time-limit diagnostics now build on. No conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/ok to test b4709b6 |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — three RISK findings affect timeout accounting and attribution; two remain unresolved from review 5637038210.
Completed the broad RESET review of all 18 changed files, including collection, capture readback, trajectory projection, CLI forwarding, aggregation, parsers, and exporters.
Verified the fixes for falsy malformed timeout containers and missing-call-ID tool timing in OpenCode and sandboxed OpenCode. Those concrete prior findings are closed.
Review complete. Static source analysis only; tests and suggested changes were not executed. Verify suggested code before adoption.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
…ts message Both OpenCode parsers skip a tool part whose owning message cannot be read, has no role, or belongs to no session. The tool was known but left out, yet the tool list read without it counted as the whole tool history. Those parts now also report tool_timing_unavailable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Three timeout-section tests ran an agent's own transcript parser (Codex, OpenClaw) from tests/unit_tests. Importing those agent modules there counted their code in the core unit job's coverage, whose tests for them run in the server suites. The tests now live in each agent's test_observability.py, with the same assertions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/ok to test 87b123c |
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — two RISK findings remain unresolved: merged-run rechecks lose timeout accounting, and configured policy-server aliases are misattributed.
Completed the broad RESET review of all 19 changed files and traced collection, capture readback, trajectory projection, CLI forwarding, aggregation, and export paths.
The latest prior OpenCode finding is fixed: both the local parser and sandboxed parser now mark tools omitted because their owner cannot be established. Projection preserves those gaps, and the timeout reader leaves tool and untracked time unknown.
Review complete. Static source analysis only; tests were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
…timed out Terminus 2 scores a rollout whose run budget (the task's agent timeout) expires: the episode is verified, and the stop is recorded only on the response it returns, as `response.metadata.terminus2_outcome == "timeout"`. No timeout signal listed that field, so the timeouts section counted such rollouts as not timed out. Add it as a legacy timeout signal, cited from the Terminus 2 agent like the other producers. Tests: a synthetic Terminus row that ended on its budget counts under that signal, and a completed one does not; the parametrized checks cover the new entry's match and its citation. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Backport the R2E parser and regression tests from NVIDIA-NeMo#5026 (upstream commit 1857c77). Strip SGR sequences before interpreting statuses and test IDs. Cover plain and colored output, including missing and nonpassing required tests. The CI setup changes from that PR are not included. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — two RISK findings remain unresolved: merged-run rechecks lose timeout accounting, and configured policy-server aliases are misattributed.
Completed the broad RESET review of all 22 changed files and traced affected collection, capture readback, trajectory projection, CLI, aggregation, scoring, and export paths.
The earlier OpenCode unknown-owner failure is fixed in both the local parser and sandboxed parser; it is not repeated here.
Static source analysis only; tests were not executed. Two delegated report payloads were unavailable, so their assigned lifecycle and agent-parser areas were inspected directly by the parent reviewer.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Record the configured policy server on landed and failed rows so offline capture joins keep policy calls separate from judges and simulators. Preserve legacy fallback and captured identities. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
/review reset=true |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — merged-run rechecks still lose failure accounting, and the live timing table misses timeout evidence inside episode responses.
Completed the broad RESET review of all 19 changed files and their affected collection, capture-readback, projection, CLI, aggregation, and export paths.
The prior configured-policy-alias finding is fixed in the inspected source. The collector-to-offline regression checks the real failure sidecar, policy/judge shares 0.4/0.1, one policy call, and 10 output tokens/s. The existing OpenCode omission guards also remain present in both parsers.
Static source analysis only; tests were not executed. Two delegated report payloads were unavailable, so their assigned timeout-calculation and parser/documentation areas were inspected directly.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| if failures_path_for(path).is_file() | ||
| for row in _jsonl_rows(failures_path_for(path)) | ||
| ) | ||
| summary["run"]["stats"]["timeouts"] = summarize_timeouts(timings, attempts).model_dump(mode="json") |
There was a problem hiding this comment.
RISK — retain source-shard provenance for default merged-run rechecks.
Failure / blast radius: Review 5650496115/comment 4376149369 remains unresolved. Aggregate two shards into a third directory: A contains one successful row; B has an empty rollout file and one failure-sidecar row with _ng_failure_class="timeout_exceeded". Both inventories owe task/repeat 0/0. Aggregation checks the original sources, reporting rollouts=2, timed_out=1. However, merging persists only landed rows, and a default recheck selects only the merged file. This assignment overwrites the section with rollouts=1, timed_out=0. Scores are unaffected, but unchanged artifacts lose timeout/failure accounting. The acknowledged dependency is not incorporated in this head.
Correction: Persist ordered original-input provenance in RolloutAggregationHelper.run_from_config; resolve it in health_check_run_dir for default rechecks and pass those paths to run_health_checks, retaining report output and persisted limits in the merged directory. Explicit rollout_file selection must bypass provenance. Preserve each source's inventory, sidecar, and source index through _read_failures: concatenating sidecars by task/repeat would let A's success suppress B's failure. Update the documented limitation. The invariant is identical accounting between aggregation and a default recheck of unchanged inputs. Manifest storage and relocation semantics require an implementation choice, so no safe drop-in patch is established.
Preservation / acceptance — proposed, not executed: The two-shard setup must retain rollouts=2, timed_out=1, by_evidence.legacy=1, and run.artifacts.failed.total=1 before and after rechecking, including reused 0/0 identities. Within one source, retain suppression of a failure followed by a landed row and selection of only the last failed attempt. With provenance present, explicitly select an alternate file containing one success and require only it to contribute (rollouts=1, timed_out=0). Retain the single-source agreement and score assertions, including mean/reward == 0.5.
| def record_result(self, result: Mapping[str, Any], seconds: float) -> None: | ||
| self._durations.append(seconds) | ||
| if result.get("timed_out") or result.get(NG_FAILURE_CLASS_KEY) in ("timeout_exceeded", AGENT_TIMEOUT): | ||
| if classify_timeout(result): |
There was a problem hiding this comment.
RISK — classify the projected episode record in both timeout reports.
Failure / blast radius: One classifier still receives different inputs. _post_subroutine calls record_result on the raw response, while collection subsequently projects materialized episode replies. A materialized task returning {episode_id: {...}, task_id: {...}, failure: {failure_reason: "Episode timed out", terminal: false}} has no recognized top-level signal in that raw envelope. The timing table therefore reports 0.0% (0) hitting the wall-clock limit. _episode_record converts the same reply to _ng_failure_message="Episode timed out", so saved health counts one timeout. This is the normal environment episode-deadline response. Signals inside an episode's successful result object have the same mismatch. Scores are unaffected, but the live deployment diagnostic can falsely report no timeouts.
Correction: Replace the record_result block in _post_subroutine at lines 4549–4550 with a timing-only projection, leaving _CompletedRollout.result raw:
if run_timing is not None and isinstance(result, Mapping):
timing_result = (
_episode_record(result)
if _materialized_taskset(row) is not None and _is_episode_response(result)
else result
)
run_timing.record_result(timing_result, time.monotonic() - started)The invariant is that table and saved-health classification see the same canonical fields. The projection constructs a separate record. Retain the request-shape guard: a legacy reply merely echoing episode-like identities must not be projected. This suggestion was traced through projection and dispatch contracts, not executed; verify it before adoption.
Preservation / acceptance — proposed, not executed: Dispatch one materialized row (task_id={"taskset":"t","task_id":"x"}, task_input={}) through a configured environment with a fake /run returning the deadline-failure envelope above. With timing enabled, require hit wall-clock limit to contain 100.0% (1) and saved health to report timed_out=1. Repeat with an episode result containing agent_timed_out=True, and with a valid typed timeout event inside result; both reports must agree. A completed episode without evidence must remain 0.0% (0). Preserve flat Codex/BrowseComp classification, the legacy identity-echo case, and run_examples returning raw replies unchanged.
giuliolovisotto
left a comment
There was a problem hiding this comment.
Thanks Adam—this addresses my earlier concern: timeout diagnostics now live under run.stats.timeouts without changing scores or health verdicts.
Two remaining requests:
- Stack this on #4996 and validate them together, so re-running health checks on merged outputs preserves timeout counts from source-shard failures.
- Document the current scope and intended follow-up: partial producer coverage and final-outcome counting are fine for this PR. Longer term, saved observability should retain all
NgFailureEvents, including failed attempts followed by successful retries. The CLI can have a narrower reporting scope; its counts don’t need to match.
No need to expand instrumentation or implement recovered-attempt accounting in this PR.
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
Thanks Giulio. I added the partial-coverage/final-outcome scope and the all-attempts observability follow-up. I also fixed the live timing table to classify episode timeout replies consistently with saved health, without changing the returned result. The published fix passes 791 focused tests. I independently tested it together with #4996: 1,213 tests pass, including the two-shard case preserving two expected rollouts and one timeout after aggregation and recheck; explicit single-file checking remains narrow. Both branches are in my fork, so I cannot target #4996 directly in the upstream PR. I will merge #4996 first once its required approval is in, then bring main into this branch and add the prepared joint regression. #4996 is still review-required; this new head also needs CI/review. No instrumentation expansion or recovered-attempt accounting is included. |
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Preserve omitted-rollout selection in the shared failure reader so timeout and health summaries use the same final outcomes. Retain both sets of collection regressions and cover interrupted versus never-started retries. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40475678899 |
Timeout diagnostics now live under
quality_summary.json::run.stats.timeouts, sogym eval health-checkcan distinguish recorded model, agent and judge/sandbox timeout evidence without changing scores or health verdicts. The live timing table also projects materialized episode replies the same way as saved health reporting; the raw reply returned to collection is unchanged.Coverage is deliberately partial and counts final outcomes. A failed attempt followed by a successful rollout is not counted as a final timeout. Retaining every failure observation, including recovered attempts, is follow-up work; CLI diagnostic counts do not need to match that broader history. This PR does not expand instrumentation or change retry policy.
#4996 and #4921 are now merged. This branch incorporates main at
4945282322ebbef8a77c90a4b937d20d412618ec, preserving shard-aware rechecking and context-overflow behavior alongside timeout diagnostics. The joint merged-shard regression and documentation update are included.Validation at
ed7e5db77180c295622199c6aa12de6eddae5bd0:Current-head hosted CI and required human review must complete before merging.