Repository navigation
fix(ci): AUT-2624 repair uv PATH and colored R2E summaries - #5026
Conversation
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
|
/ok to test 666c33b |
|
/review-approve |
|
🌿 Preview your docs: https://nvidia-preview-pr-5026-666c33bbf3fa.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP — LGTM — no reliability concerns.
The server-suite setup exposes uv to both the current shell and subsequent steps without changing dependencies or shard commands. The regression test exercises the extracted setup with installer doubles and checks both PATH contracts.
Static review completed for both changed files and directly affected CI paths. Tests and hosted workflows were not executed during this review.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40045718707 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40087667728 |
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
1857c77
|
/ok to test 1857c77 |
|
Exact-head validation update for
The updated head awaits fresh review; the prior review was dismissed after the push. |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40095930391 |
Backport the R2E parser and regression tests from NVIDIA-NeMo#5026 (upstream commit 1857c77). Strip SGR sequences before interpreting statuses and test IDs. Cover plain and colored output, including missing and nonpassing required tests. The CI setup changes from that PR are not included. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
…IA-NeMo#4841) * feat(eval): count rollouts that ran out of time in the aggregate metrics A rollout that runs out of time scores as a failure of the evaluated model, whatever made it slow. Until now a run did not say how many rollouts that was or how long they ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(eval): write the time-limit section atomically Review finding 4339034971 on NVIDIA-NeMo#4841. The section writer rewrote the aggregate metrics file in place, so a write that failed partway (a full disk) left a truncated document behind, and the next read of the metrics failed the command. The new document is now written to a temporary file in the same directory and replaces the metrics file only once it is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * feat(eval): report timed-out rollouts in the run health summary Timeouts are a run-health diagnostic, so the section moves from the aggregate metrics to quality_summary.json under run.stats.timeouts, next to the model-call error counts. gym eval run, gym eval aggregate and gym eval health-check all build it in run_health_checks from the saved rows and the failures sidecar's last attempts, so a recheck gives the same numbers. The aggregate metrics are scores only again. One typed function, classify_time_limit, decides whether a row ran out of time, which limit it hit and on what evidence: - typed: a TimeoutEvent (event_id, limit, enforcer, evidence, configured_s, elapsed_s, fired_at) in a timeouts list on the row or on an invocation, tool call or model call it carries. It wins over every other signal and is counted once per event_id. - legacy: the labels, flags, termination reason, episode-deadline message and timed-out model call that rows already carry. - unclassified: a failure with no signal whose record shows a measured interval at or past its configured limit. Timing alone never confirms a firing, so these are counted apart. A failure that none of these match is counted as failed_without_signal, never as not timed out. The section also carries the time split of the timed-out rollouts by policy model, other models, tools, time after the last call and the rest, with policy serving speed and calls per rollout. Health-check workers return only a slim view of each failed or timed-out row, so the summary does not hold payloads. The section's numbers go to the configured exporters as flat health/timeouts/... keys after the health checks run; nulls are not sent. The failure-row capture flag is not kept: it needed a capture-store read for every bare failure, which collection deliberately does not join, and the other signals still cover those rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(eval): document run.stats.timeouts on the rollout health page Moves the time-limit documentation from the aggregate metrics page to the rollout health page: the evidence order, every field of the section, the time split and the exporter keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * refactor(eval): rework the timeouts section of the run health summary The section moves to nemo_gym/health/timeouts.py and is built from one typed record per rollout instead of untyped row views. - A rollout counts as timed out on a TimeoutEvent record, or on a legacy signal that a named Gym component sets when its own deadline fires. A test checks that each component still sets its signal. Elapsed time alone no longer counts, so the unclassified evidence is gone, and so are the signals no component sets (transport_timeout and a timed-out model call). - The time split reports flat shares. The unrecorded share is now part of untracked. - gym eval run now exports the section too. Every place that computes run health exports through one helper, which logs a failure and never stops the run. - The failures sidecars are read once and shared by the run accounting and the section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(eval): describe the typed timeout record by what it is Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): keep the timeouts section safe on malformed rows and overlap A value of the wrong type in one rollout's record, such as a trajectory whose invocations are not a list, made the timeouts reader raise, and the whole health report failed with no summary. The reader now reads such a value as not measured, so the record checks report the row and the summary is written. Timeout records are read from the row's own `timeouts` list only. The trajectory's operation records have no field for them, and collection drops a trajectory that adds one. `untracked` was one minus the summed shares, which counted twice a stretch in which two kinds of call overlap, such as a tool that calls the policy model. It is now the share in which no measured call ran. The docs also say that a recheck of a merged directory does not see the shards' failure rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * docs(health): a typed timeout event goes on the row The reader and the docs page take timeout events only from the row's `timeouts` list. Two docstrings still said the event is attached to the operation it bounds, where collection would drop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): time a failed rollout from its capture file Collection joins model-call captures only into a rollout that produced a result, so a rollout that failed without one, such as an episode cut by its deadline, had no calls on its row and an empty time split, although the model server captured its calls under the same rollout id. The health check now attaches a failed row's capture file, read with the collector's own join and key, to a copy of the row before reading its timing. A row that carries its own capture is left alone, so no call counts twice; without a capture file the time stays unmeasured. Collection, `gym eval run` and `--no-serve` pass their capture dir, and `gym eval health-check` takes `--model-call-capture-dir`. A missing or invalid capture setting means no captures, never a failed check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): one timeout classifier and a safe, linear capture join The run timing table and run.stats.timeouts used different timeout evidence, so BrowseComp and Codex timeouts counted in only one of them; both now call classify_timeout. The capture join scanned the capture dir once per failed rollout, created the dir when it was missing, and stopped the report on an unusable dir or a malformed row. Partial captures, inverted or unplaceable spans and non-positive elapsed times read as measured, a resumed tracking run kept stale limit counts, and a recheck without --model-call-capture-dir lost the split silently. Each now lists the dir once, leaves the timing unmeasured, sends 0, or warns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): read gap codes safely, and count unread transcripts and empty outputs Review of the time split found three gaps: - A gap whose code was not a string, such as a list, raised while the section read the row, which stopped the whole health check. A gap code that is not a string now names no gap. - An agent that could not read its own transcript reports an `agent_artifact_unavailable` gap, and its empty tool list read as a measured zero. That gap now leaves the tool time unknown, as `tool_timing_unavailable` does. - A call that produced no output was left out of the serving speed, although it is a measured rate of zero. It now counts at zero. A call without a token count or a positive latency stays unmeasured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): leave tool time unknown when an agent observed no tools, and scope the capture rule Review found more agents whose empty tool list was read as a measured zero: - An agent that captured no observations reports `observation_capture_failed`; PinchBench, OpenClaw, Stirrup and Kilocode all emit it. That gap now leaves the tool time unknown, as tool_timing_unavailable does. - OpenClaw and OpenCode report `agent_transcript_unavailable` when they have no transcript. That gap now leaves the tool time unknown when the tool list is empty. Codex can read its tools from its native transcript when only its CLI events are missing, and tools read that way stay measured. The rollout-health page said health never reads raw captures. That holds for the integrity checks. The timeout statistics read the captured calls, and moving the capture files can change their time split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): leave the time split unknown when part of its evidence was not read The timeouts section measured from evidence it had read only in part, in two ways: - A model-call capture whose call list holds an entry that is not an object was split from its readable calls. It is now left out of the split, like a capture read only in part. - A Codex transcript record or an OpenCode part that could not be read may have been a tool call, yet the tool list read without it counted as the whole tool history. Those parsers now also report tool_timing_unavailable, so tool and untracked time stay unknown. An unreadable Codex native event, such as a token usage record, is never a tool call and does not report it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): count a malformed timeouts value, and leave an unidentified OpenCode tool untimed - An OpenCode tool part without a call id has no tool observation, yet the tool list read without it counted as the whole tool history. Both OpenCode parsers now also report tool_timing_unavailable for it. - A timeouts value that is false, 0, "" or {} read as no events. Only an absent or null value does now; any other value that is not a list counts as one invalid event. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): leave tool time unknown when OpenCode drops a tool for its message Both OpenCode parsers skip a tool part whose owning message cannot be read, has no role, or belongs to no session. The tool was known but left out, yet the tool list read without it counted as the whole tool history. Those parts now also report tool_timing_unavailable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * test(health): move the agent-parser timeout tests to the agents' suites Three timeout-section tests ran an agent's own transcript parser (Codex, OpenClaw) from tests/unit_tests. Importing those agent modules there counted their code in the core unit job's coverage, whose tests for them run in the server suites. The tests now live in each agent's test_observability.py, with the same assertions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): count a Terminus 2 rollout stopped by its run budget as timed out Terminus 2 scores a rollout whose run budget (the task's agent timeout) expires: the episode is verified, and the stop is recorded only on the response it returns, as `response.metadata.terminus2_outcome == "timeout"`. No timeout signal listed that field, so the timeouts section counted such rollouts as not timed out. Add it as a legacy timeout signal, cited from the Terminus 2 agent like the other producers. Tests: a synthetic Terminus row that ended on its budget counts under that signal, and a completed one does not; the parametrized checks cover the new entry's match and its citation. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(anyswe): parse colored R2E pytest summaries Backport the R2E parser and regression tests from NVIDIA-NeMo#5026 (upstream commit 1857c77). Strip SGR sequences before interpreting statuses and test IDs. Cover plain and colored output, including missing and nonpassing required tests. The CI setup changes from that PR are not included. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): preserve configured policy identity in timeout timing Record the configured policy server on landed and failed rows so offline capture joins keep policy calls separate from judges and simulators. Preserve legacy fallback and captured identities. Signed-off-by: Adam Rajfer <arajfer@nvidia.com> * fix(health): classify episode timeouts consistently in live timing Signed-off-by: Adam Rajfer <arajfer@nvidia.com> --------- Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Background
Scheduled run 40015918250 failed before server tests could resolve uv. Later merge-queue run 40087667728 exposed an AnySWE regression: pytest color codes broke the R2E summary assertion and parser. These focused repairs do not need a separate issue.
What changed
Details
Imported upstream main because the reported R2E setup method and regression were absent from the old PR base. No dependencies, shard commands, or failure assertions changed.
Model rollout: N/A for setup and terminal-output parsing; the regression uses real nested pytest output.
Tested
Base
f36e1265; head1857c77e.unit:lint: passed. Private guardword catalog unavailable; its hook skipped the catalog check.functional:2/8: exit 1. Fourteen modules passed; sixteen failed during offline dependency setup. AnySWE could not installswebench==4.1.0, so its new regressions remain unvalidated. Full blocker details are in the request thread.git diff --check: passed; signed head and remote SHA verified.