Skip to content

fix(ci): AUT-2624 repair uv PATH and colored R2E summaries - #5026

Merged
ko3n1g merged 4 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/server-suite-uv-path
Oct 24, 2026
Merged

ko3n1g merged 4 commits into
NVIDIA-NeMo:mainfrom
svcnemo-autobot:fix/server-suite-uv-path

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Background

Scheduled run 40015918250 failed before server tests could resolve uv. Later merge-queue run 40087667728 exposed an AnySWE regression: pytest color codes broke the R2E summary assertion and parser. These focused repairs do not need a separate issue.

What changed

  • Export uv's directory for the current shell and later steps; check its version.
  • Retain the workflow setup regression.
  • Strip SGR color codes before parsing R2E test statuses and IDs.
  • Cover plain/colored summaries and hidden-test linkage, including rejection of nonpassing or missing results.

Details

Imported upstream main because the reported R2E setup method and regression were absent from the old PR base. No dependencies, shard commands, or failure assertions changed.

flowchart LR
  A[Install uv] --> B[Expose PATH] --> C[Server tests]
  D[Colored pytest summary] --> E[Strip SGR] --> F[R2E result parser]
Loading

Model rollout: N/A for setup and terminal-output parsing; the regression uses real nested pytest output.

Tested

Base f36e1265; head 1857c77e.

  • Managed unit:lint: passed. Private guardword catalog unavailable; its hook skipped the catalog check.
  • Managed functional:2/8: exit 1. Fourteen modules passed; sixteen failed during offline dependency setup. AnySWE could not install swebench==4.1.0, so its new regressions remain unvalidated. Full blocker details are in the request thread.
  • git diff --check: passed; signed head and remote SHA verified.
  • Hosted CI on this head remains pending. Earlier green results are not claimed for this head. No GPU tests ran.

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
svcnemo-autobot requested a review from a team as a code owner October 23, 2026 23:59
@copy-pr-bot

copy-pr-bot Bot commented Oct 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:ci CI workflows, test routing, and repository automation complexity:low Localized change in one scope with a small, straightforward review surface labels Oct 23, 2026
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 666c33b

@ko3n1g

ko3n1g commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

/review-approve

@svcnvidia-nemo-ci svcnvidia-nemo-ci added ci CI, automation, test queue, or workflow infrastructure work needs-review PR is ready for code review and waiting on a reviewer labels Oct 24, 2026
@ko3n1g
ko3n1g enabled auto-merge October 24, 2026 00:02
@github-actions

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP — LGTM — no reliability concerns.

The server-suite setup exposes uv to both the current shell and subsequent steps without changing dependencies or shard commands. The regression test exercises the extracted setup with installer doubles and checks both PATH contracts.

Static review completed for both changed files and directly affected CI paths. Tests and hosted workflows were not executed during this review.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@ko3n1g
ko3n1g added this pull request to the merge queue Oct 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40045718707

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Oct 24, 2026
@ko3n1g
ko3n1g added this pull request to the merge queue Oct 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40087667728

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 24, 2026
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot svcnemo-autobot changed the title fix(ci): AUT-2624 expose uv on PATH for scheduled server tests fix(ci): AUT-2624 repair uv PATH and colored R2E summaries Oct 24, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the area:agent Agent harnesses and Responses API agent behavior label Oct 24, 2026
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 1857c77

Copy link
Copy Markdown
Contributor Author

Exact-head validation update for 1857c77e3d3701abf35712f5adc33ff58540df15:

  • Run 40091750495, attempt 1: core tests and all eight server shards passed.
  • AnySWE: 67 passed, including both plain/colored parser cases and both hidden-test-link cases. This closes the coverage gap from the local offline dependency failure.
  • Required checks Nemo_CICD_Test, Lint check, and DCO are all SUCCESS.
  • The overall workflow remains failure from optional Kilo conformance and its reporting checks: terminal model-error handling failed, and another scenario hit disk quota. These paths are unchanged by this PR. No retry requested.

The updated head awaits fresh review; the prior review was dismissed after the push.

@ko3n1g
ko3n1g added this pull request to the merge queue Oct 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40095930391

Merged via the queue into NVIDIA-NeMo:main with commit 13f5dbb Oct 24, 2026
54 of 57 checks passed
AdamRajfer pushed a commit to AdamRajfer/Gym that referenced this pull request Oct 25, 2026
Backport the R2E parser and regression tests from NVIDIA-NeMo#5026
(upstream commit 1857c77).
Strip SGR sequences before interpreting statuses and test IDs. Cover plain
and colored output, including missing and nonpassing required tests.

The CI setup changes from that PR are not included.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
thibautbar pushed a commit to thibautbar/Gym that referenced this pull request Oct 27, 2026
…IA-NeMo#4841)

* feat(eval): count rollouts that ran out of time in the aggregate metrics

A rollout that runs out of time scores as a failure of the evaluated
model, whatever made it slow. Until now a run did not say how many
rollouts that was or how long they ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(eval): write the time-limit section atomically

Review finding 4339034971 on NVIDIA-NeMo#4841. The section writer rewrote the aggregate metrics file in place, so a write that failed partway (a full disk) left a truncated document behind, and the next read of the metrics failed the command. The new document is now written to a temporary file in the same directory and replaces the metrics file only once it is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* feat(eval): report timed-out rollouts in the run health summary

Timeouts are a run-health diagnostic, so the section moves from the
aggregate metrics to quality_summary.json under run.stats.timeouts,
next to the model-call error counts. gym eval run, gym eval aggregate
and gym eval health-check all build it in run_health_checks from the
saved rows and the failures sidecar's last attempts, so a recheck gives
the same numbers. The aggregate metrics are scores only again.

One typed function, classify_time_limit, decides whether a row ran out
of time, which limit it hit and on what evidence:

- typed: a TimeoutEvent (event_id, limit, enforcer, evidence,
  configured_s, elapsed_s, fired_at) in a timeouts list on the row or
  on an invocation, tool call or model call it carries. It wins over
  every other signal and is counted once per event_id.
- legacy: the labels, flags, termination reason, episode-deadline
  message and timed-out model call that rows already carry.
- unclassified: a failure with no signal whose record shows a measured
  interval at or past its configured limit. Timing alone never
  confirms a firing, so these are counted apart.

A failure that none of these match is counted as failed_without_signal,
never as not timed out.

The section also carries the time split of the timed-out rollouts by
policy model, other models, tools, time after the last call and the
rest, with policy serving speed and calls per rollout. Health-check
workers return only a slim view of each failed or timed-out row, so
the summary does not hold payloads. The section's numbers go to the
configured exporters as flat health/timeouts/... keys after the health
checks run; nulls are not sent.

The failure-row capture flag is not kept: it needed a capture-store
read for every bare failure, which collection deliberately does not
join, and the other signals still cover those rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* docs(eval): document run.stats.timeouts on the rollout health page

Moves the time-limit documentation from the aggregate metrics page to
the rollout health page: the evidence order, every field of the
section, the time split and the exporter keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* refactor(eval): rework the timeouts section of the run health summary

The section moves to nemo_gym/health/timeouts.py and is built from one
typed record per rollout instead of untyped row views.

- A rollout counts as timed out on a TimeoutEvent record, or on a legacy
  signal that a named Gym component sets when its own deadline fires. A
  test checks that each component still sets its signal. Elapsed time
  alone no longer counts, so the unclassified evidence is gone, and so
  are the signals no component sets (transport_timeout and a timed-out
  model call).
- The time split reports flat shares. The unrecorded share is now part
  of untracked.
- gym eval run now exports the section too. Every place that computes
  run health exports through one helper, which logs a failure and never
  stops the run.
- The failures sidecars are read once and shared by the run accounting
  and the section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* docs(eval): describe the typed timeout record by what it is

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): keep the timeouts section safe on malformed rows and overlap

A value of the wrong type in one rollout's record, such as a trajectory
whose invocations are not a list, made the timeouts reader raise, and
the whole health report failed with no summary. The reader now reads
such a value as not measured, so the record checks report the row and
the summary is written.

Timeout records are read from the row's own `timeouts` list only. The
trajectory's operation records have no field for them, and collection
drops a trajectory that adds one.

`untracked` was one minus the summed shares, which counted twice a
stretch in which two kinds of call overlap, such as a tool that calls
the policy model. It is now the share in which no measured call ran.

The docs also say that a recheck of a merged directory does not see the
shards' failure rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* docs(health): a typed timeout event goes on the row

The reader and the docs page take timeout events only from the row's
`timeouts` list. Two docstrings still said the event is attached to the
operation it bounds, where collection would drop it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): time a failed rollout from its capture file

Collection joins model-call captures only into a rollout that produced a
result, so a rollout that failed without one, such as an episode cut by
its deadline, had no calls on its row and an empty time split, although
the model server captured its calls under the same rollout id.

The health check now attaches a failed row's capture file, read with the
collector's own join and key, to a copy of the row before reading its
timing. A row that carries its own capture is left alone, so no call
counts twice; without a capture file the time stays unmeasured.
Collection, `gym eval run` and `--no-serve` pass their capture dir, and
`gym eval health-check` takes `--model-call-capture-dir`. A missing or
invalid capture setting means no captures, never a failed check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): one timeout classifier and a safe, linear capture join

The run timing table and run.stats.timeouts used different timeout
evidence, so BrowseComp and Codex timeouts counted in only one of them;
both now call classify_timeout.

The capture join scanned the capture dir once per failed rollout,
created the dir when it was missing, and stopped the report on an
unusable dir or a malformed row. Partial captures, inverted or
unplaceable spans and non-positive elapsed times read as measured, a
resumed tracking run kept stale limit counts, and a recheck without
--model-call-capture-dir lost the split silently. Each now lists the dir
once, leaves the timing unmeasured, sends 0, or warns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): read gap codes safely, and count unread transcripts and empty outputs

Review of the time split found three gaps:
- A gap whose code was not a string, such as a list, raised while the
  section read the row, which stopped the whole health check. A gap code
  that is not a string now names no gap.
- An agent that could not read its own transcript reports an
  `agent_artifact_unavailable` gap, and its empty tool list read as a
  measured zero. That gap now leaves the tool time unknown, as
  `tool_timing_unavailable` does.
- A call that produced no output was left out of the serving speed,
  although it is a measured rate of zero. It now counts at zero. A call
  without a token count or a positive latency stays unmeasured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): leave tool time unknown when an agent observed no tools, and scope the capture rule

Review found more agents whose empty tool list was read as a measured
zero:
- An agent that captured no observations reports
  `observation_capture_failed`; PinchBench, OpenClaw, Stirrup and
  Kilocode all emit it. That gap now leaves the tool time unknown, as
  tool_timing_unavailable does.
- OpenClaw and OpenCode report `agent_transcript_unavailable` when they
  have no transcript. That gap now leaves the tool time unknown when the
  tool list is empty. Codex can read its tools from its native
  transcript when only its CLI events are missing, and tools read that
  way stay measured.

The rollout-health page said health never reads raw captures. That
holds for the integrity checks. The timeout statistics read the
captured calls, and moving the capture files can change their time
split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): leave the time split unknown when part of its evidence was not read

The timeouts section measured from evidence it had read only in part,
in two ways:

- A model-call capture whose call list holds an entry that is not an
  object was split from its readable calls. It is now left out of the
  split, like a capture read only in part.
- A Codex transcript record or an OpenCode part that could not be read
  may have been a tool call, yet the tool list read without it counted
  as the whole tool history. Those parsers now also report
  tool_timing_unavailable, so tool and untracked time stay unknown. An
  unreadable Codex native event, such as a token usage record, is never
  a tool call and does not report it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): count a malformed timeouts value, and leave an unidentified OpenCode tool untimed

- An OpenCode tool part without a call id has no tool observation, yet
  the tool list read without it counted as the whole tool history. Both
  OpenCode parsers now also report tool_timing_unavailable for it.
- A timeouts value that is false, 0, "" or {} read as no events. Only
  an absent or null value does now; any other value that is not a list
  counts as one invalid event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): leave tool time unknown when OpenCode drops a tool for its message

Both OpenCode parsers skip a tool part whose owning message cannot be
read, has no role, or belongs to no session. The tool was known but
left out, yet the tool list read without it counted as the whole tool
history. Those parts now also report tool_timing_unavailable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* test(health): move the agent-parser timeout tests to the agents' suites

Three timeout-section tests ran an agent's own transcript parser
(Codex, OpenClaw) from tests/unit_tests. Importing those agent modules
there counted their code in the core unit job's coverage, whose tests
for them run in the server suites. The tests now live in each agent's
test_observability.py, with the same assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): count a Terminus 2 rollout stopped by its run budget as timed out

Terminus 2 scores a rollout whose run budget (the task's agent timeout) expires: the episode is verified, and the stop is
recorded only on the response it returns, as `response.metadata.terminus2_outcome == "timeout"`. No timeout signal
listed that field, so the timeouts section counted such rollouts as not timed out.

Add it as a legacy timeout signal, cited from the Terminus 2 agent like the other producers. Tests: a synthetic Terminus
row that ended on its budget counts under that signal, and a completed one does not; the parametrized checks cover the
new entry's match and its citation.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(anyswe): parse colored R2E pytest summaries

Backport the R2E parser and regression tests from NVIDIA-NeMo#5026
(upstream commit 1857c77).
Strip SGR sequences before interpreting statuses and test IDs. Cover plain
and colored output, including missing and nonpassing required tests.

The CI setup changes from that PR are not included.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): preserve configured policy identity in timeout timing

Record the configured policy server on landed and failed rows so offline capture joins keep policy calls separate from judges and simulators. Preserve legacy fallback and captured identities.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

* fix(health): classify episode timeouts consistently in live timing

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

---------

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>

This branch was previously deployed

1 inactive deployment
public — 1857c77e Deployed Oct 24, 2026 by copy-pr-bot[bot] via release / finalize / notify #7080
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior area:ci CI workflows, test routing, and repository automation ci CI, automation, test queue, or workflow infrastructure work complexity:low Localized change in one scope with a small, straightforward review surface needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants