Skip to content

fix(eval): preserve reply observations during reverify - #5238

Merged
AdamRajfer merged 4 commits into
arajfer/failure-reportingfrom
arajfer/reverify-reply-observations
Oct 30, 2026
Merged

AdamRajfer merged 4 commits into
arajfer/failure-reportingfrom
arajfer/reverify-reply-observations

Conversation

@AdamRajfer

@AdamRajfer AdamRajfer commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

#5072 forwards the failures a Gym server observed while serving a request, including attempts a retry recovered from, and Collection saves them beside the result (FailureReportRow). gym eval reverify called /verify the same way but read only the reply body, so the failures its reply forwarded were lost:

  • A recovered row (for example, the judge answered 500 and its retry succeeded) saved none of them.
  • A row that failed again kept only the failure in the body, not the judge attempts that led to it.

Reverify now merges the reply's report into the row it writes, as Collection does for a dispatch reply. Only the observations of this re-verification are added. The original run's sidecar rows are not read or changed, a recovered row stays a success (no failure class, no failure_id), a failed row keeps its own failure_id, and events keep their IDs and are counted once.

Validation, CPU only, through rollout collection, the legacy agent relay, simple_agent, an openai_model judge and a resources server under judge_failsafe, then gym eval reverify --judge-failed-only:

  • Judge answers 500 then 200 during recovery: the /verify reply forwards 1 event. Before: the recovered row saved 0. After: 1, with the same row reward and no failure class or failure_id.
  • Judge never answers during recovery: the reply forwards 3 attempt events. Before: the row that failed again kept 1 (2 with the fix(eval): make opted-in policy timeouts final scored zeros #5220 change). After: 4, with its own failure_id and class unchanged, the same shape Collection saves for that failure.
  • The original sidecar is byte-identical after recovery in every case, and no event on a recovered row comes from it.
  • New tests fail without the source change and pass with it; reverification, collection, failure report, judge, model and relay suites pass.

Stacked on arajfer/failure-reporting at 3bedbbfc (#5072, including #5245 and the latest-outcome resume correction). The base merge preserves those changes unchanged; its only conflict was adjacent imports.

Current composed-head validation: the reverification, collection, failure-report and judge suites pass, including terminal-zero accounting, interrupted resume and the realistic body-only judge-event fixture. The fixture detects a deliberately broken overwrite merge. Repository hooks pass. This is local composition coverage, not a new live benchmark run.

Not covered: a later --resume of the same reverify output is not re-run here; it reads failure classes only.

Earlier native validation on #5072 at c69a5ff plus this fix: 802 reverify/collection/failure-report tests pass; all-file pre-commit passes. The three new tests fail without the source change. Production loopback checks confirm both recovered and failed-again behavior above. Composed separately with the #5221 identity fix: 216 reverify tests and 114 collection controls pass, identity stamps stay out of /verify requests and remain on output rows. This PR leaves that independent identity fix to #5221. No scoring or retry-policy change; no separate issue is needed for this gap in #5072.

A focused judge-only smoke, composed with #5221, also verified recovery observations and bounded-exhaustion diagnostics on saved responses while using a live verifier. The transient errors were deliberately injected; forwarded verifier responses were healthy. Grouping was preserved, original response files stayed unchanged and no policy generation occurred. The refused-connection path was outside this smoke. This smoke predates the current base merge and is not model-backed validation of the merged head.

Validation update: the branch now inherits the target's typed Harbor oracle assertions, helper/usage validation coverage tests and deterministic SWE-Together timeout fixture. This integration changes four test files and preserves the production implementation and this PR's reply-observation tests. Targeted core, Harbor oracle and SWE-Together tests pass, along with all-file pre-commit checks. Model rollouts are N/A for this test-only integration. The repository's coverage threshold is unchanged. The current signed head passes core, all eight server shards, the server aggregate and the required combined check in CI run 41002439307. Session-routing is skipped for these paths. The optional Kilocode conformance and its publish check fail separately; required CI is green.

reverify read only the /verify body, so the failures the reply forwarded
were lost: a judge attempt a retry recovered from on a recovered row, and
the judge attempts behind a row that failed again. Merge the reply's report
into the written row as collection does for a dispatch reply. A recovered
row stays a success, a failed row keeps its own failure_id, and the original
run's sidecar is neither read nor changed.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@AdamRajfer AdamRajfer added bug Something isn't working area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters labels Oct 28, 2026
@AdamRajfer AdamRajfer self-assigned this Oct 28, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@AdamRajfer
AdamRajfer marked this pull request as ready for review October 29, 2026 12:06
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 29, 2026 12:06
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 29, 2026

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

Reviewed the reply-report merge, transport contracts, downstream persistence/resume paths, and regression tests. The change preserves body-owned failure identity and recovered-row success semantics, deduplicates forwarded events, and retains omitted-report diagnostics without changing reward or retry policy.

Static review only; tests were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. The change preserves rejudging diagnostics while reusing the saved policy answers. I have one nonblocking test suggestion below.

"_ng_failure_id": judge.event_id,
"_ng_failure_events": {judge.event_id: judge.model_dump(mode="json")},
}
reply = FailureReport(events={e.event_id: e for e in (*attempts, judge)})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this fixture keep the judge event only in the body, with cause_id pointing to the final attempt, while the header contains the attempt events? That matches the judge_failsafe reply shape. Currently, replacing the merge with row.update(FailureReportRow.fields(report)) still passes all three new tests, despite losing the judge event on a realistic failed reply. Please also assert that the final cause link survives. This is a nonblocking test improvement; the implementation preserves it correctly.

@AdamRajfer AdamRajfer Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 5bc60fd0. The judge event appears only in the body, together with its final cause; the header contains the attempt events. The test now asserts that cause_id still names the final attempt. Replacing the merge with a fields update makes it fail by dropping the judge event. The owning reverification suite and all repository hooks pass. This follow-up changes only the test.

The current base is now merged in d8ad267e; the realistic fixture is unchanged. The four relevant suites and full hooks pass on the composed head. GitHub reports the conflict cleared; full CI is running.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>

@yuhezhang-ai yuhezhang-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks for addressing the test suggestion.

Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 30, 2026 04:21
@AdamRajfer
AdamRajfer merged commit a8f44a4 into arajfer/failure-reporting Oct 30, 2026
54 of 57 checks passed
@AdamRajfer
AdamRajfer deleted the arajfer/reverify-reply-observations branch October 30, 2026 11:18

This branch was previously deployed

1 inactive deployment
public — 772e1ce4 Deployed Oct 30, 2026 by copy-pr-bot[bot] via release / finalize / notify #8564
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants