Repository navigation
fix(eval): preserve reply observations during reverify - #5238
Conversation
reverify read only the /verify body, so the failures the reply forwarded were lost: a judge attempt a retry recovered from on a recovered row, and the judge attempts behind a row that failed again. Merge the reply's report into the written row as collection does for a dispatch reply. A recovered row stays a success, a failed row keeps its own failure_id, and the original run's sidecar is neither read nor changed. Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
LGTM — no concerns found.
Reviewed the reply-report merge, transport contracts, downstream persistence/resume paths, and regression tests. The change preserves body-owned failure identity and recovered-row success semantics, deduplicates forwarded events, and retains omitted-report diagnostics without changing reward or retry policy.
Static review only; tests were not executed.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Looks good overall. The change preserves rejudging diagnostics while reusing the saved policy answers. I have one nonblocking test suggestion below.
| "_ng_failure_id": judge.event_id, | ||
| "_ng_failure_events": {judge.event_id: judge.model_dump(mode="json")}, | ||
| } | ||
| reply = FailureReport(events={e.event_id: e for e in (*attempts, judge)}) |
There was a problem hiding this comment.
Could this fixture keep the judge event only in the body, with cause_id pointing to the final attempt, while the header contains the attempt events? That matches the judge_failsafe reply shape. Currently, replacing the merge with row.update(FailureReportRow.fields(report)) still passes all three new tests, despite losing the judge event on a realistic failed reply. Please also assert that the final cause link survives. This is a nonblocking test improvement; the implementation preserves it correctly.
There was a problem hiding this comment.
Updated in 5bc60fd0. The judge event appears only in the body, together with its final cause; the header contains the attempt events. The test now asserts that cause_id still names the final attempt. Replacing the merge with a fields update makes it fail by dropping the judge event. The owning reverification suite and all repository hooks pass. This follow-up changes only the test.
The current base is now merged in d8ad267e; the realistic fixture is unchanged. The four relevant suites and full hooks pass on the composed head. GitHub reports the conflict cleared; full CI is running.
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
yuhezhang-ai
left a comment
There was a problem hiding this comment.
Looks good to me. Thanks for addressing the test suggestion.
Signed-off-by: Adam Rajfer <arajfer@nvidia.com>
#5072 forwards the failures a Gym server observed while serving a request, including attempts a retry recovered from, and Collection saves them beside the result (
FailureReportRow).gym eval reverifycalled/verifythe same way but read only the reply body, so the failures its reply forwarded were lost:Reverify now merges the reply's report into the row it writes, as Collection does for a dispatch reply. Only the observations of this re-verification are added. The original run's sidecar rows are not read or changed, a recovered row stays a success (no failure class, no
failure_id), a failed row keeps its ownfailure_id, and events keep their IDs and are counted once.Validation, CPU only, through rollout collection, the legacy agent relay, simple_agent, an openai_model judge and a resources server under
judge_failsafe, thengym eval reverify --judge-failed-only:/verifyreply forwards 1 event. Before: the recovered row saved 0. After: 1, with the same row reward and no failure class orfailure_id.failure_idand class unchanged, the same shape Collection saves for that failure.Stacked on
arajfer/failure-reportingat 3bedbbfc (#5072, including #5245 and the latest-outcome resume correction). The base merge preserves those changes unchanged; its only conflict was adjacent imports.Current composed-head validation: the reverification, collection, failure-report and judge suites pass, including terminal-zero accounting, interrupted resume and the realistic body-only judge-event fixture. The fixture detects a deliberately broken overwrite merge. Repository hooks pass. This is local composition coverage, not a new live benchmark run.
Not covered: a later
--resumeof the same reverify output is not re-run here; it reads failure classes only.Earlier native validation on #5072 at c69a5ff plus this fix: 802 reverify/collection/failure-report tests pass; all-file pre-commit passes. The three new tests fail without the source change. Production loopback checks confirm both recovered and failed-again behavior above. Composed separately with the #5221 identity fix: 216 reverify tests and 114 collection controls pass, identity stamps stay out of
/verifyrequests and remain on output rows. This PR leaves that independent identity fix to #5221. No scoring or retry-policy change; no separate issue is needed for this gap in #5072.A focused judge-only smoke, composed with #5221, also verified recovery observations and bounded-exhaustion diagnostics on saved responses while using a live verifier. The transient errors were deliberately injected; forwarded verifier responses were healthy. Grouping was preserved, original response files stayed unchanged and no policy generation occurred. The refused-connection path was outside this smoke. This smoke predates the current base merge and is not model-backed validation of the merged head.
Validation update: the branch now inherits the target's typed Harbor oracle assertions, helper/usage validation coverage tests and deterministic SWE-Together timeout fixture. This integration changes four test files and preserves the production implementation and this PR's reply-observation tests. Targeted core, Harbor oracle and SWE-Together tests pass, along with all-file pre-commit checks. Model rollouts are N/A for this test-only integration. The repository's coverage threshold is unchanged. The current signed head passes core, all eight server shards, the server aggregate and the required combined check in CI run 41002439307. Session-routing is skipped for these paths. The optional Kilocode conformance and its publish check fail separately; required CI is green.