You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Re-judgment binds no judge failure saved as judge_unparseable, so the run stops incomplete #6341
A managed run whose judge failures were saved as judge_unparseable (with judge_failed kept in _ng_failure_legacy_class) never gets them re-judged. The in-place re-judgment at the end of collection reports them due, binds none, and the run stops incomplete with no eligible retry.
Where
due_judge_failures selects with _is_judge_failure, which uses matches_failure_class and so matches either class field: rollout_reverification.py#L553, #L1491.
bind_judge_failures compares only _ng_failure_class == "judge_failed", in the spent-budget check and in the returned binding: #L1432, #L1447.
is_rejudgment_of and is_rejudgment_failure_of make the same single-field comparison, so even a bound failure could not be settled: check_rejudgment raises "the re-judged result does not settle the saved judge failure". rollout_records.py#L506, #L530.
Repro
In tests/unit_tests/eval/test_rejudge_saved_judge_failures.py, build a run with _run(tmp_path, judge_failures=1, evidence={"_ng_failure_class": "judge_unparseable", "_ng_failure_legacy_class": "judge_failed"}). Then due_judge_failures(output) returns ['0-0'] while bind_judge_failures(output)["failures"] is {}.
Seen on
A real run of 1,353 rollouts with 4 such judge failures stopped at 1,349/1,353: re-judgment selected 0, recovered 0, and the job exited 76.
Activity