Skip to content

Re-judgment binds no judge failure saved as judge_unparseable, so the run stops incomplete #6341

Open

Description

@wprazuch

What happens

A managed run whose judge failures were saved as judge_unparseable (with judge_failed kept in _ng_failure_legacy_class) never gets them re-judged. The in-place re-judgment at the end of collection reports them due, binds none, and the run stops incomplete with no eligible retry.

Where

  • due_judge_failures selects with _is_judge_failure, which uses matches_failure_class and so matches either class field: rollout_reverification.py#L553, #L1491.
  • bind_judge_failures compares only _ng_failure_class == "judge_failed", in the spent-budget check and in the returned binding: #L1432, #L1447.
  • is_rejudgment_of and is_rejudgment_failure_of make the same single-field comparison, so even a bound failure could not be settled: check_rejudgment raises "the re-judged result does not settle the saved judge failure". rollout_records.py#L506, #L530.

Repro

In tests/unit_tests/eval/test_rejudge_saved_judge_failures.py, build a run with _run(tmp_path, judge_failures=1, evidence={"_ng_failure_class": "judge_unparseable", "_ng_failure_legacy_class": "judge_failed"}). Then due_judge_failures(output) returns ['0-0'] while bind_judge_failures(output)["failures"] is {}.

Seen on

A real run of 1,353 rollouts with 4 such judge failures stopped at 1,349/1,353: re-judgment selected 0, recovered 0, and the job exited 76.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions