Skip to content

feat(combibench): add CombiBench Lean 4 combinatorics benchmark - #3611

Draft
rimashahbazyan wants to merge 60 commits into
lgrigoryan/lean-catfrom
rshahbazyan/add-combibench-benchmark
Draft

rimashahbazyan wants to merge 60 commits into
lgrigoryan/lean-catfrom
rshahbazyan/add-combibench-benchmark

Conversation

@rimashahbazyan

@rimashahbazyan rimashahbazyan commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds CombiBench (paper, repository, MIT): 100 combinatorics problems formalized in Lean 4 — 45 fill-in-the-blank (the model supplies an answer in abbrev <name>_solution and proves a theorem about it) and 55 proof-only.

  • resources_servers/combibench — verifier reproducing upstream's one-stage Fine-Eval: extraction, comment stripping, the axiom/local_instance ban, the statement-tamper check, the answer equality check, and a Lean 4 compile through a Kimina Lean Server.
  • benchmarks/combibench and benchmarks/combibench_with_solution — the paper's two settings.
  • resources_servers/combibench/kimina_image/ — a pinned Dockerfile for the Lean server (Lean tarball checksummed; Mathlib, the REPL and the Kimina server each pinned by commit).

Stacked on #3400 (lgrigoryan/lean-cat), which is the base of this PR. The status vocabulary and the toolchain probe come from that PR's lean_proof; see "Compatibility" for the one shared file this change edits.

No tracking issue — this is a new benchmark port, not a fix to existing behaviour.

Reward profiling

Goedel-Prover-V2-32B (open weights), 16 rollouts per problem, temperature 1.0, 39,000-token budget, both settings end to end on this code path:

Benchmark pass@16 pass@1 (avg of 16) Harness faults
combibench 8 / 100 2.12% 0.125% (2 / 1600)
combibench_with_solution 9 / 100 1.94% 0%

Solved by family — imo and math_competitions are zero in both, as published:

Family combibench combibench_with_solution
brualdi 7 / 42 8 / 42
hackmath 1 / 10 1 / 10
imo 0 / 36 0 / 36
math_competitions 0 / 12 0 / 12

A generalist of comparable size scores at or near zero on CombiBench, which cannot distinguish a working verifier from a broken one; a Lean-native prover that closes goals can. Both settings solve almost the same problem set, and difficulty orders as published — which is the evidence that the verifier is measuring something stable rather than noise.

Caveats stated with the numbers: 3% of rollouts hit the token budget (accounting for every format_error), and the paper publishes no decoding parameters, so this is a comparison against its protocol, not a reproduction of its numbers. The rollout files themselves are not committed; the tables above and the "3%" figure were counted from them ad hoc, which the README says at the site.

Agreement with upstream's harness

Upstream's own evaluation/ code was run over the same rollouts — the paired per-item comparison, not a matching headline:

Benchmark Rollouts scored by both This verifier Upstream Agreement
combibench 1598 34 34 1598 / 1598
combibench_with_solution 1600 31 31 1600 / 1600

The same rollouts pass under both, not merely the same number. (combibench's two sandbox_error rollouts are excluded: this verifier reached no verdict on them, and counting a non-verdict as a match would flatter the result.)

Upstream's harness needs one transport-level fix to run at all. It reads res["error"] by subscript; Kimina omits that key when there was no error, so the read raises KeyError, upstream's blanket except Exception turns it into "proof invalid", and every compiling proof is reported as failed — 0/1600 unpatched. The fix is in the comparison script only, documented at its site, and changes no scoring rule. It is not a pin that could have been chosen better: upstream pins the kimina client at 0.1.1 (2025-07-24), three months before Lean v4.24.0 was released (2025-10-14), while its own statements are now on v4.24.0. No server is both contemporary with that client and able to compile the current statements.

Five deliberate departures from upstream

Documented at their sites and in the server README; the first two are configurable.

  1. The gold answer is elaborated at the abbreviation's declared type. Upstream's unascribed form elaborates for 40 of 45 fill-in-the-blank problems; five can never score under it even with the exact gold answer. Ascribed: 45 of 45. answer_check_ascription: false restores upstream's form. Both measurements come from harness_validation.py on the GitHub test split, with and without --no-ascription.
  2. Trailing whitespace is ignored in the statement check (normalize_trailing_whitespace: false restores byte-exactness).
  3. A REPL sorries entry is treated as has_sorry — stricter than upstream, whose is_error never reads sorries.
  4. An error-carrying /verify payload is refused rather than rewarded. Kimina can answer with an Error object ({"message": ...}) and no top-level error; read naively that is indistinguishable from a clean compile and scores 1.0. Guarding message is an addition over upstream; guarding error/stderr restores parity with its is_error.
  5. code_too_long — a 200,000-character wire-safety bound upstream does not have. Unreachable on the pinned corpus (longest statement 3,054; zero occurrences in 3,200 rollouts).

Harness validation

Model-free, all against Mathlib v4.24.0 through this server's code path. The reports are not committed — a resources server's data/ holds only the example trio, as it does for every other server in the repo — so the numbers are stated here and in the server README, and scripts/harness_validation.py regenerates them. Re-deriving them needs a running Lean server on Mathlib v4.24.0.

Check GitHub source (default) Hugging Face source
Statement with its sorrys compiles 100 / 100 94 / 100
"With solution" statement compiles 100 / 100 94 / 100
Published answer + ascribed check elaborates 45 / 45 41 / 45
Same with upstream's unascribed check 40 / 45 37 / 45

Negative controls through verify() over all 100 rows — empty output, statement echoed with sorry, axiom cheat : False, and the main theorem's goal replaced by True — all score 0 with the intended status.

Compatibility and benchmark impact

  • One shared file is edited. resources_servers/lean_proof/status.py gains STATUS_HAS_SORRY, and determine_proof_status returns it where it returned STATUS_COMPILE_ERROR. Its only other consumer is leancat, whose rewards and masking do not move (reward keys on STATUS_COMPLETED, masking on STATUS_SANDBOX_ERROR). What does change is the proof_status string leancat emits for a sorry-carrying build, so anything keyed on leancat's compile_error for those rows should be updated. leancat's README and test are updated in this PR.
  • Harness faults (sandbox_error, header_timeout, bad_task) set mask_sample, so they leave the quality set instead of being averaged in as model failures. A consequence worth knowing: mean/harness_failure is therefore always 0 in aggregate — coverage/masked_rollouts is the live signal, and this server promotes it into its key metrics.
  • New benchmark and server otherwise; nothing to migrate. verified: false, as a new server should be.

Validation

  • pytest resources_servers/combibench/tests — 131 passed.
  • Full stack incl. lean_proof, leancat and tests/unit_tests/test_task_data.py — 556 passed.
  • ruff check / ruff format --check clean; pre-commit clean; DCO sign-off on every commit.
  • The Kimina v4.24.0 image builds from the committed Dockerfile, non-root: the build's own gate starts a REPL as that user, loads Mathlib and asserts the version.
  • 3,200 rollouts collected and re-scored with upstream's harness; scripts/upstream_agreement.py reproduces the comparison. Neither the rollouts nor the reports are committed, per the repository's convention for data/.

Review history

Three rounds of independent review (Surveyor correctness + contracts lenses, and this repository's own pr-review rubric), each round adjudicated against upstream's and Kimina's real sources before any fix was applied. Round 2 found a false-positive reward path (departure 4 above), which no amount of agreement evidence would have caught — every one of the 3,200 rollouts is a shape where both harnesses agree anyway. Round 3 returned no blocking findings.

@copy-pr-bot

copy-pr-bot Bot commented Sep 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 22, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Sep 23, 2026
@lilithgrigoryan

Copy link
Copy Markdown
Contributor

Review (Surveyor merge-request workflow, two independent lenses + coordinator verification)

Reviewed at head 56d3b750a, merge-base 6bbc05ed9. Overall: the verifier is a faithful reproduction of upstream and the validation work is thorough. A few items below should be resolved before approval; the rest are suggestions.

Verified by execution

  • pytest resources_servers/combibench/tests: 80 passed.
  • prepare.py for test and test_with_solution from a cold cache: 100 rows each.
  • Compared fine_eval.py against upstream evaluation/util.py, one_stage_verify.py, client/lean_client.py and config/template.json5 at c67e4213: regexes, header, forbidden substrings, paragraph logic (incl. the \ntheorem replace without re-split), answer-check snippet, sorry/error semantics and prompt strings all match; the two documented departures are the only differences.
  • Compared lean_client.py against Kimina Lean Server at fb2393de: /verify request/response shape matches.
  • Recomputed the summaries of all harness_validation_*.json reports from their rows: consistent with the README/PR tables.

Blocking

  1. Reward-profiling requirements not met (CONTRIBUTING.md "Fixed evaluation benchmarks must follow the reward-profiling requirements"; fern/.../adding-a-benchmark.mdx). The guide asks for at least one open-source model in a meaningful score range and for reproducing the published numbers with the original harness first. The PR reports only gemini-2.5-pro and o3-mini (o3-mini at 0 in every setting), and no run of upstream's evaluation/ harness on the same rollouts (paired per-item agreement would be the strongest evidence the verifier matches upstream).
  2. test_surrogate_escape_in_request cannot detect its failure (resources_servers/combibench/tests/test_app.py:246). The body is str(body).encode(...), a Python repr, not JSON, so FastAPI returns 422 before verify() runs, and assert response.status_code in (200, 422) accepts that. Send json.dumps(body) with the \udcff escape embedded, assert 200, and assert on the returned status; confirm it fails when _clean_text is bypassed.
  3. test_config_type_is_the_base_one_gym_expects (tests/test_app.py:312) only restates the class header; every other test already constructs the server through _make_server. Suggest removing it.

Should fix

  1. Kimina header timeouts are charged to the model (fine_eval.py::classify_lean_result). Kimina returns "Lean REPL header command timed out in N seconds" when a cold REPL's import Mathlib exceeds the timeout (server/routers/check.py in the pinned commit). The classifier matches "timed out" and emits timeout with harness_failure=0. The model cannot cause this; matching "header" and routing it to a harness fault would fix attribution.
  2. Whitespace departure is mis-motivated for the default source (server README, "Two deliberate departures"). "Thirteen of the hundred statements contain lines consisting only of spaces" holds for the HF copy (12/100 spaces-only, 13/100 any trailing whitespace at 882ba08b). The prepared GitHub-source rows have 1/100. Worth stating which copy the number refers to.
  3. Retries and concurrency (lean_client.py:71). nemo_gym.server_utils.request retries any exception, including the client ClientTimeout, up to 3 times (_request_with_retries), so a pathological proof can cost ~3×90 s and three REPL jobs before it surfaces as lean_server_error. There is also no asyncio.Semaphore around the Lean call (AGENTS.md asks for one on external calls); rollout fan-out lands directly on LEAN_SERVER_MAX_REPLS=8. Consider _max_connection_retries=1 and a configurable semaphore.
  4. No toolchain probe or per-row Lean/Mathlib version (app.py:252). A Kimina image built for another Lean version scores every row proof_failed with nothing in the rollouts indicating the mismatch. Kimina exposes no version endpoint, but a one-time #eval Lean.versionString compile at startup, logged and echoed, would do.
  5. dataset_source is dropped (app.py:114). CombibenchRunRequest declares dataset_revision but not dataset_source, so the field written by prepare.py is discarded from every rollout record (confirmed in example_rollouts.jsonl).
  6. HF validation reports predate the committed script (data/harness_validation_hf_test*.json). They lack the answer_check_ascription key the script always writes, and their input names the GitHub file path, so nothing in the artifact ties them to --source hf or to an ascription mode. brualdi_ch7_15 appears as an HF-only gold-answer failure and is not explained in the README. Regenerating them (and recording dataset_source in the report) would make the numbers checkable.

Minor

  1. data/example.jsonl carries prompt_config_used: /home/rshahbazyan/...; no other server's example data has this key.
  2. canonical_split is combibench / combibench_with_solution; all 75 other benchmark manifests use benchmark (the dataset type --split selects).
  3. Server README scoring list omits code_too_long and lean_error, and does not say lean_error is charged to the model.
  4. open ... in directly above theorem makes the whole paragraph start with a header prefix and be skipped by the statement check (fine_eval.py:117). Upstream has the identical blind spot and the pinned corpus has zero such paragraphs, so a line under "Known blind spots" is enough.
  5. Reuse question, for the record: the PR gives one sentence on why the existing math_formal_lean sandbox path (Lean v4.12.0) was not rebuilt at v4.24.0 with its helpers reused. A short comparison (rebuilt NeMo-Skills sandbox vs Kimina, maintenance status of the pinned Kimina commit, why a commit pin rather than a release) would help future maintainers.

rimashahbazyan and others added 9 commits September 28, 2026 09:10
One-stage Fine-Eval verifier (statement-tamper check, axiom/sorry rejection,
answer check by rfl/norm_num) compiling through a Kimina Lean Server pinned to
Lean/Mathlib v4.24.0, with without-solution and with-solution benchmark assets.

Preparation defaults to the upstream repository's Lean files at c67e4213: all
100 statements compile on v4.24.0, while 6 of the Hugging Face dataset's do
not. The gold answer is elaborated at the abbrev's declared type; upstream's
unascribed check makes 5 of 45 fill-in problems unsolvable (measured, both
reports committed, upstream form kept behind a flag).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Without them the per-family metrics keyed on tag never appeared in a real
run; found by a ten-task smoke through the full Gym path. The metrics test now
builds its aggregate from actual verify() output.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Five synthetic tasks through the full Gym path (openai_model policy, Kimina
Lean 4.24 server); runtime evidence only, not a score.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
The CI scan (detect-secrets-hook over the PR diff) flags the 40-character
upstream commit hashes in the manifests' adopted_from.ref and in the README's
Kimina checkout line as HexHighEntropyString. They are public git revisions,
not credentials. Reproduced and cleared locally with the same command CI runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
…nned image

Review items from the two-lens review on #3611.

Attribution and observability:
- A Kimina *header* timeout means a cold REPL could not load `import Mathlib`
  in time. The model cannot cause or avoid it, so it is now `header_timeout`
  and a harness fault rather than a `timeout` charged to the model.
- Every response carries `lean_version`, probed once per process. A server
  built for another toolchain otherwise scores every row `proof_failed` with
  nothing in the rollouts saying why.
- `dataset_source` was declared by `prepare.py` but not by the request model,
  so it was dropped from every rollout record.

Cost and concurrency:
- Bound in-flight Lean calls by `max_concurrent_lean_requests`, defaulting to
  the server's own `LEAN_SERVER_MAX_REPLS`.
- Do not retry a compile that exhausted the client timeout: the shared client's
  3 tries cost three REPL jobs and 3x the wall clock for the same verdict.

Reproducibility:
- Add `kimina_image/`: the Lean server image is now built from a committed
  Dockerfile with the Lean tarball checksummed and Mathlib, the REPL and the
  Kimina server each pinned by commit, rather than from a clone of upstream.
  The build gate starts a REPL and requires it to load Mathlib -- `/health`
  answers `ok` long before a REPL can start, so without it a broken image is
  first visible as every rollout failing.
- `LEAN_SERVER_MAX_REPL_MEM` documented at 12G, not 8G: it becomes `RLIMIT_AS`
  on each REPL and a REPL holding Mathlib exceeds 8G.
- Add `scripts/upstream_agreement.py`, which scores collected rollouts a second
  time with upstream's own `evaluation/` code against the same Lean server and
  reports paired per-item agreement.
- `harness_validation.py` records `dataset_source`, `dataset_revision` and
  `split`, so a report can be tied to the corpus it measured.

Tests and docs:
- `test_surrogate_escape_in_request` sent a Python repr, so FastAPI returned
  422 before `verify()` ran and the assertion accepted it. It now sends real
  JSON, asserts the verdict, and fails if `_clean_text` is bypassed.
- Drop `test_config_type_is_the_base_one_gym_expects`, which only restated a
  class header.
- `canonical_split` is the dataset type Gym's `--split` selects (`benchmark`),
  not the dataset name.
- Correct the trailing-whitespace figures, which describe the Hugging Face copy
  (12/100 spaces-only) and not the default GitHub-prepared rows (1/100).
- Document `code_too_long`, `lean_error`, the `open ... in` statement-check
  blind spot, and why the existing sandbox path was not reused.
- Remove an absolute author path from the committed example data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…probe

Stacks CombiBench on lean_proof so a rollout dump reads the same across the
Lean benchmarks.

- Status names come from `lean_proof.status` wherever the concept is shared:
  `completed`, `empty_generation`, `banned_tokens`, `statement_modified`,
  `compile_error`, `has_sorry`, `timeout`, `sandbox_error`. CombiBench keeps
  `format_error`, `code_too_long`, `lean_error`, `header_timeout` and
  `bad_task` alongside them -- upstream's Fine-Eval distinguishes outcomes a
  whole-file benchmark has no equivalent for, which is the extension that
  module's docstring describes.
- Add `STATUS_HAS_SORRY` to `lean_proof.status`. A build that still declares a
  `sorry` exits zero and was reported as `compile_error`, but nothing was
  rejected -- the proof is absent. Telling the two apart separates a model
  writing wrong proofs from one writing none. Reward is unaffected: only
  `completed` earns it, so leancat's numbers do not move.
- The version probe now uses `lean_proof.toolchain`'s `TOOLCHAIN_PROBE` and
  `parse_lean_version` rather than a second copy of the same regex. The REPL
  answers in structured messages, so they are joined into the stdout shape the
  parser expects.

`proof_utils` and `lean_sandbox` are deliberately not reused; the README says
why, at the level of the individual rule. Both would change scores relative to
upstream CombiBench's published numbers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
… job

Review item 9: the Hugging Face reports predated the committed script -- they
lacked the `answer_check_ascription` key it always writes, and their `input`
named a GitHub path while measuring the Hugging Face corpus. Nothing in the
artifact tied a number to a source.

All seven reports are now produced by one run of the committed script against
one Lean server, and each records `dataset_source`, `dataset_revision`, `split`
and `answer_check_ascription`. The prepared rows go to a scratch path, which is
what the GitHub/Hugging Face mix-up came from: both sources write to the same
default output path.

Every previously published figure reproduces on the pinned v4.24.0 image:
GitHub 100/100 statements, Hugging Face 94/100 with the same six failures,
45/45 gold answers ascribed against 40/45 unascribed with the same five
problems, all four negative controls at zero, and 5/5 synthetic solutions.

The unexplained `brualdi_ch7_15` Hugging Face gold-answer failure does not
reproduce under either ascription setting, so it was an artifact of that
earlier run rather than a property of the data. The Hugging Face gold-answer
column is now reported too: its four ascribed failures are all statements that
do not compile in the first place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Review item 1 asked for a run of upstream's own evaluation/ harness on the same
rollouts, since paired per-item agreement is the strongest evidence the
re-derived verifier matches upstream. Both settings agree on every rollout:

| benchmark                 | rollouts | gym | upstream | agreement |
|---------------------------|----------|-----|----------|-----------|
| combibench                |     1600 |  34 |       34 | 1600/1600 |
| combibench_with_solution  |     1600 |  31 |       31 | 1600/1600 |

`gym_only` and `upstream_only` are both empty: the same rollouts pass under
both graders, not merely the same number of them. The two documented departures
cost nothing on this run -- they can only produce gym-accepts-upstream-rejects
cases, and there were none.

Getting there required one transport-level fix to upstream's client, which is
documented at its site and is not a scoring change. Upstream reads
`res["error"]` by subscript; Kimina omits that key when there was no error, so
the read raises KeyError, upstream's blanket `except Exception` swallows it,
and every compiling proof is reported as a failed one. Unpatched, upstream
scores 0/1600 here.

This is not a version-pinning mistake that could have been avoided. Upstream
pins the kimina client at 0.1.1 (2025-07-24), three months before Lean v4.24.0
was released (2025-10-14), while its own statements are now on v4.24.0. No
server is both contemporary with that client pin and able to compile the
current statements, so upstream's published harness cannot score the corpus in
its own repository without this fix.

Also refreshes the committed example rollouts through `gym eval reverify`, so
their verdicts name the statuses the code now emits and carry `dataset_source`
and `lean_version`. No model was re-run and every problem maps exactly under
the rename with an unchanged reward.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Review item 1: the PR reported only gemini-2.5-pro and o3-mini, with o3-mini at
0 in every setting, and no open-source model.

Goedel-Prover-V2-32B, 16 rollouts per problem, both settings end to end on this
code path: 8/100 pass@16 on combibench and 9/100 with solution, harness faults
0.125% and 0%. A generalist of comparable size scores at or near zero here,
which cannot tell a working verifier from a broken one; a prover that closes
goals can.

What makes it evidence about the harness rather than a number:

- Both settings solve almost the same problems, which is what a stable verifier
  looks like across independent runs.
- Difficulty orders as published: everything solved is brualdi or hackmath, and
  no olympiad problem is solved in 576 attempts.
- All 3,200 rollouts agree with upstream's own harness, item by item.
- Of 81 sampled statement_modified verdicts, none would pass under either
  whitespace setting, so that departure is not masking a rejection; the one
  read by hand dropped the structure and instance the statement defines.

Caveats recorded with the numbers: 3% of rollouts hit the token budget, which
accounts for every format_error, and the paper publishes no decoding parameters,
so this is a comparison against its protocol rather than a reproduction.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
lilithgrigoryan and others added 2 commits September 29, 2026 02:53
A Lean-server outage, a header timeout and an unscorable row all scored 0.0
unmasked, so an infrastructure failure was averaged into mean/reward and pass@k
as if the model had failed the problem: 2 of 1600 rollouts in the committed
profiling run already did this. They now set mask_sample and a failure_kind
(provider_unavailable for the two Lean-server faults, the namespaced
combibench:bad_task for a malformed row, since the shared vocabulary has no name
for one). harness_failure is unchanged; the masked rollouts are still in the
rollout file and are reported under coverage/.

compute_metrics returned only the per-family subset keys, so the pooled pass@k
the README's tables report was never emitted. It now merges
compute_pass_majority_metrics first, as leancat does, and layers the per-family
keys on top.

Also drops FakeLeanClient's unused raise_exc parameter: the real client converts
every exception into a transport failure itself, so nothing exercised it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
The two committed upstream_agreement_*.json carry "rows": {} and a "rows_note",
a shape no code path produced: the script wrote the full per-row map and had no
such key, so the documented command did not reproduce the committed files. The
script now writes only the disagreements plus that note, with --full-rows for
the complete map. The artifacts are unchanged apart from the note's wording,
which now matches what the script emits; no summary counter moved.

Keying is also made to fail closed. _ng_rollout_index defaults to 0 and the
per-row map was built with dict(), so a rollouts file written without that field
collapsed 1600 comparisons into 100 and still reported a complete run -- and a
collision in --rescore-with mis-paired verdicts silently. Both paths now go
through index_by_key, which raises naming the colliding keys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
@lilithgrigoryan

Copy link
Copy Markdown
Contributor

Review iteration 4 of up to 5 — at 6805fe692

Two fresh reviewers. This round they were asked to check the stated reason for every attribution row, not only the verdict, on the grounds that a correct rule with a false justification is how a maintainer later reasons it back to something wrong.

Verdicts: ship with care and comment. No blocking findings, and — for the first time — no behavioural defect. Both reviewers independently concluded every charge/mask decision is correct as a verdict. What they found is that three of the reasons are refuted by the pinned sources, including one I accepted last round.

The reason I got wrong, having just verified it

Iteration 3 moved the {"message": ...} payload from masked to charged, citing proof_utils.py::parse_error_message mapping it to severity="error". Both reviewers show that is the wrong function. Kimina's canonical per-snippet classifier is ReplResponse.analyze():

if "message" in self.response:
    return SnippetAnalysis(status=SnippetStatus.repl_error, time=self.time)
if is_error(self.response):
    return SnippetAnalysis(status=SnippetStatus.lean_error, time=self.time)

with lean_error = "Error in snippet, clearly identified by message of severity 'error'" and repl_error = "Error while running snippet, **at REPL level**". So Kimina classifies that shape as REPL-level — the opposite of what we cited it for. parse_error_message is a renderer, and its severity="error" is only the fallback when the text doesn't match ^([^:]+):\n.

I verified this directly. The verdict (charge it) is still defensible on the conservative rule — a failure the model could have caused is never masked — but not for the reason now written in the README, the docstring and two tests.

And the example we gave for the other new status is unreachable

model_header_error's stated trigger is "import Foo is the model's choice". A reviewer traced every way manager.prep can raise: REPL not started, broken pipe, unwritable stdin, JSON decode error. An unresolvable module is not among them — the REPL answers with parseable JSON, so nothing raises and the failure surfaces as ordinary body diagnostics (compile_error). The realistic trigger is a REPL that died or wrote non-JSON while loading the header, which is a much weaker claim to the model's ownership.

A caveat on the headline agreement evidence

We claim upstream's harness needs "one transport-level fix". It subscripts two keys:

res["error"] is None and not is_error(res["response"], ...)

BackwardResponse.response is NotRequired and /verify is response_model_exclude_none=True, so whenever a result carries an error — i.e. every server-side timeout — response is dropped and res["response"] raises KeyError even with our patch applied. Those rows are scored by upstream's blanket except Exception, not by its scoring logic.

The published 1598/1598 and 1600/1600 are unaffected: upstream reaches "not a success" either way and Gym scored those rows 0 too. But the claim that the patch "restores the reply shape upstream expects" is false for the timeout rows, and the comparison for them ran through an exception path rather than is_error. That needs saying where the figures are reported.

Also accepted

  • A fifth Kimina failure shape is missing from the table: a header that fails at the Lean level without raising and without timing out. check.py:89-90's if prep and prep.error is dead code — send_timeout never sets error — so Kimina runs the body on that REPL anyway and the cascading errors are charged as compile_error. Conservative direction, but absent from a table that reads as exhaustive.
  • Two documented rules are unreachable against the pinned server ("any other REPL error string", and error/stderr payload keys) and should be labelled defence-only, as the verdict-less guard already is.
  • lean_server_max_wait has a Python default and no YAML entry, though it is the knob the README tells operators to change and it feeds the HTTP budget.
  • The "no other 5xx under server/" claim is loosely scoped: server_old/healthcheck.py raises 503, though it is unmounted dead code.

Independently reproduced

A reviewer re-derived the corpus facts from the pinned tree: 100 rows, 45/55, families 42/10/36/12 matching the per-family denominators exactly, 47 distinct _solution abbrevs, hackmath_6 the only multi-answer row, one GitHub row with a spaces-only line, and 99 statements from which the weakened control can be built — matching the README's "99 (one statement's theorem is not the last declaration)".

Both reviewers list the same unverifiable set: every figure needing a live Lean server or the uncommitted rollouts. One suggests pasting the two ~20-line summary blocks into the PR body, which would make the headline numbers auditable without breaking the data/ convention.

Fixes follow in the next comment.

lilithgrigoryan and others added 4 commits September 29, 2026 10:29
…ong citation

The charge was justified with proof_utils.py::parse_error_message mapping the
payload to FinalMessage(severity="error"), i.e. "Kimina's own client reads it as
a Lean error on the snippet". That is the wrong function. Kimina's canonical
per-snippet classifier is ReplResponse.analyze() in
client/kimina_client/models.py, which tests `"message" in self.response` before
is_error and returns SnippetStatus.repl_error ("at REPL level") -- the opposite
reading. parse_error_message is a renderer whose "error" severity is only the
fallback when the text does not match ^([^:]+):\n.

The verdict stands and is unchanged; the reason is now the honest one. Kimina
would call this repl_error, and it is charged anyway because a failure the model
could have caused is never masked and masking is the direction that deletes a
model-caused failure from the denominator. Also states plainly why `message` is
split from `error`/`stderr` when Kimina groups all three as "not a command
response": `message` is the only one of the three the pinned server emits, and
it is emitted while the REPL is running the model's snippet.

Rewritten at all five sites: the README departure and the attribution table,
PAYLOAD_MODEL_ERROR_KEYS, _payload_failure's docstring, and the two test
docstrings. No parse_error_message citation remains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…each it

The row claimed `import Foo` is the model-attributable route. It is not: every
raise that reaches `manager.prep`'s handler is the REPL not started, a broken
pipe, an unwritable stdin, or a JSON decode error. An unresolvable module is
none of those — the REPL answers in parseable JSON, so nothing raises and the
failure lands in the body diagnostics as `compile_error`.

The verdict and the `header_is_harness_supplied` gate are unchanged; only the
stated trigger narrows, to a REPL that died or wrote non-JSON while loading the
header. Narrow is not empty: the REPL runs under an RLIMIT_AS cap and
`native_decide` is allowed by design.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
`compat_client` filled the absent `error` key. Upstream reads two in one
expression: `res["error"] is None and not is_error(res["response"], ...)`.
`BackwardResponse.response` is `NotRequired` and `/verify` is declared
`response_model_exclude_none=True`, so `response` is dropped whenever a result
carries an `error` — every server-side timeout — and the second subscript
raised in its turn.

Those rows were therefore scored by upstream's blanket `except Exception`
rather than by its `is_error`. Both keys are now filled, so upstream's own
logic runs on every row. The verdict for a timeout row is unchanged: `error`
is set, so `res["error"] is None` is already False, and `is_error({})` is False
— the same "not a success" by the intended route rather than by an exception.

The committed agreement figures were measured with one key filled. They do not
move, and the README now says so where they are reported rather than leaving a
reader to assume upstream's scoring logic ran on all 3,200 rows.

Also adds `lean_server_max_wait` to the shipped YAML. It was the one knob with
a Python default and no YAML entry, and it is the one the README tells an
operator to change for a server not at `LEAN_SERVER_MAX_WAIT=60` — it feeds the
HTTP budget, and too small a budget converts a model's timeout into a masked
harness fault. The YAML-vs-class-defaults test now covers it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…wo claims

Three corrections to the attribution documentation, none of which changes a
verdict.

A fifth shape reaches the model with no 5xx at all: a header that fails at the
Lean level without raising and without timing out. Kimina guards that with
`if prep and prep.error`, but `send_timeout` never sets `error`, so the guard is
dead code at this pin and the body runs against the broken environment anyway;
the cascading errors arrive as ordinary diagnostics and are scored
`compile_error`. Upstream behaves identically and the direction is conservative,
but the table read as exhaustive and omitted it.

The `model_header_error` row still described `import Foo` as its trigger in one
place the previous commit missed. An unresolvable module does not reach that
path.

"Nothing else under `server/` answers 5xx" was loosely scoped:
`server_old/healthcheck.py` raises 503. The conclusion holds — `create_app` does
not mount it, and 503 is retried as saturation — so the wording now says
"mounted routers" rather than the claim being dropped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
@lilithgrigoryan

Copy link
Copy Markdown
Contributor

Iteration 4 fixes — 6805fe692 → 9c6e4013c

Four commits. 591 tests pass, ruff and format clean. No charge/mask verdict changed — this round found none wrong.

(Note on how this landed: the fixer agent was terminated mid-task by an expired auth token, after finishing one item and part of a second. Its partial work was coherent and passing, so it was committed as-is and the remainder was completed directly.)

The justifications, re-argued

The {"message": ...} payload. Iteration 3 charged it, citing proof_utils.py::parse_error_message. That was the wrong function — it is a renderer, and its severity="error" is only the fallback for text that does not match ^([^:]+):\n. Kimina's canonical classifier is ReplResponse.analyze(), which maps the shape to SnippetStatus.repl_error, documented as "at REPL level" and explicitly distinct from lean_error. The verdict stands — we charge it because the conservative rule is that a failure the model could have caused is never masked — but it no longer claims Kimina calls it a Lean error, and it now says why message is split from error/stderr when Kimina groups all three.

model_header_error's trigger. The row said import Foo. Every raise that can reach manager.prep's handler is the REPL not started, a broken pipe, an unwritable stdin, or a JSON decode error — an unresolvable module is none of them, because the REPL answers in parseable JSON and the failure lands in the body diagnostics as compile_error. The trigger is now the narrower true one: a REPL that died or wrote non-JSON while loading the header, reachable from model code through the RLIMIT_AS cap and native_decide.

A real fix to the agreement methodology

compat_client patched one of the two keys upstream subscripts. res["response"] is dropped whenever a result carries an error — every server-side timeout — so those rows were scored by upstream's blanket except Exception rather than by its is_error. Both keys are now filled.

The committed figures do not move, and I verified why rather than asserting it: a timeout row has error set, so res["error"] is None is already False, and is_error({}) is False — the same "not a success", now by the intended route. The README says this where the numbers are reported, instead of leaving a reader to assume upstream's scoring logic ran on all 3,200 rows.

Also

  • A fifth failure shape now in the table: a header that fails at the Lean level without raising or timing out. Kimina guards it with if prep and prep.error, but send_timeout never sets error, so the guard is dead code and the body runs against the broken environment; the cascading errors are charged as compile_error. Conservative, and identical to upstream — but the table read as exhaustive.
  • Two rules labelled defence-only, as the verdict-less guard already was, since neither can fire against the pinned server.
  • lean_server_max_wait added to the shipped YAML. It was the only knob with a Python default and no YAML entry, and it is the one operators are told to change — it feeds the HTTP budget, and too small a budget converts a model's timeout into a masked harness fault. The YAML-vs-class-defaults test now covers it.
  • "Nothing else under server/ answers 5xx" scoped to the mounted routers (server_old raises 503 but is not mounted).

Where the loop stands

iteration behavioural defects character
1 3 masking reachable by the model
2 2 blocking over-corrected, both directions
3 2 two shapes still masked
4 0 right rules, wrong reasons

Four iterations, and the fourth found no behavioural defect from either reviewer. The remaining unverifiable surface is not reviewable by inspection at all: every headline figure needs a live Lean server and the uncommitted rollouts. A reviewer's suggestion is worth more than another round — pasting the two ~20-line summary blocks from harness_validation.py and upstream_agreement.py into this PR would make those numbers auditable without breaking the data/ convention.

Gym allows one benchmark dataset per workload, which had forced the "with
solution" setting into a second benchmark. Rows already carry `split`, so one
benchmark now prepares either setting (`split=test` by default,
`test_with_solution`, or `both`) into the single file its config reads, and
the server reports pass@k per `split` when a file holds both settings rather
than pooling two settings the paper never pools. Files of one setting keep
their unprefixed metric keys.

Prepared rows are byte-identical to the previous two benchmarks' files.
Drops benchmarks/combibench_with_solution and documents how the settings
differ.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Levon Grigoryan <lgrigoryan@nvidia.com>
…s author

One prepared file now holds one setting, so the per-split metric handling and
its tests are removed; the benchmark keeps a single `split` argument
(`test` by default, or `test_with_solution`).

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Levon Grigoryan <lgrigoryan@nvidia.com>
Cuts the server README from 727 to about 370 lines and trims the Kimina image
README, dropping per-line source citations, repeated explanations and the
measurement narrative while keeping the scoring rules, the failure-charging
tables, the departures from upstream and the validation numbers.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Levon Grigoryan <lgrigoryan@nvidia.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Nov 18, 2026

This branch was previously deployed

1 inactive deployment
public — dce91414 Deployed Sep 30, 2026 by lilithgrigoryan via release / finalize / notify #4038
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants