Repository navigation
feat(combibench): add CombiBench Lean 4 combinatorics benchmark - #3611
rimashahbazyan wants to merge 60 commits into
Conversation
Review (Surveyor merge-request workflow, two independent lenses + coordinator verification)Reviewed at head Verified by execution
Blocking
Should fix
Minor
|
One-stage Fine-Eval verifier (statement-tamper check, axiom/sorry rejection, answer check by rfl/norm_num) compiling through a Kimina Lean Server pinned to Lean/Mathlib v4.24.0, with without-solution and with-solution benchmark assets. Preparation defaults to the upstream repository's Lean files at c67e4213: all 100 statements compile on v4.24.0, while 6 of the Hugging Face dataset's do not. The gold answer is elaborated at the abbrev's declared type; upstream's unascribed check makes 5 of 45 fill-in problems unsolvable (measured, both reports committed, upstream form kept behind a flag). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Without them the per-family metrics keyed on tag never appeared in a real run; found by a ten-task smoke through the full Gym path. The metrics test now builds its aggregate from actual verify() output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
Five synthetic tasks through the full Gym path (openai_model policy, Kimina Lean 4.24 server); runtime evidence only, not a score. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
The CI scan (detect-secrets-hook over the PR diff) flags the 40-character upstream commit hashes in the manifests' adopted_from.ref and in the README's Kimina checkout line as HexHighEntropyString. They are public git revisions, not credentials. Reproduced and cleared locally with the same command CI runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Rima Shahbazyan <rshahbazyan@nvidia.com>
…nned image Review items from the two-lens review on #3611. Attribution and observability: - A Kimina *header* timeout means a cold REPL could not load `import Mathlib` in time. The model cannot cause or avoid it, so it is now `header_timeout` and a harness fault rather than a `timeout` charged to the model. - Every response carries `lean_version`, probed once per process. A server built for another toolchain otherwise scores every row `proof_failed` with nothing in the rollouts saying why. - `dataset_source` was declared by `prepare.py` but not by the request model, so it was dropped from every rollout record. Cost and concurrency: - Bound in-flight Lean calls by `max_concurrent_lean_requests`, defaulting to the server's own `LEAN_SERVER_MAX_REPLS`. - Do not retry a compile that exhausted the client timeout: the shared client's 3 tries cost three REPL jobs and 3x the wall clock for the same verdict. Reproducibility: - Add `kimina_image/`: the Lean server image is now built from a committed Dockerfile with the Lean tarball checksummed and Mathlib, the REPL and the Kimina server each pinned by commit, rather than from a clone of upstream. The build gate starts a REPL and requires it to load Mathlib -- `/health` answers `ok` long before a REPL can start, so without it a broken image is first visible as every rollout failing. - `LEAN_SERVER_MAX_REPL_MEM` documented at 12G, not 8G: it becomes `RLIMIT_AS` on each REPL and a REPL holding Mathlib exceeds 8G. - Add `scripts/upstream_agreement.py`, which scores collected rollouts a second time with upstream's own `evaluation/` code against the same Lean server and reports paired per-item agreement. - `harness_validation.py` records `dataset_source`, `dataset_revision` and `split`, so a report can be tied to the corpus it measured. Tests and docs: - `test_surrogate_escape_in_request` sent a Python repr, so FastAPI returned 422 before `verify()` ran and the assertion accepted it. It now sends real JSON, asserts the verdict, and fails if `_clean_text` is bypassed. - Drop `test_config_type_is_the_base_one_gym_expects`, which only restated a class header. - `canonical_split` is the dataset type Gym's `--split` selects (`benchmark`), not the dataset name. - Correct the trailing-whitespace figures, which describe the Hugging Face copy (12/100 spaces-only) and not the default GitHub-prepared rows (1/100). - Document `code_too_long`, `lean_error`, the `open ... in` statement-check blind spot, and why the existing sandbox path was not reused. - Remove an absolute author path from the committed example data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…probe Stacks CombiBench on lean_proof so a rollout dump reads the same across the Lean benchmarks. - Status names come from `lean_proof.status` wherever the concept is shared: `completed`, `empty_generation`, `banned_tokens`, `statement_modified`, `compile_error`, `has_sorry`, `timeout`, `sandbox_error`. CombiBench keeps `format_error`, `code_too_long`, `lean_error`, `header_timeout` and `bad_task` alongside them -- upstream's Fine-Eval distinguishes outcomes a whole-file benchmark has no equivalent for, which is the extension that module's docstring describes. - Add `STATUS_HAS_SORRY` to `lean_proof.status`. A build that still declares a `sorry` exits zero and was reported as `compile_error`, but nothing was rejected -- the proof is absent. Telling the two apart separates a model writing wrong proofs from one writing none. Reward is unaffected: only `completed` earns it, so leancat's numbers do not move. - The version probe now uses `lean_proof.toolchain`'s `TOOLCHAIN_PROBE` and `parse_lean_version` rather than a second copy of the same regex. The REPL answers in structured messages, so they are joined into the stdout shape the parser expects. `proof_utils` and `lean_sandbox` are deliberately not reused; the README says why, at the level of the individual rule. Both would change scores relative to upstream CombiBench's published numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
… job Review item 9: the Hugging Face reports predated the committed script -- they lacked the `answer_check_ascription` key it always writes, and their `input` named a GitHub path while measuring the Hugging Face corpus. Nothing in the artifact tied a number to a source. All seven reports are now produced by one run of the committed script against one Lean server, and each records `dataset_source`, `dataset_revision`, `split` and `answer_check_ascription`. The prepared rows go to a scratch path, which is what the GitHub/Hugging Face mix-up came from: both sources write to the same default output path. Every previously published figure reproduces on the pinned v4.24.0 image: GitHub 100/100 statements, Hugging Face 94/100 with the same six failures, 45/45 gold answers ascribed against 40/45 unascribed with the same five problems, all four negative controls at zero, and 5/5 synthetic solutions. The unexplained `brualdi_ch7_15` Hugging Face gold-answer failure does not reproduce under either ascription setting, so it was an artifact of that earlier run rather than a property of the data. The Hugging Face gold-answer column is now reported too: its four ascribed failures are all statements that do not compile in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Review item 1 asked for a run of upstream's own evaluation/ harness on the same rollouts, since paired per-item agreement is the strongest evidence the re-derived verifier matches upstream. Both settings agree on every rollout: | benchmark | rollouts | gym | upstream | agreement | |---------------------------|----------|-----|----------|-----------| | combibench | 1600 | 34 | 34 | 1600/1600 | | combibench_with_solution | 1600 | 31 | 31 | 1600/1600 | `gym_only` and `upstream_only` are both empty: the same rollouts pass under both graders, not merely the same number of them. The two documented departures cost nothing on this run -- they can only produce gym-accepts-upstream-rejects cases, and there were none. Getting there required one transport-level fix to upstream's client, which is documented at its site and is not a scoring change. Upstream reads `res["error"]` by subscript; Kimina omits that key when there was no error, so the read raises KeyError, upstream's blanket `except Exception` swallows it, and every compiling proof is reported as a failed one. Unpatched, upstream scores 0/1600 here. This is not a version-pinning mistake that could have been avoided. Upstream pins the kimina client at 0.1.1 (2025-07-24), three months before Lean v4.24.0 was released (2025-10-14), while its own statements are now on v4.24.0. No server is both contemporary with that client pin and able to compile the current statements, so upstream's published harness cannot score the corpus in its own repository without this fix. Also refreshes the committed example rollouts through `gym eval reverify`, so their verdicts name the statuses the code now emits and carry `dataset_source` and `lean_version`. No model was re-run and every problem maps exactly under the rename with an unchanged reward. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Review item 1: the PR reported only gemini-2.5-pro and o3-mini, with o3-mini at 0 in every setting, and no open-source model. Goedel-Prover-V2-32B, 16 rollouts per problem, both settings end to end on this code path: 8/100 pass@16 on combibench and 9/100 with solution, harness faults 0.125% and 0%. A generalist of comparable size scores at or near zero here, which cannot tell a working verifier from a broken one; a prover that closes goals can. What makes it evidence about the harness rather than a number: - Both settings solve almost the same problems, which is what a stable verifier looks like across independent runs. - Difficulty orders as published: everything solved is brualdi or hackmath, and no olympiad problem is solved in 576 attempts. - All 3,200 rollouts agree with upstream's own harness, item by item. - Of 81 sampled statement_modified verdicts, none would pass under either whitespace setting, so that departure is not masking a rejection; the one read by hand dropped the structure and instance the statement defines. Caveats recorded with the numbers: 3% of rollouts hit the token budget, which accounts for every format_error, and the paper publishes no decoding parameters, so this is a comparison against its protocol rather than a reproduction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
56d3b75 to
4bf6273
Compare
A Lean-server outage, a header timeout and an unscorable row all scored 0.0 unmasked, so an infrastructure failure was averaged into mean/reward and pass@k as if the model had failed the problem: 2 of 1600 rollouts in the committed profiling run already did this. They now set mask_sample and a failure_kind (provider_unavailable for the two Lean-server faults, the namespaced combibench:bad_task for a malformed row, since the shared vocabulary has no name for one). harness_failure is unchanged; the masked rollouts are still in the rollout file and are reported under coverage/. compute_metrics returned only the per-family subset keys, so the pooled pass@k the README's tables report was never emitted. It now merges compute_pass_majority_metrics first, as leancat does, and layers the per-family keys on top. Also drops FakeLeanClient's unused raise_exc parameter: the real client converts every exception into a transport failure itself, so nothing exercised it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
The two committed upstream_agreement_*.json carry "rows": {} and a "rows_note",
a shape no code path produced: the script wrote the full per-row map and had no
such key, so the documented command did not reproduce the committed files. The
script now writes only the disagreements plus that note, with --full-rows for
the complete map. The artifacts are unchanged apart from the note's wording,
which now matches what the script emits; no summary counter moved.
Keying is also made to fail closed. _ng_rollout_index defaults to 0 and the
per-row map was built with dict(), so a rollouts file written without that field
collapsed 1600 comparisons into 100 and still reported a complete run -- and a
collision in --rescore-with mis-paired verdicts silently. Both paths now go
through index_by_key, which raises naming the colliding keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Review iteration 4 of up to 5 — at
|
…ong citation
The charge was justified with proof_utils.py::parse_error_message mapping the
payload to FinalMessage(severity="error"), i.e. "Kimina's own client reads it as
a Lean error on the snippet". That is the wrong function. Kimina's canonical
per-snippet classifier is ReplResponse.analyze() in
client/kimina_client/models.py, which tests `"message" in self.response` before
is_error and returns SnippetStatus.repl_error ("at REPL level") -- the opposite
reading. parse_error_message is a renderer whose "error" severity is only the
fallback when the text does not match ^([^:]+):\n.
The verdict stands and is unchanged; the reason is now the honest one. Kimina
would call this repl_error, and it is charged anyway because a failure the model
could have caused is never masked and masking is the direction that deletes a
model-caused failure from the denominator. Also states plainly why `message` is
split from `error`/`stderr` when Kimina groups all three as "not a command
response": `message` is the only one of the three the pinned server emits, and
it is emitted while the REPL is running the model's snippet.
Rewritten at all five sites: the README departure and the attribution table,
PAYLOAD_MODEL_ERROR_KEYS, _payload_failure's docstring, and the two test
docstrings. No parse_error_message citation remains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…each it The row claimed `import Foo` is the model-attributable route. It is not: every raise that reaches `manager.prep`'s handler is the REPL not started, a broken pipe, an unwritable stdin, or a JSON decode error. An unresolvable module is none of those — the REPL answers in parseable JSON, so nothing raises and the failure lands in the body diagnostics as `compile_error`. The verdict and the `header_is_harness_supplied` gate are unchanged; only the stated trigger narrows, to a REPL that died or wrote non-JSON while loading the header. Narrow is not empty: the REPL runs under an RLIMIT_AS cap and `native_decide` is allowed by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
`compat_client` filled the absent `error` key. Upstream reads two in one
expression: `res["error"] is None and not is_error(res["response"], ...)`.
`BackwardResponse.response` is `NotRequired` and `/verify` is declared
`response_model_exclude_none=True`, so `response` is dropped whenever a result
carries an `error` — every server-side timeout — and the second subscript
raised in its turn.
Those rows were therefore scored by upstream's blanket `except Exception`
rather than by its `is_error`. Both keys are now filled, so upstream's own
logic runs on every row. The verdict for a timeout row is unchanged: `error`
is set, so `res["error"] is None` is already False, and `is_error({})` is False
— the same "not a success" by the intended route rather than by an exception.
The committed agreement figures were measured with one key filled. They do not
move, and the README now says so where they are reported rather than leaving a
reader to assume upstream's scoring logic ran on all 3,200 rows.
Also adds `lean_server_max_wait` to the shipped YAML. It was the one knob with
a Python default and no YAML entry, and it is the one the README tells an
operator to change for a server not at `LEAN_SERVER_MAX_WAIT=60` — it feeds the
HTTP budget, and too small a budget converts a model's timeout into a masked
harness fault. The YAML-vs-class-defaults test now covers it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…wo claims Three corrections to the attribution documentation, none of which changes a verdict. A fifth shape reaches the model with no 5xx at all: a header that fails at the Lean level without raising and without timing out. Kimina guards that with `if prep and prep.error`, but `send_timeout` never sets `error`, so the guard is dead code at this pin and the body runs against the broken environment anyway; the cascading errors arrive as ordinary diagnostics and are scored `compile_error`. Upstream behaves identically and the direction is conservative, but the table read as exhaustive and omitted it. The `model_header_error` row still described `import Foo` as its trigger in one place the previous commit missed. An unresolvable module does not reach that path. "Nothing else under `server/` answers 5xx" was loosely scoped: `server_old/healthcheck.py` raises 503. The conclusion holds — `create_app` does not mount it, and 503 is retried as saturation — so the wording now says "mounted routers" rather than the claim being dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Iteration 4 fixes —
|
| iteration | behavioural defects | character |
|---|---|---|
| 1 | 3 | masking reachable by the model |
| 2 | 2 blocking | over-corrected, both directions |
| 3 | 2 | two shapes still masked |
| 4 | 0 | right rules, wrong reasons |
Four iterations, and the fourth found no behavioural defect from either reviewer. The remaining unverifiable surface is not reviewable by inspection at all: every headline figure needs a live Lean server and the uncommitted rollouts. A reviewer's suggestion is worth more than another round — pasting the two ~20-line summary blocks from harness_validation.py and upstream_agreement.py into this PR would make those numbers auditable without breaking the data/ convention.
Gym allows one benchmark dataset per workload, which had forced the "with solution" setting into a second benchmark. Rows already carry `split`, so one benchmark now prepares either setting (`split=test` by default, `test_with_solution`, or `both`) into the single file its config reads, and the server reports pass@k per `split` when a file holds both settings rather than pooling two settings the paper never pools. Files of one setting keep their unprefixed metric keys. Prepared rows are byte-identical to the previous two benchmarks' files. Drops benchmarks/combibench_with_solution and documents how the settings differ. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Levon Grigoryan <lgrigoryan@nvidia.com>
…s author One prepared file now holds one setting, so the per-split metric handling and its tests are removed; the benchmark keeps a single `split` argument (`test` by default, or `test_with_solution`). Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Levon Grigoryan <lgrigoryan@nvidia.com>
Cuts the server README from 727 to about 370 lines and trims the Kimina image README, dropping per-line source citations, repeated explanations and the measurement narrative while keeping the scoring rules, the failure-charging tables, the departures from upstream and the validation numbers. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Levon Grigoryan <lgrigoryan@nvidia.com>
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
What does this PR do?
Adds CombiBench (paper, repository, MIT): 100 combinatorics problems formalized in Lean 4 — 45 fill-in-the-blank (the model supplies an answer in
abbrev <name>_solutionand proves a theorem about it) and 55 proof-only.resources_servers/combibench— verifier reproducing upstream's one-stage Fine-Eval: extraction, comment stripping, theaxiom/local_instanceban, the statement-tamper check, the answer equality check, and a Lean 4 compile through a Kimina Lean Server.benchmarks/combibenchandbenchmarks/combibench_with_solution— the paper's two settings.resources_servers/combibench/kimina_image/— a pinned Dockerfile for the Lean server (Lean tarball checksummed; Mathlib, the REPL and the Kimina server each pinned by commit).Stacked on #3400 (
lgrigoryan/lean-cat), which is the base of this PR. The status vocabulary and the toolchain probe come from that PR'slean_proof; see "Compatibility" for the one shared file this change edits.No tracking issue — this is a new benchmark port, not a fix to existing behaviour.
Reward profiling
Goedel-Prover-V2-32B (open weights), 16 rollouts per problem, temperature 1.0, 39,000-token budget, both settings end to end on this code path:
combibenchcombibench_with_solutionSolved by family —
imoandmath_competitionsare zero in both, as published:combibenchcombibench_with_solutionbrualdihackmathimomath_competitionsA generalist of comparable size scores at or near zero on CombiBench, which cannot distinguish a working verifier from a broken one; a Lean-native prover that closes goals can. Both settings solve almost the same problem set, and difficulty orders as published — which is the evidence that the verifier is measuring something stable rather than noise.
Caveats stated with the numbers: 3% of rollouts hit the token budget (accounting for every
format_error), and the paper publishes no decoding parameters, so this is a comparison against its protocol, not a reproduction of its numbers. The rollout files themselves are not committed; the tables above and the "3%" figure were counted from them ad hoc, which the README says at the site.Agreement with upstream's harness
Upstream's own
evaluation/code was run over the same rollouts — the paired per-item comparison, not a matching headline:combibenchcombibench_with_solutionThe same rollouts pass under both, not merely the same number. (
combibench's twosandbox_errorrollouts are excluded: this verifier reached no verdict on them, and counting a non-verdict as a match would flatter the result.)Five deliberate departures from upstream
Documented at their sites and in the server README; the first two are configurable.
answer_check_ascription: falserestores upstream's form. Both measurements come fromharness_validation.pyon the GitHubtestsplit, with and without--no-ascription.normalize_trailing_whitespace: falserestores byte-exactness).sorriesentry is treated ashas_sorry— stricter than upstream, whoseis_errornever readssorries./verifypayload is refused rather than rewarded. Kimina can answer with anErrorobject ({"message": ...}) and no top-levelerror; read naively that is indistinguishable from a clean compile and scores 1.0. Guardingmessageis an addition over upstream; guardingerror/stderrrestores parity with itsis_error.code_too_long— a 200,000-character wire-safety bound upstream does not have. Unreachable on the pinned corpus (longest statement 3,054; zero occurrences in 3,200 rollouts).Harness validation
Model-free, all against Mathlib v4.24.0 through this server's code path. The reports are not committed — a resources server's
data/holds only the example trio, as it does for every other server in the repo — so the numbers are stated here and in the server README, andscripts/harness_validation.pyregenerates them. Re-deriving them needs a running Lean server on Mathlib v4.24.0.sorrys compilesNegative controls through
verify()over all 100 rows — empty output, statement echoed withsorry,axiom cheat : False, and the main theorem's goal replaced byTrue— all score 0 with the intended status.Compatibility and benchmark impact
resources_servers/lean_proof/status.pygainsSTATUS_HAS_SORRY, anddetermine_proof_statusreturns it where it returnedSTATUS_COMPILE_ERROR. Its only other consumer isleancat, whose rewards and masking do not move (reward keys onSTATUS_COMPLETED, masking onSTATUS_SANDBOX_ERROR). What does change is theproof_statusstring leancat emits for a sorry-carrying build, so anything keyed on leancat'scompile_errorfor those rows should be updated.leancat's README and test are updated in this PR.sandbox_error,header_timeout,bad_task) setmask_sample, so they leave the quality set instead of being averaged in as model failures. A consequence worth knowing:mean/harness_failureis therefore always 0 in aggregate —coverage/masked_rolloutsis the live signal, and this server promotes it into its key metrics.verified: false, as a new server should be.Validation
pytest resources_servers/combibench/tests— 131 passed.lean_proof,leancatandtests/unit_tests/test_task_data.py— 556 passed.ruff check/ruff format --checkclean;pre-commitclean; DCO sign-off on every commit.scripts/upstream_agreement.pyreproduces the comparison. Neither the rollouts nor the reports are committed, per the repository's convention fordata/.Review history
Three rounds of independent review (Surveyor correctness + contracts lenses, and this repository's own
pr-reviewrubric), each round adjudicated against upstream's and Kimina's real sources before any fix was applied. Round 2 found a false-positive reward path (departure 4 above), which no amount of agreement evidence would have caught — every one of the 3,200 rollouts is a shape where both harnesses agree anyway. Round 3 returned no blocking findings.