Repository navigation
feat(formal-conjectures): add the Formal Conjectures benchmark on lean_proof - #3767
Draft
lilithgrigoryan wants to merge 4 commits into
Draft
lilithgrigoryan wants to merge 4 commits into
lilithgrigoryan wants to merge 4 commits into
Conversation
…n_proof Ports #3404 onto the `lean_proof` library that #3400 (LeanCat) introduces, rather than carrying its own copy of the Lean plumbing. #3404 was written against `main`, so it shipped a second `lean_proof` package (its own `proof_utils`/`sandbox_client`/`toolchain`) and patched `math_formal_lean` to share it. #3400 has since landed that library properly, backed by `nemo_gym.sandbox` instead of an out-of-band NeMo-Skills HTTP sandbox. This is the same benchmark rebuilt on top of it: the duplicated modules and the `math_formal_lean` edits are gone, and `math_formal_lean` stays byte-identical to `main`. Formal Conjectures is 1,527 tasks from 323 upstream files (test 1290, API 191, textbook 26, research solved 20) against Lean/Mathlib v4.33.1. Each row is a whole Lean file with one proof removed; the model returns the file with that hole filled. What FC needs that LeanCat does not, and why the shared library grew three seams rather than being forked again: - An FC file legitimately keeps other `sorry`s -- it pairs a provable lemma with the open conjecture it sanity-checks, and the task keeps the declarations above the target because proofs lean on them. So `determine_proof_status` takes `sorry_is_error=False`, and `find_banned_declarations` takes a token list plus `declarations_only` (`sorry`/`admit` are legal here; `axiom`/`unsafe` are banned as declarations, not as words). - `#print axioms <target>` stands in for the dropped `sorry` rule. The server appends the probe itself and scores 1.0 only when the target is free of `sorryAx`, which also catches a proof leaning on a sorry'd lemma earlier in the file. - `check_target_statement_preserved` is the target-signature variant of the preservation check; splitting the reference on `sorry` the way LeanCat does throws out honest answers when the hole is not textually unique. - `has_unterminated_block_comment` moves into the shared module: a mangled `-/` swallows the file, and without this it is misreported as a weakened statement rather than malformed output. Also adds the v4.33.1 pins to `lean_image/versions.py`, a `check_sandbox.py` that is both the pre-flight gate and the sweep defining `verified_tasks.json` (#3404 referred to that sweep but never shipped it), and a real README in place of #3404's stub. Every existing `lean_proof`/`leancat` test still passes unchanged. Validation: 108 tests pass (`gym env test --resources-server formal_conjectures`, plus the lean_proof and leancat suites); `pre-commit run --files ...` clean; `gym env resolve` resolves the benchmark config; `prepare.py` re-derives the rows at the pin and reproduces the committed `data/example.jsonl` byte-identically. No model rollouts -- that needs a v4.33.1 Lean image, which is built by `lean_image/build.sh v4.33.1` and is not available in this environment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…3.1 gate Built the Mathlib v4.33.1 image and ran the validation sweep against it. The gate failed 12 of the first 25 tasks, which turned out to be three real bugs inherited from #3404 rather than a bad image. 1. `#print axioms` has two output forms, and only one was parsed. 'thm' depends on axioms: [propext, Classical.choice] 'thm' does not depend on any axioms <-- fully constructive proof The second was read as "no axiom line found", i.e. a missing declaration, so the STRONGEST possible result -- a proof citing no axioms at all -- scored 0.0. Confirmed against Lean 4.33.1 in the built image before fixing. The parser is now also anchored to the target's name and takes the last match. The submission is a whole file and may contain `#print axioms` calls of its own; without the anchor a reply that prints the axioms of some proved Mathlib lemma next to its own sorry'd theorem was scored as a proof. 2. The declaration-name regex truncated Unicode identifiers. `[A-Za-z0-9_.'!?]` stops at the first subscript, so `not_lt₂` became `not_lt`. That name goes into `#print axioms <target>`, so the probe named a declaration that does not exist, Lean errored, and the task could never be passed by any model. Measured at the pin: 49 of 5,248 declarations. 3. Task ids were `<path>::<short name>`, which is not unique -- truncation made it worse, but the collision is real regardless: `sIncreasingrTuples.lean` has five `not_lt₂*` lemmas. `prepare.py` built a dict on the id, so several distinct tasks silently collapsed into whichever came last, which need not be the one the sweep validated. Ids are now namespace-qualified, and both `prepare.py` and `check_sandbox.py` raise on a duplicate rather than letting one task displace another. Because the ids changed form and the old list was built with the truncating regex, `verified_tasks.json` was regenerated from scratch by the full sweep against the real image: 1,526 of 1,788 candidates verified (was 1,527 on ambiguous ids). The 262 rejects are dominated by files whose syntax needs FormalConjecturesUtil after all, and by proofs that exceed the kernel's budget -- both would otherwise be unanswerable tasks scoring a flat 0. Validation: - Image built (lean-image-build-fc.sub): gym-lean:v4.33.1, sha256-pinned Lean tarball, Mathlib 0df444a, `lake build Mathlib` offline, in-image `import Mathlib` smoke test. - Sandbox reports Lean/Mathlib 4.33.1 as expected. - Full sweep over all 1,788 candidates, 10m38s, exit 0. - 110 tests pass; pre-commit clean; prepare.py regenerates example.jsonl. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
…arisons Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
Closed
5 tasks
…a plain-text list Matches the supported_instance_ids.txt convention used by the SWE servers. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
This was referenced Oct 22, 2026
yaoyu-33
marked this pull request as draft
November 18, 2026 01:56
Contributor
|
We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft. If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Reimplements #3404 (Formal Conjectures) on top of #3400's
lean_prooflibrary, and targetslgrigoryan/lean-catso it stacks on LeanCat rather than duplicating it.#3404 was written against
main, before that library existed. To get shared Lean plumbing it shipped asecond
lean_proofpackage (its ownproof_utils/sandbox_client/toolchain) and patchedmath_formal_leanto use it. #3400 has since landed the real one, backed bynemo_gym.sandboxrather than anout-of-band NeMo-Skills HTTP sandbox. This is the same benchmark rebuilt on it:
math_formal_leanedits —math_formal_leanisbyte-identical to
main, preserving that property of feat(leancat): add LeanCat formal category-theory benchmark #3400;LeanSandbox(one sandbox per process,lake env lean, toolchain probe), so it runson OpenSandbox / enroot / docker unchanged with nothing started out of band;
lean_proof/lean_image, with v4.33.1 pins added toversions.py(Mathlib commit0df444a, resolved from the tag; Lean tarball sha256 verified by download; the tag's ownlean-toolchainconfirms the pair).
Formal Conjectures is a Google DeepMind library of formalized mathematics. Most of it is open — the headline
conjectures carry a
sorrynobody can fill, which is useless as a benchmark. What is scored is the other half:theorems that ship with real Lean proofs.
extract.pystrips a proof off; the model gets the whole file back withone hole. 1,526 tasks from 323 upstream files (of 1,788 extractable candidates), Lean/Mathlib v4.33.1:
categorytestAPItextbookresearch solvedWhere FC parts company with LeanCat, and why the library grew seams instead of being forked
An FC file legitimately keeps other
sorrys — it pairs a provable lemma with the open conjecture itsanity-checks, and the task keeps the declarations above the target because the proofs lean on them (dropping them
broke 14 of a 32-task sample). Three consequences, each an additive change to
lean_proof:determine_proof_status(..., sorry_is_error=False). LeanCat is right to call Lean's "declaration usessorry"warning an error; applying that here would score every real FC task 0.
find_banned_declarations(code, DECLARED_SHORTCUT_TOKENS, declarations_only=True).sorry/admitare legal inthese files, and
axiom/unsafeare banned only as declarations the model added.check_target_statement_preserved— the target-signature variant of the preservation check. Splitting thereference on
sorrythe way LeanCat does assumes the hole is textually unique; here it is not, and it throws outhonest answers (measured on FC: 23 of 100).
Replacing the dropped
sorryrule,#print axioms <target>decides the reward. The server appends the probeitself — never trusted to the model — and pays out only when the target is free of
sorryAx. That also catchesindirection a textual scan cannot: a proof leaning on a sorry'd lemma earlier in the file. During dataset
validation it rejected upstream "proofs" that were not proofs.
has_unterminated_block_commentalso moves into the shared module. A mangled-/swallows the rest of the file,so every later check sees an empty one; without this the server reports "statement modified" — blaming the model
for weakening a theorem when it actually just produced malformed output (48 times in a 12k-rollout run).
unprovedis the one status this server adds, and it sits next to the shared vocabulary rather than redefining it.lean_proof/README.mdgains a table contrasting the two task shapes so the next Lean benchmark picks the right column.Known limitation, stated rather than hidden
The statement check is containment, so a submission that appends to the conclusion (
True→True ∨ False)passes it. Tightening it risks the false rejections the rule was tuned to avoid over a 12k-rollout run, so it is
left as is, asserted by a named test, and documented in the README with
#check/#print axiomsas the strictertools.
statement_preservedis reported on every response, so a stricter criterion can be applied to storedrollouts without rerunning.
Beyond a port
check_sandbox.py— add formal conjestures #3404'sprepare.pypointed at a validation sweep "see the README", but neither thesweep nor a README existed. This is both: the pre-flight gate (
--limit 25) and the sweep that definesverified_tasks.json(--all --write-verified), sharingtarget_is_provedwith the server so the two rule alike.Data links: ?stub.extract.py: it claimed tasks contain exactly one declaration, which the codestopped doing — it keeps the prefix, which is the whole reason the verifier is scoped with
#print axioms.Validation
I built the Mathlib v4.33.1 image and ran the sweep, which is how the three bugs below were found.
Image (
lean-image-build-fc.sub, 5m28s): Lean tarball sha256-pinned, Mathlib0df444a,lake build Mathliboffline, in-image
import Mathlibsmoke test. The sandbox reports Lean/Mathlib 4.33.1, as expected.Full sweep over all 1,788 candidates (10m38s, exit 0) → 1,526 verified,
verified_tasks.jsonregenerated.Re-gate of the committed list: 1526/1526 pass, exit 0.
The 262 rejects are dominated by files whose syntax needs
FormalConjecturesUtilafter all, and by proofs thatexceed the kernel's budget — both would otherwise be unanswerable tasks scoring a flat 0 for every model.
Also: 110 tests pass (
gym env test --resources-server formal_conjectures, pluslean_proofandleancat, allpre-existing ones unchanged);
pre-commitclean;gym env resolveresolves the benchmark config;prepare.pyre-derives the rows from the pin.
Model rollouts: still N/A. The gate proves the environment and the dataset; it does not measure a model. Happy
to run a baseline if useful — say which model and k.
Three bugs the gate caught, all inherited from #3404
The first gate run failed 12 of 25 tasks. Not a bad image — three real bugs:
#print axiomshas two output forms and only one was parsed. A fully constructive proof reports'thm' does not depend on any axiomsrather than an axiom list. That was read as "no axiom line found", i.e. amissing declaration, so the strongest possible result scored 0.0. Verified against Lean 4.33.1 in the built
image before changing anything.
The parser is now anchored to the target's name and takes the last match, which also closes a gaming vector:
the submission is a whole file, so it can print the axioms of any proved Mathlib lemma next to its own sorry'd
theorem, and an unanchored scan accepted the decoy.
[A-Za-z0-9_.'!?]stops at the first subscript,so
not_lt₂becamenot_lt. That name goes into#print axioms <target>, so the probe named a declarationthat does not exist, Lean errored, and the task could never be passed by any model. 49 of 5,248 declarations
at the pin.
<path>::<short name>collides —sIncreasingrTuples.leanhas fivenot_lt₂*lemmas.
prepare.pybuilt a dict on the id, so distinct tasks silently collapsed into whichever came last,which need not be the one the sweep validated. Ids are now namespace-qualified, and
prepare.py/check_sandbox.pyboth raise on a duplicate.Because ids changed form and the old list was built with the truncating regex,
verified_tasks.jsonwasregenerated from scratch rather than migrated.
No tracking issue — a new benchmark port, not a fix to existing behaviour.
Review order
resources_servers/lean_proof/first (three additive, opt-in seams), thenformal_conjectures/app.py.Checklist
pre-commit run --all-files).git commit -s).🤖 Generated with Claude Code