Skip to content

feat(formal-conjectures): add the Formal Conjectures benchmark on lean_proof - #3767

Draft
lilithgrigoryan wants to merge 4 commits into
lgrigoryan/lean-catfrom
lgrigoryan/formal-conjectures-leancat
Draft

lilithgrigoryan wants to merge 4 commits into
lgrigoryan/lean-catfrom
lgrigoryan/formal-conjectures-leancat

Conversation

@lilithgrigoryan

@lilithgrigoryan lilithgrigoryan commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Reimplements #3404 (Formal Conjectures) on top of #3400's lean_proof library, and targets
lgrigoryan/lean-cat so it stacks on LeanCat rather than duplicating it.

#3404 was written against main, before that library existed. To get shared Lean plumbing it shipped a
second lean_proof package (its own proof_utils / sandbox_client / toolchain) and patched
math_formal_lean to use it. #3400 has since landed the real one, backed by nemo_gym.sandbox rather than an
out-of-band NeMo-Skills HTTP sandbox. This is the same benchmark rebuilt on it:

  • the duplicated modules are gone, and so are add formal conjestures #3404's math_formal_lean edits — math_formal_lean is
    byte-identical to main, preserving that property of feat(leancat): add LeanCat formal category-theory benchmark #3400;
  • verification goes through LeanSandbox (one sandbox per process, lake env lean, toolchain probe), so it runs
    on OpenSandbox / enroot / docker unchanged with nothing started out of band;
  • the image comes from lean_proof/lean_image, with v4.33.1 pins added to versions.py (Mathlib commit
    0df444a, resolved from the tag; Lean tarball sha256 verified by download; the tag's own lean-toolchain
    confirms the pair).

Formal Conjectures is a Google DeepMind library of formalized mathematics. Most of it is open — the headline
conjectures carry a sorry nobody can fill, which is useless as a benchmark. What is scored is the other half:
theorems that ship with real Lean proofs. extract.py strips a proof off; the model gets the whole file back with
one hole. 1,526 tasks from 323 upstream files (of 1,788 extractable candidates), Lean/Mathlib v4.33.1:

category Tasks
test 1,285
API 196
textbook 25
research solved 20

Where FC parts company with LeanCat, and why the library grew seams instead of being forked

An FC file legitimately keeps other sorrys — it pairs a provable lemma with the open conjecture it
sanity-checks, and the task keeps the declarations above the target because the proofs lean on them (dropping them
broke 14 of a 32-task sample). Three consequences, each an additive change to lean_proof:

  • determine_proof_status(..., sorry_is_error=False). LeanCat is right to call Lean's "declaration uses sorry"
    warning an error; applying that here would score every real FC task 0.
  • find_banned_declarations(code, DECLARED_SHORTCUT_TOKENS, declarations_only=True). sorry/admit are legal in
    these files, and axiom/unsafe are banned only as declarations the model added.
  • check_target_statement_preserved — the target-signature variant of the preservation check. Splitting the
    reference on sorry the way LeanCat does assumes the hole is textually unique; here it is not, and it throws out
    honest answers (measured on FC: 23 of 100).

Replacing the dropped sorry rule, #print axioms <target> decides the reward. The server appends the probe
itself — never trusted to the model — and pays out only when the target is free of sorryAx. That also catches
indirection a textual scan cannot: a proof leaning on a sorry'd lemma earlier in the file. During dataset
validation it rejected upstream "proofs" that were not proofs.

has_unterminated_block_comment also moves into the shared module. A mangled -/ swallows the rest of the file,
so every later check sees an empty one; without this the server reports "statement modified" — blaming the model
for weakening a theorem when it actually just produced malformed output (48 times in a 12k-rollout run).

unproved is the one status this server adds, and it sits next to the shared vocabulary rather than redefining it.
lean_proof/README.md gains a table contrasting the two task shapes so the next Lean benchmark picks the right column.

Known limitation, stated rather than hidden

The statement check is containment, so a submission that appends to the conclusion (True → True ∨ False)
passes it. Tightening it risks the false rejections the rule was tuned to avoid over a 12k-rollout run, so it is
left as is, asserted by a named test, and documented in the README with #check / #print axioms as the stricter
tools. statement_preserved is reported on every response, so a stricter criterion can be applied to stored
rollouts without rerunning.

Beyond a port

  • check_sandbox.py — add formal conjestures #3404's prepare.py pointed at a validation sweep "see the README", but neither the
    sweep nor a README existed. This is both: the pre-flight gate (--limit 25) and the sweep that defines
    verified_tasks.json (--all --write-verified), sharing target_is_proved with the server so the two rule alike.
  • A real README, in place of add formal conjestures #3404's Data links: ? stub.
  • Stale docstring fixed in extract.py: it claimed tasks contain exactly one declaration, which the code
    stopped doing — it keeps the prefix, which is the whole reason the verifier is scoped with #print axioms.

Validation

I built the Mathlib v4.33.1 image and ran the sweep, which is how the three bugs below were found.

Image (lean-image-build-fc.sub, 5m28s): Lean tarball sha256-pinned, Mathlib 0df444a, lake build Mathlib
offline, in-image import Mathlib smoke test. The sandbox reports Lean/Mathlib 4.33.1, as expected.

Full sweep over all 1,788 candidates (10m38s, exit 0) → 1,526 verified, verified_tasks.json regenerated.
Re-gate of the committed list: 1526/1526 pass, exit 0.

The 262 rejects are dominated by files whose syntax needs FormalConjecturesUtil after all, and by proofs that
exceed the kernel's budget — both would otherwise be unanswerable tasks scoring a flat 0 for every model.

Also: 110 tests pass (gym env test --resources-server formal_conjectures, plus lean_proof and leancat, all
pre-existing ones unchanged); pre-commit clean; gym env resolve resolves the benchmark config; prepare.py
re-derives the rows from the pin.

Model rollouts: still N/A. The gate proves the environment and the dataset; it does not measure a model. Happy
to run a baseline if useful — say which model and k.

Three bugs the gate caught, all inherited from #3404

The first gate run failed 12 of 25 tasks. Not a bad image — three real bugs:

  1. #print axioms has two output forms and only one was parsed. A fully constructive proof reports
    'thm' does not depend on any axioms rather than an axiom list. That was read as "no axiom line found", i.e. a
    missing declaration, so the strongest possible result scored 0.0. Verified against Lean 4.33.1 in the built
    image before changing anything.
    The parser is now anchored to the target's name and takes the last match, which also closes a gaming vector:
    the submission is a whole file, so it can print the axioms of any proved Mathlib lemma next to its own sorry'd
    theorem, and an unanchored scan accepted the decoy.
  2. The declaration-name regex truncated Unicode identifiers. [A-Za-z0-9_.'!?] stops at the first subscript,
    so not_lt₂ became not_lt. That name goes into #print axioms <target>, so the probe named a declaration
    that does not exist, Lean errored, and the task could never be passed by any model. 49 of 5,248 declarations
    at the pin.
  3. Task ids were not unique. <path>::<short name> collides — sIncreasingrTuples.lean has five not_lt₂*
    lemmas. prepare.py built a dict on the id, so distinct tasks silently collapsed into whichever came last,
    which need not be the one the sweep validated. Ids are now namespace-qualified, and prepare.py /
    check_sandbox.py both raise on a duplicate.

Because ids changed form and the old list was built with the truncating regex, verified_tasks.json was
regenerated from scratch rather than migrated.

No tracking issue — a new benchmark port, not a fix to existing behaviour.

Review order

resources_servers/lean_proof/ first (three additive, opt-in seams), then formal_conjectures/app.py.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally.
  • Pre-commit checks pass locally (pre-commit run --all-files).
  • All commits have DCO sign-off (git commit -s).

🤖 Generated with Claude Code

…n_proof

Ports #3404 onto the `lean_proof` library that #3400 (LeanCat) introduces,
rather than carrying its own copy of the Lean plumbing.

#3404 was written against `main`, so it shipped a second `lean_proof` package
(its own `proof_utils`/`sandbox_client`/`toolchain`) and patched
`math_formal_lean` to share it. #3400 has since landed that library properly,
backed by `nemo_gym.sandbox` instead of an out-of-band NeMo-Skills HTTP
sandbox. This is the same benchmark rebuilt on top of it: the duplicated
modules and the `math_formal_lean` edits are gone, and `math_formal_lean`
stays byte-identical to `main`.

Formal Conjectures is 1,527 tasks from 323 upstream files (test 1290, API 191,
textbook 26, research solved 20) against Lean/Mathlib v4.33.1. Each row is a
whole Lean file with one proof removed; the model returns the file with that
hole filled.

What FC needs that LeanCat does not, and why the shared library grew three
seams rather than being forked again:

- An FC file legitimately keeps other `sorry`s -- it pairs a provable lemma
  with the open conjecture it sanity-checks, and the task keeps the
  declarations above the target because proofs lean on them. So
  `determine_proof_status` takes `sorry_is_error=False`, and
  `find_banned_declarations` takes a token list plus `declarations_only`
  (`sorry`/`admit` are legal here; `axiom`/`unsafe` are banned as
  declarations, not as words).
- `#print axioms <target>` stands in for the dropped `sorry` rule. The server
  appends the probe itself and scores 1.0 only when the target is free of
  `sorryAx`, which also catches a proof leaning on a sorry'd lemma earlier in
  the file.
- `check_target_statement_preserved` is the target-signature variant of the
  preservation check; splitting the reference on `sorry` the way LeanCat does
  throws out honest answers when the hole is not textually unique.
- `has_unterminated_block_comment` moves into the shared module: a mangled
  `-/` swallows the file, and without this it is misreported as a weakened
  statement rather than malformed output.

Also adds the v4.33.1 pins to `lean_image/versions.py`, a `check_sandbox.py`
that is both the pre-flight gate and the sweep defining `verified_tasks.json`
(#3404 referred to that sweep but never shipped it), and a real README in
place of #3404's stub. Every existing `lean_proof`/`leancat` test still
passes unchanged.

Validation: 108 tests pass (`gym env test --resources-server
formal_conjectures`, plus the lean_proof and leancat suites);
`pre-commit run --files ...` clean; `gym env resolve` resolves the benchmark
config; `prepare.py` re-derives the rows at the pin and reproduces the
committed `data/example.jsonl` byte-identically. No model rollouts -- that
needs a v4.33.1 Lean image, which is built by `lean_image/build.sh v4.33.1`
and is not available in this environment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…3.1 gate

Built the Mathlib v4.33.1 image and ran the validation sweep against it. The
gate failed 12 of the first 25 tasks, which turned out to be three real bugs
inherited from #3404 rather than a bad image.

1. `#print axioms` has two output forms, and only one was parsed.

   'thm' depends on axioms: [propext, Classical.choice]
   'thm' does not depend on any axioms          <-- fully constructive proof

   The second was read as "no axiom line found", i.e. a missing declaration, so
   the STRONGEST possible result -- a proof citing no axioms at all -- scored
   0.0. Confirmed against Lean 4.33.1 in the built image before fixing.

   The parser is now also anchored to the target's name and takes the last
   match. The submission is a whole file and may contain `#print axioms` calls
   of its own; without the anchor a reply that prints the axioms of some proved
   Mathlib lemma next to its own sorry'd theorem was scored as a proof.

2. The declaration-name regex truncated Unicode identifiers. `[A-Za-z0-9_.'!?]`
   stops at the first subscript, so `not_lt₂` became `not_lt`. That name goes
   into `#print axioms <target>`, so the probe named a declaration that does not
   exist, Lean errored, and the task could never be passed by any model.
   Measured at the pin: 49 of 5,248 declarations.

3. Task ids were `<path>::<short name>`, which is not unique -- truncation made
   it worse, but the collision is real regardless: `sIncreasingrTuples.lean` has
   five `not_lt₂*` lemmas. `prepare.py` built a dict on the id, so several
   distinct tasks silently collapsed into whichever came last, which need not be
   the one the sweep validated. Ids are now namespace-qualified, and both
   `prepare.py` and `check_sandbox.py` raise on a duplicate rather than letting
   one task displace another.

Because the ids changed form and the old list was built with the truncating
regex, `verified_tasks.json` was regenerated from scratch by the full sweep
against the real image: 1,526 of 1,788 candidates verified (was 1,527 on
ambiguous ids). The 262 rejects are dominated by files whose syntax needs
FormalConjecturesUtil after all, and by proofs that exceed the kernel's budget
-- both would otherwise be unanswerable tasks scoring a flat 0.

Validation:
- Image built (lean-image-build-fc.sub): gym-lean:v4.33.1, sha256-pinned Lean
  tarball, Mathlib 0df444a, `lake build Mathlib` offline, in-image
  `import Mathlib` smoke test.
- Sandbox reports Lean/Mathlib 4.33.1 as expected.
- Full sweep over all 1,788 candidates, 10m38s, exit 0.
- 110 tests pass; pre-commit clean; prepare.py regenerates example.jsonl.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 29, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Sep 30, 2026
…arisons

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
@lilithgrigoryan lilithgrigoryan mentioned this pull request Sep 30, 2026
5 tasks
…a plain-text list

Matches the supported_instance_ids.txt convention used by the SWE servers.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: Lilit Grigoryan <lgrigoryan@nvidia.com>
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 01:56
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:medium Single-domain change with interacting parts or a moderate review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants