diff --git a/news/2026-10-05-hma-mle-bench.md b/news/2026-10-05-hma-mle-bench.md new file mode 100644 index 0000000..e7b2753 --- /dev/null +++ b/news/2026-10-05-hma-mle-bench.md @@ -0,0 +1,66 @@ +--- +title: "78.2% medal rate on MLE-bench in six hours, by making two agents take turns" +description: "Two coding agents alternating over one workspace, each starting fresh, beat both of them working alone on 75 MLE-bench tasks: 78.2% any-medal against 72.4% and 68.0%. Self-reported, and not on the official leaderboard." +date: 2026-10-05 +authors: + - Changye Li +tag: HKA +--- + +Take two strong coding agents. Give each the same machine-learning task and six hours, and +Claude Opus 5 wins a medal on **72.4%** of [MLE-bench](https://github.com/openai/mle-bench)'s +75 Kaggle tasks while GPT-5.6-sol wins one on **68.0%**. Now give them *one* six-hour budget +and make them take turns in the same workspace. Together they medal on **78.2%**. + +That is [Humanize MLE Agents (HMA)](https://github.com/humanfia/hma), and the number comes +from its [results table](https://github.com/humanfia/hma#main-results). + +| Workflow | Budget | Any medal | Gold | Above human median | +| --- | ---: | ---: | ---: | ---: | +| Claude Opus 5, native `/goal` | 6 h | 72.4% | 50.7% | 85.3% | +| GPT-5.6-sol, native `/goal` | 6 h | 68.0% | 47.6% | 81.3% | +| **HMA: Opus 5 ↔ GPT-5.6-sol** | **6 h** | **78.2%** | **53.8%** | **90.7%** | +| HMA: GPT-5.6-sol ↔ Opus 5 | 6 h | 73.3% | 50.7% | 85.3% | +| HMA: GPT-5.6-sol ↔ DeepSeek V4.1 Flash | 6 h | 72.0% | 50.7% | 85.3% | +| HMA: GPT-5.6-sol ↔ Kimi K3 | 6 h | 70.7% | 50.7% | 85.3% | + +Against the mean of its two single-agent runs, the best pairing is **8.0 points** higher on +any-medal rate. Across all six pairings evaluated, the average gain is +[6.4 points](https://github.com/humanfia/hma#results-summary). + +## Taking turns, with a clean slate + +The arrangement is simple, and almost all of it is about what is *not* carried over: + +1. One agent works the task in its own native harness and submits candidate solutions. +2. After at most **five accepted submissions**, it stops. The other agent starts a **fresh + session** in the same workspace: same code, models and candidates, but none of the first + agent's context. +3. They alternate until the six hours are nearly up. The last **15 minutes** go to picking one + of the candidates already accepted. Private test scores are never shown to either agent. + +The repository credits [two effects at once](https://github.com/humanfia/hma#figure-1-hma-overview): +**context renewal**, because a long-running agent improves more and more slowly and a fresh +session restarts that curve, and **complementary capabilities**, because the second model +searches differently from the first. The handoff keeps everything the first agent built and drops only its +train of thought. + +## What to be careful with + +- **Order matters a lot.** The same two models score 78.2% with Opus going first and 73.3% + with GPT-5.6-sol going first. Choosing the best order after the fact flatters the headline. + The average over pairings, 6.4 points, is the more conservative figure. +- **This is self-reported, and it is not on the official leaderboard.** The + [MLE-bench leaderboard](https://github.com/openai/mle-bench#leaderboard) has not reviewed + these runs, and it has not taken new submissions since April. For scale only, its highest "All" entry today is 64.44%, with a 24-hour budget + and an older model. Different models, budgets and repeat counts make that a different + experiment, not a lower bar. +- **The paper is not out yet.** The arXiv link will be + [added with the paper release](https://github.com/humanfia/hma#citation). Until then, the + full ablations exist only in the repository's + [experiment map](https://github.com/humanfia/hma/blob/main/docs/experiments.md). + +The whole suite reruns from the repository: the baselines, HMA, and the cap and starting-order +ablations. + +[humanfia/hma](https://github.com/humanfia/hma) · [HKA](/projects/hka) diff --git a/news/2026-10-05-kaggle-biohub-final.md b/news/2026-10-05-kaggle-biohub-final.md new file mode 100644 index 0000000..631d3b0 --- /dev/null +++ b/news/2026-10-05-kaggle-biohub-final.md @@ -0,0 +1,63 @@ +--- +title: "Biohub closed at #188 of 3,947: top 5%, not the #8 the public board showed" +description: "In August an agent workflow sat 8th on the Biohub cell-tracking public leaderboard. The private board has now spoken: 188th of 3,947, top 4.8%. A real finish, a smaller one, and what the Kaggle tally looks like when only final ranks count." +date: 2026-10-05 +authors: + - Changye Li +tag: HKA +--- + +[On 15 August](/news/2026-08-15-kaggle-nineteen-competitions), one of our Kaggle agent workflows was **8th of 2,378** on the public leaderboard +of [Biohub – Cell Tracking During Development](https://www.kaggle.com/competitions/biohub-cell-tracking-during-development). +That was the top 0.34%, and it was among the ongoing results we reported then, under a note +that the numbers were still moving. + +The competition has closed. The authenticated final private rank is **188th of 3,947**, the +**top 4.76%**. That is a genuine top-5% finish on a closed board. It is also 188th, not 8th. + +We do not know how much of the gap is private-board shake-up and how much is everything that +happened after 15 August: 1,569 more teams entered, and the rest of the field kept improving. +The point stands either way. **A public rank in the middle of a competition is a weather +report.** We said so in August, and this is what it looks like. + +## The other boards that closed + +| Competition | Best tracked, 15 August, public board | Best tracked, final private board | +| --- | --- | --- | +| [Biohub – Cell Tracking](https://www.kaggle.com/competitions/biohub-cell-tracking-during-development) | 8 / 2,378 · top 0.34% | **188 / 3,947 · top 4.76%** | +| [PTCG AI Battle](https://www.kaggle.com/competitions/pokemon-tcg-ai-battle) | 955 / 6,829 · top 14.0% | **654 / 6,807 · top 9.61%** | +| [CUHK-X Small Model Track](https://www.kaggle.com/competitions/cuhk-x-competition-small-model-track) | 31 / 199 · top 15.6% | **56 / 326 · top 17.2%** | +| [Hyperspectral Object Detection](https://www.kaggle.com/competitions/hyperspectral-object-detection-challenge-2026) | 10 / 40 · top 25% | **77 / 309 · top 24.9%** | + +One more closed with a striking number we cannot yet count. +[Predicting Smartphone Addiction](https://www.kaggle.com/competitions/playground-series-s6e8) +shows an account at **2nd of 3,531**, but on the *public* board as of the snapshot. That +account's final private rank has not been recorded. The best authenticated private rank we +have there is 564th. Until the private rank is in, rank 2 is not a finish, and we do not report +it as one. + +## The tally, counting only final ranks + +The [AgentKaggle Team Radar](https://agentkaggle.github.io/leaderboard/visualizations/) +(snapshot 2026-10-05 09:18 UTC) tracks **39** completed competitions. Taking the best tracked +result in each, **16** are in the top 5%. Most of those are not finishes: + +- **3** are authenticated final private ranks: Predicting Student Health Risk (top 2.00%), + ROGII Wellbore Geology (top 2.24%) and Biohub (top 4.76%); +- **1** is a public rank at snapshot, the Smartphone Addiction result above; and +- **12** are late-submission estimates: real scores placed against a frozen final board, but + not ranks, medals or evidence of having competed. + +The audit's +[charts](https://github.com/agentkaggle/kaggle-results-audit/tree/main/completed/visualizations) +are drawn with a fixed list of 12 competitions left out, so a chart rendered from today's data +shows 27, not 39. All 12 are outside the top 15%, so leaving them out flatters the picture. The +counts above use all 39. Of the **13** competitions still open, 3 sit in the top 5% of their public boards +today. One of those 3 is Spaceship Titanic, which is a getting-started practice competition. + +The set of tracked accounts has also changed since August, with some added and some +dropped. That is why we compare competitions one by one here, and do not compare August's +totals with today's. + +[The audit](https://github.com/agentkaggle/kaggle-results-audit) · +[Team Radar](https://agentkaggle.github.io/leaderboard/) · [HKA](/projects/hka) diff --git a/news/2026-10-05-kda-upstream.md b/news/2026-10-05-kda-upstream.md new file mode 100644 index 0000000..10878f2 --- /dev/null +++ b/news/2026-10-05-kda-upstream.md @@ -0,0 +1,83 @@ +--- +title: "Six KDA pull requests land in SGLang, the best adding 8.7% end to end" +description: "Since August, SGLang has merged six pull requests carrying kernels our agents wrote, and the MLSys contest kernels are now public with a benchmark anyone can rerun. One upstream merge was reverted the next day." +date: 2026-10-05 +authors: + - Xiaoyu Zhang + - Dongyun Zou +tag: KDA +--- + +A kernel that wins a benchmark is a claim. A kernel a maintainer merges into the engine people +serve models with is something else. That is the bar this post is about. Since August, six +pull requests carrying [KDA](/projects/kda)-written kernels have been merged into +[SGLang](https://github.com/sgl-project/sglang), all from Xiaoyu Zhang. We also count one +merge elsewhere that did not stay. + +## Into SGLang + +| Pull request | What it does | Measured | +| --- | --- | --- | +| [#36865](https://github.com/sgl-project/sglang/pull/36865) | NVFP4 GEMM for Qwen3.x on SM120 | 1.319× kernel geomean over 16 production shapes; Qwen3.5-4B output throughput **+8.73%** at concurrency 1, +6.5% at 8 | +| [#38082](https://github.com/sgl-project/sglang/pull/38082) | FP8 skinny GEMM on SM120 | end to end +0.2% to **+4.06%** across five models | +| [#41305](https://github.com/sgl-project/sglang/pull/41305) | SANA-Video residual-gate add on H200 | **2.54×** on the two real SANA-Video layouts; −1.7% end-to-end time | +| [#41459](https://github.com/sgl-project/sglang/pull/41459) | LTX-2.3 QK norm + split RoPE on H200 | **2.58×** to 2.61× kernel geomean; −1.0% to −1.2% end-to-end time | +| [#36845](https://github.com/sgl-project/sglang/pull/36845) | QSA decode on SM121 (GB10) | a correctness fix: the old path silently corrupted long-context decode | +| [#37385](https://github.com/sgl-project/sglang/pull/37385) | registers the merged agent kernels under a `KernelBackend.KDA` | provenance, so the engine records which kernels an agent wrote | + +Two things in that table are worth reading slowly. + +**A kernel speedup is not a model speedup.** The two diffusion kernels are about 2.5× faster +on their own and save one to two percent of a whole video generation, because they are a +small part of it. Both pull requests report both numbers, and #41305 kept a kernel only if +end-to-end time improved by at least 1.5% in both of its measurement groups. + +**A pull request can get smaller under review.** #41305 began with 40 kernel families. It +merged with one, after every family that showed a measured regression, or had no established +end-to-end benefit, was taken out. The other 39 are listed in its +[selection record](https://github.com/BBuf/sglang/blob/42d1d19e5f876e7b5977fb55d90073230bce1655/diffusion-prs/kda-residual-gate-h200-20260927/selection.json). + +Three of the six (#36845, #41305 and #41459) credit the kernel to KDA running with Codex and +**Kimi K3**. + +## The contest kernels, public + +The MLSys 2026 FlashInfer contest kernels are now +[public](https://github.com/mit-han-lab/mlsys2026-flashinfer-contest-solution) (Dongyun Zou), +as release **KDA 0.5**, with a benchmark that pins the human winners' own repositories and +refuses to run if they are modified. On B200, as the geometric mean over the official +workloads: + +| Track | KDA 0.5 over the FlashInfer baseline | KDA 0.5 over the best human entry | +| --- | ---: | ---: | +| GDN prefill | 10.36× | **1.688×** | +| MoE, FP8 block scale | 1.57× | **1.173×** | +| DSA sparse attention | 38.33× | **1.408×** | + +The repository [states its own caveats](https://github.com/mit-han-lab/mlsys2026-flashinfer-contest-solution#results), +and they apply here: + +- The timing is CUPTI kernel spans. The contest was scored by the organizers' wall-clock + harness, so these are not contest scores. +- The kernels are tuned to the official shapes, so do not expect the same speedups elsewhere. +- The pinned human repositories are a few commits past the contest deadline. +- The repository numbers its releases from 0.1, the original contest entry. Our [August post](/news/2026-08-02-kda-15-past-human-sota) + called the same two generations KDA 1.0 and 1.5, and reported 1.25× to 1.39× over the human + entries on B300 under a different protocol. The two tables are not comparable, and the MoE margin is smaller in this one. + +## One merge that did not stay + +On 27 August, [svg-project/flash-kmeans#23](https://github.com/svg-project/flash-kmeans/pull/23) +merged a KDA-generated CuTe backend for Flash-KMeans, measured at **7.31×** on a real +LongCat-Video workload on B200. The next day the maintainer +[reverted it](https://github.com/svg-project/flash-kmeans/pull/24) to keep the repository +simple and asked for it to go to FlashLib instead. That has not happened yet. So, for now, it +is not upstream. + +## And the long version + +How our agents wrote Kimi Delta Attention kernels up to 2.96× faster than FlashKDA on B300, and +how they tried to cheat along the way, is in +[KDA²: KDA optimizes KDA](/blog/2026-09-27-kda-for-kda). + +[KDA](/projects/kda) · [kernel-design-agents](https://github.com/mit-han-lab/kernel-design-agents) diff --git a/news/2026-10-05-lean-eval-v1.md b/news/2026-10-05-lean-eval-v1.md new file mode 100644 index 0000000..6b15a65 --- /dev/null +++ b/news/2026-10-05-lean-eval-v1.md @@ -0,0 +1,57 @@ +--- +title: "30 first solves on Lean-Eval v1, the most of anyone — and second by one on total" +description: "Lean-Eval froze its first 128-problem release and moved everything older to an archive. On v1 our GPT-5.6 flow has the most first solves and is one problem behind the total leader; on the archive it has 170 of 171." +date: 2026-10-05 +authors: + - Zhengyang Zhang + - Hongzhou Lin + - Jui-Hui Chung +tag: HOA +--- + +In August this flow [took first place](/news/2026-08-18-lean-eval-first) on the [Lean-Eval](https://lean-lang.org/eval/) leaderboard. +That board no longer exists in the same shape. The maintainers have +[frozen a first release](https://lean-lang.org/eval/formalization-evaluation/), **LeanEval v1**, +of **128** problems (published 20 August 2026), and moved the older problems to an +**Archive** of **171**. The leaderboard now shows one scope at a time, and a single +"first place" no longer describes it. Here is where we stand in each. + +All numbers below come from the board's own +[published data](https://lean-lang.org/eval/site-data/v2/groups/formalization-evaluation.json), +generated 2026-10-05 08:56 UTC. Our entry is listed as "Humanifa + GPT 5.6 sol" (sic). + +## LeanEval v1: 128 problems, 88 solved by anyone + +| Entry | Unique | First solves | Total | +| --- | ---: | ---: | ---: | +| Axiom Prover (Axiom Math) | 3 | 21 | 74 | +| NEAR AI, with DeepSeek V4 | 3 | 12 | **80** | +| **Humanfia, GPT-5.6** | 1 | **30** | 79 | + +- **First solves: first, with 30.** We were the first to have an accepted proof on 30 of + the 88 problems anybody has solved. +- **Total: second, with 79**, one behind NEAR AI's 80. +- **Unique: fourth.** Only one of our 79 has no other solver. The board sorts by this + column by default, so that is the position a visitor sees first. + +The three columns reward different things: being first, being alone, or being broad. We lead +on the first and are close on the third. Others have more problems nobody else can do. + +## Archive: 170 of 171 + +On the 171 archived problems we have accepted proofs for **170**, tied for the most with two +other entries. Counted across both scopes, **249** distinct problems carry an accepted +Humanfia proof, the most of any entry on the board. That count is ours, made from the +published data; the site does not show a combined column. The board also notes that one +model may appear under more than one name while names are consolidated. + +## What counts + +Lean-Eval accepts a solution only when it passes +[Comparator](https://github.com/leanprover/comparator) against the problem's statement. There +are no partial credits and no `sorry`. Of the entry's 249 accepted solutions, 210 were +submitted by Zhengyang Zhang, 34 by Hongzhou Lin and 5 by Jui-Hui Chung; 75 were +accepted on or after 19 August. + +[The leaderboard](https://lean-lang.org/eval/formalization-evaluation/) · +[its source](https://github.com/leanprover/lean-eval-leaderboard) · [HOA](/projects/hoa) diff --git a/news/2026-10-05-olympiads.md b/news/2026-10-05-olympiads.md new file mode 100644 index 0000000..82fae8a --- /dev/null +++ b/news/2026-10-05-olympiads.md @@ -0,0 +1,103 @@ +--- +title: "Full marks on five science exams, and the fine print on every one" +description: "IPhO 2026 theory 30 of 30, IChO 2026 formalized 68 of 68, IOI 2026 six of six, IBO 2024 theory 100 of 100, and quantum information theory 40 of 40. What each score is, who graded it, and what it does not show." +date: 2026-10-05 +authors: + - Jing Xiong + - Zhengyang Zhang + - Ligeng Zhu +tag: HOA +--- + +Since July, the same open harness has finished IPhO and the quantum run +[we reported then](/news/2026-07-29-physics-and-quantum), and +has been pointed at three more olympiads. Each one reached the top of its own scale. **None of these scores came from an official +jury,** and the scales are different, so each comes with a note on what was graded and by +whom. The notes matter as much as the numbers. + +| Exam | Result | Graded by | +| --- | --- | --- | +| IPhO 2026, theory | 30.00 / 30.00, both GPT-5.6 Sol and Kimi K3 Max | our audit against the official marking scheme | +| IChO 2026, theory | 68 / 68 subquestions formalized in Lean, both models | Lean, plus our own review gates | +| IOI 2026 | 6 / 6 problems at 100% | Codeforces | +| IBO 2024, theory | 100 / 100 tasks; 400 / 400 verdicts match the key | the official answer key | +| QIT | 40 / 40 tasks proved in Lean | Lean, plus automated semantic review | + +## IPhO 2026: 30 of 30 on theory + +Both the GPT-5.6 Sol and the Kimi K3 Max runs answered all 23 theory subparts and scored +**30.00 / 30.00** ([GPT](https://github.com/humanfia/ipho2026/blob/main/NaturalLanguage/GRADING_REPORT.md), +[Kimi](https://github.com/humanfia/ipho2026/blob/main/Kimi/NaturalLanguage/GRADING_REPORT.md)). +The workers never saw the official solutions or marking schemes. Those came in only after the +runs had finished, for grading. + +The caveat is the grader. These are +[our own estimates](https://github.com/humanfia/ipho2026#results) against the official +solutions and itemized marking schemes, not a jury's adjudication. + +The Lean side went further than in July. The formal release splits the paper more finely, into +**41 parts**, and every one of them now has a Lean formalization. Formalizing is harder than answering, and the two models separate here: the +GPT-5.6 Sol formalizations are estimated at +[47.45 / 50](https://github.com/humanfia/ipho2026/blob/main/GRADING_REPORT.md), Kimi's at +[29.90 / 50](https://github.com/humanfia/ipho2026/blob/main/Kimi/GRADING_REPORT.md). Every +proof elaborates without `sorry`, `admit` or custom axioms. + +## IChO 2026: 68 of 68 formalized, and a baseline that shows why it matters + +All **68** numbered theory subquestions across the nine problems now have Lean formalizations +that passed both formalization review and proof review, for +[GPT-5.6 Sol](https://github.com/humanfia/icho2026/tree/main/gpt-5.6-sol-full68-formalization) +and for Kimi-K3 ([32 answer-blind](https://github.com/humanfia/icho2026/tree/main/kimi-k3-answer-blind) +plus [36 more](https://github.com/humanfia/icho2026/tree/main/kimi-k3-nl-36-formalization)). +The comparison that matters is the same models without the review loop. Run as a plain +Codex `/goal` on the same 68 targets, they had +[32 / 68](https://github.com/humanfia/icho2026/tree/main/gpt-5.6-sol-native-goal68) and +[31 / 68](https://github.com/humanfia/icho2026/tree/main/kimi-k3-native-goal68) accepted. + +The fine print: + +- **68 / 68 is formalization coverage, not answer accuracy.** Lean checks deductions from the + encoded inputs. It does not certify that the encoding matches the official chemistry. +- **Not every target came from the original inputs alone.** GPT has 66 targets from the + original inputs and 2 conditional ones. One of the conditional targets, T8-A8, uses two + disclosed contest-model axioms that we authorized. +- **Kimi's 68 is two campaigns.** One is 32 answer-blind targets; the other formalizes 36 + natural-language answers Kimi had already written. Three of those 36 are conditional and two + carry authorized answer corrections. +- **The answer scores are generous and unofficial.** Graded against the official key with + credit for equivalent forms and reasonable rounding, GPT scores + [424.5 / 437](https://github.com/humanfia/icho2026/blob/main/gpt-5.6-sol-full68-formalization/grading/GRADING.md) + and Kimi [417.5 / 437](https://github.com/humanfia/icho2026/blob/main/kimi-k3-max/GRADING.md). + +## IOI 2026: six of six + +All six problems over both days +[passed at 100%](https://github.com/humanfia/ioi2026#results) when the final submissions were +judged on [Codeforces](https://codeforces.com/). The +[annotated solutions](https://github.com/humanfia/ioi2026/blob/main/solutions/README.md) state +the guarantee each one meets. What you can check yourself is narrower: the repository's +`verify.sh` compiles all six and replays every released example. Codeforces' hidden tests +remain the external check, and the verdicts are not linked publicly. + +## IBO 2024: 100 of 100 theory tasks + +All 100 Theory A and B tasks were solved. All **400** true/false verdicts +[match the official answer key](https://github.com/humanfia/ibo2024/blob/main/GRADING.md), +with zero extraction errors. Two limits apply. The practical exams are excluded entirely, and +this is agreement with an answer key, not an official points total. It is also the 2024 paper, +because IBO withholds each paper for two years. A paper that has been public that long may +appear in a model's training data. + +## Quantum information theory: 37 of 40 is now 40 of 40 + +In July, the blind QIT run stood at 37 of 40 end to end. After repair it is +[**40 / 40**](https://huggingface.co/datasets/humanfia-lab/QIT), alongside 36 / 36 on quantum +algorithms ([QAlg](https://huggingface.co/datasets/humanfia-lab/QAlg)). Every task compiles, +has a complete proof with no `sorry` or `admit`, and passes a full `lake build`. The caveat +is the one we drew in July: the semantic review, which judges whether a formal statement says +what the problem says, is automated, not an independent human audit. +[Both sets rebuild with one script](https://github.com/humanfia/lean-qit-qlg). + +[IPhO 2026](https://github.com/humanfia/ipho2026) · [IChO 2026](https://github.com/humanfia/icho2026) · +[IOI 2026](https://github.com/humanfia/ioi2026) · [IBO 2024](https://github.com/humanfia/ibo2024) · +[HOA](/projects/hoa) diff --git a/news/2026-10-05-putnambench-672.md b/news/2026-10-05-putnambench-672.md new file mode 100644 index 0000000..288106e --- /dev/null +++ b/news/2026-10-05-putnambench-672.md @@ -0,0 +1,63 @@ +--- +title: "672 of 672: we closed PutnamBench's last two, and three others got there too" +description: "GPT-5.6 in a worker–reviewer loop proved all 672 PutnamBench statements in Lean, at $44.50 a problem. The top line is now a four-way tie, and what separates the four is cost." +date: 2026-10-05 +authors: + - Ligeng Zhu + - Zhengyang Zhang +tag: HOA +--- + +In June this solver [stood at 670 of 672](/news/2026-06-26-putnambench). It has now closed the last two. Every one of PutnamBench's **672** formal statements has a Lean 4 proof that the +PutnamBench team accepted, and the +[official leaderboard](https://trishullab.github.io/PutnamBench/leaderboard.html) lists +Humanfia at 672 ([trishullab/PutnamBench#350](https://github.com/trishullab/PutnamBench/pull/350), +merged 1 September 2026). + +We are not alone up there. Four entries now report 672 in the Lean column with answers given, in the +[leaderboard data](https://github.com/trishullab/PutnamBench/blob/main/docs/results.json): + +| Entry | Added | Reported cost | +| --- | --- | --- | +| Aleph Prover (Logical Intelligence) | 2026-08-26 | average $74 per problem, maximum $1,468 | +| **Humanfia** — GPT-5.6, `xhigh` reasoning | **2026-08-27** | **average $44.50 per problem** | +| NEAR AI, with DeepSeek V4 | 2026-09-04 | mean $0.17, median $0.04 per problem | +| Forall (Astrio), with Claude Opus 5 | 2026-09-24 | mean $3.52 actor-side per problem | + +The costs come from each team's own note on the leaderboard, and they are not measured the +same way: one is total spend, one is actor-side only. Read them as orders of magnitude, not +as a ranking. Read that way, the benchmark is saturated, and **what separates the leaders now +is the price of a proof, which is where we are not first.** Two of the four got to 672 for +a tenth of our cost or less. + +The board also has a separate "no answer" column, where the prover is not given the answer to +problems that ask for one. There, [Midas Prover](https://mi.inc/blog/midas-prover-solves-putnambench) +reports 672 at an average of $6.17 per problem. Our run is in the column with answers, so the +two are not the same test. + +## What counts as a proof + +A problem counts only when the Lean file passes every gate in the +[solver's verification method](https://github.com/humanfia/hoa-qed#verification-method), and +the model that wrote the proof never decides whether it is accepted: + +- the statement is byte-identical to the pinned benchmark statement; +- no `sorry`, `admit`, `axiom` or `native_decide`; +- it compiles against the pinned Lean 4.27.0 and Mathlib; +- [Comparator](https://github.com/leanprover/comparator) checks the statement and replays the + proof through the Lean kernel; and +- a separate reviewer, which cannot edit the file, gets `okay: true` back from the + [AXLE](https://axle.axiommath.ai/) verification API. + +The worker's shell has no network access, and no existing PutnamBench solution is ever +mounted into either model's workspace. + +## Checking it + +At the PutnamBench authors' request the full proof set is not public. The +[first 12 proofs](https://huggingface.co/datasets/humanfia-lab/putnambench-solution-preview) +are, along with the whole pipeline, and checking them through AXLE takes Python 3 and a +network connection: the steps are in the +[repository](https://github.com/humanfia/hoa-qed#start-here-verify-our-proofs-through-the-axle-api). + +[humanfia/hoa-qed](https://github.com/humanfia/hoa-qed) · [HOA](/projects/hoa)