Skip to content
Merged
66 changes: 66 additions & 0 deletions news/2026-10-05-hma-mle-bench.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
title: "78.2% medal rate on MLE-bench in six hours, by making two agents take turns"
description: "Two coding agents alternating over one workspace, each starting fresh, beat both of them working alone on 75 MLE-bench tasks: 78.2% any-medal against 72.4% and 68.0%. Self-reported, and not on the official leaderboard."
date: 2026-10-05
authors:
- Changye Li
tag: HKA
---

Take two strong coding agents. Give each the same machine-learning task and six hours, and
Claude Opus 5 wins a medal on **72.4%** of [MLE-bench](https://github.com/openai/mle-bench)'s
75 Kaggle tasks while GPT-5.6-sol wins one on **68.0%**. Now give them *one* six-hour budget
and make them take turns in the same workspace. Together they medal on **78.2%**.

That is [Humanize MLE Agents (HMA)](https://github.com/humanfia/hma), and the number comes
from its [results table](https://github.com/humanfia/hma#main-results).

| Workflow | Budget | Any medal | Gold | Above human median |
| --- | ---: | ---: | ---: | ---: |
| Claude Opus 5, native `/goal` | 6 h | 72.4% | 50.7% | 85.3% |
| GPT-5.6-sol, native `/goal` | 6 h | 68.0% | 47.6% | 81.3% |
| **HMA: Opus 5 ↔ GPT-5.6-sol** | **6 h** | **78.2%** | **53.8%** | **90.7%** |
| HMA: GPT-5.6-sol ↔ Opus 5 | 6 h | 73.3% | 50.7% | 85.3% |
| HMA: GPT-5.6-sol ↔ DeepSeek V4.1 Flash | 6 h | 72.0% | 50.7% | 85.3% |
| HMA: GPT-5.6-sol ↔ Kimi K3 | 6 h | 70.7% | 50.7% | 85.3% |

Against the mean of its two single-agent runs, the best pairing is **8.0 points** higher on
any-medal rate. Across all six pairings evaluated, the average gain is
[6.4 points](https://github.com/humanfia/hma#results-summary).

## Taking turns, with a clean slate

The arrangement is simple, and almost all of it is about what is *not* carried over:

1. One agent works the task in its own native harness and submits candidate solutions.
2. After at most **five accepted submissions**, it stops. The other agent starts a **fresh
session** in the same workspace: same code, models and candidates, but none of the first
agent's context.
3. They alternate until the six hours are nearly up. The last **15 minutes** go to picking one
of the candidates already accepted. Private test scores are never shown to either agent.

The repository credits [two effects at once](https://github.com/humanfia/hma#figure-1-hma-overview):
**context renewal**, because a long-running agent improves more and more slowly and a fresh
session restarts that curve, and **complementary capabilities**, because the second model
searches differently from the first. The handoff keeps everything the first agent built and drops only its
train of thought.

## What to be careful with

- **Order matters a lot.** The same two models score 78.2% with Opus going first and 73.3%
with GPT-5.6-sol going first. Choosing the best order after the fact flatters the headline.
The average over pairings, 6.4 points, is the more conservative figure.
- **This is self-reported, and it is not on the official leaderboard.** The
[MLE-bench leaderboard](https://github.com/openai/mle-bench#leaderboard) has not reviewed
these runs, and it has not taken new submissions since April. For scale only, its highest "All" entry today is 64.44%, with a 24-hour budget
and an older model. Different models, budgets and repeat counts make that a different
experiment, not a lower bar.
- **The paper is not out yet.** The arXiv link will be
[added with the paper release](https://github.com/humanfia/hma#citation). Until then, the
full ablations exist only in the repository's
[experiment map](https://github.com/humanfia/hma/blob/main/docs/experiments.md).

The whole suite reruns from the repository: the baselines, HMA, and the cap and starting-order
ablations.

[humanfia/hma](https://github.com/humanfia/hma) · [HKA](/projects/hka)
63 changes: 63 additions & 0 deletions news/2026-10-05-kaggle-biohub-final.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
title: "Biohub closed at #188 of 3,947: top 5%, not the #8 the public board showed"
description: "In August an agent workflow sat 8th on the Biohub cell-tracking public leaderboard. The private board has now spoken: 188th of 3,947, top 4.8%. A real finish, a smaller one, and what the Kaggle tally looks like when only final ranks count."
date: 2026-10-05
authors:
- Changye Li
tag: HKA
---

[On 15 August](/news/2026-08-15-kaggle-nineteen-competitions), one of our Kaggle agent workflows was **8th of 2,378** on the public leaderboard
of [Biohub – Cell Tracking During Development](https://www.kaggle.com/competitions/biohub-cell-tracking-during-development).
That was the top 0.34%, and it was among the ongoing results we reported then, under a note
that the numbers were still moving.

The competition has closed. The authenticated final private rank is **188th of 3,947**, the
**top 4.76%**. That is a genuine top-5% finish on a closed board. It is also 188th, not 8th.

We do not know how much of the gap is private-board shake-up and how much is everything that
happened after 15 August: 1,569 more teams entered, and the rest of the field kept improving.
The point stands either way. **A public rank in the middle of a competition is a weather
report.** We said so in August, and this is what it looks like.

## The other boards that closed

| Competition | Best tracked, 15 August, public board | Best tracked, final private board |
| --- | --- | --- |
| [Biohub – Cell Tracking](https://www.kaggle.com/competitions/biohub-cell-tracking-during-development) | 8 / 2,378 · top 0.34% | **188 / 3,947 · top 4.76%** |
| [PTCG AI Battle](https://www.kaggle.com/competitions/pokemon-tcg-ai-battle) | 955 / 6,829 · top 14.0% | **654 / 6,807 · top 9.61%** |
| [CUHK-X Small Model Track](https://www.kaggle.com/competitions/cuhk-x-competition-small-model-track) | 31 / 199 · top 15.6% | **56 / 326 · top 17.2%** |
| [Hyperspectral Object Detection](https://www.kaggle.com/competitions/hyperspectral-object-detection-challenge-2026) | 10 / 40 · top 25% | **77 / 309 · top 24.9%** |

One more closed with a striking number we cannot yet count.
[Predicting Smartphone Addiction](https://www.kaggle.com/competitions/playground-series-s6e8)
shows an account at **2nd of 3,531**, but on the *public* board as of the snapshot. That
account's final private rank has not been recorded. The best authenticated private rank we
have there is 564th. Until the private rank is in, rank 2 is not a finish, and we do not report
it as one.

## The tally, counting only final ranks

The [AgentKaggle Team Radar](https://agentkaggle.github.io/leaderboard/visualizations/)
(snapshot 2026-10-05 09:18 UTC) tracks **39** completed competitions. Taking the best tracked
result in each, **16** are in the top 5%. Most of those are not finishes:

- **3** are authenticated final private ranks: Predicting Student Health Risk (top 2.00%),
ROGII Wellbore Geology (top 2.24%) and Biohub (top 4.76%);
- **1** is a public rank at snapshot, the Smartphone Addiction result above; and
- **12** are late-submission estimates: real scores placed against a frozen final board, but
not ranks, medals or evidence of having competed.

The audit's
[charts](https://github.com/agentkaggle/kaggle-results-audit/tree/main/completed/visualizations)
are drawn with a fixed list of 12 competitions left out, so a chart rendered from today's data
shows 27, not 39. All 12 are outside the top 15%, so leaving them out flatters the picture. The
counts above use all 39. Of the **13** competitions still open, 3 sit in the top 5% of their public boards
today. One of those 3 is Spaceship Titanic, which is a getting-started practice competition.

The set of tracked accounts has also changed since August, with some added and some
dropped. That is why we compare competitions one by one here, and do not compare August's
totals with today's.

[The audit](https://github.com/agentkaggle/kaggle-results-audit) ·
[Team Radar](https://agentkaggle.github.io/leaderboard/) · [HKA](/projects/hka)
83 changes: 83 additions & 0 deletions news/2026-10-05-kda-upstream.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
title: "Six KDA pull requests land in SGLang, the best adding 8.7% end to end"
description: "Since August, SGLang has merged six pull requests carrying kernels our agents wrote, and the MLSys contest kernels are now public with a benchmark anyone can rerun. One upstream merge was reverted the next day."
date: 2026-10-05
authors:
- Xiaoyu Zhang
- Dongyun Zou
tag: KDA
---

A kernel that wins a benchmark is a claim. A kernel a maintainer merges into the engine people
serve models with is something else. That is the bar this post is about. Since August, six
pull requests carrying [KDA](/projects/kda)-written kernels have been merged into
[SGLang](https://github.com/sgl-project/sglang), all from Xiaoyu Zhang. We also count one
merge elsewhere that did not stay.

## Into SGLang

| Pull request | What it does | Measured |
| --- | --- | --- |
| [#36865](https://github.com/sgl-project/sglang/pull/36865) | NVFP4 GEMM for Qwen3.x on SM120 | 1.319× kernel geomean over 16 production shapes; Qwen3.5-4B output throughput **+8.73%** at concurrency 1, +6.5% at 8 |
| [#38082](https://github.com/sgl-project/sglang/pull/38082) | FP8 skinny GEMM on SM120 | end to end +0.2% to **+4.06%** across five models |
| [#41305](https://github.com/sgl-project/sglang/pull/41305) | SANA-Video residual-gate add on H200 | **2.54×** on the two real SANA-Video layouts; −1.7% end-to-end time |
| [#41459](https://github.com/sgl-project/sglang/pull/41459) | LTX-2.3 QK norm + split RoPE on H200 | **2.58×** to 2.61× kernel geomean; −1.0% to −1.2% end-to-end time |
| [#36845](https://github.com/sgl-project/sglang/pull/36845) | QSA decode on SM121 (GB10) | a correctness fix: the old path silently corrupted long-context decode |
| [#37385](https://github.com/sgl-project/sglang/pull/37385) | registers the merged agent kernels under a `KernelBackend.KDA` | provenance, so the engine records which kernels an agent wrote |

Two things in that table are worth reading slowly.

**A kernel speedup is not a model speedup.** The two diffusion kernels are about 2.5× faster
on their own and save one to two percent of a whole video generation, because they are a
small part of it. Both pull requests report both numbers, and #41305 kept a kernel only if
end-to-end time improved by at least 1.5% in both of its measurement groups.

**A pull request can get smaller under review.** #41305 began with 40 kernel families. It
merged with one, after every family that showed a measured regression, or had no established
end-to-end benefit, was taken out. The other 39 are listed in its
[selection record](https://github.com/BBuf/sglang/blob/42d1d19e5f876e7b5977fb55d90073230bce1655/diffusion-prs/kda-residual-gate-h200-20260927/selection.json).

Three of the six (#36845, #41305 and #41459) credit the kernel to KDA running with Codex and
**Kimi K3**.

## The contest kernels, public

The MLSys 2026 FlashInfer contest kernels are now
[public](https://github.com/mit-han-lab/mlsys2026-flashinfer-contest-solution) (Dongyun Zou),
as release **KDA 0.5**, with a benchmark that pins the human winners' own repositories and
refuses to run if they are modified. On B200, as the geometric mean over the official
workloads:

| Track | KDA 0.5 over the FlashInfer baseline | KDA 0.5 over the best human entry |
| --- | ---: | ---: |
| GDN prefill | 10.36× | **1.688×** |
| MoE, FP8 block scale | 1.57× | **1.173×** |
| DSA sparse attention | 38.33× | **1.408×** |

The repository [states its own caveats](https://github.com/mit-han-lab/mlsys2026-flashinfer-contest-solution#results),
and they apply here:

- The timing is CUPTI kernel spans. The contest was scored by the organizers' wall-clock
harness, so these are not contest scores.
- The kernels are tuned to the official shapes, so do not expect the same speedups elsewhere.
- The pinned human repositories are a few commits past the contest deadline.
- The repository numbers its releases from 0.1, the original contest entry. Our [August post](/news/2026-08-02-kda-15-past-human-sota)
called the same two generations KDA 1.0 and 1.5, and reported 1.25× to 1.39× over the human
entries on B300 under a different protocol. The two tables are not comparable, and the MoE margin is smaller in this one.

## One merge that did not stay

On 27 August, [svg-project/flash-kmeans#23](https://github.com/svg-project/flash-kmeans/pull/23)
merged a KDA-generated CuTe backend for Flash-KMeans, measured at **7.31×** on a real
LongCat-Video workload on B200. The next day the maintainer
[reverted it](https://github.com/svg-project/flash-kmeans/pull/24) to keep the repository
simple and asked for it to go to FlashLib instead. That has not happened yet. So, for now, it
is not upstream.

## And the long version

How our agents wrote Kimi Delta Attention kernels up to 2.96× faster than FlashKDA on B300, and
how they tried to cheat along the way, is in
[KDA²: KDA optimizes KDA](/blog/2026-09-27-kda-for-kda).

[KDA](/projects/kda) · [kernel-design-agents](https://github.com/mit-han-lab/kernel-design-agents)
57 changes: 57 additions & 0 deletions news/2026-10-05-lean-eval-v1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
title: "30 first solves on Lean-Eval v1, the most of anyone — and second by one on total"
description: "Lean-Eval froze its first 128-problem release and moved everything older to an archive. On v1 our GPT-5.6 flow has the most first solves and is one problem behind the total leader; on the archive it has 170 of 171."
date: 2026-10-05
authors:
- Zhengyang Zhang
- Hongzhou Lin
- Jui-Hui Chung
tag: HOA
---

In August this flow [took first place](/news/2026-08-18-lean-eval-first) on the [Lean-Eval](https://lean-lang.org/eval/) leaderboard.
That board no longer exists in the same shape. The maintainers have
[frozen a first release](https://lean-lang.org/eval/formalization-evaluation/), **LeanEval v1**,
of **128** problems (published 20 August 2026), and moved the older problems to an
**Archive** of **171**. The leaderboard now shows one scope at a time, and a single
"first place" no longer describes it. Here is where we stand in each.

All numbers below come from the board's own
[published data](https://lean-lang.org/eval/site-data/v2/groups/formalization-evaluation.json),
generated 2026-10-05 08:56 UTC. Our entry is listed as "Humanifa + GPT 5.6 sol" (sic).

## LeanEval v1: 128 problems, 88 solved by anyone

| Entry | Unique | First solves | Total |
| --- | ---: | ---: | ---: |
| Axiom Prover (Axiom Math) | 3 | 21 | 74 |
| NEAR AI, with DeepSeek V4 | 3 | 12 | **80** |
| **Humanfia, GPT-5.6** | 1 | **30** | 79 |

- **First solves: first, with 30.** We were the first to have an accepted proof on 30 of
the 88 problems anybody has solved.
- **Total: second, with 79**, one behind NEAR AI's 80.
- **Unique: fourth.** Only one of our 79 has no other solver. The board sorts by this
column by default, so that is the position a visitor sees first.

The three columns reward different things: being first, being alone, or being broad. We lead
on the first and are close on the third. Others have more problems nobody else can do.

## Archive: 170 of 171

On the 171 archived problems we have accepted proofs for **170**, tied for the most with two
other entries. Counted across both scopes, **249** distinct problems carry an accepted
Humanfia proof, the most of any entry on the board. That count is ours, made from the
published data; the site does not show a combined column. The board also notes that one
model may appear under more than one name while names are consolidated.

## What counts

Lean-Eval accepts a solution only when it passes
[Comparator](https://github.com/leanprover/comparator) against the problem's statement. There
are no partial credits and no `sorry`. Of the entry's 249 accepted solutions, 210 were
submitted by Zhengyang Zhang, 34 by Hongzhou Lin and 5 by Jui-Hui Chung; 75 were
accepted on or after 19 August.

[The leaderboard](https://lean-lang.org/eval/formalization-evaluation/) ·
[its source](https://github.com/leanprover/lean-eval-leaderboard) · [HOA](/projects/hoa)
Loading
Loading