Record: Bank QAT + seq4096 + SWA w=256 + QK-Gain 2.5 + PKO — val_bpb 1.1117 (3-seed mean)#1512

Open

Itssshikhar wants to merge 77 commits intoopenai:mainfrom

Itssshikhar:shikhar

Commits on Mar 21, 2026

Commits on Mar 31, 2026

Record: 11L + Efficient Partial XSA (val_bpb: 1.1307) — NEW SOTA

unnir
authored and
Itssshikhar
committed
Record: 11L XSA + EMA + Int6 MLP3x + WD=0.04 (val_bpb: 1.1271)

jfprincz
authored and
Itssshikhar
committed
Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248)

jfprincz
authored and
Itssshikhar
committed
Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233, 3-seed mean)

signalrush
authored and
Itssshikhar
committed
Update README leaderboard with merged records

cocohearts
authored and
Itssshikhar
committed
Use GitHub usernames in new leaderboard rows

cocohearts
authored and
Itssshikhar
committed
Describe leaderboard entries by base-run diff

cocohearts
authored and
Itssshikhar
committed
Record: LeakyReLU² + Legal TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean)

authored and
Itssshikhar
committed
Fix pre-TTT BPB, TTT gains, and steps to match logs exactly

abaybektursun
authored and
Itssshikhar
committed
Fix author attributions: PR #493 @parinzee, PR #461 @Christopher-Lee-McClendon

abaybektursun
authored and
Itssshikhar
committed
Update README.md

valerio-oai
authored and
Itssshikhar
committed
Update README.md

valerio-oai
authored and
Itssshikhar
committed
Notable Non-Record Submission: 1.1239 BPB - 106.2M Binary Asymmetric U-Net + NeoMuon + 4xrelu²MLP + Smear + Fact Tied Emb + Poly5 Softcap + YaRN2048 + 8192BPE + FP8 + Bit-packing LZMA + Stride-16 E…

authored and
Itssshikhar
committed
Update README.md

0hq
authored and
Itssshikhar
committed
Record Submission: 1.1570 BPB - 73.7M Ternary U-Net (10L 768d 8192BPE relu² 4xMLP FP8) (#640)

authored and
Itssshikhar
committed
Update README.md

0hq
authored and
Itssshikhar
committed
Update README.md

0hq
authored and
Itssshikhar
committed
Update README.md

0hq
authored and
Itssshikhar
committed
Update README.md

0hq
authored and
Itssshikhar
committed
Update README.md

0hq
authored and
Itssshikhar
committed
Non-record: Depth Recurrence in Parameter-Constrained Transformers — What Works, What Doesn't, and Why (#363)

authored and
Itssshikhar
committed
Add Gram Newton-Schulz integration on top of #1 submission

Itssshikhar
and
claude
committed

Commits on Apr 2, 2026

Commits on Apr 3, 2026

Commits on Apr 4, 2026

Add sliding window attention experiment + full debugging journey

Itssshikhar
and
claude
committed

Commits on Apr 6, 2026

Commits on Apr 7, 2026

Commits on Apr 8, 2026

Document NanoGPT technique stacking: QK gain 2.5 + sigmoid rescale + PKO

Itssshikhar
and
claude
committed

Commits on Apr 9, 2026

Commits on Apr 27, 2026

Commits on Apr 28, 2026

Commits on Apr 29, 2026

Commits on Apr 30, 2026