Non-record: 11L + 30-Epoch Legal TTT (BPB 1.14252)

[Christopher-Lee-McClendon]

11L Depth Recurrence + 30-Epoch Legal Score-First TTT

val_bpb = 1.14252 | Pre-TTT: 1.1609 | TTT gain: −0.0184 | Artifact: 15.48 MB

Key Finding: SGD with 30 epochs per chunk yields large TTT gains

This submission builds on PR #461 (BPB 1.14458) with a single change: increasing TTT epochs from 3 to 30. A sweep from 3→30 epochs showed general BPB improvement, though not strictly monotonic at every point:

TTT Epochs	BPB	Δ vs 3ep	Notes
3	1.14458	baseline	PR #461
5	1.14399	−0.00059
7	1.14378	−0.00080
10	1.14295	−0.00163
15	1.14335	−0.00123	Non-monotonic (worse than 10ep)
20	1.14292	−0.00166
30	1.14252	−0.00206	This submission

All results are single runs (no error bars). 40-epoch and 50-epoch runs are in progress at time of submission.

Why does more SGD help for legal TTT?

Legal score-first TTT applies SGD per chunk (32K tokens): score → train → advance. With frozen early layers (freeze=2), the remaining 19.9M parameters benefit from extended optimization. The cosine LR schedule across chunks provides natural regularization, preventing overfitting even at 30 epochs.

Key experimental findings:

• SGD >> AdamW for legal TTT (0.027 BPB gap) — Adam's moment estimates don't converge with only ~30 steps per chunk
• freeze=2 is essential — freeze=0 causes catastrophic per-chunk overfitting in legal TTT
• 40-epoch and 50-epoch runs in progress — results pending at time of submission

Comparison to prior submissions

The TTT gain of −0.0184 represents 2.7× more TTT improvement than our prior 1-epoch AdamW approach (−0.0068 in PR #456), and 12% more than the 3-epoch SGD baseline (−0.0165 in PR #461).

Architecture (unchanged from PR #461)

• 11 logical layers (10 unique BlockCores with depth recurrence)
• dim=512, 8 heads (64 dim/head), 4 KV heads (GQA)
• MLP 3× expansion (1536), ReLU² activation; SmearGate
• Partial RoPE (16/64 dims), Value Embeddings (128d on layers 9-10)
• Layer-Norm Scale, XSA last 4, BigramHash(2048)
• SWA, Late QAT, int6+zstd quantization
• 15,479,992 bytes total (520KB headroom under 16MB limit)
• Trained on 4×A100-40GB, 5200 steps (~41 min), eval 3662s on 1×A100

Credits

This submission builds on work from many contributors to the parameter-golf competition:

• Muon optimizer — Baseline (modded-nanogpt); Newton-Schulz orthogonal preconditioning
• BigramHash embeddings — PR Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65 (aquariouseworkman)
• SmearGate — PR Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65 (aquariouseworkman)
• XSA — PR [Closed] EMA + Multi-Order N-gram Backoff + PE Confidence (val_bpb=0.9393) #187 (Idan3011); GQA-aware variant in PR Record: 11L + Efficient Partial XSA (val_bpb: 1.1307)  #265 (unnir)
• U-Net skip connections — PR Record: Mixed Quant Int6/FP16 + SmearGate + OrthoInit + MLP 3x + Sliding Window, val_bpb=1.1556 #65 (aquariouseworkman), PR SubSixteen v2: Int6 QAT + MLP 3x + SWA + Sliding Window (val_bpb 1.1708) #69 (TevBenji)
• Mixed int5/int6 quantization — PR 12L Int5-MLP + SmearGate + BigramHash + SWA (val_bpb 1.1433) #76 (unixmadtoonslab / Will DePue)
• SWA — PR SubSixteen v2: Int6 QAT + MLP 3x + SWA + Sliding Window (val_bpb 1.1708) #69 (TevBenji)
• Late QAT — PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 (jfprincz), PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• Sliding window evaluation — PR Record: Sliding Window Eval (stride=64), val_bpb=1.1925 #50 (mattqlf / Matthew Li)
• Value Embeddings — PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• Partial RoPE — PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 (jfprincz), PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• LN Scale — PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315 (jfprincz), PR Record: 11L + Tight SWA + Shared VE128 + Partial RoPE + LN Scale + XSA4 (val_bpb: 1.1246) #374 (unnir)
• Legal TTT framework — PR [record bpb=1.195] sliding window + LoRA TTT #77 (samacqua); full-model variant in our PR Non-record submission: Depth Recurrence + Legal Score-First TTT (10L, 1.1532 BPB) #456
• Score-first protocol + SGD TTT — Our prior work (PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461)

Built on the parameter-golf starter code by Beren Millidge & Keller Jordan.

[MatoTeziTanka]

Community Review — Non-record: 11L + 30-Epoch Legal TTT (BPB 1.14252)

BPB: 1.14252 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 6edbca9ad123, file records/track_non_record_16mb/2026-03-22_11L_VE128_PartialRoPE_LegalTTT/train_gpt.py):

The TTT path at line 842 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=11, vocab=1024, code=71738 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=11, vocab=1024, code=71738 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.