Record: 0.4416 BPB -- Complementary Training + Backoff N-gram Mixer

[pentxayc]

Summary

• 0.4416 BPB (3-seed mean, std 0.0001)
• Seeds: 42 (0.4416), 1337 (0.4416), 2024 (0.4417)
• 11L transformer (26.99M params) with VRL, LeakyReLU(0.5)², XSA-4
• Artifact: 15,875,857 bytes (under 16MB)
• Training: 4648 steps in 600s on 8xH100 SXM
• Eval: 458s / 600s budget

Key Innovation: Complementary Training

During training, tokens predictable by bigram statistics receive lower loss weight (COMPLEMENT_ALPHA=0.5). The model specializes on tokens that n-gram caches can't predict — novel word choices, long-range dependencies, semantic surprises.

This enables higher eval-time n-gram alpha (20-75% vs standard 5-60%) because the model is deliberately weak where n-grams are strong. The synergy:

Config	BPB
Base model only	1.139
+ Standard backoff (alpha=0.05)	0.700
+ Complementary training + alpha=0.20	0.442

Eval Stack

• BackoffNgramMixer: orders 2-10, 4M flat hash buckets, greedy cascade (highest matching order wins)
• Entropy-adaptive alpha: 0.20 + 0.55 * sigmoid(2*(H - 3.0)) — per-token blending based on model uncertainty
• AdamW TTT: lr=5e-4, 4 epochs/chunk, Polyak EMA 0.998, freeze first 9/11 blocks
• Sliding window: stride=64

Legality

1. Complementary training uses only training-data bigram statistics. No validation data during training.
2. N-gram cache built from already-scored tokens only (backward-looking, score-first).
3. Alpha formula is a fixed function of model entropy — target-independent, committed before scoring.
4. TTT is standard score-first legal TTT.
5. Committed distribution: (1-α)·P_neural + α·P_ngram — proper mixture, all tokens have nonzero probability.

Credits & Acknowledgments

This submission builds on techniques from several prior PRs:

• Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779 (@BackoffNgramMixer author) — BackoffNgramMixer architecture, flat hash table design, entropy-adaptive alpha formula, drift-free TTT recipe. Our backoff cascade and alpha blending directly adapt their approach.
• Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 — AdamW TTT with Polyak EMA, byte-weighted loss, adaptive cosine LR scheduling.
• 11-Layer Int6 + WD=0.04 + SWA + FA3 (val_bpb: 1.1318) #198 — Base 11L architecture with XSA, SWA/EMA, weight decay tuning.
• Record: 11L XSA + EMA + Int6 MLP3x + WD=0.04 (val_bpb: 1.1271) #287 — EMA + XSA co-dependency discovery, int6 MLP quantization.
• Non-record: Value Residual (-0.015 BPB) + Gated Attention (-0.003 BPB) with ablations #413 — Value Residual Learning (VRL) technique (-0.01 BPB).
• Record: 11L LeakyReLU² + Full GPTQ + QAT Alignment (val_bpb: 1.1204) #535 — LeakyReLU(0.5)² activation function.
• Record: int5 GPTQ + Soft-Round QAT (3-seed mean 1.1162) #606/Record: Residual Input Mixing + mixed int6 GPTQ + grouped TTT + MLP 3.5x #615 — TTT recipe improvements (AdamW, freeze blocks, Polyak averaging).

The novel contribution is complementary training: reweighting the training loss by bigram predictability so the neural model specializes on tokens the n-gram cache can't handle, enabling significantly higher eval-time n-gram weight.

Test plan

• Seed 42: 0.4416 BPB
• Seed 1337: 0.4416 BPB
• Seed 2024: 0.4417 BPB

🤖 Generated with Claude Code

[MatoTeziTanka]

Community Review — Record: 0.4416 BPB -- Complementary Training + Backoff N-gram Mixer

BPB: 0.4416 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA d9cf80d9d26e, file records/track_10min_16mb/2026-03-26_ComplementaryBackoff_0.4416/train_gpt.py):

The TTT path at line 1029 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.25s, dim=512, layers=11, vocab=1024, code=94053 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.25s, dim=512, layers=11, vocab=1024, code=94053 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.