11L XSA4 + SmearGate + BigramHash + SWA + RoPE50K (mean val_bpb=1.1565, 3 seeds)

[mahsumaktas]

Summary

Mean val_bpb: 1.1565 (3 seeds) | Best: 1.1538 (seed 1337) | Artifact: ~15.9 MB

23 GPU runs on 8xH100 SXM5. Systematic exploration of XSA, EMA vs SWA, depth recurrence, seq curriculum, LR/WD sweep, and MLP scaling.

Techniques

• 11 transformer layers + XSA on last 4 layers
• SmearGate + BigramHash(2048) + OrthoInit
• INT6 per-row quantization + zstd-22 + FP16 tied embedding + Late-K FP16
• SWA every 50 steps (fp32 accumulation) — bf16 causes catastrophic loss
• Muon WD=0.04 + grad clip 0.3 + RoPE base 50K
• Overtone SVD init + Phase-transition residual mixing
• MLP 2.75x — sweet spot (3x exceeds 16MB with SmearGate at 11L)

Results (3 seeds)

Seed	Sliding BPB	Post-quant BPB	Artifact
1337	1.1538	1.1766	15.99 MB
42	1.1565	1.1790	15.87 MB
7	1.1593	1.1820	15.93 MB
Mean	1.1565

Key Findings from 23 Runs

• EMA(0.997) causes 0.14 BPB quant gap — SWA far better for our stack
• 11L MLP 3x exceeds 16MB with SmearGate+BigramHash
• SmearGate removal loses more than MLP 3x gains — bigram context matters
• XSA needs GQA-compatible v expansion (repeat_interleave, bug found and fixed)
• Seq curriculum doesn't work — SWA checkpoint incompatibility across seq lengths
• Depth recurrence works but dim=640 too narrow; dim=768+ exceeds 16MB
• Higher LR (0.03) improves BPB but worsens compression (larger weights)
• Late QAT (75%) reduces quant gap (0.023 -> 0.006) but fewer steps

Run command

NUM_LAYERS=11 XSA_LAST_N=4 BIGRAM_VOCAB_SIZE=2048 \
TRAIN_SEQ_LEN=2048 TRAIN_BATCH_TOKENS=524288 MLP_MULT=2.75 \
MATRIX_LR=0.025 SCALAR_LR=0.025 TIED_EMBED_LR=0.035 \
MUON_MOMENTUM=0.99 MUON_WEIGHT_DECAY=0.04 WARMDOWN_ITERS=3000 \
SWA_ENABLED=1 SWA_EVERY=50 ROPE_BASE=50000 EVAL_STRIDE=64 \
python3 -m torch.distributed.run --standalone --nproc_per_node=8 train_gpt.py

Test plan

• Runs reproducibly on 8xH100 SXM in under 10 minutes
• Artifact under 16 MB (15.87-15.99 MB)
• 3-seed validation (mean 1.1565, std 0.0028)
• Sliding window eval completes within 10 minutes

Built with Claude Code

[MatoTeziTanka]

Community Review — 11L XSA4 + SmearGate + BigramHash + SWA + RoPE50K (mean val_bpb=1.1565, 3 seeds)

BPB: 1.1565 | Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

What I found in the code (head SHA 871fd864e7b3, file records/track_10min_16mb/2026-03-20_CombinedSOTA_INT6_SmearGate_BigramHash_SWA/train_gpt.py):

Static code review found no TTT adaptation function, no SLOT optimization loop, no n-gram-cache class, and no pre-quant val-token fine-tune. The eval path uses the standard sliding-window stride-64 pattern. The submission is a pure-neural architecture iteration on the standard SP1024/SP4096/SP8192 baseline.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=10, vocab=1024, code=66074 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the classification pass — this looks like a clean pure-neural iteration on the standard baseline.

Auto-classification caveat: this review was drafted by the AST-based classifier. If there's a non-standard eval mechanism (logit postprocessing, hedge mixing, etc.) that I missed because it's factored into a helper file or a non-standard function name, please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=10, vocab=1024, code=66074 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.