Record: 11L XSA4 + EMA + Batch524K + zstd fallback (val_bpb: 1.1357)

[dennisimoo]

Record: 11L XSA4 + EMA + Batch524K + zstd fallback (val_bpb: 1.1357)

val_bpb: 1.1357 (sliding window, stride=64) | 15.67 MB | 8xH100 SXM, 600s

Key changes

Change	Impact
TRAIN_BATCH_TOKENS=524288	Better fixed-budget step count than the larger-batch 11-layer XSA+EMA setting
SDPA fallback for flash_attn_interface	Runs cleanly when FA3 Python bindings are unavailable in the official image
torch.compile behind an env flag	Reliable eager smoke tests, faster compiled full run
zstd Python-or-CLI fallback	Keeps int6 export under 16MB without depending on a specific Python package in the image

Results

Metric	Value
Pre-quant val_bpb	1.1529
Int6 roundtrip val_bpb	1.1580
Int6 sliding val_bpb (s64)	1.1357
Steps completed (600s cap)	8,202
Step time	73.37ms
Model params	26,829,913
Artifact size	15,669,953 bytes

Single-seed run. Below the current merged README SOTA (1.1428).

Run command

NUM_LAYERS=11 MODEL_DIM=512 NUM_HEADS=8 NUM_KV_HEADS=4 MLP_MULT=3 \
TRAIN_BATCH_TOKENS=524288 TRAIN_SEQ_LEN=2048 EVAL_SEQ_LEN=2048 EVAL_STRIDE=64 \
BIGRAM_VOCAB_SIZE=2048 BIGRAM_DIM=128 \
XSA_LAST_N=4 EMA_ENABLED=1 EMA_DECAY=0.997 SWA_ENABLED=0 TTT_ENABLED=0 \
MUON_WD=0.04 ADAM_WD=0.04 MATRIX_LR=0.025 SCALAR_LR=0.025 TIED_EMBED_LR=0.035 \
MUON_MOMENTUM=0.99 MUON_MOMENTUM_WARMUP_START=0.92 MUON_MOMENTUM_WARMUP_STEPS=1500 \
WARMDOWN_ITERS=3000 WARMUP_STEPS=20 ENABLE_TORCH_COMPILE=1 \
ITERATIONS=9000 MAX_WALLCLOCK_SECONDS=600 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

[mohosy]

nice score, the zstd fallback is smart for portability. how many steps did you get in 600s? trying to see if theres a speed difference between fa2 and fa3 setups

[dennisimoo]

8202 steps in ~600s, 73.37 ms/step avg from the training loop. I didn’t do a clean FA2 vs FA3 benchmark here, so I wouldn’t read too much into it beyond that.

[mohosy]

@dennisimoo 73ms per step on 8x is solid, we were geting similar numbers. good to know fa2 isnt holding us back much

[MatoTeziTanka]

Community Review — Record: 11L XSA4 + EMA + Batch524K + zstd fallback (val_bpb: 1.1357)

BPB: 1.1357 | Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

What I found in the code (head SHA ca19512a49e8, file records/track_10min_16mb/2026-03-21_11L_XSA4_EMA_B524K_ZstdFallback/train_gpt.py):

Static code review found no TTT adaptation function, no SLOT optimization loop, no n-gram-cache class, and no pre-quant val-token fine-tune. The eval path uses the standard sliding-window stride-64 pattern. The submission is a pure-neural architecture iteration on the standard SP1024/SP4096/SP8192 baseline.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=9, vocab=1024, code=67690 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the classification pass — this looks like a clean pure-neural iteration on the standard baseline.

Auto-classification caveat: this review was drafted by the AST-based classifier. If there's a non-standard eval mechanism (logit postprocessing, hedge mixing, etc.) that I missed because it's factored into a helper file or a non-standard function name, please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=9, vocab=1024, code=67690 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.