Record: SP8192 MP-SGD TTT (4 phases) + QK-Gain 5.25 — val_bpb 1.07217 (3-seed mean)

[yahya010]

Summary

3-seed mean val_bpb = 1.07217 (std 0.00114) on Track A (10min/16MB), beating merged SOTA (1.0810, @bigbag PR #1493) by 0.00883 BPB / ~0.00612 nats, clearing the 0.005-nat / ~0.0072 BPB threshold.

This PR extends the Multi-Phase Global SGD + Phased LoRA TTT stack from open PR #1700 (@jorge-asenjo) with two hyperparameter tunes:

1. PHASED_TTT_NUM_PHASES=4 (up from PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700's 3) — one additional multi-phase SGD adaptation pass at eval time. PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700's 3-phase eval used ~352s out of the 600s budget; adding a 4th phase adds ~25-70s and fits comfortably (observed eval times: 349-396s across all 3 seeds).
2. QK_GAIN_INIT=5.25 (up from PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700's 5.0) — matches merged SOTA PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493.

train_gpt.py is byte-identical to PR #1700's. The only changes are the two environment variables above.

Results (8×H100 SXM, torch 2.9.1+cu128)

Seed	val_bpb	artifact bytes
42	1.07310	15,933,641
0	1.07090	15,938,690
1234	1.07250	15,930,318
mean	1.07217
std	0.00114

Comparison with PR #1700 (same code, same seeds, phases=3):

Seed	#1700 val_bpb	this PR (phases=4)	delta
42	1.07332	1.07310	-0.00022
0	1.07115	1.07090	-0.00025
1234	1.07211	1.07250	+0.00039
mean	1.07219	1.07217	-0.00002

The extra 4th phase yields a very small (within-noise) mean delta versus PR #1700. The submission clears the merged-SOTA threshold through PR #1700's stack; the novelty this PR contributes is (a) independent reproduction on different hardware, (b) verification that an additional 4th MP-SGD phase is legal and fits the budget, and (c) the QK-Gain 5.25 import from merged SOTA.

Compliance (Issue #1017 Track A)

• Condition 1 (Causality): standard causal attention
• Condition 2 (Normalized): standard softmax over full vocab
• Condition 3 (Score-before-update): each of the 4 phases is scored under torch.no_grad() before any SGD update
• Condition 4 (Single pass): each token is scored exactly once per phase, no rescoring across phases
• No SLOT, no n-gram cache, no ETLB, no pre-quant TTT
• All 3 artifacts < 16 MB (max 15,938,690 B)
• All 3 trainings < 600s (wallclock-capped at 596s)
• All 3 evals < 600s (348s / 349s / 396s)

Reproduction

pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
pip install flash_attn_3 --no-deps --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291/
pip install brotli sentencepiece python-minifier numpy huggingface-hub zstandard einops ninja

MATCHED_FINEWEB_REPO_ID=kevclark/parameter-golf \
  python3 data/cached_challenge_fineweb.py --variant sp8192

for seed in 42 0 1234; do
  SEED=$seed \
  QK_GAIN_INIT=5.25 \
  PHASED_TTT_ENABLED=1 PHASED_TTT_PREFIX_DOCS=2000 PHASED_TTT_NUM_PHASES=4 \
  MLP_CLIP_SIGMAS=12.0 ATTN_CLIP_SIGMAS=13.0 EMBED_BITS=7 EMBED_CLIP_SIGMAS=15.0 \
  MATRIX_LR=0.026 GPTQ_RESERVE_SECONDS=4 GPTQ_CALIBRATION_BATCHES=16 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py
done

Attribution

Full credit to @jorge-asenjo (PR #1700) for the base stack: Multi-Phase Global SGD TTT + Phased LoRA TTT, VarLen flash attention, fused Triton MLP, SP8192 pipeline, and all quantization/optimization choices. train_gpt.py in this submission is byte-identical to PR #1700's.

Extended lineage:

• @bigbag — PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 (merged SOTA): QK-Gain 5.25
• @samacqua — PR Record: Varlen attention + fused MLP + doc-independent TTT (1.07336) #1530 (VarLen + fused MLP base)
• @clarkkev — PR Record: SP8192 + GPTQ Embeddings + Depth Recurrence + MuonEq-R + SDClip — val_bpb 1.08563 (5 seed mean) #1394 (SP8192 + GPTQ)
• @abaybektursun — PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 (legal TTT framework)
• @dexhunter — PR Record: VarLen Attention + Fused MLP + Multi-Phase Global SGD TTT — val_bpb 1.07193 (3-seed mean) #1626 (Multi-Phase Global SGD TTT concept)

Test plan

• 3-seed training on 8×H100 SXM (seeds 42, 0, 1234)
• All artifacts under 16 MB
• All trainings and evals within budget
• Score-first TTT preserved exactly as in PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700

🤖 Generated with Claude Code