Record: 11L XSA4 + Tight SWA + FA3 + Two-Phase TTT (val_bpb=1.1216)#415

Closed

EthanYangTW wants to merge 17 commits intoopenai:mainfrom

EthanYangTW:submission/fa3-twophase-ttt

Commits on Mar 21, 2026

Commits on Mar 22, 2026

Add #315/#388 full stack: 11L, XSA4, Partial RoPE, LN Scale, EMA, Late QAT, TTT
EthanYangTW
committed
Fix speed and artifact size: disable FA3, reduce BigramHash, EMA every 10 steps
EthanYangTW
committed
Reduce BigramHash to 2048, increase pruning to 10% to fit under 16MB
EthanYangTW
committed
Remove BigramHash, increase pruning to 15% — must fit under 16MB
EthanYangTW
committed
New record: 11L XSA4 + Tight SWA + TTT (based on PR #374)
EthanYangTW
committed
Fix TTT: compile + DDP + 3 epochs + batch 64 for speed
EthanYangTW
committed
Fix FA3 import: add fallback to flash_attn and SDPA
EthanYangTW
committed
New record: 11L XSA4 + Tight SWA + Two-Phase TTT (1.1258 BPB)
EthanYangTW
committed
Update: FA3 Hopper + aggressive two-phase TTT (val_bpb=1.1216)
EthanYangTW
committed