[Record Candidate] SP8192 · GatedAttn + Phased TTT + LQER · 10 min / 16 MB

[IanniMuliterno]

This PR combines three components from existing accepted PRs with a new integration and QuantGate export path:
gated attn + quantgate (lineage #1769, Qwen (arXiv:2505.06708))
phased TTT+global SGD (lineage PR #1727)
mixed GPTQ + LQER (lineage #1855 )

Score-first compliance is preserved throughout: tokens are always scored before any update that could have seen them, both for the local LoRA adapter updates (chunk level) and the
global SGD updates (phase level).

Architecture

• Tokenizer: SP8192 sentencepiece, vocab 8192
• Layers: 11, dim 512, 8 heads / 4 KV heads
• Depth recurrence: loop over layers 3–5, NUM_LOOPS=2
• Parallel residual from layer 7
• GatedAttn on all layers (GATED_ATTN_ENABLED=1, N(0, 0.01) init)
• QK gain init: 5.25, EMA decay: 0.9965

Eval path

Primary scored metric: quantized_ttt_phased

Eval sequence:

1. pre-quantization post-ema
2. quantized
3. quantized_ttt_phased — primary scored path

Full 3-seed results (seeds 42 / 314 / 999, 8×H100, 10-min wall clock) will be added as train logs once the runs complete.

Reproduction

See records/track_10min_16mb/2026-04-29_SP8192_AttnGate_PhasedTTT_LoRA_LaCT/README.md for full setup, smoke test command, and 8×H100 launch instructions.

SEED=42 bash run.sh 2>&1 | tee logs/seed42.log
SEED=314 bash run.sh 2>&1 | tee logs/seed314.log
SEED=999 bash run.sh 2>&1 | tee logs/seed999.log

References

• PR Record: SP8192 MP-SGD TTT (4 phases) + QK-Gain 5.25 — val_bpb 1.07217 (3-seed mean) #1727 — phased TTT + global SGD controller
• PR Record: SP8192 + CaseOps + GatedAttn + QuantGate + Loop4-5 + PhasedTTT + MLPClip12 — val_bpb 1.06453 (5-seed mean) #1769 — dense GatedAttn + QuantGate
• PR Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855 — mixed GPTQ + LQER export
• Zhang et al., Test-Time Training Done Right, arXiv:2505.23884