Record: Casefold V4 + AttnOutGate + Multi-Phase Global SGD TTT — val_bpb 1.05733 (3-seed mean)

[dexhunter]

Summary

• val_bpb: 1.05733 (3-seed mean, std 0.00035) | 3.04721 nats | ~15.21 MB | 8xH100 SXM, 600s | Phased TTT
• Stacks per-head Attention Output Gate (PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667 @MarioPaerle) and SmearGate on top of our Casefold V4 + Multi-Phase Global SGD TTT base (PR Record: Casefold V4 Tokenizer + Multi-Phase Global SGD TTT — val_bpb 1.05970 (3-seed mean) #1670)
• Both gates are zero-initialized (identity at init) and add 1,056 + 13 parameters total
• Clears the 0.005-nat bar vs the casefold leader (PR Record: Casefold Tokenizer + Parallel Residuals + Systems Optimization — val_bpb 1.0639 (3-seed mean) #1585) by 3.4x; improves on PR Record: Casefold V4 Tokenizer + Multi-Phase Global SGD TTT — val_bpb 1.05970 (3-seed mean) #1670 by -0.00237 BPB / -0.00680 nats

Key Innovation

Attention Output Gate (from PR #1667 @MarioPaerle)

Per-head multiplicative gate on the attention output, weight-initialized to zero so all heads pass through at scale 1.0 at init. Implemented as an inline-safe op with .contiguous() barriers so it composes with fullgraph=True torch.compile:

def _apply_attn_out_gate_inline(y, x_orig, gate_w):
    gate_in = x_orig[:, :, :12].contiguous()
    gate = (2.0 * torch.sigmoid(F.linear(gate_in, gate_w.to(gate_in.dtype)))).contiguous()
    return y * gate.unsqueeze(-1)

• 12 x 8 heads x 11 layers = 1,056 new parameters
• Applied in all three attention paths (standard, parallel-residual, depth-recurrent)
• Negligible throughput cost (<2%)

SmearGate

Input-dependent per-channel residual mixer (13 parameters). Mixes the current token with the previous token (strictly backward-looking, so causal). Zero-initialized lambda means it starts as the identity on the residual stream.

Casefold V4 + Multi-Phase Global SGD TTT (from PR #1670)

Same tokenizer and TTT protocol as PR #1670. Casefold legality is pending organizer review at Issue #1604.

Results

Seed	Pre-TTT BPB	Post-TTT BPB	val_loss (nats)	Artifact
42	1.06633	1.05693	3.04604	15,936,269
0	1.06674	1.05730	3.04712	15,937,514
1234	1.06714	1.05777	3.04846	15,938,772
Mean	1.06674	1.05733	3.04721	15,937,518
Std		0.00035

Record Comparison

Submission	val_bpb	Delta BPB vs prior	Delta nats vs prior
Merged SOTA (PR #1493)	1.08100	-	-
PR #1530 @samacqua	1.07336	-	-
PR #1585 @codemath3000 (casefold leader)	1.06390	-	-
PR #1670 @dexhunter (casefold v4 + phased TTT)	1.05970	-0.00420 vs #1585	-
This	1.05733	-0.00657 vs #1585 / -0.00237 vs #1670	-0.01697 vs #1585 / -0.00680 vs #1670

Compliance (Issue #1017 Track B)

• Condition 1 (Causality): Sliding-window eval is strictly causal. AttnOutGate is a position-local sigmoid of the current token's channels. SmearGate mixes with the previous token only (backward-looking).
• Condition 2 (Normalized distribution): Standard softmax over full vocab. Gates modulate hidden states, not logits.
• Condition 3 (Score before update): Each phase fully scored under torch.no_grad() before any SGD update.
• Condition 4 (Single pass): Each token scored exactly once per phase.

AttnOutGate and SmearGate are pure training-time architectural additions with trained weights; no eval-time effect beyond the trained weights. Analogous gating constructs have precedent in skip gates (PR #549 family), parallel-lane gating (PR #1204 family), and SmearGate (modded-nanogpt).

Casefold tokenizer: pending organizer review at Issue #1604. The tokenizer is retrained from scratch on casefolded FineWeb — it is not a modification of the standard SP8192 tokenizer. Byte-level BPB is computed correctly via the sentencepiece piece table on the original (non-casefolded) validation bytes.

Lineage

PR #1530 (@samacqua) -> PR #1626 (@dexhunter, multi-phase SGD TTT) -> PR #1670 (@dexhunter, casefold v4 + phased TTT) -> this PR (+ AttnOutGate from PR #1667 @MarioPaerle + SmearGate)

Credits

• @samacqua — PR Record: Varlen attention + fused MLP + doc-independent TTT (1.07336) #1530 base architecture (11L/512d/4x MLP, depth recurrence, parallel residuals, MuonEq-R, GPTQ SDClip, VarLen attention, fused MLP)
• @MarioPaerle — Attention Output Gate (PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667), SmearGate reintroduction to parameter-golf
• @KellerJordan — SmearGate concept (modded-nanogpt)
• @mikeapedia — Casefold tokenizer concept (PR Record: Custom Casefold Tokenizer — 1.0668 BPB #1578)
• @romeerp — Phased TTT concept (PR Record: VarLenAttn + PhasingTTT - val_bpb 1.0728 (3-seed mean) #1610)
• @abaybektursun — Score-first TTT framework (PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549, merged precedent)
• @dexhunter — Casefold V4 retokenization + BOS fix, multi-phase global SGD TTT, trimmed GPTQ tuning, inline-safe gate implementation

Test plan

• 3-seed training on 8xH100 SXM (seeds 42, 0, 1234) — all under 600s train, all under 600s eval
• All artifacts under 16 MB (max 15,938,772 B / 16,000,000 B)
• Code-size and hyperparameter consistency verified across all 3 seed logs
• Score-before-update ordering held across all 3 phases (phase boundaries [666, 1333, 2000])
• Clears the 0.005-nat bar vs merged SOTA and vs casefold leader
• Judges verify reproducibility
• Judges confirm casefold legality (Issue Clarify which text normalizations are allowed for custom tokenizers #1604)