FP8 + Arithmetic Coding + SWA (1.1511 BPB)

[cruz-andr]

Summary

• val_bpb: 1.1511 on 8xH100, 10-minute wallclock
• FP8 training via TransformerEngine (E4M3 fwd / E5M2 bwd) for ~1.3-1.5x throughput
• Custom arithmetic coder replacing zstd-22, using per-tensor empirical histograms approaching Shannon entropy
• Early SWA start (step 4500) for more weight averaging during warmdown
• TF32 matmul precision for remaining non-FP8 ops
• Architecture: 10L, 512dim, MLP 3x, SmearGate + BigramHash(10240)

Run Command

NUM_LAYERS=10 MODEL_DIM=512 MLP_MULT=3 VAL_LOSS_EVERY=0 MAX_WALLCLOCK_SECONDS=600 \
TTT_ENABLED=0 BIGRAM_VOCAB_SIZE=10240 WEIGHT_DECAY=0.04 \
DATA_PATH=/dev/shm/fineweb10B_sp1024/ \
torchrun --standalone --nproc_per_node=8 train_gpt.py

Built On

• SOTA submission by thwu1 (1.1428 BPB): Int5-MLP + BigramHash(10240) + SWA
• LoRA TTT submission by samacqua (1.1928 BPB)

[MatoTeziTanka]

Community Review — FP8 + Arithmetic Coding + SWA (1.1511 BPB)

Compliance: LOOKS CLEAN — legal score-first-per-chunk TTT (PR #1413 pattern)

PR #538 — FP8 + Arithmetic Coding + TTT (1.1511 BPB)

Author: cruz-andr
Head SHA: a0b5478
File: records/track_10min_16mb/2026-03-20_FP8_ArithCoding_TTT/train_gpt.py (1887 lines)

---

Check 1: N-gram family bug (CLOSE trigger)

Result: CLEAN

The n-gram component is BigramHashEmbedding (class at line 678). The hash function (line 695):

out[..., 1:] = torch.bitwise_xor(36313 * t[..., 1:], 27191 * t[..., :-1]) % mod

This mixes t[i] (current input position) and t[i-1] (previous input position) — both are context tokens. Critically, bigram_hash is called inside forward() with input_ids only (line 786), not target_ids. The target is never present in the hash key. This matches the legal BigramHash pattern explicitly exempted from the CLOSE rule.

---

Check 2: Pre-Quant TTT bug (CLOSE trigger)

Result: CLEAN — Legal score-first TTT

The TTT evaluation is in eval_val_ttt_lora (line 1311). The per-chunk loop (lines 1364–1409) follows this exact order:

1. Forward pass on chunk ci → produces ptl (per-token loss tensor)
2. Score accumulation via _accumulate_bpb (lines 1393–1400) — reads current chunk loss into loss_sum/byte_sum before any gradient update
3. Train step (lines 1402–1409) — only executes when needs_train is True (i.e., not the final chunk), using ptl from the same chunk

Verdict: LOOKS CLEAN — legal TTT implementation.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source. If this review misread your code, please call it out so I can re-audit manually.