Record: SwiGLU + MLP 3x + Int6 + LoRA TTT, val_bpb=1.1670 (8xH100)

[polarizedfortnite-cpu]

No description provided.

[MatoTeziTanka]

Community Review — Record: SwiGLU + MLP 3x + Int6 + LoRA TTT, val_bpb=1.1670 (8xH100)

Compliance: LOOKS CLEAN — legal score-first-per-chunk TTT (PR #1413 pattern)

PR #81 — flux_depth_recurrence_int6

Head SHA: 942a986
Claimed BPB: 1.1670 (raw 1.2004, TTT improvement 0.0334)
Techniques: SwiGLU, 10 layers, int6 quantization, zstd22, LoRA TTT eval, QAT last 25%, document-isolated eval

---

Check 1 — N-gram family bug (CLOSE trigger: target token in hash key)

No n-gram or hash-based models present anywhere in train_gpt.py. No BigramHash, no n-gram LUT, no token-keyed hash lookup. This check does not apply. CLEAN.

Check 2 — Pre-Quant TTT (CLOSE trigger: multi-epoch AdamW on val_tokens without score-first)

The TTT optimizer is torch.optim.Adam (not AdamW — no weight decay parameter). Single-pass per document, no epoch loop. The optimizer is reset per document batch via _reset_ttt_optimizer / lora.reset() before each document group. No multi-epoch loop over val_tokens. CLEAN.

Check 3 — Legal TTT / Score-first-per-chunk

The TTT loop in eval_val_ttt_lora (lines 967–1016) follows score-first discipline correctly:

1. Forward pass runs on chunk ci (lines 991–996).
2. Score is accumulated immediately after forward, unconditionally (lines 998–1006, _accumulate_bpb called before any gradient step).
3. Train step only runs if needs_train (lines 1008–1016), which requires ci < nc - 1 — i.e., the model only trains on chunks that are not the final chunk of a document. The final chunk's loss is scored but never trained on.

This is the correct legal pattern matching PR #1413. Score happens before training for every chunk. CLEAN.

Check 4 — Scored-region SLOT

No scored-region manipulation detected. The BPB accumulation (_accumulate_bpb) operates over the natural document boundaries found by _find_docs. No selective region skipping or windowed re-scoring of only favorable regions. Eval covers all val documents assigned to each rank. HOLD not triggered. CLEAN.

Verdict: LOOKS CLEAN — legal TTT implementation matching the PR #1413 (dexhunter) pattern: each chunk scored under torch.no_grad() before optimizer.step(), with is_last_chunk guard preventing adaptation on the final scored chunk.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). TTT implementation follows the legal score-first discipline.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.