Non-record: 33.6M Int5 GPTQ + Legal s_0-only TTT (val_bpb=1.1182)

[ibarrajo]

Summary

val_bpb: 1.1182 (s_0 score only, single seed — additional seeds pending)

Resubmission addressing PR #991's closure. Key fix: reports ONLY the cumulative s_0 score from the first scoring pass. No post-TTT re-evaluation. No temperature calibration on re-scored tokens.

What changed from PR #991

• Removed illegal post-TTT re-eval — PR Record: 33.6M Int5 GPTQ + Score-First TTT (val_bpb=1.1145, 3-seed) #991 reported s_1 (re-scored after training). This PR reports s_0 (scored before training on each chunk).
• Removed temperature calibration — T=0.98 on re-scored tokens was illegal. Removed entirely.
• Increased pruning 3%→5% — ensures artifact <16MB across all seeds.
• All assertions pass: train+gptq < 600s, artifact < 16MB, eval < 600s.

Results

Metric	Value
Base (no TTT, sliding window)	1.1246
Legal s_0 TTT	1.1182
TTT improvement	-0.0064
Artifact	15,535,414 bytes (465KB headroom)
Train+GPTQ	593.8s / 600s
Eval	~414s / 600s

Rule compliance

• s_0 only — each token scored BEFORE training, cumulative loss reported
• No re-scoring — no second eval pass after TTT
• No temperature calibration — removed
• GPTQ within training budget — 593.8s total
• Artifact < 16MB — 15.5MB with 465KB headroom
• Eval < 600s — ~414s
• Assertions enforce all constraints at runtime

Architecture

33.6M params (d=576, MLP 3.5x=1792, 11L), int5 GPTQ, XSA-all(11), BigramHash(8192), EMA(0.997), 5% magnitude pruning. Based on PR #576 by @cmcdnd.

🤖 Generated with Claude Code

[ibarrajo]

Updated with B6 run: val_bpb = 1.1179 (s_0 TTT, seed 1337).

Changes from previous:

• 10% magnitude pruning (5% and 7% both overflowed 16MB on some seeds)
• BigramHash reduced 8192→6144 for artifact headroom
• Artifact: 15.5MB (516KB headroom)
• Train+GPTQ: 593.7s / 600s
• inference_temp = 1.0 (no temperature calibration)
• Reports s_0 only — no re-scoring

Non-record submission (SOTA is 1.1147).

[ibarrajo]

Updated results summary — B6 is our best legal approach at val_bpb=1.1179 (s_0 TTT, seed 1337).

We tested 12 approaches this session exploring the design space:

Approach	Best BPB	Key technique	Finding
B6	1.1179	s_0 TTT, 8KV, int5, 10% prune	Best overall
J (GQA+SLOT)	1.1240	GQA 4KV + SLOT eval	GQA hurts TTT/SLOT capacity
I (GQA+TTT)	1.1264	GQA 4KV + LZMA + TTT	Faster steps but weaker TTT
E (SLOT)	1.1179	SLOT delta opt, QK-Gain 4.0	Ties B6, QK-Gain 4.0 slightly hurts
L (online GPTQ)	1.1349	Online Hessian accumulation	Overhead > savings
D (TurboQuant)	1.1521	Mixed int4/int5 per role	Weight quant ≠ activation quant
H (focal loss)	1.1460	P2 focal loss γ=2	Too aggressive, hurts training

Key findings:

• GQA (4 KV heads) improves base BPB but reduces TTT/SLOT capacity
• SLOT gives ~0.009 BPP improvement (similar to TTT)
• Online Hessian GPTQ adds too much per-step overhead to be net positive
• Fused Triton MLP kernel didn't help because torch.compile already fuses the ops

All approaches use GPTQ within the training budget. Non-record (SOTA is 1.1147). Needs 2 more seeds for statistical validation.

[MatoTeziTanka]

Community Review — Non-record: 33.6M Int5 GPTQ + Legal s_0-only TTT (val_bpb=1.1182)

Compliance: LOOKS CLEAN — legal score-first-per-chunk TTT (PR #1413 pattern)

GPTQ on train data, score-first TTT with is_last_chunk guard. Clean.

Verdict: LOOKS CLEAN — legal TTT implementation matching the PR #1413 (dexhunter) pattern: each chunk scored under torch.no_grad() before optimizer.step(), with is_last_chunk guard preventing adaptation on the final scored chunk.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). TTT implementation follows the legal score-first discipline.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.