Record: Score-First TTT + Multi-Order N-gram Backoff (3-seed mean val_bpb=0.9581)

[antaloaalonso]

Summary

3-seed mean val_bpb: 0.9581 (seeds 1337, 42, 7 → 0.9576 / 0.9581 / 0.9585)

Beats current leaderboard by combining two independently-validated techniques:

1. Score-First TTT — tokens scored under inference_mode before the model trains on them (compliant with #677)
2. Multi-Order N-gram Backoff (orders 2–7) with entropy-adaptive alpha mixing — backward-looking cache, no target-aware gating

Architecture

• 11L, 512d, GQA (8H/4KV), MLP 3×, U-Net skip connections
• LeakyReLU(0.5)² + XSA (all layers) + Value Residual + Gated Attention
• EMA(0.997), warmdown=3000, int6 per-row + zstd-16, ~15.7MB artifact

Results

Seed	val_bpb	Artifact
1337	0.9576	15,721,728 B
42	0.9581	15,702,393 B
7	0.9585	15,768,158 B
Mean	0.9581

Hardware: 8× H100 SXM 80GB, ~6406 steps in 600s (~93.67 ms/step)

Compliance

• TTT is score-first: each token evaluated under inference_mode before the model ever trains on it — no forward-looking information
• N-gram cache is backward-looking only: built from previously-scored tokens, not target-aware
• Training stays strictly within 600s wall-clock; all eval within allotted time
• No training data accessed at eval time

Reproduction

python3 data/cached_challenge_fineweb.py --variant sp1024
SEED=1337 TTT_ENABLED=1 NGRAM_CACHE=1 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

Full details: records/track_10min_16mb/2026-03-26_ScoreFirst_TTT_Ngram_Backoff/README.md

[MatoTeziTanka]

Community Review — Score-First TTT + Multi-Order N-gram Backoff

BPB: 0.9581 (3-seed mean) | Seeds: 3 (1337 / 42 / 7) | Artifact: ~15.7 MB | Compliance: FLAG

What this does: Combines a score-first TTT adapter (4-epoch AdamW on already-scored chunks) with a multi-order hashed n-gram backoff cache (orders 2-7, entropy-adaptive alpha mixing). Neural BPB is ~1.13; the drop to 0.9581 comes almost entirely from the n-gram cache during the sliding-window eval.

What I found in the code (head SHA 682797376f06e5c2297f4ffcc6fe45aaeba5c108, file records/track_10min_16mb/2026-03-26_ScoreFirst_TTT_Ngram_Backoff/train_gpt.py):

1. TTT ordering is correct. ttt_adapt() (lines 1226+) scores each chunk under torch.inference_mode() before the AdamW step updates weights — this part is consistent with Issue Illegal submissions megathread #677 / Invalid submissions due to information leakage during TTT #402 / A Field Guide to Valid Submissions #1017.

2. N-gram cache update is after scoring on the timing axis. In eval_val_sliding() the per-order ctx_counts / full_counts lookup and mixing at lines 1173-1189 happen before the np.add.at(...) update at lines 1197-1198. So on the "when does the update run" axis, this is score-first.

3. However, the lookup key on line 1164 is target-aware:

   full_key = ((ctx_hash ^ (tgt_np * ng_primes[ctx_w % len(ng_primes)])) & ng_mask).astype(np.int64)

   tgt_np = val_np[jv] (line 1163) is the true next-token at each position. That token is XORed into the hash that indexes full_tables[oi], and then full_counts / ctx_counts is used as the predicted probability for that same token (line 1179). So at scoring time the cache is not answering "what's the next-token distribution given the prefix?" — it's answering "is the true next token in this bucket?"

4. Empirical signature is consistent with the family bug. From logs/p23_s42.txt:

   • step:6403/20000 val_loss:1.9306 val_bpb:1.1434 (end of training, neural only)
   • final_int6_roundtrip val_loss:1.9190 val_bpb:1.1365 (after TTT, no cache)
   • final_int6_sliding_window val_loss:1.6178 val_bpb:0.9581 (TTT + n-gram cache, stride=64)

   The cache contributes ~0.178 BPB — larger than TTT's contribution — which is the pattern we've seen across the target-hashed n-gram family.

Questions/flags:

• Per @valerio-oai's ruling on PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779 (comment 4145781641, 2026-03-27), hashed n-gram caches that hash the target token into the lookup key are disallowed for leaking eval tokens. Mechanism in comment 4146407380. The full_key construction at line 1164 here matches that pattern: the hash is a function of the target token, so the lookup is implicitly P(target | prefix, target) rather than P(next | prefix). I believe this submission falls under the same ruling.

• One gentle note on the phrase "Score-First" in the title and README, because several PRs in the n-gram family have hit this same confusion: score-first ordering of update vs lookup (i.e. deferring update() until after score()) is necessary but not sufficient. The bug is not that the update runs too early — it's that the lookup key itself is a function of the target token, regardless of when the update runs. This code is score-first on the timing axis, but the key construction on line 1164 is the separate issue the ruling addresses.

• Per Issue A Field Guide to Valid Submissions #1017 Condition 1, p_t may depend only on the artifact and x_1...x_{t-1}. Because full_key is a function of x_t, the best_p_ng mixing at line 1189 depends on x_t, not just the prefix.

Real engineering worth preserving (regardless of what happens to the cache):

• The score-first TTT pipeline itself (lines 1226+) looks clean and carries over unchanged to a legal resubmission.
• 11L/512d with XSA all layers + LeakyReLU(0.5)^2 + VR + GA + EMA(0.997) + int6 per-row + zstd-16 → ~15.7 MB artifact, 6406 steps in 600 s is tight and well-tuned.
• Multi-order backoff with entropy-adaptive alpha is a nice framing. A context-only version (drop the tgt_np XOR from the full_key and instead emit a full-vocab probability vector built from the bucket's observed targets) would sidestep the ruling entirely and is a reasonable path forward.

Path forward: I'd be happy to take another look if the author drops the target token from the lookup key (or switches to a context-only cache that returns a full-vocab distribution from observed targets in each bucket) and re-runs the 3 seeds. The neural stack + TTT half of the submission appears independently solid.

Verdict: COMPLIANCE FLAG — target token is hashed into the n-gram lookup key (line 1164).

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE (without prejudice) pending removal of target-aware key construction in the n-gram cache, per @valerio-oai's #779 ruling. The TTT and neural architecture are independently sound and would be welcome in a resubmission.

---

Reviewed by @MatoTeziTanka — The Agora. CPU gauntlet skipped for this PR — review is code-level + log-level (the compliance question is answered by a single line of source). AI tooling: review drafted with Claude Code (Sonnet/Opus) using an internal review template; all citations, file paths, and compliance audits were verified against the PR's actual code at SHA 682797376f06e5c2297f4ffcc6fe45aaeba5c108.