Record: DualClock PPM token-context mixture — 1.0803 BPB seed 42

[okezue]

Summary

Score-first eval-time mixture over the neural model and two causal PPM experts (global stream + document-local) using a fixed-share Bayesian mixer. GPU-vectorized via FNV rolling hashes, hash-bucketed count tables, and a prefix-rank counter for chunk-local causal scoring.

Final val_bpb on seed 42: 1.080326 (8×H100 SXM, brotli artifact 15,977,914 bytes).

Folder: `records/track_10min_16mb/2026-04-27_DualClockPPM_TokenContextMixture/`

Result breakdown

component	val_bpb	eval_time
pre-quantization post-EMA	1.087045	6.8s
quantized	1.098230	8.6s
quantized + sliding window	1.081633	91.1s
quantized + TTT + DualClock mixture	1.080326	361.9s

Final mixture weights converged to `[0.99950, 0.00035, 0.00015]` over (neural, global PPM, local PPM).

Implementation

At evaluation time, every chunk is scored under `torch.no_grad()` before any TTT update on that chunk. As each chunk is scored, the per-token neural NLL is gathered across DDP ranks via `all_reduce(MIN)` on a buffer initialized to `+∞`, since each rank scores a disjoint window subset and the score buffer is later consumed by a single deterministic mixture pass that runs on every rank.

Per chunk, we construct two causal PPM experts. Each expert maintains, for every order `k = 1..K`, a hash-bucketed count table `cnt[k]` of shape `(B·V,)` (int32) holding continuation counts and a context-mass table `tot[k]` of shape `(B,)`. The order-`k` context for a target at chunk position `i` is the FNV-rolling hash of the last `k` source tokens, computed in a vectorized loop over `k`. The hash modulo `B` gives the bucket id; the resulting `(ctx_hash, target_token)` pair indexes into `cnt[k]`. Predictions are produced by the recursive Dirichlet-backoff smoothing chain `q_k(a) = (cnt_k(a) + α·q_{k-1}(a)) / (tot_k + α)` where `q_0` is a causal running unigram prior over the full SP8192 vocabulary, so each expert is by construction a fully normalized distribution over the official token alphabet.

Within a chunk, the strictly causal contribution from already-seen positions is enforced by a vectorized prefix-rank counter (`dc_prank`): for the `n` positions in the chunk it returns, for each position, the count of identical earlier-position keys, which is the chunk-local correction that lets us score the entire chunk in parallel without ever leaking a token's identity into its own predicted probability. After the chunk's score is finalized, the count tables and the unigram base prior are updated with that chunk's tokens via GPU `index_add`. The global expert never resets; the document-local expert resets at chunk boundaries as a doc-boundary proxy.

The three experts (neural, global PPM, local PPM) are blended through a fixed-share Bayesian mixer. Each chunk yields a per-expert mean log-loss `L_i`; the chunk-end posterior is `post_i ∝ w_i · exp(-(L_i - L_min))`, then `w_new = (1 - share) · post + share · prior` with prior `(0.90, 0.07, 0.03)` and `share = 0.005`. This bounds cumulative log-loss against the best switching expert across chunks while keeping a small floor of probability on the slower-moving experts so they can re-engage if the dominant expert ever changes. Mixture weights for chunk `c+1` are a deterministic function of chunks `0..c`'s losses, which is the same legality property the neural TTT update path relies on.

Hyperparameters

```
DC_ENABLED=1 DC_ORDER_G=6 DC_ORDER_L=8 DC_BUCKETS_G=2048 DC_BUCKETS_L=2048
DC_ALPHA_G=1.0 DC_ALPHA_L=0.5 DC_EPS_UNI=0.25 DC_SHARE=0.005
DC_PRIOR='0.90,0.07,0.03' DC_DOC_RESET=1
```

Test plan

• seed 42 on 8×H100 SXM, brotli artifact 15,977,914 bytes
• full normalized distribution over the official SP8192 vocabulary verified by construction
• score-before-update legality verified: counts and mixer weights update only after the chunk is scored

[okezue]

@cocohearts @willdepue could you take a look when you get a chance? This is a record submission at 1.0803 BPB seed 42, beating the prior 1.0810 leader. The full normalized-distribution + score-before-update legality is preserved per the rules (details in the PR description). Happy to run additional seeds if useful.

[h1beee]

You need to be lower than 1.076 bpb

https://github.com/openai/parameter-golf#submission-process

[okezue]

Update — broader context on PR #1879

Beyond the DualClock submission in this PR, I separately ran a study reproducing the open SOTA stack from PR #1855. Posting here just for context — the DualClock contribution and the reproduction work are independent.

Independent reproduction of PR #1855 on a fresh 8×H100 SXM pod (cu129/torch 2.9.1/FA3 cu129_torch291) with the env vars from #1855's hparam table plus SMEAR_GATE_ENABLED=1 SPARSE_ATTN_GATE_ENABLED=1 EMBED_BITS=7 MIN_LR=0.1 GPTQ_RESERVE_SECONDS=0.5 PHASED_TTT_NUM_PHASES=3, 600 s wallclock:

seed	post-TTT	compressor	artifact
42	1.05965	brotli	16,112,007 (over cap)
314	1.06041	brotli	~16.11 M (over cap)
999	1.06124	brotli	~16.11 M (over cap)
mean	1.06043
42	1.06052	pergroup	15,902,285 ✅

3-seed brotli mean 1.06043 vs PR #1855's reported 1.06108 — within 1σ. Posted the same data on #1855 as a reproduction confirmation.

The DualClock submission in this PR (1.0803 BPB seed 42) remains a separate contribution. To be fully transparent: the Bayesian mixer auto-collapses to ~99.95% on the neural model after the first few chunks, so DualClock is empirically neutral on a strong neural baseline — the safety guarantee holds, but the small-vocab PPM experts don't add information the neural model is missing. Documented as-is.