Non-record: Audited Byte-Level Neural/PPM-D Mixture BPB = 1.5221 (Full Validation) — Framework for Legal Score-First PPM-D Mixtures

[Christopher-Lee-McClendon]

Non-record: Audited Byte-Level Neural/PPM-D Mixture BPB = 1.5221 (Full Validation) — Framework for Legal Score-First PPM-D Mixtures

Key Results

Metric	Value	Status
Full-val Path B mixture BPB	1.5221 (151,078,222 bytes)	✅ Audited, claim_ready=true
Full-val Path B neural-only BPB	1.5430	✅ Audited
Full-val Path B PPM-D-only BPB	2.0319	✅ Audited
Neural-only sliding BPB (token-level)	1.0830	✅ Valid
Neural-only TTT BPB (token-level)	1.0812	✅ Valid
Invalid in-source PPM mixture	0.9949	❌ Invalid — see formal proof

PPM-D mixture gain: −0.021 BPB over neural-only byte-level baseline (1.5430 → 1.5221). Note: This result is sliding-window only; no corrected Path B TTT result exists yet.

This is a non-record methodology submission documenting a rigorous framework for computing valid byte-level BPB with neural + PPM-D mixtures. The headline result is a full-validation, provably normalized, audited byte-level mixture BPB of 1.5221 — the first such audited number in the contest.

---

What's new in this update

1. Full-validation Path B result (previously only had 8M-token subset):

   • mixture_bpb = 1.5221 (was 1.5459 on 8M subset)
   • neural_only_bpb = 1.5430 (was 1.5619 on 8M subset)
   • PPM-D improves by −0.021 BPB on full val (vs −0.016 on subset)
   • Runtime: ~9.4 hours offline CPU postpass

2. Comparison with PR Report: PPM-D byte-level scoring is not a valid probability distribution, and why it appears to gain #1905 by @leon2k2k2k:

   • Both independently discovered the same normalization invalidity
   • Our Path B mixture improves (−0.021 BPB); PR Report: PPM-D byte-level scoring is not a valid probability distribution, and why it appears to gain #1905's conditional mixture worsens (+0.038 BPP)
   • Key differences: we use PPM-D with exclusion (theirs lacks exclusion), different gating direction

3. Formal mathematical description of byte-level vs token-level BPB metrics

4. Path A archived as computationally intractable (O(V=8192) per position)

5. Score-First Legal TTT evidence section citing PRs Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461, Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549, Record: SP8192 + Parallel Pre-Quant TTT — val_bpb 1.0429 (3-seed mean) #1735, Record: val_bpb = 1.06128 SmearGate BOS Fix + PR #1787 Base + Smear Gate + LQER Asymmetric + Phased TTT (indirect 3 seed mean) #1851, Record: SmearGate BOS Fix 3-Seed Compliance Re-run — val_bpb 1.06141 (3-seed mean) #1868, Record: Coprime-Stride Loader + Full GPTQ + Score-First TTT — val_bpb 1.08008 (3-seed mean) #1876

---

Formal Math

Token-Level BPB (standard contest metric)

$$\text{BPB}_{\text{token}} = \frac{\sum_{t=1}^{T} -\log p_{\text{NN}}(v_t \mid v_{\lt t})}{\log 2 \cdot B}$$

Byte-Level Neural BPB (Path B — trie marginalization over continuable mass)

$$p_{\text{NN}}^{\text{byte}}(b_k \mid \pi_{\lt k}) = \frac{\sum_{v \in C(\pi_{\lt k} b_k)} p_{\text{NN}}(v \mid v_{\lt t})}{\sum_{v \in C(\pi_{\lt k})} p_{\text{NN}}(v \mid v_{\lt t})}$$

where $C(\pi)$ denotes tokens whose byte sequences strictly extend $\pi$ (excluding tokens terminating exactly at $\pi$). This is a proper conditional distribution by construction.

Mixture

$$p_{\text{mix}}(b) = (1 - \lambda) \cdot p_{\text{NN}}^{\text{byte}}(b) + \lambda \cdot p_{\text{PPM}}(b)$$

$$\lambda = \begin{cases} 0.90 & \text{if } \max_b p_{\text{PPM}}(b) \geq 0.90 \ 0.05 & \text{otherwise} \end{cases}$$

Full-val BPB

$$\text{BPB}_{\text{byte}} = \frac{\sum_{i=1}^{B} -\log_2 p_{\text{mix}}(b_i \mid h_{\lt i})}{B} = \frac{\text{total NLL bits}}{151{,}078{,}222} = 1.5221$$

Why neural_only_bpb (1.5430) ≠ token-level sliding BPB (1.0830)

Token-level BPB distributes each token's cross-entropy uniformly across its bytes. Byte-level BPB asks "given the byte prefix emitted so far, what is the conditional probability of the next byte?" — a fundamentally harder task requiring the model to resolve within-token byte ambiguity.

---

Comparison with PR #1905

Both this submission and PR #1905 (by @leon2k2k2k) independently discovered the same normalization invalidity in geometric-mean byte decomposition. Both implement correct trie-based conditional byte distributions. Yet the mixture effect diverges:

Aspect	This submission (Path B)	PR #1905
PPM-D variant	With exclusion (order 5)	Without exclusion
Confidence gating	PPM-D confidence-based (both use PPM-D side)	PPM-D confidence-based
Mixture effect on own byte-level baseline	−0.021 BPB (improvement)	+0.038 BPB (degradation)
PPM-D helps?	Yes	No (hurts)

Note: The raw neural baselines (1.5430 vs 1.08335) are not directly comparable — ours is byte-level, theirs appears to be token-level. The meaningful comparison is the direction of the mixture effect on each submission's own baseline.

Both independently confirmed: uniform-spread (geometric mean) byte decomposition is NOT a valid probability distribution (sums > 1). The key difference is PPM-D with exclusion (ours) vs without (theirs) — exclusion produces sharper predictions and provably normalizes.

---

Score-First Legal TTT

PR	Contribution
#461 (MERGED)	Introduced score-first TTT, proved legal under Issue #1017 C1–C4
#549 (MERGED)	Extended score-first TTT
#1735	Parallel TTT, 21 epochs in eval budget
#1851	SmearGate BOS + score-first TTT, post-TTT BPB 1.06128
#1868	Clean neural baseline
#1876	Coprime-Stride + Full GPTQ + Score-First TTT, BPB 1.08008
#1881	PPM-D mixture 0.9019 BPB (invalid uniform-spread)

---

Path A: Computationally Intractable

Path A (token-normalized PPM mixture) required O(V=8192) PPM-D evaluations per token position — projected at ~38 days CPU-only. Even with C++ backend (17-50× speedup), it exceeds practical budgets. Archived with full materials.

---

Red-team investigation

The full formal legality proof (docs/legality/ppmd-legality-proof.md) includes 6 theorems:

• Theorems 1, 4, 5, 6: ✅ Verified (PPM-D normalization, score-before-update, denominators, coverage)
• Theorems 2, 3: ❌ Disproved (geometric-mean neural bytes, mixture — NOT valid distributions)

Full-val legality proof: docs/legality/ppmd-legality-proof-fullval-result.md

---

Files changed

• README.md — Major update: headline full-val result, formal math, PR Report: PPM-D byte-level scoring is not a valid probability distribution, and why it appears to gain #1905 comparison, score-first TTT evidence
• submission.json — Added full-val Path B fields
• results/exp_1876_ppmd/path_b_prod_8gpu_fullval_local_score/path_b_sliding_full.json — Full-val result JSON
• docs/legality/ppmd-legality-proof-fullval-result.md — Full-val legality proof
• scripts/fast_score.py — Fast scoring utility
• docs/path_a_archive/ — Archived Path A materials with intractability note

Artifact: 15,975,706 bytes (model) + 20,220 bytes (code) = 15,995,926 bytes (under 16 MB cap)