N-gram Cache + Entropy-Adaptive Alpha: 1.0945 BPB#1026

Closed

danielxmed wants to merge 14 commits intoopenai:mainfrom

danielxmed:submission/ngram-cache-entropy-adaptive-1.0945

Commits on Mar 24, 2026

feat: integrate PolyGLU per-neuron activation routing
Daniel Nobrega Medeiros
committed

Commits on Mar 27, 2026

feat: rebase PolyGLU onto SOTA stack with per-neuron slope specialization

danielxmed
and
claude
committed
infra: add EMA_DECAY, EMA_ENABLED, GATED_MLP env vars
danielxmed
committed
feat: add N-gram cache eval + gated MLP infrastructure
danielxmed
committed
opt: two-pass N-gram cache with precomputed rolling hashes
danielxmed
committed
feat: optimized N-gram cache eval — reuses sliding window NLL, no duplicate inference
danielxmed
committed
update: CLAUDE.md with N-gram eval, results tracking, ngram test script
danielxmed
committed
feat: entropy-adaptive alpha for N-gram cache (scales with model uncertainty)
danielxmed
committed
results: adaptive N-gram alpha = -0.043 BPB on 500K subset (2x better than fixed)
danielxmed
committed
opt: N-gram cache tuned to alpha=0.50, 2x speed with bytes() keys

danielxmed
and
claude
committed
results: N-gram on full 62M val set: 1.2011 BPB (-0.065 from 1.2665)

danielxmed
and
claude
committed
opt: concurrent N-gram via background thread (overlaps with GPU eval)

danielxmed
and
claude
committed

Commits on Mar 28, 2026