fix: account for TurboQuant rotation tensors in KV cache context pool#1

Closed

gmsimmons wants to merge 49 commits into

TheTom:feature/turboquant-kv-cachefrom

gmsimmons:fix/turbo-kv-cache-pool-size

Commits on Mar 25, 2026

WIP: add TurboQuant KV cache types (turbo3, turbo4)

TheTom
and
claude
committed
feat: Metal kernels for TurboQuant KV cache (turbo3, turbo4) #21

TheTom
and
claude
committed
feat: full TurboQuant with rotation matrices in Metal kernels #21

TheTom
and
claude
committed
feat: inline rotation matrices in Metal shader + C round-trip test #21

TheTom
and
claude
committed
fix: remove thread static from Metal dequantize, fix stale code #23

TheTom
and
claude
committed
feat: replace dense 128x128 matvec with Fast Walsh-Hadamard rotation #26

TheTom
and
claude
committed
docs: detailed speed investigation plan for TurboQuant Metal shader #23

TheTom
and
claude
committed
docs: log simd_broadcast attempt — no speed improvement #23

TheTom
and
claude
committed
docs: log threadgroup attempt — no speed improvement, rethinking #23

TheTom
and
claude
committed
docs: CRITICAL — dequant is NOT the bottleneck, no-op still 2.4 tok/s #23

TheTom
and
claude
committed
fix: inline turbo-wht.h — was causing CPU fallback, not Metal! #23

TheTom
and
claude
committed
docs: real Metal benchmarks after #include fix — 8× gap not 35× #23

TheTom
and
claude
committed
docs: final investigation summary + upstream tracking #23 #27

TheTom
and
claude
committed
docs: upstream competitive intel — pre-rotate-queries is the key #28

TheTom
and
claude
committed
docs: speed ceiling test — 49 tok/s without dequant rotation (4.6× gain) #23

TheTom
and
claude
committed
docs: pre-rotate-queries implementation plan + speed ceiling 49 tok/s

TheTom
and
claude
committed
feat: pre-rotate-queries optimization — 51.4 tok/s (5× speedup) #23

TheTom
and
claude
committed
docs: final investigation summary — 2.4 → 51.4 tok/s journey complete #23

TheTom
and
claude
committed
feat: MSE-only mode — drop QJL, all 3 bits to PolarQuant #23

TheTom
and
claude
committed
docs: Change 2 not needed — Q rotation overhead is negligible

TheTom
and
claude
committed
docs: block size is the bottleneck — q4_0 at block 32 = 100% of q8_0

TheTom
and
claude
committed
feat: block size 32 — 77.7 tok/s MoE (91% of q8_0), 17.0 Qwopus (97%) 🎉

TheTom
and
claude
committed
fix: TURBO_D=128 independent of QK_TURBO3, file turbo4 bugs #29

TheTom
and
claude
committed
docs: final investigation log — 77.7 tok/s, 91% of q8_0

TheTom
and
claude
committed
CRITICAL: turbo3 perplexity is 165.6 vs q8_0 6.1 — quality broken #30

TheTom
and
claude
committed
CRITICAL: found TWO root causes for PPL=165 #30

TheTom
and
claude
committed
docs: bisect confirms block size innocent, rotation access is the bug #31

TheTom
and
claude
committed
fix: restore inverse rotation in dequant — PPL 6.19 (1.2% of q8_0) #31 #30

TheTom
and
claude
committed
docs: perplexity 6.194 confirmed — 1.4% of q8_0 #30

TheTom
and
claude
committed
docs: complete quality benchmark summary + lessons learned #30

TheTom
and
claude
committed
perf: fp16 WHT dequant + SIMD cooperative dequant — 45% speedup

TheTom
and
claude
committed
chore: move turboquant docs to turboquant_plus repo

TheTom
and
claude
committed
perf: vectorized half4 WHT butterfly — 31% speedup (1074 → 1411 tok/s)

TheTom
and
claude
committed
perf: pre-packed half4 sign arrays — minor speedup (1411 → 1424 tok/s)

TheTom
and
claude
committed
perf: graph-side WHT rotation — 2095 tok/s (0.78x q8_0, was 0.53x)

TheTom
and
claude
committed
perf: block-32 + graph WHT — 2747 tok/s (1.02x q8_0!!!)

TheTom
and
claude
committed
feat: layer-adaptive KV cache — q8_0 quality with 80% turbo3 compression

TheTom
and
claude
committed
fix: address Codex review on layer-adaptive — thread safety + underflow guard

TheTom
and
claude
committed

Commits on Mar 26, 2026