Follow-up: TurboQuant upstream landscape + independent implementation feasibility

[dogkeeper886]

Context

Follow-up to #99 (survey merged in #100, see `docs/research/google-kv-survey.md`). Before committing to a prototype, we need to understand the state of upstream llama.cpp work and decide whether to port an existing PR or implement independently from the paper.

Upstream PR landscape (as of 2026-04-18)

Three TurboQuant PRs were opened against `ggml-org/llama.cpp`:

PR	Scope	State	Notes
#21089	CPU-only, TBQ3_0 + TBQ4_0	Open, stalled	Only PR that followed the contribution guidelines (CPU-first, disclosed AI usage, used template). Author: elusznik.
#21062	Full CUDA, TBQ2/3/4	Closed	Closed for AI-generated content policy violation. Also had bad merge + multi-backend-in-one-PR policy violations. Targets SM 12.1 (GB10). Segfaults on CUDA 13.2.
#21010	Vulkan, TQ3_0	Closed	Also closed for AI policy violation.

Maintainer sentiment on #21089

• ggerganov (project lead): called the PR "pure slop" on 2026-04-03. Tone later softened but skepticism remained.
• JohannesGaessler (core maintainer): "tbq4_0 isn't even really better than the existing naive q4_0 type... we should not invest maintainer time here."
• CISC (core maintainer): pointed out the speed is half of q4_0 and that latest master should be retested.
• pwilkin, ericcurtin: supportive, note the PR follows guidelines properly.
• Overall: the PR is not moving. Author (elusznik) expressed frustration at the reception.

The landscape-changing finding

"Master now already applies rotations on the KV cache before quantization, improving the quality of all the regular quantization types. And since TurboQuant is barely any better than the regular quants without rotation, latest master might actually show better performance with the regular quants." — Mushoz, 2026-04-03

This substantially changes our calculus. After upstream's pre-rotation change:

Cache type	PPL (Qwen3.5-4B, post-rotation master)	Gen t/s	Compression
f16	9.027	14.22	1.00×
q4_0	9.047 (+0.22%)	14.09	3.56×
tbq4_0	9.046 (+0.21%)	6.68	3.94×
tbq3_0	9.178 (+1.67%)	6.74	5.19×

tbq4_0 is now quality-equivalent to q4_0 with only ~10% more compression, at half the speed. tbq3_0 remains the only compelling option (5.2× compression, +1.67% PPL).

Community extensions not in any upstream PR

From TheTom (TurboQuant+) and animehacker:

• Asymmetric K/V quantization is critical — symmetric turbo3 on Qwen2.5-7B Q4_K_M yields catastrophic PPL (3,556). Asymmetric q8_0-K / turbo3-V gives 6.71 (+2.0%).
• V compression is near-free; K compression stacks badly with weight quantization on sensitive models.
• Asymmetric WHT normalization: 1/√32 on K-side, unnormalized on Q-side. Using symmetric 1/32 produces plausible but broken output.
• QJL residual correction hurts attention workloads in practice — implementations are dropping it.

None of the open/closed PRs include these findings, and the paper doesn't cover Q4_K_M-weighted models.

Can we implement without porting a PR?

Yes — and it's arguably the better path. The algorithm is published (arXiv:2504.19874), the implementation patterns from open PRs are in the public record for reference (GGML type registration, block layouts, KV cache integration points), and we have no incentive to chase upstream compatibility since ollama37 is a fork.

Three concrete paths, in order of cost:

Path A — Adopt upstream's pre-rotation only (cheapest, biggest ROI)

• Cherry-pick llama.cpp master's "apply rotations on KV cache before quantization" change (the one Mushoz cited).
• Use existing q4_0/q8_0 KV types, now with rotation.
• Result: most of TurboQuant's quality benefit with zero new kernels, zero new types, zero K80-specific kernel work.
• This may be sufficient for the K80 use case.

Path B — Implement tbq3_0 only, from the paper, for K80 vec path (medium)

• Skip tbq4_0 entirely (now pointless per upstream numbers).
• Implement TBQ3_0 directly from arXiv:2504.19874, following the integration patterns observed in #21089 (type registration in `ggml.c`, block layout in `ggml-common.h`, CPU from_float + vec_dot for correctness reference, CUDA fattn-vec template instance for K80).
• Include asymmetric K/V support from day one (community finding, not paper).
• Skip QJL residual correction.
• No code copied from #21062 (closed/AI-violation) or #21089 (upstream attribution cleaner if we re-derive).

Path C — Port #21089 CPU code as attribution-disclosed baseline, add CUDA ourselves (highest risk)

• Take #21089 as upstream reference under its license, disclose attribution in commit messages.
• Add our own sm_37 CUDA kernels.
• Risk: if #21089 never merges, we inherit a stale base.

Recommendation

Prototype Path A first as a baseline. If the upstream pre-rotation change (now in llama.cpp master) gives ollama37 most of the quality win with existing q4_0/q8_0 on K80, we ship that and revisit tbq3_0 later. If not, proceed to Path B (independent tbq3_0 implementation) — Path C is the worst of both worlds.

Tasks

• Identify the specific upstream commit(s) that introduced "rotation before KV quantization" in llama.cpp master
• Assess whether ollama37's current `ml/backend/ggml/` vendor tree is ahead of or behind that commit
• Benchmark q4_0 KV with and without pre-rotation on K80 (existing infra, minimal work)
• Only if Path A is insufficient: open a prototype issue for Path B (independent tbq3_0)

Out of scope

• Path C (porting #21089 verbatim). Rejected.
• tbq4_0 / TBQ2. No longer compelling given upstream's rotation change.
• QJL residual correction. Community evidence is against it.

References

• Study: `docs/research/google-kv-survey.md` (this repo)
• Upstream PR #21089: ggml : add CPU TurboQuant KV cache types (TBQ3_0 / TBQ4_0) ggml-org/llama.cpp#21089 (CPU, open, stalled)
• Upstream PR #21062: (AI policy violation) ggml: add TurboQuant KV cache compression (2/3/4-bit) with CUDA support ggml-org/llama.cpp#21062 (CUDA, closed, AI-policy)
• Upstream PR #21010: (close due to violation AI policy) vulkan: add TQ3_0 (TurboQuant) 3.5-bit KV cache quantization ggml-org/llama.cpp#21010 (Vulkan, closed, AI-policy)
• Paper: https://arxiv.org/abs/2504.19874
• TurboQuant+ (extension findings): https://github.com/TheTom/turboquant_plus
• animehacker writeup: https://oliverchurch.com/turboquant-for-ggml-achieving-4.57x-kv-cache-compression-in-llama.cpp.html

Related to #99.

[dogkeeper886]

Two-engine distinction + revised priority

ollama37 has two inference engines

Engine	Path	Attention impl	KV cache impl	Has PR #21038 rotation?
llamarunner (legacy)	`runner/llamarunner/` + vendored `llama/llama.cpp/`	C++ (llama.cpp graph builder)	llama.cpp KV cache	No — vendored copy predates 2026-04-01
ollamarunner (Go-native, newer)	`runner/ollamarunner/` + `ml/nn/attention.go` + `kvcache/causal.go`	Go	Go	No — never had it

Both engines share `ml/backend/ggml/` (tensor lib + CUDA kernels). K80 support lives in the shared backend, so it benefits both engines — but the rotation change has to be applied separately in each engine's attention layer.

Most new/active models (gemma3, gemma4, qwen3, qwen35, llama4, gptoss, mistral3 …) run on ollamarunner. That's the priority target.

Revised plan — two phases

Phase 1: Rotation-before-quant (cheap, high ROI, covers most of the win)

• Port PR #21038's rotation logic into `ml/nn/attention.go` (ollamarunner path). Apply WHT-style rotation to Q/K/V before `cache.Put` and matching unrotation on `cache.Get`.
• For llamarunner: either bump vendored `llama/llama.cpp/` past commit `744c0c73` or cherry-pick the change as a patch in `llama/patches/`.
• Benchmark q4_0 KV with rotation on K80: target +0.22% PPL vs fp16, matching the upstream CPU numbers on a GPU path.
• Gate behind a runtime flag initially (e.g. `OLLAMA_KV_ROTATE=1`).

Phase 2: tbq3_0 (only if Phase 1 context headroom is insufficient)

• Independent implementation of TBQ3_0 from arXiv:2504.19874 (skip TBQ4/TBQ2, skip QJL).
• Asymmetric K/V from day one (K norms 50-180× V norms — symmetric was catastrophic on Qwen per TheTom's data).
• `fattn-vec` template instance for sm_37 scalar path.
• Gate behind `--cache-type-k tbq3_0 --cache-type-v tbq3_0` (following llama.cpp CLI convention).

Rejected: tbq4_0 (redundant with rotated q4_0), QJL (hurts quality per community testing), porting #21089 verbatim (stalled, CPU-only).

[dogkeeper886]

Closing — analysis done, Phase 1 implementation tracked in #102. If Phase 1 results show we still need TBQ3_0 for extra compression headroom, a fresh Phase 2 issue will be opened then rather than kept alive as a placeholder here.