Study Google's recent KV cache compression/quantization research for K80 adoption

[dogkeeper886]

Goal

Survey Google's recent research on KV cache compression and quantization, and assess what could be adopted into ollama37 to reduce VRAM pressure on Tesla K80 (compute capability 3.7, 2x12 GB VRAM per card).

Why this matters for 3.7

K80 is VRAM-bound: long-context inference is capped by KV cache size well before compute becomes the bottleneck. If Google's techniques land model quality at a fraction of the KV footprint, we can:

• fit longer contexts on a single K80 pair
• run larger models without offloading KV to CPU
• reduce PCIe Gen3 x16 traffic during decode

Scope of study

Target: published Google Research / DeepMind work on KV cache reduction. Candidates to investigate (non-exhaustive — expand during the survey):

• KV cache quantization (per-channel / per-token schemes)
• KV cache eviction / sparsity (e.g., attention-score-based retention)
• Grouped-query and multi-query attention variants
• Low-rank / shared KV projections
• Hybrid local+global attention (sliding window) as it interacts with KV footprint
• Any FP8 / INT4 KV work compatible with sm_37 (no native FP8 — must emulate or skip)

Deliverables

• docs/research/google-kv-survey.md — summary of each technique: method, reported accuracy/perf, implementation complexity
• Feasibility table: each technique vs. sm_37 constraints (no tensor cores, no FP8, no bfloat16 native, CUDA 11.4)
• Recommendation: 1-2 techniques worth prototyping, with rationale
• Integration sketch: where in ollama37 / llama.cpp / GGML the change would land (attention kernels, KV buffer allocation, graph ops)

Out of scope

• Actual implementation — this is a study issue. Prototype work will be tracked in a follow-up issue based on the recommendation.

References to seed the search

• Google Research blog (research.google/blog) — filter for KV cache, attention, long context
• DeepMind publications on efficient inference
• arXiv: search authors affiliated with Google/DeepMind on KV compression

Acceptance criteria

• Survey doc merged
• Recommendation is specific enough to open a prototype issue with a concrete implementation target

[dogkeeper886]

Starting work on branch issue-99-google-kv-survey. Survey-only — deliverable is docs/research/google-kv-survey.md plus a feasibility table and prototype recommendation.

[dogkeeper886]

Survey complete. PR #100 opened with docs/research/google-kv-survey.md. Recommendation: prototype 3-bit TurboQuant on the K80 fattn-vec path — VRAM win, not throughput. See PR for integration sketch.

[dogkeeper886]

Study update — upstream landscape + ollama handling

Upstream PR status (ggml-org/llama.cpp)

PR	Scope	State
#21089	CPU TBQ3_0 + TBQ4_0	Open, stalled. Maintainer ggerganov called it "pure slop"; JohannesGaessler said "we should not invest maintainer time here." Author frustrated.
#21062	Full CUDA	Closed — AI-policy violation.
#21010	Vulkan	Closed — AI-policy violation.

The landscape-changer: upstream PR #21038

Merged 2026-04-01 by ggerganov himself: "llama: rotate activations for better quantization" (commit 744c0c73). Adds Walsh-Hadamard-style rotation to Q/K/V before KV-cache quantization, lifting the quality of existing q4_0 / q8_0 KV types. After this, TurboQuant's tbq4_0 is essentially tied with q4_0 on PPL at half the CPU speed — only tbq3_0 (5.2× compression, +1.67% PPL) remains compelling.

How does ollama handle this?

It doesn't — automatically. PR #21038 lives in src/llama-graph.cpp and src/llama-kv-cache.cpp. Ollama vendors only ggml/src/ (the tensor library); it does not vendor llama.cpp/src/. Ollama builds attention graphs in Go (ml/nn/attention.go) and has its own KV cache (kvcache/causal.go). The upstream rotation change has to be re-implemented on the Go side to land in ollama37.

That's actually good news for us: we don't chase llama.cpp releases, and we can add the rotation directly where our K80 path runs.

Most benefit is for q4? Yes.

Cache type	Compression	PPL vs f16 (post-rotation)	Gen t/s (CPU)
f16	1.00×	—	14.22
q8_0	1.88×	+0.00% (already close)	14.12
q4_0	3.56×	+0.22% (huge recovery)	14.09
tbq4_0	3.94×	+0.21% (tied w/ q4_0)	6.68
tbq3_0	5.19×	+1.67%	6.74

q8_0 was already near-lossless so rotation had little room to help. q4_0 had headroom and gained the most — it's now within a fraction of a percent of f16. The rotation change collapses the quality gap that made TurboQuant interesting in the first place.

What this means for ollama37

• Most of the K80 win is reachable by porting PR #21038's rotation logic into ml/nn/attention.go + using existing q4_0 KV. No new quant types needed.
• Only after that should we consider tbq3_0 for the extra 1.5× compression, if more context is needed.
• Full recommendation tracked in Follow-up: TurboQuant upstream landscape + independent implementation feasibility #101.