UPSTREAM PR #21089: ggml : add CPU TurboQuant KV cache types (TBQ3_0 / TBQ4_0)

[loci-dev]

Note

Source pull request: ggml-org/llama.cpp#21089

Summary

This PR adds CPU-only TurboQuant KV-cache support for two new cache types:

• tbq3_0
• tbq4_0

The scope is intentionally narrow for the first PR:

• CPU-only
• KV-cache types only
• TBQ only (TBQP / Q-prod is left for follow-up work)

That keeps the initial landing aligned with the contributor guidance for new features and new ggml_type additions: start with CPU support first, keep the PR reviewable, and add backend support in follow-up PRs.

What changed

• add GGML_TYPE_TBQ3_0 and GGML_TYPE_TBQ4_0
• add block layouts and CPU quantize / dequantize support
• add CPU vec_dot support so CPU flash attention can consume the new KV types
• wire the new types into ggml type traits and quantization entry points
• allow tbq3_0 / tbq4_0 in CLI KV-cache arguments
• add llama-bench and quantize support for the new types
• add CPU regression coverage in test-quantize-fns
• add backend-op coverage for GET_ROWS, SET_ROWS, CPY, and FLASH_ATTN_EXT

Why this scope

I started from a broader TurboQuant implementation, but for the first upstream PR I cut the surface down to the part that is strongest on the current CPU-only evaluation:

• tbq4_0 is the best-balanced TurboQuant point here
• tbq3_0 is the memory-first option
• the wider TBQP / split-outlier path is better handled as follow-up work after the plain TBQ CPU base lands

Block layout

• tbq3_0: 98 bytes / 256 elements = 3.0625 bits / element
• tbq4_0: 130 bytes / 256 elements = 4.0625 bits / element

CPU results

Model: Qwen3.5-4B-Q4_K_M.gguf

Settings:

• CPU only
• 4 threads
• flash_attn=on
• llama-bench with pp32/tg8
• llama-perplexity on wikitext-2-raw/wiki.test.raw
• ctx=256, chunks=4

Cache type	Prompt t/s	Gen t/s	KV MiB	Compression vs f16	PPL	KLD	RMS Δp	Same top p
f16	50.67	15.72	64.00	1.00x	13.8387	0.00000	0.000%	100.000%
q8_0	50.63	15.67	34.00	1.88x	13.8348	0.00320	1.510%	97.835%
q4_0	50.46	15.64	18.00	3.56x	13.8400	0.00912	2.179%	93.898%
tbq3_0	46.19	8.29	12.25	5.22x	14.3198	0.02647	4.471%	91.732%
tbq4_0	45.84	8.31	16.25	3.94x	13.8323	0.00960	2.892%	94.094%

Key takeaways:

• tbq4_0 is the best-balanced TurboQuant point in this CPU-only sweep.
• tbq4_0 reduces KV cache below stock q4_0 while keeping similar KLD and slightly better perplexity in this run.
• tbq3_0 pushes KV memory lower again, with the expected quality tradeoff.

Plots

KV cache memory usage

Throughput

Compression vs speed

Ablation: KV size vs KLD

Validation

Built locally:

• cmake -S . -B build-cpu-pr -DCMAKE_BUILD_TYPE=Release
• cmake --build build-cpu-pr --target test-quantize-fns test-backend-ops llama-bench llama-cli llama-perplexity -j4

Checks run:

• ./build-cpu-pr/bin/test-quantize-fns
• ./build-cpu-pr/bin/test-backend-ops test -b CPU -o GET_ROWS,SET_ROWS,CPY,FLASH_ATTN_EXT -p 'tbq'
• llama-bench CPU comparison vs f16, q8_0, q4_0
• llama-perplexity + KL divergence comparison vs f16

Follow-up work

Planned follow-ups after this CPU base:

• TBQP / Q-prod variants
• split outlier path
• ROCm backend support
• CUDA backend support

Acknowledgements

This work was informed by:

• TurboQuant paper: https://arxiv.org/abs/2504.19874
• Google Research blog: https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/
• discussion: TurboQuant - Extreme KV Cache Quantization ggml-org/llama.cpp#20969
• issue: Feature Request: TurboQuant support ggml-org/llama.cpp#20977
• mudler/llama.cpp feat/turbo-quant
• Aaryan-Kapoor/llama.cpp turboquant-tq3_0
• TheTom/turboquant_plus
• tonbistudio/turboquant-pytorch

AI usage disclosure

AI tools were used in an assistive capacity for exploration, mechanical refactoring, test/benchmark scripting, and draft review text. The code and measurements in this PR were manually reviewed locally, the relevant checks were run manually, and I can explain the submitted changes and benchmark setup in detail.