(close due to violation AI policy) vulkan: add TQ3_0 (TurboQuant) 3.5-bit KV cache quantization

[crayolaconsumer]

Summary

Adds GGML_TYPE_TQ3_0 — a 3.5-bit KV cache quantization type based on Google's TurboQuant algorithm (arXiv:2504.19874, ICLR 2026). Provides ~4.6x KV cache compression vs FP16 with near-zero quality loss.

Tested on: AMD Radeon RX 7800 XT (RDNA3, Vulkan backend) with Qwen 3.5 27B (hybrid DeltaNet+Attention architecture).

Results

KV Cache	Context	KV Size	Speed	VRAM Free
q4_0	65K	1,152 MB	11.6 tok/s	36 MB
tq3_0	98K	1,344 MB	10.8 tok/s	79 MB

• 50% more context in the same VRAM budget
• 7% speed reduction (within noise for bandwidth-limited workloads)
• More stable VRAM (79 MB headroom vs 36 MB)
• Coherent output verified on Qwen 3.5 9B and 27B models

Algorithm

TurboQuant uses a 32-element Walsh-Hadamard Transform (WHT) rotation to gaussianize input vectors, then applies a 2-bit Max-Lloyd codebook quantization + 1-bit QJL residual sign:

• Block format: 14 bytes per 32 elements (3.5 bits/value)

  • qs[8] — 32 × 2-bit codebook indices
  • qr[4] — 32 × 1-bit QJL residual signs (stored, not used in MSE-only reconstruction)
  • gamma — FP16 per-block scale

• WHT Forward (quantize): sign preconditioning → 5 butterfly stages → normalize

• WHT Inverse (dequantize): 5 butterfly stages → normalize + undo signs

• Codebook: Max-Lloyd optimal 4-level for N(0,1): {-1.510, -0.4528, +0.4528, +1.510}

The WHT is self-inverse and operates in-register (no shared memory needed for the per-block quantize/dequant path).

Changes

GGML core:

• ggml.h — GGML_TYPE_TQ3_0 = 41 enum, increment GGML_TYPE_COUNT
• ggml-common.h — block_tq3_0 struct (14 bytes), QK_TQ3_0 = 32
• ggml.c — type_traits entry
• ggml-quants.{h,c} — CPU reference quantize/dequantize with WHT rotation
• ggml-cpu/{ggml-cpu.c,quants.{h,c},ops.cpp} — CPU backend wrappers

Vulkan backend:

• types.glsl — block struct + packed16 for flash attention
• copy_to_quant.comp — GPU quantizer (single-threaded WHT + codebook per block)
• dequant_funcs.glsl — unscaled dequant for mul_mat path + get_dm()
• flash_attn_base.glsl — scaled dequant for flash attention inner loop
• dequant_tq3_0.comp — standalone dequant shader
• vulkan-shaders-gen.cpp — shader compilation registration (FA, cpy, set_rows)
• ggml-vulkan.cpp — pipeline registration, FA allowlist, supports_op

CLI:

• arg.cpp — tq3_0 added to kv_cache_types allowlist

Usage

llama-server -m model.gguf -ngl 99 -c 131072 \
  --cache-type-k tq3_0 --cache-type-v tq3_0 \
  --flash-attn on

Flash attention is required (KV cache quantization depends on FA).

References

• TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Google Research, ICLR 2026
• unixsysdev/llama-turboquant — C + HIP reference implementation (validated +5.8% PPL vs FP16)
• Discussion #20969 — TurboQuant community discussion
• Issue #20977 — Feature request

Known Limitations

• QJL residual correction (qr bits) is stored but not used in the dequant path — MSE-only reconstruction. A fused MMVQ kernel could use the QJL bits for unbiased inner product estimation (future work).
• Dequant in dequant_funcs.glsl uses Approach A (full 32-element WHT per thread in registers). Cooperative shared-memory WHT would be more efficient but requires calling convention changes.
• Formal perplexity validation pending — integration testing confirmed coherent output on Qwen 3.5 9B and 27B.
• No CUDA/Metal kernels — Vulkan only. CPU fallback works for all backends.

Testing

• cmake --build passes with zero errors
• llama-server --help shows tq3_0 in cache type options
• Server starts with --cache-type-k tq3_0 --cache-type-v tq3_0
• Coherent text generation on Qwen 3.5 9B (42 tok/s)
• Coherent text generation on Qwen 3.5 27B (10.8 tok/s)
• Multi-turn conversations work correctly
• Formal perplexity comparison (pending wikitext-2 dataset)

[ggml-gh-bot]

Hi @crayolaconsumer, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[Aaryan-Kapoor]

Interesting work! A few technical observations:

• This implementation uses a 4-level (2-bit) Max-Lloyd codebook with 1-bit QJL residual signs stored in qr[4] - but the QJL bits are never used in the dequant path. This means you're getting 2-bit reconstruction quality at 3.5-bit storage cost. The paper's Algorithm 1 (TurboQuant_mse) specifies an 8-level (3-bit) Lloyd-Max codebook for the 3-bit case:

centroids = {-2.1519, -1.3439, -0.7560, -0.2451, +0.2451, +0.7560, +1.3439, +2.1519}

This uses all 3 bits for reconstruction with theoretical MSE of ~0.0346 for N(0,1). Without the QJL correction path actually implemented, the 4-level codebook gives substantially worse distortion for the same 14-byte block. The qr[4] bytes could be repurposed for 32 additional codebook bits, giving a proper 8-level quantizer that uses all storage meaningfully.

• vec_dot is set to NULL in type_traits_cpu, which means CPU flash attention won't work with TQ3_0 :)

  • Quantized V cache requires FA, so this limits CPU users to K-cache only. A vec_dot_tq3_0_q8_0 implementation (dequant block to float, dot with Q8_0) would close this gap.

• Also. The amax / 1.510 scaling normalizes by the max centroid magnitude. The paper uses RMS-based scaling (sqrt(sum(x_i^2) / n)), which minimizes MSE across the full block rather than anchoring to the single largest element. Worth comparing the two on a perplexity benchmark.

I have a CPU-only implementation with the 8-level codebook + vec_dot on my fork (branch turboquant-tq3_0) if you want to compare approaches. The Vulkan shaders are a great addition - that's something my branch doesn't cover.

If you'll use the 8-level codebook or da vec_dot implementation, feel free to cherry pick from 1fb1fb3ab on my branch :)

[crayolaconsumer]

Thank you for the pointer! i'm very new to this, just chasing what i find but your comment has made a big improvement! i'll outline that AI helped me create this in another PR, fingers crossed!

[dpblnt]

i can't build it

/bin/glslc -fshader-stage=compute --target-env=vulkan1.3 /opt/llama.cpp_21010/ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_cm2.comp -o /opt/llama.cpp_21010/vulkan/ggml/src/ggml-vulkan/vulkan-shaders.spv/flash_attn_f32_f16_tq3_0_cm2.spv -O -DACC_TYPE=float -DACC_TYPEV4=vec4 -DBLOCK_SIZE=QUANT_K_TQ3_0 -DDATA_A_TQ3_0=1 -DDEQUANTFUNC=dequantFuncTQ3_0 -DD_TYPE=float -DD_TYPEV4=vec4 -DFLOAT16=1 -DFLOAT_TYPE=float16_t -DFLOAT_TYPEV4=f16vec4 -DFLOAT_TYPE_MAX=float16_t(65504.0) -DQ_TYPE=float 

/opt/llama.cpp_21010/ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_cm2.comp:233: error: 'dequantFuncTQ3_0' : undeclared identifier
/opt/llama.cpp_21010/ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_cm2.comp:233: error: 'coopMatLoadTensorNV' : no matching overloaded function found
/opt/llama.cpp_21010/ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_cm2.comp:294: error: 'coopMatLoadTensorNV' : no matching overloaded function found
3 errors generated.

there is more
cannot compile flash_attn_f32_f16_tq3_0_f16acc_cm2
cannot compile matmul_tq3_0_f16_cm2
cannot compile matmul_tq3_0_f16_aligned_cm2

and anything that has tq3_0 in it fails here,
mesa-26.0.1
shaderc-2025.5