Vulkan TG Performance Degradation at Large Context Sizes on RDNA4

[DerDudeLOL]

Vulkan TG Performance Degradation at Large Context Sizes on RDNA4

Summary

Text generation (TG) throughput degrades ~36% when increasing context from short prompt to 50k tokens on AMD RX 9070 XT (RDNA4) using Vulkan backend. The degradation is not caused by KV-cache bandwidth alone — analysis shows KV-cache is only 21% of total memory reads at 50k context for this MoE model. The root cause appears to be a combination of L2 cache thrashing, split_k overhead scaling, and memory latency — likely exacerbated by RADV driver maturity on RDNA4.

Note: This issue was written with AI assistance (model: opencode/mimo-v2.5-free). All measurements and code analysis were verified independently by multiple AI agents. I'm a hobbyist user, not a professional GPU programmer — community help in diagnosing and fixing this would be greatly appreciated.

Hardware

• GPU: AMD Radeon RX 9070 XT (gfx1201), 16 GB GDDR6
• CPU: AMD Ryzen 7 7700 (iGPU disabled)
• OS: Arch Linux (CachyOS), kernel 7.0.11-1-cachyos
• Driver: Mesa 26.1.2 (RADV), Vulkan 1.4.350
• VRAM Bandwidth: 640 GB/s (peak theoretical)

Model

• Model: Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (Mixture-of-Experts, 35B total, ~3.3B active per token)
• Architecture: 10 hybrid attention layers (GQA 16Q:2KV, head_dim=256) + 30 DeltaNet recurrent layers + 256 routed experts (9 active)
• Weight quant: IQ3_XXS (3.25 bits/param)
• KV cache: q8_0 K (1 byte/element) + turbo3 V (WHT-compressed, ~0.5 bytes/element, 75% VRAM savings vs fp16)
• Context length: 262k max

Reproduction

Benchmark command

GGML_VULKAN=1 ./llama-bench -m Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf \
  -ngl 99 -fa 1 -ctk q8_0 -ctv turbo3 \
  -p <PROMPT_SIZE> -n 200 -t 16 -b 4096 -dev Vulkan0

Measurements

Prompt Size	TG (t/s)	Change vs Short
Short (~100 tokens)	102.9	—
~10,000 tokens	79.6	-22.6%
~50,000 tokens	66.2	-35.7%

Exclusion tests (all at ~10k context)

Test	TG (t/s)	Conclusion
Baseline	79.6	—
FA disabled (-fa 0)	79.7	NOT FA overhead
RADV_PERFTEST=nogttspill	79.4	NOT GTT spilling
Different source copy (backup)	79.4	NOT code regression
Short prompt (same binary)	102.9	NOT kernel regression

Analysis

Memory bandwidth math

Per-token memory reads for this MoE model:

Source	Size	Notes
Model weights	1.34 GB	3.3B active × 0.406 bytes/param (IQ3_XXS)
DeltaNet state	30 MB	Fixed, 16 heads × 128² × 4B × 30 layers
KV cache	7.68 KB/token	K: 512B + V: 256B × 10 layers

Context	KV Cache	Total Read	Theoretical Max TG	KV % of Total
2k	15 MB	1.39 GB	462 t/s	1.1%
10k	75 MB	1.45 GB	441 t/s	5.2%
50k	375 MB	1.77 GB	363 t/s	21.2%

Key finding: KV-cache bandwidth only explains ~21% of the theoretical drop (462→363 = -21.4%). But the observed drop is 35.7%. The remaining ~14% comes from other factors.

Flash Attention split_k behavior

The FA split_k parameter scales with context size (verified by tracing the code):

Context	split_k	Dispatch size (x × y × z)
2k	32	512 × 2 × 1 = 1,024 workgroups
10k	53	848 × 2 × 1 = 1,696 workgroups
50k	61	976 × 2 × 1 = 1,952 workgroups

split_k increases from 32→61, meaning:

• 2× more pipeline barriers between split_k invocations
• 2× more split_k_reduce dispatches
• Each barrier is a full VkMemoryBarrier (not buffer-specific), potentially evicting L2 cache for all buffers

What's NOT the cause

1. GTT spilling — nogttspill makes no difference
2. Code changes — Different source copies produce identical performance
3. FA overhead — Disabling FA makes no difference at 10k context
4. Kernel regression — Short prompt still achieves 102.9 t/s

Hypothesis

The degradation is caused by a combination of:

1. L2 cache thrashing: At 2k context, KV cache (15 MB) partially fits in RDNA4's L2 cache (2-4 MB per CU). At 50k (375 MB), it doesn't fit → all KV reads go to VRAM with higher latency.

2. split_k overhead scaling: More splits = more barriers = more synchronization overhead. Each barrier is a global VkMemoryBarrier instead of a buffer-specific VkBufferMemoryBarrier, potentially evicting L2 cache for unrelated buffers (weights, DeltaNet state).

3. Memory latency stalls: Larger KV cache = more random access patterns during attention computation, increasing average memory latency.

4. Partial compute-boundness: Bandwidth utilization is only ~25% (actual 114 t/s vs theoretical 462 t/s), meaning significant time is spent on compute (MoE FFN, attention matmuls, WHT butterfly). When memory latency increases, the compute pipeline stalls more often.

Questions for the community

1. Is this expected behavior for MoE models on Vulkan? The 35.7% degradation from 2k→50k context seems high compared to dense models.

2. Could the FA split_k barrier implementation be optimized? Using VkBufferMemoryBarrier instead of VkMemoryBarrier might reduce L2 cache eviction.

3. Is there a way to hint the driver about buffer access patterns? RDNA4's L2 cache is small (2-4 MB per CU), and the current code doesn't seem to optimize for this.

4. Would this improve with a newer RADV driver? The RX 9070 XT (gfx1201) is relatively new, and driver maturity might affect memory latency behavior.

5. Should split_k be capped for TG mode? With only 1 query token (N=1→8 after GQA), 32-61 splits seems excessive.

Environment

• llama.cpp: commit e95345cf from TheTom/llama-cpp-turboquant fork (analysis focuses on upstream Vulkan FA code)
• Note: This issue was also posted to ggml-org/llama.cpp#24483 for broader visibility, and to TheTom/llama-cpp-turboquant#180 for the fork-specific context
• Build: -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
• Server flags: --cache-type-k q8_0 --cache-type-v turbo3 -fa on -ngl -1 --poll 0 --poll-batch 0

Additional context

Other llama.cpp projects on RDNA4 (e.g., TurboQuant KV-cache optimization) have been paused due to similar RADV driver maturity concerns. The Vulkan backend's performance on RDNA4 may need driver-level improvements for optimal large-context behavior.

TG throughput is highly unstable on RDNA4

Text generation throughput varies dramatically between runs and is not reproducible. Measurements taken with the same binary, same model, same flags, same GPU state show large variance:

Run	Context	TG (t/s)
Previous measurement (same binary)	~50k	66.2
Current measurement (same binary, same flags)	~50k	40.0
Short prompt (baseline)	short	107.9

The TG throughput at 50k context dropped from 66.2 to 40.0 t/s — a 39% variance between runs with identical configuration. This is not a thermal or clock issue: the GPU was idle between runs, and no other workloads were running.

Combined with the previously measured context-size degradation (102.9 → 66.2 t/s, -35.7%), the worst-case TG at 50k context is now 62% slower than short-prompt TG (107.9 → 40.0 t/s).

This instability makes it impossible to reliably benchmark or optimize the Vulkan backend on RDNA4. Any optimization work is obscured by the non-deterministic performance variance.

TurboQuant KV-cache is SLOWER than FP16/Q8 — Help needed

We are developing TurboQuant, a WHT-based (Walsh-Hadamard Transform) KV-cache quantization format that achieves 75% VRAM savings compared to FP16 (turbo3: ~0.5 bytes/element vs FP16: 2 bytes/element). However, on RDNA4/Vulkan, turbo3 V-cache is significantly slower than both FP16 and Q8_0, which defeats the purpose of the compression.

Performance comparison (Qwen3.6-35B-A3B, Vulkan, RX 9070 XT)

KV-cache config	TG (t/s)	VRAM savings vs FP16	Status
K=f16, V=f16	125.75	— (baseline)	Fastest
K=q8_0, V=q8_0	124.08	~50%	Nearly same as FP16
K=q8_0, V=turbo3	114.41	~75%	9% slower than FP16
K=q8_0, V=turbo4	111.70	~80%	11% slower than FP16

What we've already optimized (turbo3 V-cache)

• WHT butterfly in shared memory (barrier-safe)
• Precomputed scaled centroids in FA dequant macros
• attn * norm scalar precompute in V-dequant inner loop
• SHMEM_STAGING for AMD RDNA
• 2-barrier WHT via subgroupShuffleXor

The paradox

Turbo3 reads 3× less data from VRAM than FP16 (0.5 bytes/element vs 2 bytes/element), yet it is 9% slower. This suggests the bottleneck is not memory bandwidth but rather:

1. WHT decompression overhead: The Walsh-Hadamard Transform adds compute (butterfly operations + shared memory barriers) that costs more than the bandwidth savings.
2. Vulkan pipeline overhead: Quantized KV-cache dispatch may have more pipeline barriers, descriptor set bindings, or specialization constant overhead.
3. RDNA4-specific: On ROCm/HIP, turbo3 achieves 91% of FP16 speed (86.5 vs 95 t/s). The Vulkan backend may not be optimizing the quantized path as well.

Questions for the community

6. Is there a way to reduce WHT decompression cost in the FA V-dequant path? The WHT butterfly requires shared memory barriers that stall the pipeline.
7. Could the quantized V-cache path use cooperative matrix operations? Currently, turbo3 dequant is scalar — cooperative matrix might be faster.
8. Is the Vulkan backend's quantized KV-cache path optimized for RDNA4? The HIP backend achieves better relative performance (91% vs 89% of FP16).
9. Would a simpler compression scheme (e.g., Q4_0 V-cache) be faster than WHT-based turbo3? The WHT overhead might not be worth the extra compression.

Thank you for any help or insights the community can provide.

[Eiion]

While I do not have such nice documentation as the user above, I can confirm that certain models do degrade in their token generation speed using an RX9070XT from e.g. 12t/s down down to 7t/s over the course of about 4 requests, for large context windows (e.g. 128k) used for large token contexts (starting with about 32k - i.e. about 3600 lines of JavaScript).

But as mentioned, this is not true for all models. While I unfortunately cannot name affected models from the top of my head, about two weeks ago I did test several models and not all of them showed the same behavior. (Since I have deleted most of them again, I cannot check which ones degraded and which ones did not.)
My advice would be to test other models yourself to find those that do not degrade. While I'm no expert in LLMs, I assume that this is an effect of how certain models are setup to work. (Feel freer to correct me if my assumption is wrong.)