Eval bug: -ctk turbo3 -ctv turbo3 corrupts decode on Gemma 4 31B (Vulkan / RDNA 3.5 / GFX1151) — likely K head_dim 512 path, not ISWA

[Saturcolor]

TL;DR

On AMD Strix Halo (Radeon 8060S Graphics, RADV GFX1151) with Vulkan, -ctk turbo3 -ctv turbo3 produces gibberish from the very first decode tokens on Gemma 4 31B. Reproduces with both a stock unsloth Q4_K_M and a custom 6-bit GGUF, so the weight quant is not the variable. Same binary, same hardware, same flags works cleanly on Qwen 3.6 35B-A3B (qwen35moe, head_dim K = 256) and on Gemma 4 31B with f16 KV.

Initial guess was Gemma 4's ISWA hybrid cache. After a quant-control test pass, head_dim K = 512 on Gemma 4 global-attention layers looks like a more likely trigger than the SWA/full alternation itself.

This is potentially the same family as #88 (Vulkan turbo3 incoherent decode while HIP works), but with a different repro (different model, different RDNA generation), so I'm filing it separately.

Environment

• Hardware: AMD Ryzen AI Max+ 395 (Strix Halo APU), Radeon 8060S Graphics (RADV GFX1151), 124 GiB unified memory free
• OS: Fedora 43, kernel 6.19.14-200.fc43.x86_64
• Vulkan: API 1.4.328, driver RADV, Mesa 25.3.6
• Build: 2e81dc5f6 (current default branch of AtomicBot-ai/atomic-llama-cpp-turboquant, which forks this repo)
• CMake: -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release -DGGML_RPC=ON
• Coopmat: both GL_KHR_cooperative_matrix and GL_NV_cooperative_matrix2 reported supported by glslc at configure time
• HIP not available on this APU, so I cannot run the HIP-vs-Vulkan A/B that Eval bug: Vulkan turbo3 KV produces incoherent decode while HIP on same model is fine (7900 XTX, head_dim=128) #88 uses.

Repro matrix

All four runs share the same binary, hardware, and flags -c 8192..16384 -ngl 999 -fa on --host 127.0.0.1 --port 8088 plus -ctk turbo3 -ctv turbo3 unless noted.

#	Model	Arch	Weight quant	KV cache layout	turbo3 KV result
1	gemma-4-31B-it (custom Mercury 6-bit)	gemma4	6-bit	ISWA: 10×16384 global + 50×4608 SWA	❌ gibberish from token 30
2	gemma-4-31B-it-Q4_K_M (unsloth, stock)	gemma4	Q4_K_M	ISWA: 10×16384 global + 50×4608 SWA	❌ gibberish from token 30
3	Qwen3.6-35B-A3B-Uncensored (Q8-grade)	qwen35moe	Q8-grade	uniform 40×8192, head_dim K = 256	✅ coherent (short, 1k, 2.5k prompts)
4	gemma-4-31B-it-Q4_K_M with -ctk f16 -ctv f16	gemma4	Q4_K_M	ISWA same as #2 but f16	✅ coherent

Run #2 is the key control: stock Q4_K_M, file type printed as Q4_K - Medium, general.quantization_version = 2. Same gibberish as the custom 6-bit. So this is not a custom-quant interaction.

Run #3 is the second key control: same backend (Vulkan), same binary, same flags, same hardware, including turbo3 KV, on a Q8-grade Qwen MoE. Coherent across short / 1k / 2.5k-token prompts. So this is not "Vulkan turbo3 general bug on GFX1151".

Sample broken outputs (Gemma 4 31B stock Q4_K_M + turbo3)

Prompt: "The capital of France is", n_predict=30, temp=0.7:

B evenCL de ownL неBL-LBK неL неCCSCCBBSCCSSH

Prompt ~1021 tokens of coherent text + summarize instruction, n_predict=50, temp=0.3:

 smallC smallCC la smallC smalledCPC laCC deC laCBC laCB deCBCCCC orCCCBTCeC smallCCPH laCCK

Prompt ~2533 tokens (well past the 1024-token SWA window):

SSSK queCC queCSCCCBClappingSS que smallC laS la smallSLH la laLS laSS queSBCCCCC1S0 small small laC

Symptom: garbage tokens (Latin + Cyrillic + CJK fragments) from the very first decoded token. No transient correctness then drift — broken from t=0.

Sample working outputs

Gemma 4 31B Q4_K_M, drop turbo3, keep everything else:

 Paris.

Qwen 3.6 35B-A3B, same binary, same -ctk turbo3 -ctv turbo3, short prompt:

 Paris.\nA. 2nd & 3rd\nB. 1st & 2nd\nC. 2 & 4

Same Qwen on a 2.9k-token prompt: coherent summary, EOS at 32 tokens.

Mitigations attempted (none fix it)

Variant	Result
-fa off (disable flash-attn)	still gibberish
GGML_VK_DISABLE_COOPMAT=1 GGML_VK_DISABLE_COOPMAT2=1	still gibberish
-ctk turbo3 -ctv f16 (asymmetric)	still gibberish
-ctk f16 -ctv turbo3 (asymmetric)	still gibberish
-ctk turbo4 -ctv turbo4	GGML_ABORT (expected per README — Vulkan only ships turbo3 attention kernels)

Suspected cause — K head_dim 512 on Gemma 4 global, not ISWA per se

I started out blaming the ISWA hybrid cache, but the Qwen control above narrows it. The discriminating axis between the broken Gemma runs and the working Qwen run is not archi-vs-archi or quant-vs-quant in the obvious way:

	Gemma 4 31B (broken)	Qwen 3.6 35B-A3B (working)
Backend / hardware / driver	Vulkan, GFX1151, Mesa 25.3.6	identical
-ctk/-ctv turbo3 -fa on	yes	yes
KV layout	2 caches (ISWA), interleaved SWA(1024)/global	1 uniform cache
K head_dim	512 on global, 256 on SWA	256 uniform
V head_dim	256	128

WHT-rotated KV kernels typically operate on power-of-2 blocks (128 is the canonical block size for the turbo3 rotation in this fork's CUDA path; Vulkan coopmat is 16x16 / 32x16 tiled inside). A K head_dim of 512 forces the rotation to chain over four 128-blocks, which is unusual — most models (Llama-family, Qwen, Mistral) sit at 128 K head_dim, some Gemma variants and a few others sit at 256, but 512 is rare. If the Vulkan turbo3 K path is correct at 128 and 256 but mistiles or under-reduces at 512, the Gemma 4 global layers would corrupt write-side immediately, which is what we see.

ISWA itself is probably not the trigger: the SWA layers in Gemma 4 use K head_dim 256, same as Qwen, and Qwen is fine. So the broken layers in this scheme would be the 10 global-attention layers, head_dim 512, not the 50 SWA ones.

I have not bisected this — happy to instrument the K-write path if you can point me at it.

Relation to existing issues

• Eval bug: Vulkan turbo3 KV produces incoherent decode while HIP on same model is fine (7900 XTX, head_dim=128) #88 (Vulkan turbo3 incoherent decode on Qwen3-8B-Q4_K_M, RX 7900 XTX gfx1100, RADV Mesa 25.x): same family of symptom (Vulkan turbo3 → garbage tokens), different repro. My Qwen 3.6 35B-A3B run on a similar driver stack (RDNA 3.5 instead of RDNA 3) does not reproduce Eval bug: Vulkan turbo3 KV produces incoherent decode while HIP on same model is fine (7900 XTX, head_dim=128) #88 — so the trigger there is something else (possibly Q4_K_M weight × turbo3 V on RDNA 3, or a Qwen 3 dense path that diverges from Qwen 3.6 MoE).
• Eval bug: broken vulkan on Bazzite Linux #64 (Gemma-4-26B-A4B + turbo3 + flash-attn assertion failure on RX 9070 XT / Bazzite): same model family, but a hard GGML_ASSERT at warmup, not silent corruption. Probably a different code path (FA assert vs. K-write rotation).

What I'd find useful

• A pointer to where K head_dim is dispatched in the Vulkan turbo3 KV write/read path, so I can patch-test with explicit 128-block chaining at head_dim 512.
• Or a known-good Gemma 4 31B + turbo3 run on Vulkan from the maintainers (any GPU) — would confirm whether this is GFX1151 / Mesa 25.3.6 specific, vs. Vulkan-wide for Gemma 4 global layers.
• Whether the CUDA turbo3 K path special-cases head_dim 512 in a way the Vulkan path doesn't.

Happy to run any test build / patch — Strix Halo is the only piece of hardware here so iteration is fast.

Other notes

• MTP works perfectly on the same Gemma 4 31B model (--mtp-head + --spec-type mtp), ~2.3× speedup over baseline, with -ctk f16 -ctv f16. So the bug is purely on the -ctk turbo3 write-or-read path on Vulkan, not in MTP, not in the GGUF, not in flash-attn-on-Gemma in general.
• Repro logs (load logs with the llama_kv_cache: block confirming head dims, full curl outputs, all four runs) available on request — kept under /tmp/turbo3-repro/ on my box.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.