vulkan: restore turbo_wht op + turbo3/4 FA dispatch (regression from b9190 upstream sync)

[TheTom]

Summary

Restores Vulkan turbo3/4 KV cache support that was clobbered during the b9190 upstream-sync merge (e30bbcfe5). Brings back the wiring originally landed in PR #62 (ff8bb7394), adapts it to the post-sync Vulkan API, and adds turbo2/turbo4 FA dispatch + the requiredSubgroupSize=32 constraint that's required on RDNA4.

Tracking: #159 (regression discovery). Closes: #88, #89.

What was broken on current main

• pipeline_turbo_wht member, ggml_vk_turbo_wht dispatch fn, and GGML_OP_TURBO_WHT case handler all missing — graph-level WHT rotation for Q (and inverse rotation post-FA) silently fell back or failed
• CREATE_FA(GGML_TYPE_TURBO3_0, ...) SPIR-V variants not built — FA dispatch had no turbo pipeline to find
• fa_kv_ok rejected turbo types in supports_op(FLASH_ATTN_EXT)
• flash_attn_dequant.glsl had no turbo cases in dequantize4()
• SET_ROWS pipelines never registered for turbo2/turbo3/turbo4 → hard assert at cache allocation

Net effect: --cache-type-k turbo3 --cache-type-v turbo3 on Vulkan failed at startup, and asymmetric configs produced incoherent output.

Fix

Three commits:

1. SET_ROWS pipeline registration for turbo2/3/4

• Adds turbo types to vulkan-shaders-gen.cpp set_rows generation loop
• Registers set_rows_turbo*_0_{i32,i64} pipelines in ggml-vulkan.cpp
• Adds head_dim % 128 == 0 guard in supports_op(SET_ROWS) matching the 128-element block constraint
• Special-cases the dispatch math: turbo set_rows shaders run 128 threads per WG handling one full QK=128 block, so ne = ne / 128 rather than the generic CEIL_DIV(ne, 32 * blck_size)
• requiredSubgroupSize = 32 on the three turbo set_rows pipelines. Without this the RDNA4 driver picked wave64 by default, breaking the subgroupBallot.x 32-lane sign-packing inside the turbo set_rows shader. This is THE root cause of the incoherent decode in Eval bug: Vulkan turbo3 KV produces incoherent decode while HIP on same model is fine (7900 XTX, head_dim=128) #88

2. ggml_turbo_wht Vulkan op

• pipeline_turbo_wht member + creation reusing the existing turbo_wht.comp shader
• ggml_vk_turbo_wht dispatch function (reads direction, group_size from op_params, dispatches with explicit ggml_vk_sync_buffers barriers around for compute-to-compute RAW/WAW ordering)
• GGML_OP_TURBO_WHT case in the build-graph dispatch switch
• supports_op(GGML_OP_TURBO_WHT) returns true for F32 src with ne[0] % 128 == 0

3. Flash Attention dispatch for turbo3 K/V

• Adds fa_kv_ok cases for turbo2/3/4 in supports_op(FLASH_ATTN_EXT)
• Adds turbo K/V SSBO bindings to flash_attn_dequant.glsl (restrict std430 to avoid alias-stride issues on RDNA when the same binding has mismatched-stride aliases)
• Adds FA_DEQUANT4_TURBO{2,3,4}_0 macros: per-element centroid lookup, no iWHT in the kernel because the graph applies WHT to Q pre-attention and inverse WHT to output post-attention (orthogonality: Q · K_rot = Q_rot · K_rot)
• Adds fa_block_elems cases for turbo2/3/4 → QUANT_K_TURBO*_0
• Dedicated DATA_A_TURBO3_0 SPIR-V variant (scalar + cm1 coopmat paths) that gates away the generic f16/q4/q5/q8 binding aliases — eliminates the SSBO alias collision on RDNA where mismatched stride aliases at the same binding caused mis-strided loads
• memoryBarrierShared() added in flash_attn_cm1.comp at the shared-memory load sites for kvsh to fix the cm1 coopmat read-before-write race
• flash_attn.comp and flash_attn_cm1.comp gate the f16 K/V aliases under #if !defined(DATA_A_TURBO3_0) so the turbo3 variant only has turbo bindings at slots 1/2

Note: cm2 (NV cooperative matrix v2) FA path is intentionally skipped in this PR — turbo3 is f32-accumulator only. NV users get the cm1 KHR-coopmat path. cm2 specialization can be a follow-up.

Two real root causes (out of ~8 hypotheses)

1. requiredSubgroupSize = 32 on turbo set_rows — RDNA4 driver picks wave64 by default for compute pipelines unless explicitly constrained. The turbo set_rows shader uses subgroupBallot to pack 32 sign bits into a single uvec4.x word; on wave64 the ballot covered 64 lanes and the sign packing was off by 32. This silently produced garbage in the cache writes. Pinning to wave32 via VkPipelineShaderStageRequiredSubgroupSizeCreateInfo makes the ballot semantics deterministic.

2. ne = ne / 128 dispatch math — turbo set_rows runs one full QK=128 block per workgroup with 128 threads. The generic quant path uses 32 threads per WG with CEIL_DIV(ne, 32 * blck_size). Without the special case, the turbo path was under-dispatched by 32×, leaving most of the cache uninitialized.

The second was the missing piece from the original April implementation — present in ff8bb7394, dropped during the b9190 sync. The first is RDNA4-new: didn't matter on the hardware the April PR was developed on, surfaces on wave-flexible hardware.

Verification

All on RX 9070 XT (RDNA4, gfx1201), Qwen2.5-7B Q4_K_M, current branch HEAD:

Config	pp512	tg128	Status
--cache-type-k f16 --cache-type-v f16 (baseline)	2636 t/s	48.4 t/s	no regression
--cache-type-k q8_0 --cache-type-v turbo4 (recommended)	1951 t/s	49.2 t/s	coherent
--cache-type-k q8_0 --cache-type-v turbo3	~446 t/s	~45.6 t/s	coherent
--cache-type-k q8_0 --cache-type-v turbo2	~453 t/s	~45.2 t/s	coherent

All decode against HIP turbo3 baseline produces coherent output. Prefill on Vulkan now lands at ~3.3× the previously-measured HIP throughput on the same hardware. Decode is ~56% of HIP; there's headroom for follow-up tuning (workgroup-size sweep, possibly turbo3-specific FA pipeline tuning).

#137 (RDNA 3.5 GFX1151 case) not retested here — requires Strix Halo hardware. Likely fixed by the same requiredSubgroupSize=32 mechanism since RDNA 3.5 has the same wave-flexible scheduling, but unverified.

Files touched

• ggml/src/ggml-vulkan/ggml-vulkan.cpp (host-side wiring)
• ggml/src/ggml-vulkan/vulkan-shaders/flash_attn.comp (turbo3 binding guards)
• ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_base.glsl (fa_block_elems cases)
• ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_cm1.comp (turbo3 binding guards + memoryBarrierShared)
• ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_dequant.glsl (turbo K/V bindings + dequant macros + DATA_A_TURBO3_0 specialization)
• ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp (turbo3 FA SPIR-V variants + set_rows turbo + turbo_wht)

212 insertions / 4 deletions across 6 files.

Follow-ups

• vulkan: turbo3/4 decode perf vs HIP gap (~44% headroom on RX 9070 XT) — workgroup-size sweep, possibly cm2 path
• vulkan: cm2 (NV cooperative matrix v2) path for turbo3 FA — NV users currently on cm1, optimal would be cm2
• vulkan: extend DATA_A_TURBO2_0 / DATA_A_TURBO4_0 SPIR-V specialization — turbo2/4 currently use the generic FA SPIR-V which has all binding aliases declared; if RDNA alias issues surface there, dedicated variants would be the fix
• Eval bug: -ctk turbo3 -ctv turbo3 corrupts decode on Gemma 4 31B (Vulkan / RDNA 3.5 / GFX1151) — likely K head_dim 512 path, not ISWA #137 RDNA 3.5 verification on Strix Halo hardware

[TheTom]

@Titaniumtown — heads up, this PR restores most of your PR #62 (ff8bb7394) work that got clobbered during the b9190 upstream sync. Co-authored on the commit. Two additions on top: requiredSubgroupSize=32 on the turbo set_rows pipelines (needed on RDNA4 where the driver picks wave64 by default and breaks the 32-lane subgroupBallot sign-packing) and the ne = ne / 128 dispatch math for the turbo set_rows path. Happy to adjust attribution if you'd prefer it framed differently.