vulkan: Tweak Xe2 warptile configuration

[TheBlueMatt]

    vulkan: Tweak Xe2 warptile configuration

    On native float matmul shaders, the existing warptile configuration
    for Xe2 ended up spilling quite some registers. By tweaking the
    warptile config we can drive spills to zero and we get a
    substantial speedup in BF16 models, and a small one in others.

    Using the mesa anv driver with the load combining and LICM fix from
    https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15162 and the
    spill-reduction improvements from
    https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/40796 on
    mesa 26.0.3,

    On a single Arc Pro B60:
     * gpt-oss 20B MXFP4 MoE
       pp512: 1356.08 ± 34.83 -> 1378.53 ± 15.17
       pp2048: 1311.92 ± 1.20 -> 1331.65 ± 4.11 (+2%)
       matmul_f16_l spill 75 -> 0, cycles 237414 -> 97336
       tg128: 52.01 ± 0.01 -> 51.88 ± 0.23
     * qwen35moe 35B.A3B Q4_K - Medium:
       pp512: 899.38 ± 16.84 -> 903.65 ± 14.92
       pp2048: 897.72 ± 1.91 -> 900.93 ± 1.83
       matmul_f32_f32_aligned_l spill 66 -> 0, cycles 159052 -> 58102
       matmul_f16_aligned_l spill 68 -> 0, cycles 158332 -> 55054
       matmul_f16_f32_f16acc_aligned_l spill 0 -> 0, cycles 80040 -> 54872
       tg128: 49.31 ± 0.02 -> 49.50 ± 0.01
     * qwen35 9B BF16:
       pp512: 509.34 ± 79.17 -> 844.24 ± 64.5 (+66%)
       pp2048: 564.64 ± 0.95 -> 949.35 ± 1.39 (+68%)
       matmul_bf16_aligned_l spill 47 -> 0, cycles 127438 -> 39124
       tg128: 22.12 ± 0.02 -> 22.12 ± 0.02

    Across four Arc Pro B60s:
     * qwen35moe 122B.A10B Q5_K - Small
       pp512: 268.06 ± 8.07 -> 269.08 ± 7.45
       pp2048: 318.88 ± 4.69 ->  320.80 ± 1.98
       matmul_f32_f32_aligned_l spill 66 -> 0, cycles 159052 -> 58102
       matmul_f16_aligned_l spill 68 -> 0, cycles 158332 -> 55054
       matmul_f16_f32_f16acc_aligned_l spill 0 -> 0, cycles 80040 -> 54872
       tg128: 26.20 ± 0.01 -> 26.40 ± 0.01
     * gemma4 31B BF16
       pp512: 141.92 ± 4.77 -> 222.61 ± 4.58 (+57%)
       pp2048: 162.35 ± 1.42 -> 268.07 ± 6.41 (+65%)
       matmul_bf16_aligned_l spill 48 -> 0, cycles 116834 -> 39124
       tg128: 6.40 ± 0.00 -> 6.41 ± 0.00

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Claude suggested this as a part of a larger re-tuning of the parameters, but after doing my own benchmarks Claude's larger suggestions weren't as good as the more minimal fix here, and its first fix was invalid anyway.

[TheBlueMatt]

Heh, my first attempt at DIYing it was wrong, claude was right though and ultimately its still a material win with it fixed.

[TheBlueMatt]

Apologies, this required a little more tuning but should be ready to go now. This will presumably need more tuning after the next mesa (where coopmat2 will be enabled for more hardware, including Intel devices), but sadly current mesa git is not running properly for me so I can't tune that.

[virajwad]

Hi @TheBlueMatt

The PR is changing coopmat2 warptile, but that isn't supported by Intel, please check it. Can you also please run test-backend-ops and show if all unit tests pass?

[TheBlueMatt]

Yes, the intent of the first commit is to unify which warptile variable is used in the same situation across the coopmat/coopmat2/non-coopmat paths so that its possible to configure them separately in the second commit. The first commit doesn't actually change behavior for any platform.

test-backend-ops passes all unit tests for me on an Xe2 device with mesa 26.0.3 (coopmat, no coopmat2).

edit: Dropped the first commit instead for ease of review. Also fixed an issue which led to bunk bench results, but still a 60% win on BF16 models.

[0cc4m]

I can't check the performance claim, but the change looks good.

[virajwad]

Seeing some significant regression on Panther Lake Xe3 BF16 specifically

[TheBlueMatt]

Hmm, maybe we need to split xe3 from xe2? Can you provide the GGML_VK_PIPELINE_STATS spill/fill output where you're seeing issues? Xe3 has pretty different register allocation options from Xe2 afaiu so I guess it makes sense that those need different tuning.

[TheBlueMatt]

Also I don't have an XE3 device handy, any suggestions for a simple way to detect the arch as XE3 (or vulkaninfo output)?

[0cc4m]

You can often grab that kind of info from https://vulkan.gpuinfo.org/

[virajwad]

Hi @0cc4m Thanks, unfortunately I didn't see any Panther Lake (B390) in the vulkan.gpuinfo, maybe not uploaded yet.

@TheBlueMatt We did some checks on various SKUs, I think the best shot to deliminate Xe2/Xe3 is with the maxComputeWorkgroupSubgroups field from vulkaninfo. On Xe2 it is 64 (for Xe2 IGPU like Lunar Lake and Xe2 DGPU like Battlemage) and on Xe3 it is 80 (Panther Lake, other SKUs). I've uploaded some vulkaninfo logs as examples.

B580_xe2_vulkaninfo.txt
panther_lake_xe3_vulkaninfo.txt
lunar_lake_xe2_vulkaninfo.txt

The current check is like this:

I would do minSubgroupSize == 16 && maxComputeWorkgroupSubgroups == 64 for XE2
minSubgroupSize == 16 && maxComputeWorkgroupSubgroups == 80 for XE3

Since 8->16 was a defining feature for Xe2+ architecture, I would keep that check.

So my suggestion would be we can create 2 separate tunings for Xe2 and Xe3. Your tuning you found can be set for Xe2 to reduce register spill (with check device->architecture == INTEL_XE2) and the current tuning in master can be kept for Xe3 (device->architecture == INTEL_XE3)

Meanwhile I can do some more testing on Xe3 to check the perf drops on your proposed tuning.

[TheBlueMatt]

Easy enough, added the separate detection and only updated the config for xe2.

[virajwad]

Thanks, please give me a bit of time, I want to review and test and will get back

[rillomas]

There's actually another issue I've noticed recently related to BF16. Windows driver does not enable VK_KHR_shader_bfloat16 for Xe2/Xe3 at the moment which is causing scalar kernels to run on Windows (BF16 extension is available for Xe2/Xe3 on Linux). This is causing the following issue:

• Current tile config code does not handle the case where coopmat is enabled but bf16 coopmat is unavailable. As a result, the scalar kernel (matmul_bf16_aligned_fp32.spv) gets executed in a slower pipeline config matmul_bf16_aligned_l, while a faster pipeline matmul_bf16_aligned_m is executed when coopmat is disabled (via set GGML_VK_DISABLE_COOPMAT=1).

We can see this with test-backend-ops.exe perf -o MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1)

[TheBlueMatt]

That seems like a separate issue that we could tune for, or, really, the windows driver could be fixed to expose the actual hardware's functionality.

[rillomas]

That seems like a separate issue that we could tune for, or, really, the windows driver could be fixed to expose the actual hardware's functionality.

I agree. The driver fix is WIP so we may create a different PR to fix the BF16 pipeline mismatch issue for the time being.

[virajwad]

Hi @TheBlueMatt Thank you for your patience!

I did some testing (Windows Xe2/Xe3). The summary is:

• Xe3 perf didn't change with this PR (great, since we kept the master tuning)
• Xe2 perf decreases for BF16 cases when BF16 windows driver support is enabled.

Here is the perf data below:

Actually, the BF16 perf on Xe2 Windows (B580) prefill does significantly better using the current master tuning. For this reason, I can't really recommend this change for Windows. I wonder if this is due to driver differences between OS (windows intel public driver vs mesa driver)

The only solution I can otherwise think of, is the Xe2 tuning is enabled only for Ubuntu (you can set a check via device->driver_id). In all other cases (Xe2 Windows, Xe3 Windows + Ubuntu) the master tuning is kept.

But the mesa driver PR you linked in the description that reduces register spill, it seems is still open on mesa project and not merged into the driver. So if we merge this change, it won't be reflected yet for mesa end users. Do we know what happens to perf if we use the new proposed tuning on an earlier mesa driver (will perf decrease)?

[TheBlueMatt]

Xe2 perf decreases for BF16 cases when BF16 windows driver support is enabled.

Interesting! Can you provide the GGML_VK_PIPELINE_STATS output for the shaders I listed in my description, as well as maybe the performance for one of the models I listed so that we can compare them? I'm curious whether this is a mesa issue or an intel windows driver issue.

But the mesa driver PR you linked in the description that reduces register spill, it seems is still open on mesa project and not merged into the driver. So if we merge this change, it won't be reflected yet for mesa end users. Do we know what happens to perf if we use the new proposed tuning on an earlier mesa driver (will perf decrease)?

Those changes shouldn't have a material impact here (if anything they should improve things as they reduce register pressure somewhat), I believe, I only mention it because it is in the mesa I was using.

[TheBlueMatt]

Just to confirm, I did revert back to upstream mesa 26.0.4 and got the ~same results as I quoted with the patches for qwen35 9B BF16 (though the upstream version regressed in performance somewhat due to a further increase in register spills).

[0cc4m]

LGTM

[virajwad]

@TheBlueMatt Yes I'll get those for you
@0cc4m @TheBlueMatt Before this gets merged, I'd suggest we change Xe2 warptile to Ubuntu only. My current data is showing Windows side is losing perf for BF16

[virajwad]

@TheBlueMatt I'm working to get access to a B60 Arc Pro system, I should have soon, then I will get the perf numbers for Windows so we can compare. Generally I have seen better Vulkan perf on Intel-Windows compared to Intel-Mesa.

The Qwen3.5-9B BF16 model you tried can't fully fit in the B580's VRAM, but I can give GGML_VK_PIPELINE_STATS for the 2 BF16 models I collected data for in my above screenshot.

For Llama3.2 1B BF16 w/ B580 BF16 enabled:
(left is master tuning, right side is this PR tuning) - both on Windows

For Qwen3 4B BF16 w/ B580 BF16 enabled:
(left is master tuning, right side is this PR tuning) - both on Windows

So it looks like there is better perf (specifically Intel-Windows) at the master warptile tuning

[TheBlueMatt]

Wow, yea, mesa is pretty bad by comparison lol. I see matmul_bf16_aligned_l instruction count (the only obviously-comparable metric, though some of that may still just be NOPs so unclear) on current master of 3490 vs your windows of 2080 (qwen3 4B BF16).

What is "Scratch Memory Size" in the windows driver stats, would that include register spills?

Any chance you can pull the full compiled shader from the windows driver? I'd love to compare it to what mesa built. Of course ideally we get this fixed in mesa rather than here but if we can't do comparisons like that I'll update this to detect windows. I'll need to update the tuning anyway since the latest mesa release candidate enables coopmat2 so will need to apply the tuning there.

edit: also, is there any way to tell if the windows driver is compiling this shader with additional registers/halving thread count? I believe mesa refuses to do so but windows might not?

[virajwad]

Hi @TheBlueMatt

I got temporary access to an Arc Pro B60 Windows system, unfortunately I don't have a driver for this card that will enable BF16. So I could only test the driver disabled BF16 scalar path.

Nevertheless, I collected some data:

And some GGML_VK_PIPELINE_STATS:

Qwen3 4B BF16 (driver disabled) for Windows
(left is master tuning, right side is this PR tuning)

Llama 3.2 8B Q4_K_M for Windows (different model this time)
(left is master tuning, right side is this PR tuning)

You could compare at least the Windows non-BF16 data / pipeline_stats VS Ubuntu-Mesa non-BF16 data / pipeline_stats to get an idea of the performance gap b/w the two drivers. I suspect you will see the windows perf better.

To address your Qs -

I am not sure about the Scratch Memory Size - but the statistic is reported via this extension.

I understand why you requested it, but the dump out of Windows GPU ASM would be confidential and I could not share it. I agree ideally if improvements could be made to Intel-Mesa that would be great. I'm also not aware of any way to tell regarding the register/thread count thing that could be exposed.

Specifically in context of this PR:
I think this PR has merit, especially the Xe2/Xe3 differentiation which makes the architectural identification much clearer, but I still think the Xe2 tuning this PR is proposing should be gated to Mesa only until there's evidence it helps on Windows side (so we need to modify this PR) @0cc4m. Right now B580 Windows perf would decrease.

I request we please make that change :)

[virajwad]

Ultimately I am suggesting:

• Xe2 Mesa --> get this PR warptile tuning
• Xe2 Windows + Xe3 (Win / Mesa both) --> get the master tuning

[virajwad]

I think this may not be correct, my understanding from #18178 is the following:

"the mul_mm shader calculates NUM_WARPS = WGSIZE / WARP_SIZE. Say if WGSIZE=512 and WARP=32, then NUM_WARPS=16. Since BM * BN is the tile size that a workgroup operates on, and WM * WN is the tile size that a subgroup/warp operates on, then 2D-wise the number of warps also needs to satisfy (BM * BN)/(WM * WN) = NUM_WARPS (contained within a workgroup)."

Here WN is changed from 32->16. Originally the calculation would be WGSIZE (512) / WARP_SIZE = NUM_WARPS = (128 * 128) / (WARP_SIZE * 32)

In my BMG Xe2 case, WARP_SIZE (same as subgroup_size_8) here is 32. So then NUM_WARPS = 512 / 32 = (128 * 128) / (32 * 32) so it's same on both sides of the equation.

But if it is changed to (128 * 128) / (WARP_SIZE * 16), then it would be NUM_WARPS = 512 / 32 != (128 * 128) / (32 * 16)

I suspect the unit tests are passing because the other changes in the PR remove the usage of l_warptile and instead use l_warptile_mmq / mmq_id, which isn't changed.

@0cc4m can comment better on this since he has deeper understanding of the mul_mm shader.

[virajwad]

Hi @TheBlueMatt

I saw you modified the BN=128->64 and l_wg_denoms 2nd dim to 64, so now the change functionally looks fine to me. I spot checked some perf numbers also on Arc B580, but couldn't check BF16 as not supported in my driver.

b8779 -> your PR: (pp1024)
GPT_OSS 20B Q4_K_M: 2088.64 ± 14.61 --> 2095.83 ± 13.16
DeepSeek R1 Qwen3 8B Q4_K_M: 1754.27 ± 2.09 --> 1743.96 ± 1.25

I'll also check on Xe3 and respond back quickly

[virajwad]

@TheBlueMatt
Ok I tried on Panther Lake Xe3 system also:

b8779 -> your PR: (pp1024)
Qwen3 30B A3B 2507 Q4_K_M: 776.16 ± 6.22 --> 781.41 ± 8.55
Llama3.1 8B Instruct Q4_K_M: 780.75 ± 1.23 --> 899.17 ± 2.65
GPT_OSS 20B Q4_K_M: 962.53 ± 13.37 --> 974.88 ± 16.12
Gemma4 26B A4B Q4_K_M: 789.18 ± 8.15 --> 824.66 ± 41.84
Phi4 Mini Instruct BF16: 1683.59 ± 56.05 --> 1366.33 ± 19.79
Qwen3.5 9B BF16: 875.77 ± 17.37 --> 515.90 ± 4.97

I'm seeing perf regression on Panther Lake BF16 :(

[virajwad]

1. So I am suggesting this new tuning add a check to device->driver_id to set it only for Mesa

[virajwad]

2. Please also add a check for INTEL_XE2 to pass through with this tuning in the case of Windows (both Xe2 and Xe3 use this tuning)

[TheBlueMatt]

Found the issue in mesa - https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15311 and https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15312 together bring the qweb3.5 9b parameter model to 1906 tok/s (from ~550 current master, past the 950 of this PR, with the patch in 15311 alone going to 1700). That is obviously a much better way to go about this, so going to close this.

@virajwad if you wouldn't mind nagging the intel-internal folks who work on mesa about those as well as https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15162 and https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/40798 I'd appreciate it.

[TheBlueMatt]

Oh, and just to confirm, I re-ran this PR against mesa patched with the patches from 15311 and 15312 and it is only very marginally faster than this PR + upstream mesa (955 tok/s instead of 850/950), so skipping this PR and fixing mesa is definitely better.

[TheBlueMatt]

Also, thanks @virajwad for all your testing here! I wouldn't have dug into mesa without it :)

[TheBlueMatt]

And finally while everyone is still looking at this, friendly reminder that #21751 is maybe even simpler even if its a smaller win :)

[virajwad]

Also, thanks @virajwad for all your testing here! I wouldn't have dug into mesa without it :)

No problem. Looking forward to your improvements in Mesa :)