Misc. bug: vulkan prompt processing suddenly slows down once I reach a certain prompt size

[netrunnereve]

This is a weird issue that I'm currently looking into, and I'll keep this open to share whatever I can find and also to see if anyone else is seeing this!

I'm running llama.cpp Vulkan on two GPUs, a FirePro W8100 and a RX 470 using Ubuntu 24 and RADV. On certain models prompt processing speed drops significantly once the prompt size reaches a certain amount. Text generation is perfectly fine and this only affects prompt processing.

For example here's Mistral Small 22B quantized with Q4_K_M and Q4_K_S, fully offloaded with -sm layer. Once the prompt hits 385 tokens for Q4_K_M or 202 tokens for Q4_K_S it slows down by more than half.

model	size	params	backend	ngl	threads	main_gpu	test	t/s
llama ?B Q4_K - Medium	12.42 GiB	22.25 B	Vulkan	100	8	1	pp384	40.37 ± 0.00
llama ?B Q4_K - Medium	12.42 GiB	22.25 B	Vulkan	100	8	1	pp385	16.00 ± 0.00
llama ?B Q4_K - Small	11.79 GiB	22.25 B	Vulkan	100	8	1	pp201	32.42 ± 0.00
llama ?B Q4_K - Small	11.79 GiB	22.25 B	Vulkan	100	8	1	pp202	9.17 ± 0.00

My GPUs aren't running out of VRAM and they're not swapping to GTT. Also the GPUs normally run right up to their power and tdc limits during prompt processing, but in this case as soon as the prompt size hits that amount the entire prompt processing segment runs at much lower power. For example my 470 runs at 120W during pp384 but it only hits around 80W or so during pp385. The memory clock also runs at a lower speed during pp385 but locking it at its maximum value didn't do a thing. It's almost as if the GPU is somehow running a different shader there.

Now if I decrease -ub then magically the prompt processing speeds return to normal. However that's not a real fix since if I increase the prompt size some more it breaks again at pp577.

model	size	params	backend	ngl	threads	n_ubatch	main_gpu	test	t/s
llama ?B Q4_K - Medium	12.42 GiB	22.25 B	Vulkan	100	8	256	1	pp385	35.52 ± 0.00
llama ?B Q4_K - Medium	12.42 GiB	22.25 B	Vulkan	100	8	256	1	pp576	39.88 ± 0.00
llama ?B Q4_K - Medium	12.42 GiB	22.25 B	Vulkan	100	8	256	1	pp577	21.13 ± 0.00

This also happens regardless if I set my main gpu to something else or if I have flash attention on or off.

[jeffbolznv]

There are cutoffs at certain sizes where we switch between different tile sizes. When I run -p 60-70 for example there's a cutoff at 65 where it switches from the medium to large tile size. I'm seeing a really long compile time (minutes) which I think is for the aligned large shader. This isn't specific to multigpu, so I'm not sure it's the same thing you're seeing, but some of the sizes do line up (e.g. 577 with batch size of 512 will run the matmul shader with N=65).

Once the compile succeeds the shader itself isn't terribly slow, but it does have some wasted work (N=128 worth of work when only N=65 is needed) so running as low as half speed isn't too surprising.

I tested this with coopmat1/2 disabled. The coopmat1/2 shaders don't have this problem (they still have some wasted work, but even that is not as severe, and coopmat2 specifically has some code to minimize it).

[IMbackK]

The fact that the power and memory clocks fall suggests that the shader IS terribly slow and is stalling the cus for some reason, as opposed to wasted work.

[netrunnereve]

This is happening on a single GPU as well with the same Mistral Small model quantized to iq1_s so that it actually fits on my card. Smaller models like Llama 7B/8B and Mistral 7B simply don't have this issue which is strange.

model	size	params	backend	ngl	threads	main_gpu	sm	test	t/s
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp640	49.20 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp641	17.90 ± 0.00

Now if I run the new vulkan perf feature (that's why it's useful!) on this model the results are interesting.

640 tokens:

Vulkan Timings:
ADD: 110 x 357.371 us
CONT: 56 x 297.145 us
CPY: 112 x 250 us
MUL: 166 x 1194.9 us
MUL_MAT m=1024 n=512 k=6144: 112 x 3189.58 us
MUL_MAT m=128 n=512 k=512: 56 x 2088.48 us
MUL_MAT m=16384 n=512 k=6144: 110 x 45203.3 us
MUL_MAT m=512 n=512 k=128: 56 x 3399.48 us
MUL_MAT m=6144 n=512 k=16384: 55 x 44590.9 us
MUL_MAT m=6144 n=512 k=6144: 112 x 16881.2 us
RMS_NORM: 111 x 277.055 us
ROPE: 112 x 157.384 us
SILU: 55 x 636.325 us
SOFT_MAX: 56 x 1046.07 us
----------------
Vulkan Timings:
ADD: 112 x 197.301 us
CONT: 56 x 138.054 us
CPY: 112 x 34.905 us
GET_ROWS: 2 x 5.36 us
MUL: 169 x 199.987 us
MUL_MAT m=1024 n=128 k=6144: 112 x 1210.99 us
MUL_MAT m=128 n=128 k=640: 56 x 949.517 us
MUL_MAT m=16384 n=128 k=6144: 110 x 10046.7 us
MUL_MAT m=6144 n=128 k=16384: 55 x 12518.6 us
MUL_MAT m=6144 n=128 k=6144: 112 x 4243.68 us
MUL_MAT m=640 n=128 k=128: 56 x 1401.96 us
MUL_MAT_VEC m=16384 k=6144: 2 x 620 us
MUL_MAT_VEC m=32768 k=6144: 1 x 2145.6 us
MUL_MAT_VEC m=6144 k=16384: 1 x 641.76 us
RMS_NORM: 113 x 162.4 us
ROPE: 112 x 77.804 us
SILU: 56 x 113.794 us
SOFT_MAX: 56 x 650.117 us
----------------
Vulkan Timings:
ADD: 110 x 357.473 us
CONT: 56 x 280.377 us
CPY: 112 x 243.92 us
MUL: 166 x 1176.3 us
MUL_MAT m=1024 n=512 k=6144: 112 x 3204.76 us
MUL_MAT m=128 n=512 k=512: 56 x 2030.11 us
MUL_MAT m=16384 n=512 k=6144: 110 x 45319 us
MUL_MAT m=512 n=512 k=128: 56 x 3402.19 us
MUL_MAT m=6144 n=512 k=16384: 55 x 44904.7 us
MUL_MAT m=6144 n=512 k=6144: 112 x 16352 us
RMS_NORM: 111 x 276.465 us
ROPE: 112 x 139.425 us
SILU: 55 x 670.248 us
SOFT_MAX: 56 x 1000.97 us
----------------
Vulkan Timings:
ADD: 112 x 194.527 us
CONT: 56 x 140.802 us
CPY: 112 x 34.638 us
GET_ROWS: 2 x 4.56 us
MUL: 169 x 186.51 us
MUL_MAT m=1024 n=128 k=6144: 112 x 1204.85 us
MUL_MAT m=128 n=128 k=640: 56 x 955.408 us
MUL_MAT m=16384 n=128 k=6144: 110 x 9888.52 us
MUL_MAT m=6144 n=128 k=16384: 55 x 12586.3 us
MUL_MAT m=6144 n=128 k=6144: 112 x 4225.4 us
MUL_MAT m=640 n=128 k=128: 56 x 1399.33 us
MUL_MAT_VEC m=16384 k=6144: 2 x 298.56 us
MUL_MAT_VEC m=32768 k=6144: 1 x 1050.4 us
MUL_MAT_VEC m=6144 k=16384: 1 x 319.36 us
RMS_NORM: 113 x 160.157 us
ROPE: 112 x 77.42 us
SILU: 56 x 105.188 us
SOFT_MAX: 56 x 653.211 us
| llama ?B IQ1_S - 1.5625 bpw    |   4.50 GiB |    22.25 B | Vulkan     | 100 |       8 |          1 |  none |           pp640 |         49.09 ± 0.00 |

641 tokens:

Vulkan Timings:
ADD: 110 x 7702.49 us
CONT: 56 x 4634.3 us
CPY: 112 x 593.861 us
MUL: 166 x 9982.53 us
MUL_MAT m=1024 n=512 k=6144: 112 x 4667.16 us
MUL_MAT m=128 n=512 k=512: 56 x 23931.1 us
MUL_MAT m=16384 n=512 k=6144: 110 x 48303.7 us
MUL_MAT m=512 n=512 k=128: 56 x 16557.5 us
MUL_MAT m=6144 n=512 k=16384: 55 x 260782 us
MUL_MAT m=6144 n=512 k=6144: 112 x 18025.7 us
RMS_NORM: 111 x 7727.9 us
ROPE: 112 x 2571.81 us
SILU: 55 x 12222.3 us
SOFT_MAX: 56 x 28192.6 us
----------------
Vulkan Timings:
ADD: 112 x 1916.04 us
CONT: 56 x 1192.18 us
CPY: 112 x 141.38 us
GET_ROWS: 2 x 13.28 us
MUL: 169 x 2494.66 us
MUL_MAT m=1024 n=129 k=6144: 112 x 1793.79 us
MUL_MAT m=128 n=129 k=672: 56 x 6004.53 us
MUL_MAT m=16384 n=129 k=6144: 110 x 15793.1 us
MUL_MAT m=6144 n=129 k=16384: 55 x 16848.2 us
MUL_MAT m=6144 n=129 k=6144: 112 x 6023.53 us
MUL_MAT m=672 n=129 k=128: 56 x 5500.91 us
MUL_MAT_VEC m=16384 k=6144: 2 x 612.88 us
MUL_MAT_VEC m=32768 k=6144: 1 x 4177.92 us
MUL_MAT_VEC m=6144 k=16384: 1 x 628.32 us
RMS_NORM: 113 x 1883.86 us
ROPE: 112 x 674.748 us
SILU: 56 x 3045.2 us
SOFT_MAX: 56 x 6858.14 us
----------------
Vulkan Timings:
ADD: 110 x 7723.2 us
CONT: 56 x 4634.3 us
CPY: 112 x 593.87 us
MUL: 166 x 9990.22 us
MUL_MAT m=1024 n=512 k=6144: 112 x 4674.77 us
MUL_MAT m=128 n=512 k=512: 56 x 23731.9 us
MUL_MAT m=16384 n=512 k=6144: 110 x 48324.9 us
MUL_MAT m=512 n=512 k=128: 56 x 16554.4 us
MUL_MAT m=6144 n=512 k=16384: 55 x 257358 us
MUL_MAT m=6144 n=512 k=6144: 112 x 18009.1 us
RMS_NORM: 111 x 7724.54 us
ROPE: 112 x 2566.66 us
SILU: 55 x 12206.2 us
SOFT_MAX: 56 x 28156.9 us
----------------
Vulkan Timings:
ADD: 112 x 1916.01 us
CONT: 56 x 1191.44 us
CPY: 112 x 140.857 us
GET_ROWS: 2 x 13.2 us
MUL: 169 x 2495.33 us
MUL_MAT m=1024 n=129 k=6144: 112 x 1782.46 us
MUL_MAT m=128 n=129 k=672: 56 x 5993.36 us
MUL_MAT m=16384 n=129 k=6144: 110 x 15807 us
MUL_MAT m=6144 n=129 k=16384: 55 x 16929.8 us
MUL_MAT m=6144 n=129 k=6144: 112 x 6017.56 us
MUL_MAT m=672 n=129 k=128: 56 x 5508.23 us
MUL_MAT_VEC m=16384 k=6144: 2 x 610.8 us
MUL_MAT_VEC m=32768 k=6144: 1 x 4172.16 us
MUL_MAT_VEC m=6144 k=16384: 1 x 640 us
RMS_NORM: 113 x 1886.26 us
ROPE: 112 x 675.057 us
SILU: 56 x 3041.49 us
SOFT_MAX: 56 x 6855.54 us
| llama ?B IQ1_S - 1.5625 bpw    |   4.50 GiB |    22.25 B | Vulkan     | 100 |       8 |          1 |  none |           pp641 |         17.68 ± 0.00 |

640 tokens:

MUL_MAT m=1024 n=512 k=6144: 112 x 3189.58 us
MUL_MAT m=128 n=512 k=512: 56 x 2088.48 us
MUL_MAT m=16384 n=512 k=6144: 110 x 45203.3 us
MUL_MAT m=512 n=512 k=128: 56 x 3399.48 us
MUL_MAT m=6144 n=512 k=16384: 55 x 44590.9 us
MUL_MAT m=6144 n=512 k=6144: 112 x 16881.2 us

641 tokens:

MUL_MAT m=1024 n=512 k=6144: 112 x 4667.16 us
MUL_MAT m=128 n=512 k=512: 56 x 23931.1 us
MUL_MAT m=16384 n=512 k=6144: 110 x 48303.7 us
MUL_MAT m=512 n=512 k=128: 56 x 16557.5 us
MUL_MAT m=6144 n=512 k=16384: 55 x 260782 us
MUL_MAT m=6144 n=512 k=6144: 112 x 18025.7 us

If this is a cutoff issue this wouldn't make sense since both these sections are run with the exact same matrix sizes and should be calculated the same way in the code. Here's a test-backend-ops on on a m=6144 k=16384 matrix with different ns:

  MUL_MAT(type_a=iq1_s,type_b=f32,m=6144,n=200,k=16384,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):                      42 runs - 24452.67 us/run -  40.27 GFLOP/run -   1.65 TFLOPS
  MUL_MAT(type_a=iq1_s,type_b=f32,m=6144,n=300,k=16384,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):                      32 runs - 32773.28 us/run -  60.40 GFLOP/run -   1.84 TFLOPS
  MUL_MAT(type_a=iq1_s,type_b=f32,m=6144,n=512,k=16384,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):                      19 runs - 55025.42 us/run - 103.08 GFLOP/run -   1.87 TFLOPS
  MUL_MAT(type_a=iq1_s,type_b=f32,m=6144,n=600,k=16384,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):                      15 runs - 67378.93 us/run - 120.80 GFLOP/run -   1.79 TFLOPS
  MUL_MAT(type_a=iq1_s,type_b=f32,m=6144,n=700,k=16384,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):                      14 runs - 75372.29 us/run - 140.93 GFLOP/run -   1.87 TFLOPS

This time the numbers are pretty stable.

[0cc4m]

I can't see the same effect on Radeon Pro VII.

[netrunnereve]

I can't see the same effect on Radeon Pro VII.

Interesting. Which Mistral Small quant did you try and did you run with FP16 and DP4A turned off? AFAIK you're also on Ubuntu 24 as well but I think you're on Mesa 25 whereas I'm still on the default 24.2.8.

[0cc4m]

It was a Q4_K_S I believe, and I tried with those features off, yeah. Ubuntu 24 and Mesa 25 is also correct, I think. I won't be around for the next week to try again.

[netrunnereve]

Unfortunately upgrading to Mesa 25.1.1 didn't help, and I also tried switching to FP16 and forcing the small and large mul mat block sizes. RGP traces don't work on Polaris and older cards and I don't have the time to slowly compare radv assembly dumps. I give up for now 😢

Anyways I forgot to mention this earlier but the speed jumps back up abruptly once the number of tokens passes a certain point.

model	size	params	backend	ngl	threads	main_gpu	sm	test	t/s
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp200	41.59 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp400	46.50 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp600	47.42 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp640	49.38 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp641	17.82 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp800	18.78 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp1000	15.97 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp1200	48.72 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp1400	46.36 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp1600	45.28 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp1800	47.83 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp2000	45.19 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp2200	44.31 ± 0.00
llama ?B IQ1_S - 1.5625 bpw	4.50 GiB	22.25 B	Vulkan	100	8	1	none	pp2400	44.17 ± 0.00

[netrunnereve]

Turns out this is probably a RADV driver issue as AMDVLK doesn't have this problem. Maybe it only affects Polaris and older cards?

It's slower than RADV in general but the speeds are actually consistent across different prompt sizes.

[netrunnereve]

I've submitted this up to Mesa as this is beyond me. Also RGP works now, yay! Well, when it doesn't crash or fail at least.

https://gitlab.freedesktop.org/mesa/mesa/-/issues/13282

If anyone's curious the other case where I can trigger this memory clock drop is when I run the n=512 bf16 mat mul test along with the other ones.

> GGML_VK_PERF_LOGGER=1 ./bin/test-backend-ops perf -o MUL_MAT -p "((a=f32)|(a=f16)|(a=bf16)).*n=512"
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):      ----------------
Vulkan Timings:
...
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 36828 us
          28 runs - 36507.61 us/run -  60.13 GFLOP/run -   1.65 TFLOPS
  MUL_MAT(type_a=f16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):      ----------------
...
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 26724.2 us
          40 runs - 26132.70 us/run -  60.13 GFLOP/run -   2.30 TFLOPS
  MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):             ----------------
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 1 x 323548 us
----------------
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 322892 us
----------------
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 324457 us
           4 runs - 323810.25 us/run -  60.13 GFLOP/run - 185.69 GFLOPS

And if I run that same test on its own nothing happens.

> GGML_VK_PERF_LOGGER=1 ./bin/test-backend-ops perf -o MUL_MAT -p "((a=bf16)).*n=512"
  MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0):             ----------------
...
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 24435.3 us
----------------
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 24923.3 us
----------------
Vulkan Timings:
MUL_MAT m=4096 n=512 k=14336: 2 x 25111.5 us
          42 runs - 24381.52 us/run -  60.13 GFLOP/run -   2.47 TFLOPS

[IMbackK]

Presumably this is down to llvm as a shader compiler vs aco as the shader compiler, did you try llvm with radv?

RADV_DEBUG=llvm

[netrunnereve]

Oh RADV_DEBUG=llvm has never worked with llama.cpp, it only works with aco. It crashes instantly.

[IMbackK]

hmm ok it certainly worked in the early days of the vulkan backend. AMDVLK uses llvm unless you used the pro shader compiler, so it not working with radv is a bit strange.

[netrunnereve]

Yeah I've tried it a couple times to see if it would run faster but it always crashes on me. Maybe it's a GCN thing as well.

[netrunnereve]

Welp turns out this is a GTT issue after all and there's not much we can do about it from our end. For some reason the driver is allocating us GTT instead of VRAM even when we have the space for it.

Anyways I have a PR at Mesa to add a debug flag to turn off GTT spilling, but for now I'm just using a modified RADV which doesn't spill memory to GTT.