Misc. bug: Huge performance degradation with MTP on Vulkan

[piotrp88]

Name and Version

version: 9265 (0be84685b)
built with GNU 15.2.0 for Linux x86_64

version: 9354 (9777256c3)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

# Change this variables to suite your case
PORT="8080"
DOCKER_IMAGE="ghcr.io/ggml-org/llama.cpp:server-vulkan"
MODEL_DIR_ON_HOST="/mnt/ext_ssd1tb/llamacpp_models/qwen/"
MODEL_FILENAME="Qwen3.6-35B-A3B-MTP-UD-Q5_K_XL.gguf"

# Run the llama.cpp Vulkan Docker on AMD R9700 AI PRO
docker run --rm \
    --privileged \
    --device /dev/kfd --device /dev/dri \
    --name "llama-server-$PORT" \
    --group-add video \
    --ipc=host \
    --shm-size 16g \
    -v "$MODEL_DIR_ON_HOST":"$MODEL_DIR_ON_HOST" \
    -p "$PORT":"$PORT" \
    "$DOCKER_IMAGE" \
    --host 0.0.0.0 --port "$PORT" \
    --model "$MODEL_DIR_ON_HOST$MODEL_FILENAME" \
    --ctx-size 262144 \
    --n-gpu-layers auto \
    --flash-attn on \
    --batch-size 16384 \
    --ubatch-size 3072 \
    --threads 8 \
    --threads-batch 16 \
    --cont-batching \
    --cache-prompt \
    --temp 0.2 \
    --top-k 20 \
    --top-p 0.95 \
    --min-p 0.01 \
    --presence-penalty 0.00 \
    --repeat-penalty 1.0 \
    --ctx-checkpoints 64 \
    --no-mmap \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    --rope-scale 2 \
    --rope-scaling yarn

Problem description & steps to reproduce

I'm running on Ubuntu Server 24 LTS on a AMD R9700 AI PRO.

Comparing the Vulkan docker release b9354 against b9265 I observed a huge performance degradation with MTP models like Qwen3.6 (~5% in prompt and 22% to 33% in generation).

I'm running a simple prompt from the llama.cpp web UI and saw this kind of results:

Model	release	prompt toc/s	gen toc/s	Toc/s drop %prompt, %gen
Qwen3.6-35B-A3B-MTP-UD-Q5_K_XL.gguf	❌ b9354	748	63	-4.7%, -22.2%
Qwen3.6-35B-A3B-MTP-UD-Q5_K_XL.gguf	✅ b9265	785	81	baseline
Qwen3.6-35B-A3B-MTP-IQ4_XS-4.19bpw.gguf	❌ b9354	976	102	-5.9%, -33.3%
Qwen3.6-35B-A3B-MTP-IQ4_XS-4.19bpw.gguf	✅ b9265	1037	153	baseline

Using the same prompt I also noticed a very strange behaviour with Qwen3.5-9B-Q4_K_M.gguf which is a non-MTP model: running the b9354 provides a very verbose and sometimes never-ending output, while the old b9265 release generates a very fast and concise response. The two releases doesn't change with the non-MTP model.

First Bad Commit

No response

Relevant log output

No response

[jeffbolznv]

Please try to bisect the change that caused the slowdown.

[piotrp88]

Please try to bisect the change that caused the slowdown.

I'm not into debugging. Can you tell me a easy way to provide more useful info? I saw other similar issues published today, so maybe the performance regression bug is common to other backends and also to non-MTP models (as I said, I just checked a small non-MTP model that fits entirely on my GPU).

Thank you for your time

[robquill]

Can you give a link to the models you used? Or the full llama-server command? The smallest model you've seen the regression in will make it easier for me/someone to try to reproduce.

I wonder if this is related to #23772 (which has tighter bounds to bisect)

[ahoplock]

I narrowed down the same builds as #23772; b9297 works fine but b9305 is 2.5x slower on TG for both Vulkan and ROCm on my 9070 XT.

I'm guessing it's due to #23485 as my context size goes from 40704 on b9297 down to 4096 on b305 with --fit on. If I play with --fit-target, performance returns.

[ahoplock]

...was about to reply on #23681 with my findings before I came across #23772 (which lead me here).

[piotrp88]

Can you give a link to the models you used? Or the full llama-server command? The smallest model you've seen the regression in will make it easier for me/someone to try to reproduce.

I wonder if this is related to #23772 (which has tighter bounds to bisect)

• Qwen3.6-35B-A3B-MTP-UD-Q5_K_XL: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
• Qwen3.6-35B-A3B-MTP-IQ4_XS-4.19bpw.gguf: https://huggingface.co/byteshape/Qwen3.6-35B-A3B-MTP-GGUF/blob/main/Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf

Here's the full command (I've also updated the original post):

# Change this variables to suite your case
PORT="8080"
DOCKER_IMAGE="ghcr.io/ggml-org/llama.cpp:server-vulkan"
MODEL_DIR_ON_HOST="/mnt/ext_ssd1tb/llamacpp_models/qwen/"
MODEL_FILENAME="Qwen3.6-35B-A3B-MTP-UD-Q5_K_XL.gguf"

# Run the llama.cpp Vulkan Docker on AMD R9700 AI PRO
docker run --rm \
    --privileged \
    --device /dev/kfd --device /dev/dri \
    --name "llama-server-$PORT" \
    --group-add video \
    --ipc=host \
    --shm-size 16g \
    -v "$MODEL_DIR_ON_HOST":"$MODEL_DIR_ON_HOST" \
    -p "$PORT":"$PORT" \
    "$DOCKER_IMAGE" \
    --host 0.0.0.0 --port "$PORT" \
    --model "$MODEL_DIR_ON_HOST$MODEL_FILENAME" \
    --ctx-size 262144 \
    --n-gpu-layers auto \
    --flash-attn on \
    --batch-size 16384 \
    --ubatch-size 3072 \
    --threads 8 \
    --threads-batch 16 \
    --cont-batching \
    --cache-prompt \
    --temp 0.2 \
    --top-k 20 \
    --top-p 0.95 \
    --min-p 0.01 \
    --presence-penalty 0.00 \
    --repeat-penalty 1.0 \
    --ctx-checkpoints 64 \
    --no-mmap \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    --rope-scale 2 \
    --rope-scaling yarn

[ahoplock]

I'm guessing it's due to #23485 as my context size goes from 40704 on b9297 down to 4096 on b305 with --fit on. If I play with --fit-target, performance returns.

#23646 (b9318) also could have ate up a good chunk of context for those living on the edge of their VRAM compared to the older builds as well.

[frenzybiscuit]

I notice you’re using docker. I’ve run into issues on docker where only half my GPU is being used. Bare metal runs then at 100%.

Can you verify GPU utilization is at 100%?

[piotrp88]

I notice you’re using docker. I’ve run into issues on docker where only half my GPU is being used. Bare metal runs then at 100%.

Can you verify GPU utilization is at 100%?

As far as I know, MoE models don't necessarily run at 100% on GPU because some experts may reside on RAM and the load is splitted between GPU and CPU, but I may be wrong.

Anyway, these are the results for the Qwen3.6-35B-A3B-MTP-UD-Q5_K_XL model:

release	% GPU	% CPU	% diff (GPU)	% diff (CPU)
❌ b9354	51%	58%	-21.5%	-17.2%
✅ b9265	65%	48%	baseline	baseline

This seems to be aligned with the -22.2% drop in prompt toc/s in my first post.

NOTE: My cpu is a Intel i9-13900K and gave 22 cores and 80GB of RAM to the machine (I run on a Proxmox environment with multi GPU passthrough).

[hschougaard]

FWIW I tried the two releases b9265 and b9354 with a Strix Halo and the Qwen3.6-35B-A3B-MTP-UD-Q6_K_XL model under Vulkan with a coding prompt. Virtually no difference in token gen 71-72 t/s.

[MGAndreasen]

I also think i whent from around 70tps down to low 50ish high 40ish on gemma-4-26B-A4B IQ4. within the last few weeks. I use both ngram-mod and mtp on my startup, too be able to load different models like qwen 3.6 35B-A3B with and without MTP. I use vulkan on a rx9060xt 16GB card. and if I compare it to an older build on a spare rx580 with gemma-4-e4b q4 the gap is way smaller than it used to be.

[MGAndreasen]

I also think i whent from around 70tps down to low 50ish high 40ish on gemma-4-26B-A4B IQ4. within the last few weeks. I use both ngram-mod and mtp on my startup, too be able to load different models like qwen 3.6 35B-A3B with and without MTP. I use vulkan on a rx9060xt 16GB card. and if I compare it to an older build on a spare rx580 with gemma-4-e4b q4 the gap is way smaller than it used to be.

Looking more into my own issue, I pulled the repo from commit: 1348f67 its the last on the 15 maj, before any MTP stuff. and toyed around for a bit, still had issues, then i started looking at mesa updated on my fedora 44 and what not. refetched the latest version of llama.cpp, checking back and forth until it hit me. I added a few args to my cmd line within the last week or so, I added:
--presence_penalty 0.01
--repeat-penalty 1.1

When I add this, my TPS drops around 25% for multiple models, even small models like Gemma 4 E2B Q4 and E4B Q4 drops by 30-33%.

I know it takes extra horse power to check for repeated stuff, but that amount just seams wild. I dont know if poster also uses this, but this was the culpit i had with the performance i lost, so I have no problems with the newest version as it performs the same with my ngram-mod.

my cmd for reference:
--host 0.0.0.0
--port 8080
--models-dir /models
--spec-type ngram-mod
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 2
--spec-ngram-mod-n-max 12
--spec-draft-ngl 99
-fa on
--no-mmap
--no-direct-io
--models-max 1
--parallel 1
--threads 1
--threads-batch 2
--no-warmup
-c 50000
--cache-reuse 128
--batch-size 2048
--split-mode layer
--cache-ram 128
-kvu
--context-shift
--jinja
--no-mmproj
-ctk q4_0
-ctv q4_0
--fit on
--fit-target 512
--fit-ctx 50000
-lv 4
--reasoning-budget 16384
--reasoning-budget-message "OK, I've thought long enough. Let's answer."
--repeat_last-n 64
--sleep-idle-seconds 300
--device vulkan0
--temp 0.7
--top_p 0.95
--min_p 0.05
--top_k 25
--presence_penalty 0.01
--repeat-penalty 1.1

[AlexanderBartash]

FWIW I tried the two releases b9265 and b9354 with a Strix Halo and the Qwen3.6-35B-A3B-MTP-UD-Q6_K_XL model under Vulkan with a coding prompt. Virtually no difference in token gen 71-72 t/s.

• ProArt PX13 HN7306EAC (Strix Halo 395+, 128G)
• Memory allocation is dynamic — I tested in the past, and allocating some specific amount does not improve performance, so I just set it to auto and do not bother since then.
• Windows 11 26200.8457
• G-Helper:
  • Windows power mode — Balanced
  • CPU Boost — Efficient Enabled
  • Power Limits — Default
• LM Studio 0.4.15 (Build 2)
• Vulkan v2.18.0

I have tested this on all versions since when MTP was released with all imaginable settings combinations.

Confirm the same for unsloth/qwen3.6-35b-a3b-mtp and bartowski/qwen_qwen3.6-35b-a3b0 compared to qwen/qwen3.6-35b-a3b. MTP either makes things slower or makes no difference in the best case.

• q8_0 ~42.45 t/s ⇒ ~41.04 t/s
• q4_0_k_m it is ~50-60 t/s with regular and with MTP similarly the same or slower.

Sometimes it can be faster, e.g. q8_0 45 t/s with or without MTP. It seems to depend on power efficiency optimizations being applied at a specific moment in time. Often I use it even on silent power mode, because it is about 10 degrees cooler, much quieter and speed decrease is negligible.

For unsloth/qwen-3.6-27b and bartowski/qwen_qwen3.6-27b, I do see an improvement compared to qwen/qwen3.6-27b. The difference between unsloth and bartowski is negligible — noise.

• q8_0 regular ~7.37 t/s ⇒ MTP ~11.73 t/s
• q4_0_k_m regular ~11.61 t/s ⇒ MTP ~17 t/s

Not sure if MTP is supposed to improve MOE.
Maybe it is bottlenecked by something else.