Tool calls with self-hosted Qwen3-Coder fail (almost always)

[ruudniew]

What happened?

Self hosting Qwen3-Coder-480B-A35B-Instruct was a success, using something like this, but tool calls don't work:

Without a jinja template

 lama-box --host 0.0.0.0 --embeddings --gpu-layers 63 --parallel 4 --ctx-size 8192 --port 40003 --model /mnt/nfs/models/Qwen3-Coder-480B-A35B-Instruct-Q4_K_M-00001-of-00006.gguf --alias Qwen3-Coder-480B-A35B-Instruct --no-mmap --no-warmup --tensor-split 182335,182335 --ctx-size 262144 --flash-attn --parallel 4 --mmap --mlock --verbose --top-k 20 --temp 0.7 --top-p 0.8 --repeat-penalty 1.05 --min-p 0.00

With a jinja template

 lama-box --host 0.0.0.0 --embeddings --gpu-layers 63 --parallel 4 --ctx-size 8192 --port 40003 --model /mnt/nfs/models/Qwen3-Coder-480B-A35B-Instruct-Q4_K_M-00001-of-00006.gguf --alias Qwen3-Coder-480B-A35B-Instruct --no-mmap --no-warmup --tensor-split 182335,182335 --ctx-size 262144 --flash-attn --parallel 4 --mmap --mlock --verbose --top-k 20 --temp 0.7 --top-p 0.8 --repeat-penalty 1.05 --jinja --min-p 0.00 --chat-template-file /mnt/nfs/models/chat_template.jinja

Qwen Code appears to call tools, but it doesn't actually execute them. It just says something like [calling tool x with argument y and reasoning z].

What did you expect to happen?

I'd expect the tool calls to work.

Client information

Details

Login information

No response

Anything else we need to know?

I tried various quantizations and jinja files

[aaronnewsome]

ReadFile tool seems to work but I can't seem to get any write file tool to work. Very strange, so hopefully someone has run into this and knows how to fix it.

[bbartels]

running into the same issue here!

[vs0533]

插个眼

[bbartels]

I think this is down to the wrong tool-call parser being used. Once i switched to qwen3-coder tool call parser in vllm this disappeared

[7f87254a77e8483b7d8]

Have you tried serving it in FP-8 format using vllm (https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8)? I found that this seems to give a boost in tool calling accuracy (even when compared to bf16) and I used the qwen3_coder tool parser as well from the newest vllm/vllm-openai container.

I served it using vllm's guide across 64 A100s as well in case that makes a difference.
https://docs.vllm.ai/en/latest/serving/distributed_serving.html

[pomelo-nwu]

This issue has been open for over 6 months with no recent activity. It appears to be related to an early version of Qwen Code.
If you can still reproduce this issue with the latest version, please provide updated reproduction steps and we'll investigate.
Closing for now - feel free to reopen with fresh information.