Eval bug: qwen3-asr not transcribing correctly

[cora4]

Name and Version

version b9103

Fixed in 72e60f5, b9173

Models

Qwen3-ASR

Problem description & steps to reproduce

llama-server -hf ggml-org/Qwen3-ASR-0.6B-GGUF
llama-server -m Qwen3-ASR-0.6B-Q8_0.gguf --mmproj mmproj-Qwen3-ASR-0.6B-Q8_0.gguf

llama-mtmd-cli -m Qwen3-ASR-0.6B-Q8_0.gguf --mmproj mmproj-Qwen3-ASR-0.6B-Q8_0.gguf --audio test.wav -p a

Relevant log output

b9103 Wrong transcription:
language English<asr_text>Why aren't we doing it? We cannot be the only ones. This is the future. We want to be the ones. You cannot be the ones. We want to be the ones. You want to be the ones. We want to be the ones. You want to be the ones. We want to be the ones. You want to be the ones. We want to be the ones. You want to be the ones. ...

Correct transcription:
language English<asr_text>Why aren't we doing it? Could we come up with a better plan for everyone by doing so? If you think so, please comment.

[cora4]

Correct transcription:
language English<asr_text>If you look at the AI sector landscape more broadly, what you've seen is OpenAI's early dominance being challenged increasingly by Anthropic. Anthropic is drawing investor interest at a valuation of 800 billion dollars or higher, making it close to, if not the most valuable, startup.

b8931 Wrong transcription:
language English<asr_text>If you look at the AI sector landscape more broadly, what you see is that AI is already dominating, increasing by dropping, dropping, dropping investment, valuation, billions and billions more.

b9103 Wrong transcription:
language English<asr_text>If you look at the AI sector landscape more broadly, what you see is open AI is a dominant challenge for people trying to create infrastructure and valuation of AI systems. What is the cost of AI systems? What is the cost of AI systems? What is the cost of AI systems? ...

test.wav

[nebulakid]

Same for me. I have a script that even splits the wav file in smaller chunks. Regardless how long they are it start outputing random stuff after a few tokens of processing

<chunk 1/33> 0s-30s: Today, more children are finding it difficult, and they are really trying. They are really trying, and they are really trying to find the right one. Unfortunately, the right one is very difficult to find, and they are really trying. So today, more children are finding it difficult, and they are really trying.
<chunk 2/33> 26s-56s: Their ability to focus. So the first one is the child's ability to focus. The second is the child's ability to concentrate. The third is the child's ability to follow instructions. The fourth is the child's ability to follow rules. The fifth is the child's ability to follow directions. The sixth is the child's ability to follow rules. The seventh is the child's ability to follow directions. The eighth is the child's ability to follow rules. The ninth is the child's ability to follow instructions. The tenth is the child's ability to follow rules. The eleventh is the child's ability to follow directions. The twelfth is the child's ability to follow rules. The thirteenth is the child's ability to follow instructions. The fourteenth is the child's ability to follow rules. The fifteenth is the child's ability to follow directions. The sixteenth is the child's ability to follow rules. The seventeenth is the child's ability to follow instructions. The eighteenth is the child's ability to follow rules. The nineteenth is the child's ability to follow directions. The twentieth is the child's ability to follow rules. The twenty-first is the child's ability to follow instructions. The twenty-second is the child's ability to follow rules. The twenty-third is the child's ability to follow directions. The twenty-fourth is the child's ability to follow rules. The twenty-fifth is the child's ability to follow instructions. The twenty-sixth is the child's ability to follow rules. The twenty-seventh is the child's ability to follow directions. The twenty-eighth is the child's ability to follow rules. The twenty-ninth is the child's ability to follow instructions. The thirtieth is the child's ability to follow rules.
<chunk 3/33> 52s-82s: Hey, stop! There's something serious now. When this is truly true, and when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but it's true. Now, when you are doing the right thing, the hard thing to do is to be true. It's really hard, but ......

Tried the ggml and the mradermacher ggufs

[muzhi1991]

Confirming this on a fresh setup: AMD Radeon RX 6600 XT (gfx1030 via HSA_OVERRIDE_GFX_VERSION=10.3.0) running upstream ghcr.io/ggml-org/llama.cpp:server-rocm with Qwen3-ASR-1.7B-Q8_0 GGUF.

A diagnostic that strongly localizes the bug to the audio encoder, not the LLM decoder or sampler:

• Synthetic TTS audio (macOS say, 16 kHz mono wav, three different topics — recipe / travel / movie review): all three transcribe accurately, including topic-specific vocabulary that the model couldn't have hallucinated by chance.
• Real human-voice audio (16 kHz mono wav, 72 s Chinese e-commerce vlog opener): transcript is entirely fabricated content with no relation to the actual audio. Crucially, the hallucinated text is bit-deterministic across cold container restarts — same input, identical garbage out. Rules out KV cache / sampling contamination. Sampling hacks (--repeat-penalty, --dry-multiplier 0.8, etc.) only reduce the length of the loop, never recover the real content.

After rebuilding from cora4/llama.cpp@b8931-qwen3a (PR #22343) using:

docker build --target server --build-arg ROCM_DOCKER_ARCH=gfx1030 \
  -f .devops/rocm.Dockerfile -t llama-cpp-server-rocm-cora4 .

the same real-voice audio transcribes correctly end-to-end, with all named entities intact (eBay-equivalent terms, country names, regulatory document names — none of which are in the typical "Chinese short-video" prior the upstream image samples from). Compute buffer also drops from 552 MiB → 62 MiB during warmup, consistent with the windowed audio encoding the missing n_window / n_window_infer / conv_chunksize hparams in cora4's patch enable.

So the fix in #22343 demonstrably addresses the user-visible bug, but per @ngxson's review the implementation needs reshaping into an independent mtmd_audio_preprocessor_qwen3a rather than branching inside the existing whisper preprocessor. A clean reimplementation by someone familiar with mtmd internals would be appreciated — there's a steady stream of users hitting this on real-world Chinese audio while the upstream image hallucinates silently.

[cora4]

Someone has broken the conversion script:
python3 convert_hf_to_gguf.py --outfile mmproj-Qwen3-ASR-0.6B.gguf --mmproj Qwen3-ASR-0.6B

File "./llama.cpp/convert_hf_to_gguf.py", line 14238, in
main()
File "./llama.cpp/convert_hf_to_gguf.py", line 14232, in main
model_instance.write()
File "./llama.cpp/convert_hf_to_gguf.py", line 945, in write
self.prepare_tensors()
File "./llama.cpp/convert_hf_to_gguf.py", line 805, in prepare_tensors
for new_name, data_torch in (self.modify_tensors(data_torch, name, bid)):
File "./llama.cpp/convert_hf_to_gguf.py", line 5141, in modify_tensors
yield from Qwen25AudioModel.modify_tensors(self, data_torch, name, bid)
File "./llama.cpp/convert_hf_to_gguf.py", line 4397, in modify_tensors
yield from MmprojModel.modify_tensors(self, data_torch, name, bid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./llama.cpp/convert_hf_to_gguf.py", line 579, in modify_tensors
new_name = self.map_tensor_name(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "./llama.cpp/convert_hf_to_gguf.py", line 572, in map_tensor_name
raise ValueError(f"Can not map tensor {name!r}")
ValueError: Can not map tensor 'thinker.audio_tower.conv2d1.bias'

[cora4]

test2.wav

0.6B:
呢個宋朝嘅文學家蘇洵咧，有篇文章上皇帝書裏面啊：，有官就必有貨，有貨就必有想法，有官而無貨，是無官也；有貨而無想法，是無貨也。
1.7B:
呢個宋朝嘅文學家蘇洵咧，有篇文章上皇帝書裏邊啊，有官就必有貨，有貨就必有上佛，有官而無貨是無官也，有貨而無上佛是無貨也。

@ngxson's implementation:
0.6B:
呢個宋朝嘅文學家蘇舜啊，有篇文章上皇帝書裏邊啊：，有官就必有貨，有貨就必有想法，有官而無貨，是無官人；有貨而無想法，是無貨人。
1.7B:
呢個宋朝嘅文學家蘇洵咧，有篇文章上皇帝書咧，叫《有官就必有貨，有貨就必有想法。有官而無貨，是無官也；有貨而無想法，是無貨也。》

[ngxson]

Please try 7161371

language Cantonese<asr_text>呢個宋朝嘅文學家蘇洵咧，有篇文章上皇帝書裏邊啊，有官就必有貨，有貨就必有上佛，有官而無貨是無官也，有貨而無上佛是無貨也。

For ref, here is the output from python version (1.7B): https://huggingface.co/spaces/hlevring/Qwen3-ASR

呢個宋朝嘅文學家蘇洵咧，有篇文章上皇帝書裏邊啊，有官就必有貨，有貨就必有上佛，有官而無貨是無官也，有貨而無上佛是無貨也。