Skip to content

mtmd: add chunks and fix preproc for qwen3a#23073

Merged
ngxson merged 6 commits into
ggml-org:masterfrom
ngxson:xsn/qwen3a_fix
May 15, 2026
Merged

mtmd: add chunks and fix preproc for qwen3a#23073
ngxson merged 6 commits into
ggml-org:masterfrom
ngxson:xsn/qwen3a_fix

Conversation

@ngxson

@ngxson ngxson commented May 14, 2026

Copy link
Copy Markdown
Collaborator

Overview

Fix #22357

Fixes for qwen3-asr:

  • fix preprocessing (add padding and normalization)
  • add chunking + attn mask
  • remove the unused clip_has_whisper_encoder API

Testing

audio file from: #22357 (comment)

language English<asr_text>If you look at the AI sector landscape more broadly, what you've seen is OpenAI's early dominance being challenged increasingly by Anthropic. Anthropic is drawing investor interest at a valuation of 800 billion dollars or higher, making it close to, if not the most valuable startup.

audio file from "I have a dream", truncated to 2 minutes: i-have-a-dream-2m.mp3

language English<asr_text>I have a dream that one day every valley shall be exalted, every hill and mountain shall be made low. The rough places will be made plain, and the crooked places will be made straight. And the glory of the Lord shall be revealed, and all flesh shall see it together. This is our hope. This is the faith that I go back to the south with. With this faith, we will be able to chew out of the mountain of despair a stone of hope. With this faith, we will be able to transform the jangling discord of our nation into a beautiful symphony of brotherhood. With this faith, we will be able to work together, to pray together, to struggle together, to go to jail together, to stand up for freedom together, knowing that we will be free one day. This will be the day when all the God's children will be able to sing with new meaning. My country tears at thee. Sweet land of liberty, we shall sing. Land where my fathers died, land of the pilgrim's pride. From every mountain side, let freedom ring. And America is to be a great nation. This must become true. And so let freedom ring. From the prodigious hill tops of New Hampshire, let freedom ring. From the mighty mountains of New York, let freedom ring. From the heightening Alleghanies of Pennsylvania, let freedom ring. From the snow cap Rockies of Colorado, let freedom ring. From the curvaceous slopes of California. Not only that, let freedom ring from Stone Mountain of Georgia. Let freedom ring from Lookout Mountain of Tennessee.

vietnamese audio about the city of Hue, mixed between music and speech: audio_hue.mp3

language Vietnamese<asr_text>Thành phố Huế là vùng đất cổ thuộc tỉnh Thừa Thiên Huế. Thành phố tựa lưng vào dãy Trường Sơn và nằm cách biển không xa. Nơi đây đã được chọn làm kinh đô của triều Ngụy từ năm một nghìn tám trăm linh hai đến năm một nghìn chín trăm bốn mươi lăm. Cổ đô Huế nổi tiếng với các kiến trúc cung đình, thành quách, đền miếu lăng tẩm của các vua Chúa triều Ngụy và được công nhận là di sản văn hóa thế giới vào năm một nghìn chín trăm chín mươi ba. nét đặc sắc của văn hóa Huế là kết hợp hài hòa giữa văn hóa dân gian và văn hóa cung đình. Nhã nhạc cung đình Huế năm hai nghìn lẻ ba đã được UNESCO công nhận là di sản văn hóa phi vật thể và quyền hội của nhân loại. Vẻ đẹp của Huế và lịch sử mà nó là nhân chứng đã đem đến cho Huế một sức hấp dẫn về văn hóa và du lịch và trở thành thành phố festival ở Việt Nam.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: yes, for investigating bugs, writing the preprocessor and debug my cgraph code

@ngxson
ngxson requested a review from a team as a code owner May 14, 2026 21:39
@SmartestWashingMachine

Copy link
Copy Markdown
Contributor

Btw, is this correct or should qwen3-asr be <|audio_start|> / <|audio_end|>?

aud_beg = "<|audio_bos|>";
aud_end = "<|audio_eos|>";

because qwen3-asr template uses those

and in pytorch we get:
<|im_start|>system\n<|im_end|>\n<|im_start|>user\n<|audio_start|>...<|audio_end|>.....

@ngxson

ngxson commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

thanks, it's fixed in the latest commit

@ngxson
ngxson requested a review from a team May 15, 2026 14:50
@ngxson

ngxson commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

asking for a re-approval, thanks! @ggerganov @ServeurpersoCom

@ngxson

ngxson commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

sorry, one more approval please 😅

@ngxson
ngxson merged commit 72e60f5 into ggml-org:master May 15, 2026
49 of 50 checks passed
dandm1 pushed a commit to dandm1/llama.cpp that referenced this pull request May 16, 2026
* mtmd: add chunks and fix preproc for qwen3a

* add attn_mask

* limit mtmd_chunk size (avoid blow up memory)

* correct audio tokens

* re-order the set_input case

* remove attn_mask
ArberSephirotheca pushed a commit to ArberSephirotheca/llama.cpp that referenced this pull request May 19, 2026
* mtmd: add chunks and fix preproc for qwen3a

* add attn_mask

* limit mtmd_chunk size (avoid blow up memory)

* correct audio tokens

* re-order the set_input case

* remove attn_mask
baramofme pushed a commit to baramofme/llama-cpp-turboquant that referenced this pull request May 23, 2026
* mtmd: add chunks and fix preproc for qwen3a

* add attn_mask

* limit mtmd_chunk size (avoid blow up memory)

* correct audio tokens

* re-order the set_input case

* remove attn_mask
winstonma pushed a commit to winstonma/llama.cpp that referenced this pull request May 27, 2026
* mtmd: add chunks and fix preproc for qwen3a

* add attn_mask

* limit mtmd_chunk size (avoid blow up memory)

* correct audio tokens

* re-order the set_input case

* remove attn_mask
fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request May 30, 2026
* mtmd: add chunks and fix preproc for qwen3a

* add attn_mask

* limit mtmd_chunk size (avoid blow up memory)

* correct audio tokens

* re-order the set_input case

* remove attn_mask
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval bug: qwen3-asr not transcribing correctly

4 participants