[hmellor]

Overview

Adds /v1/audio/transcriptions to the Transformers modeling backend, so VibeVoice-ASR is served without the native implementation in #62398. Needs Transformers 5.23 for tokenize on apply_transcription_request (huggingface/transformers#49944), and huggingface/transformers#50417 for the sync check, so it stays a draft until those are released.

Claims

• New TransformersMultiModalForTranscription, selected when the HF processor class has apply_transcription_request, serves json, text and diarized_json. Also checked through the API: GLM-ASR, Audio Flamingo 3 and Qwen/Qwen3-ASR-1.7B-hf transcribe, and Granite Speech has no endpoint.
• post_process_output and parse_diarized_transcript now take model_config, and the latter parses each chunk's raw output, with timestamps shifted by the chunk offset.
• Transcription models measure their audio token limit from the processor, as config values can be per encoder window. Those that don't truncate are sized by the context length up to the audio decode limits (--limit-mm-per-prompt changes it), and other audio models keep their budget.
• Different-length audios in one encoder batch no longer crash, and VibeVoice-ASR's padding_mask stays on the CPU so that path passes VLLM_GPU_SYNC_CHECK=error with [Bugfix][Model Runner V2] Restore multimodal draft capability detection #50417.

Validation

B200, bf16, Transformers 5.23.0.dev0. Native numbers are from #62398 on different hardware. The long clips differ: here it is plain WER on 40 min of concatenated LibriSpeech.

WER	#62398 native	This PR, via the API
LibriSpeech test-clean, 500 utts	2.14 %	2.07 %
40 min, one request	1.56 %	1.64 %

Noise: the backend keeps HF's default VAE sampling, which gave 2.03 % and 2.08 % across two seeds through LLM before this rebase. With that default, 5 min at the default limits gave 2.35 %, and 40 min with the three audio limits raised gave 1.58 % at --max-model-len 32768, on this PR's code applied to 9b34fec. --pipeline-parallel-size 2 gave 2.48 % on 20 LibriSpeech utterances and 2.47 % on 5 min. A LoRA adapter on the text model loads and changes the output, and a zero adapter leaves it unchanged. VoxPopuli-de, cpWER, 60 min, vllm bench serve and TP from #62398 were not rerun.

vllm serve microsoft/VibeVoice-ASR-HF --dtype bfloat16 --max-model-len 32768 --gpu-memory-utilization 0.5 --seed 0
pytest tests/models/multimodal/processing/test_transformers_audio.py \
  tests/models/multimodal/processing/test_moss_transcribe_diarize.py \
  tests/models/multimodal/processing/test_ming_v3_asr.py \
  tests/entrypoints/speech_to_text/transcription/test_transcription_inter_chunk_spacing.py
pytest tests/models/multimodal/generation/test_transformers_audio.py -k "VibeVoice or mixed_lengths"
pytest tests/entrypoints/speech_to_text/transcription/test_transcription_validation.py -k transformers_backend_audio
pytest tests/models/test_initialization.py -k "VibeVoice or TransformersMultiModalForTranscription"

All pass (86 passed, 1 xfailed). test_transformers_audio_mixed_lengths fails without the fix with AttributeError: 'list' object has no attribute 'sum'. Native Qwen/Qwen3-ASR-1.7B still transcribes. MossAudioModel initialization fails with and without this PR. The tests needing 5.23 skip in CI until the pin is bumped.

Details

• Streaming returns the raw model output and streamed usage excludes audio tokens (no get_num_audio_tokens). language is accepted but unused. translate raises a ValueError, as in [Model][Frontend] Add native VibeVoice-ASR with transcription API support #62398.
• Transcripts are not deterministic across seeds, unlike [Model][Frontend] Add native VibeVoice-ASR with transcription API support #62398, which uses the VAE mean.
• diarized_json needs VibeVoice's or MOSS's segment format, so other models get a 400.
• Qwen/Qwen3-ASR-1.7B-hf stays on the native class by default and uses this backend with --model-impl transformers.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[mergify]

Documentation preview: https://vllm--65735.org.readthedocs.build/en/65735/

[sebfelix]

Thanks @hmellor, this covers everything #62398 was for. I ran the evaluation from #62398 against this PR (head f071d7d881, Transformers main @ b55e076, 5.23.0.dev0) on the same hardware as the native numbers: Quadro RTX 6000 (Turing), fp16, TP=2, --enforce-eager.

	Native (#62398)	This PR, seed 0	This PR, seed 1	This PR + sample=False
LibriSpeech test-clean, 500 utts, WER	2.14 %	2.13 %	2.18 %	2.14 %
VoxPopuli-de, 300 utts, WER (cleaned, 290)	17.23 % (12.97 %)	17.24 % (13.00 %)	–	17.24 % (12.98 %)
20 synthetic 2–3 speaker conversations, WER / cpWER / speaker count	1.98 / 4.25 % / 18 of 20	2.06 / 4.38 % / 18 of 20	2.00 / 4.27 % / 18 of 20	1.98 / 4.25 % / 18 of 20
40 min, 8 speakers, one request, WER / cpWER	1.56 / 7.86 %	1.50 / 7.79 %	–	1.53 / 7.83 %

Accuracy and diarization are on par in every set, including single-pass long audio. Three notes:

1. Determinism. With VAE sampling, the transcript depends on the seed: only 471/500 LibriSpeech and 8/20 conversation transcripts are identical between seeds 0 and 1. Passing sample=False to get_audio_features ([VibeVoice] Add sample argument to disable VAE noise in audio features huggingface/transformers#50179, also in 5.23) makes the outputs identical to the native implementation (500/500, 20/20, 299/300 VoxPopuli) at no accuracy cost. It also keeps embeddings that the encoder or prefix cache reuses independent of which request computed them. I tested this by setting kwargs["sample"] = False in _process_audio_input. A signature check won't find the parameter, because the top-level model's get_audio_features is wrapped as (*args, **kwargs).
2. Streamed usage already includes the audio tokens. A 4.03 s clip reports prompt_tokens: 90, which is 60 text tokens plus 30 audio placeholder tokens: prompt_token_ids contain the expanded placeholders. So the "Details" line about usage could go. Adding get_num_audio_tokens would count the audio twice, which [Model][Frontend] Add native VibeVoice-ASR with transcription API support #62398 actually did.
3. Long audio. In [Model][Frontend] Add native VibeVoice-ASR with transcription API support #62398 runs, the output degraded beyond about 40 min of audio (60-min clips loop after ~36–50 min). It's a decoder long-context limit above ~40k tokens: fp32 behaves the same, and those minutes transcribe cleanly on their own. A docs note recommending client-side splitting beyond ~40 min might help users who raise the decode limits.

[Isotr0py]

Overall look reasonable

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hmellor.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hmellor]

/ci run

[github-actions]

❌ This PR is 136 commits behind upstream main. The bot could not merge upstream main into this branch (API returned 422: merge conflict between base and head). No new CI build was started. Either merge or rebase onto the latest main yourself and rerun /ci run, or use /ci run --allow-stale to test this branch as is.

[hmellor]

/ci run

[github-actions]

✅ Triggered Buildkite CI #101068 for commit 4ac5a05965d3.

[hmellor]

Checked that pre-commit passes locally on the PR head (4ac5a05), running pre-commit run --from-ref <merge-base with main> --to-ref HEAD. All hooks pass (ruff, mypy, typos, markdownlint, docstring and config checks, etc.).