[jblox26]

What does this fix?

Running video inference with any Qwen3VL model raises StopIteration during model.generate():

File ".../transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1126, in get_rope_index
    grid_thw = next(grid_iters[modality_type])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

Root cause

Qwen3VL's processor (processing_qwen3_vl.py) formats each video frame individually, prefixing it with a timestamp and wrapping it in its own <|vision_start|>…<|vision_end|> pair:

<0.0 seconds><|vision_start|>[880 video tokens]<|vision_end|>
<0.5 seconds><|vision_start|>[880 video tokens]<|vision_end|>
...

This produces T separate type-2 groups in mm_token_type_ids (one per frame, interleaved with type-0 text groups for the timestamps).

get_rope_index builds an iterator over video_grid_thw and calls next() once per type-2 group it encounters. However, video_grid_thw has one entry per video with shape [T, H, W] — not one entry per frame. For a 14-frame video, the iterator is exhausted after the very first frame and every subsequent call raises StopIteration.

This is a mismatch introduced when the per-frame timestamp feature was added to Qwen3VL's processor: the processor now emits T groups per video, but get_rope_index (inherited from the Qwen2VL pattern) still expects a single contiguous block per video.

Fix

Before building grid_iters, expand video_grid_thw from per-video entries [T, H, W] into per-frame entries [1, H, W], giving the iterator exactly as many elements as there are type-2 groups in mm_token_type_ids:

if video_grid_thw is not None:
    expanded_video_grid = torch.cat(
        [thw[1:].unsqueeze(0).expand(thw[0].item(), -1) for thw in video_grid_thw], dim=0
    )
    ones = torch.ones(expanded_video_grid.shape[0], 1, dtype=video_grid_thw.dtype, device=video_grid_thw.device)
    expanded_video_grid = torch.cat([ones, expanded_video_grid], dim=1)
else:
    expanded_video_grid = None
grid_iters = {
    1: iter(image_grid_thw) if image_grid_thw is not None else None,
    2: iter(expanded_video_grid) if expanded_video_grid is not None else None,
}

For a video with video_grid_thw = [[14, 80, 44]], the expansion produces 14 rows of [1, 80, 44], matching the 14 type-2 groups the processor emits. Multiple videos in a batch are handled correctly: each [T_i, H, W] entry expands independently.

Testing

Verified by running end-to-end video inference with Qwen3VL-based models (e.g. Cosmos-Reason2-8B) on a real video file — inference now completes successfully and produces coherent output.

A regression test should check that get_rope_index does not raise when video_grid_thw has T > 1 and mm_token_type_ids has T separate type-2 groups (one per frame).

---

Fix diagnosed and developed with assistance from Claude Code (Anthropic).

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: qwen3_vl

[jblox26]

This was pretty much all Claude, for the record. I just woke up to my stuff not working when running Cosmos-Reason-2B on a fresh install.

[Rocketknight1]

cc @zucchini-nlp

[zucchini-nlp]

Fixed by #44474, by qwen team. I'll close this PR as duplicate and since it's code agent generated mostly