[Bugfix] fix video inference of qwen3vl and qwen3.5 series by JJJYmmm · Pull Request #44474 · huggingface/transformers

JJJYmmm · 2026-03-05T15:46:09Z

What does this PR do?

Fix QwenLM/Qwen3.8#58. In the latest code, Qwen3VL and Qwen3.5 use the same get_rope_index func of Qwen2VL. But they should be different since Qwen3VL/Qwen3.5 introduce text timestamps.

This PR fixs it follow the original impl

transformers/src/transformers/models/qwen3_vl/modular_qwen3_vl.py

Lines 867 to 870 in 753d611

    
           # Since we use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split 
        
           if video_grid_thw is not None: 
        
               video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0) 
        
               video_grid_thw[:, 0] = 1

.

cc @zucchini-nlp 🤗

zucchini-nlp

Nice, also saw the error yesterday! Can you add integration tests in qwen3-vl for image and video inference, which will help us track this kind of issues before merging PRs? There are no tests atm because model was released day-0 and we prob forgot to add later

JJJYmmm · 2026-03-06T10:57:56Z

@zucchini-nlp Hi, I have added the test for qwen3.5 and fix the previous qwen3vl test(now it can catch this type of error). I also add a slow qwen3.5-0.8b test. All added tests pass locally. 🫡

zucchini-nlp · 2026-03-06T11:26:24Z

run-slow: qwen3_5, qwen3_5_moe, qwen3_vl, qwen3_vl_moe

github-actions · 2026-03-06T11:27:34Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/qwen3_5", "models/qwen3_5_moe", "models/qwen3_vl", "models/qwen3_vl_moe"]
quantizations: []

github-actions · 2026-03-06T20:11:44Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	8f1685bc	workflow commit (merge commit)
PR	05affcec	branch commit (from PR)
main	4f91111b	base commit (on `main`)

Model CI Report

❌ 1 new failed tests from this PR 😭

qwen3_5:
tests/models/qwen3_5/test_modeling_qwen3_5.py::Qwen3_5IntegrationTest::test_model_video_generation (✅ ⟹ ❌)

zucchini-nlp · 2026-03-09T13:10:07Z

run-slow: qwen3_5

github-actions · 2026-03-09T13:11:27Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/qwen3_5"]
quantizations: []

zucchini-nlp · 2026-03-09T13:11:48Z

SSH-ing into runners isn't possible so I triggered CI again, let's copy from the error message when it fails

HuggingFaceDocBuilderDev · 2026-03-09T13:19:30Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-03-09T13:55:46Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	30314db8	workflow commit (merge commit)
PR	aa318218	branch commit (from PR)
main	b8d6b845	base commit (on `main`)

Model CI Report

❌ 1 new failed tests from this PR 😭

qwen3_5:
tests/models/qwen3_5/test_modeling_qwen3_5.py::Qwen3_5IntegrationTest::test_model_video_generation (✅ ⟹ ❌)

github-actions · 2026-03-09T15:22:24Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: qwen3_5, qwen3_5_moe, qwen3_vl, qwen3_vl_moe

zucchini-nlp

Should have approved so it can be auto-merged 🤦🏻

…face#44474) Expand video_grid_thw to per-frame rows before iterating in get_rope_index(). Qwen3.5 uses timestamps to separate video frames (<t1><frame1><t2><frame2>...), creating one mm_token_type_ids segment per frame. grid_iters needs one grid_thw entry per segment. Fix: repeat_interleave(grid_thw, grid_thw[:,0]) then set T=1 per row. Applied to: qwen3_vl, qwen3_5, qwen3_vl_moe, qwen3_5_moe modeling files. The modular_qwen3_vl.py already contains this fix but generated files were not regenerated.

* fix video inference of qwen3vl and qwen3.5 series * fix ci * get_rope_index inhert from qwen2vl * add test for qwen3vl,qwen3.5 * fix ci * use assertListEqual instead * update CI result

fix video inference of qwen3vl and qwen3.5 series

77ba753

JJJYmmm mentioned this pull request Mar 5, 2026

Huggingface transformers generation issue with video input on Qwen 3.5-9B QwenLM/Qwen3.8#58

Closed

1 task done

fix ci

729ff7c

zucchini-nlp reviewed Mar 6, 2026

View reviewed changes

Comment thread src/transformers/models/qwen3_5/modeling_qwen3_5.py Resolved

Comment thread src/transformers/models/qwen3_vl/modular_qwen3_vl.py Resolved

zucchini-nlp mentioned this pull request Mar 6, 2026

[bug] v5.3.0 video input regression for qwen2_5_vl, qwen3_vl, qwen3_5, and qwen3_5_moe #44479

Closed

2 of 4 tasks

hmellor mentioned this pull request Mar 6, 2026

Update to transformers v5 vllm-project/vllm#30566

Merged

JJJYmmm added 2 commits March 6, 2026 17:44

get_rope_index inhert from qwen2vl

f79cfb8

add test for qwen3vl,qwen3.5

e0879ec

fix ci

05affce

zucchini-nlp mentioned this pull request Mar 6, 2026

Fix Qwen3VL get_rope_index StopIteration with per-frame video tokens #44472

Closed

JJJYmmm commented Mar 7, 2026

View reviewed changes

Comment thread tests/models/qwen3_5/test_modeling_qwen3_5.py Resolved

JJJYmmm mentioned this pull request Mar 9, 2026

Pre-processing error QwenLM/Qwen3-VL#2041

Open

use assertListEqual instead

aa31821

update CI result

7022b34

Merge branch 'main' into main

c5d1cb3

zucchini-nlp enabled auto-merge (squash) March 9, 2026 16:08

zucchini-nlp approved these changes Mar 10, 2026

View reviewed changes

zucchini-nlp added this pull request to the merge queue Mar 10, 2026

zucchini-nlp mentioned this pull request Mar 10, 2026

Qwen3-vl-embedding Video Error "StopIteration" in transformers 5.3.0 #44560

Closed

2 of 4 tasks

Merged via the queue into huggingface:main with commit 5a098a1 Mar 10, 2026
22 checks passed

wuxibin89 mentioned this pull request Mar 11, 2026

[megatron, model] feat: qwen3.5 example verl-project/verl#5381

Merged

8 tasks done

Artessay mentioned this pull request Mar 14, 2026

feat: add Qwen3.5 model implementation and registration EvolvingLMMs-Lab/lmms-eval#1249

Merged

1 of 7 tasks

DragonAura mentioned this pull request Mar 14, 2026

Weird evaluation results on Video-MME for Qwen 3.5-9B with Huggingface transformers QwenLM/Qwen3.8#86

Closed

1 task

Jintao-Huang mentioned this pull request Mar 18, 2026

Qwen3.5 4.0.2 docker 镜像训练如果有视频数据训练中断（packing) modelscope/ms-swift#8362

Closed

1 task done

JJJYmmm mentioned this pull request Mar 27, 2026

[inference_fusion] convert conv3d patch embed to linear #45041

Merged

	# Since we use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split
	if video_grid_thw is not None:
	video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0)
	video_grid_thw[:, 0] = 1

Conversation

JJJYmmm commented Mar 5, 2026

What does this PR do?

Uh oh!

zucchini-nlp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JJJYmmm commented Mar 6, 2026

Uh oh!

zucchini-nlp commented Mar 6, 2026

Uh oh!

github-actions Bot commented Mar 6, 2026

Uh oh!

github-actions Bot commented Mar 6, 2026

CI Results

Commit Info

Model CI Report

Uh oh!

Uh oh!

zucchini-nlp commented Mar 9, 2026

Uh oh!

github-actions Bot commented Mar 9, 2026

Uh oh!

zucchini-nlp commented Mar 9, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Mar 9, 2026

Uh oh!

github-actions Bot commented Mar 9, 2026

CI Results

Commit Info

Model CI Report

Uh oh!

github-actions Bot commented Mar 9, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zucchini-nlp left a comment •

edited

Loading