Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 7 additions & 18 deletions src/transformers/models/qwen3_5/modeling_qwen3_5.py
Original file line number Diff line number Diff line change
Expand Up @@ -1472,24 +1472,8 @@ def get_rope_index(
**kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Calculate the 3D rope index based on image and video's sizes. The utility expects a `vision + text`
sequence and will error out otherwise. For pure text sequence, please rely on model's auto-inferred
position ids. In a mixed vision + text sequence, vision tokens use 3D RoPE (temporal, height, width)
while text tokens use standard 1D RoPE.

Example:
Temporal patches: 3; Height patches: 2; Width patches: 2
Each vision input results in (temporal x height × width) positions. Here: 3 x 2 × 2 = 12 positions total.

Temporal position IDs are spaced by:
`interval = tokens_per_second * temporal_patch_size / fps`

If fps = 1; tokens_per_second = 25; temporal_patch_size = 2, temporal IDs increase by 50 for each temporal patch:
`[0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100]`

Height IDs repeat per row: `[0, 0, 1, 1, ...]`
Width IDs alternate per column: `[0, 1, 0, 1, ...]`
Text tokens follow standard 1D RoPE and the position IDs grow consequently with a step of `1`
Difference from Qwen2VL/Qwen2.5VL's get_rope_index:
- Since Qwen3.5 use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split too.

Args:
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
Expand All @@ -1511,6 +1495,11 @@ def get_rope_index(
position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`)
mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`)
"""

# Separate video grid thw into multiple grids because timestamps are used to seperate videos.
if video_grid_thw is not None:
video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0)
video_grid_thw[:, 0] = 1
Comment thread
JJJYmmm marked this conversation as resolved.
Resolved
spatial_merge_size = self.config.vision_config.spatial_merge_size

mrope_position_deltas = []
Expand Down
25 changes: 7 additions & 18 deletions src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py
Original file line number Diff line number Diff line change
Expand Up @@ -1597,24 +1597,8 @@ def get_rope_index(
**kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Calculate the 3D rope index based on image and video's sizes. The utility expects a `vision + text`
sequence and will error out otherwise. For pure text sequence, please rely on model's auto-inferred
position ids. In a mixed vision + text sequence, vision tokens use 3D RoPE (temporal, height, width)
while text tokens use standard 1D RoPE.

Example:
Temporal patches: 3; Height patches: 2; Width patches: 2
Each vision input results in (temporal x height × width) positions. Here: 3 x 2 × 2 = 12 positions total.

Temporal position IDs are spaced by:
`interval = tokens_per_second * temporal_patch_size / fps`

If fps = 1; tokens_per_second = 25; temporal_patch_size = 2, temporal IDs increase by 50 for each temporal patch:
`[0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100]`

Height IDs repeat per row: `[0, 0, 1, 1, ...]`
Width IDs alternate per column: `[0, 1, 0, 1, ...]`
Text tokens follow standard 1D RoPE and the position IDs grow consequently with a step of `1`
Difference from Qwen2VL/Qwen2.5VL's get_rope_index:
- Since Qwen3.5 use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split too.

Args:
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
Expand All @@ -1636,6 +1620,11 @@ def get_rope_index(
position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`)
mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`)
"""

# Separate video grid thw into multiple grids because timestamps are used to seperate videos.
if video_grid_thw is not None:
video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0)
video_grid_thw[:, 0] = 1
spatial_merge_size = self.config.vision_config.spatial_merge_size

mrope_position_deltas = []
Expand Down
25 changes: 7 additions & 18 deletions src/transformers/models/qwen3_vl/modeling_qwen3_vl.py
Original file line number Diff line number Diff line change
Expand Up @@ -1032,24 +1032,8 @@ def get_rope_index(
**kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Calculate the 3D rope index based on image and video's sizes. The utility expects a `vision + text`
sequence and will error out otherwise. For pure text sequence, please rely on model's auto-inferred
position ids. In a mixed vision + text sequence, vision tokens use 3D RoPE (temporal, height, width)
while text tokens use standard 1D RoPE.

Example:
Temporal patches: 3; Height patches: 2; Width patches: 2
Each vision input results in (temporal x height × width) positions. Here: 3 x 2 × 2 = 12 positions total.

Temporal position IDs are spaced by:
`interval = tokens_per_second * temporal_patch_size / fps`

If fps = 1; tokens_per_second = 25; temporal_patch_size = 2, temporal IDs increase by 50 for each temporal patch:
`[0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100]`

Height IDs repeat per row: `[0, 0, 1, 1, ...]`
Width IDs alternate per column: `[0, 1, 0, 1, ...]`
Text tokens follow standard 1D RoPE and the position IDs grow consequently with a step of `1`
Difference from Qwen2VL/Qwen2.5VL's get_rope_index:
- Since Qwen3.5 use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split too.

Args:
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
Expand All @@ -1071,6 +1055,11 @@ def get_rope_index(
position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`)
mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`)
"""

# Separate video grid thw into multiple grids because timestamps are used to seperate videos.
if video_grid_thw is not None:
video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0)
video_grid_thw[:, 0] = 1
spatial_merge_size = self.config.vision_config.spatial_merge_size

mrope_position_deltas = []
Expand Down
37 changes: 37 additions & 0 deletions src/transformers/models/qwen3_vl/modular_qwen3_vl.py
Original file line number Diff line number Diff line change
Expand Up @@ -780,6 +780,43 @@ def __init__(self, config):
self.visual = Qwen3VLVisionModel._from_config(config.vision_config)
self.language_model = Qwen3VLTextModel._from_config(config.text_config)

def get_rope_index(
self,
video_grid_thw: torch.LongTensor | None = None,
Comment thread
JJJYmmm marked this conversation as resolved.
Resolved
**super_kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Difference from Qwen2VL/Qwen2.5VL's get_rope_index:
- Since Qwen3.5 use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split too.

Args:
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide
it.
mm_token_type_ids (`torch.IntTensor` of shape `(batch_size, sequence_length)`):
Token type ids matching each modality to a different value in the input sequence, i.e. text (0), image (1), video (2).
image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
The temporal, height and width of feature shape of each image in LLM.
video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
The temporal, height and width of feature shape of each video in LLM.
attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:

- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.

Returns:
position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`)
mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`)
"""

# Separate video grid thw into multiple grids because timestamps are used to seperate videos.
if video_grid_thw is not None:
video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0)
video_grid_thw[:, 0] = 1

return super().get_rope_index(video_grid_thw=video_grid_thw, **super_kwargs)

@can_return_tuple
@auto_docstring
def get_image_features(
Expand Down
25 changes: 7 additions & 18 deletions src/transformers/models/qwen3_vl_moe/modeling_qwen3_vl_moe.py
Original file line number Diff line number Diff line change
Expand Up @@ -1161,24 +1161,8 @@ def get_rope_index(
**kwargs,
) -> tuple[torch.Tensor, torch.Tensor]:
"""
Calculate the 3D rope index based on image and video's sizes. The utility expects a `vision + text`
sequence and will error out otherwise. For pure text sequence, please rely on model's auto-inferred
position ids. In a mixed vision + text sequence, vision tokens use 3D RoPE (temporal, height, width)
while text tokens use standard 1D RoPE.

Example:
Temporal patches: 3; Height patches: 2; Width patches: 2
Each vision input results in (temporal x height × width) positions. Here: 3 x 2 × 2 = 12 positions total.

Temporal position IDs are spaced by:
`interval = tokens_per_second * temporal_patch_size / fps`

If fps = 1; tokens_per_second = 25; temporal_patch_size = 2, temporal IDs increase by 50 for each temporal patch:
`[0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100]`

Height IDs repeat per row: `[0, 0, 1, 1, ...]`
Width IDs alternate per column: `[0, 1, 0, 1, ...]`
Text tokens follow standard 1D RoPE and the position IDs grow consequently with a step of `1`
Difference from Qwen2VL/Qwen2.5VL's get_rope_index:
- Since Qwen3.5 use timestamps to seperate videos, like <t1> <vision_start> <frame1> <vision_end> <t2> <vision_start> <frame2> <vision_end>, the video_grid_thw should also be split too.

Args:
input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
Expand All @@ -1200,6 +1184,11 @@ def get_rope_index(
position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`)
mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`)
"""

# Separate video grid thw into multiple grids because timestamps are used to seperate videos.
if video_grid_thw is not None:
video_grid_thw = torch.repeat_interleave(video_grid_thw, video_grid_thw[:, 0], dim=0)
video_grid_thw[:, 0] = 1
spatial_merge_size = self.config.vision_config.spatial_merge_size

mrope_position_deltas = []
Expand Down
Loading