Skip to content

fix: bump transformers lower bound to 5.5.4 to exclude broken Qwen2.5-VL RoPE releases - #3704

Closed
yushuosun wants to merge 1 commit into
huggingface:mainfrom
yushuosun:claude/awesome-cannon-UB1Hs
Closed

yushuosun wants to merge 1 commit into
huggingface:mainfrom
yushuosun:claude/awesome-cannon-UB1Hs

Conversation

@yushuosun

Copy link
Copy Markdown

Summary / Motivation

Closes #3648.

The transformers-dep extra currently allows transformers>=5.4.0,<5.6.0.
Releases 5.4.0–5.5.3 contain a bug in
Qwen2_5_VLForConditionalGeneration.get_rope_index where still images are
incorrectly routed through the video temporal-scaling path. Concretely,
the variable second_per_grid_t is set to a non-zero value for image
tokens, causing temporal_start to be multiplied by tokens_per_second
rather than remaining zero-scaled. The result is that every image token
receives a shifted temporal position encoding, degrading policy performance
on all Qwen2.5-VL-backed policies.

The bug was fixed in transformers 5.5.4 via
huggingface/transformers#45330 (tracking issue
huggingface/transformers#45325).

Root cause

lerobot/policies/eo1/modeling_eo1.py (line 566) calls
vlm_backbone.model.get_rope_index(...) directly through the transformers
public API. Any environment that resolves transformers>=5.4.0 to a
release between 5.4.0 and 5.5.3 (inclusive) will silently receive wrong
position IDs for still-image observations.

wall_x maintains its own local copy of get_rope_index and already sets
second_per_grid_t = 0 for images unconditionally, so it is not affected
by this dependency range.

What changed

The transformers-dep extra in pyproject.toml:

-transformers-dep = ["transformers>=5.4.0,<5.6.0"]
+transformers-dep = ["transformers>=5.5.4,<5.6.0"]

The upper bound is unchanged; only the lower bound is raised to exclude the
broken 5.4.0–5.5.3 range. All extras that pull in transformers-dep
(eo1, wallx, sarm, smolvla, pi, hilserl, xvla, peft,
libero, etc.) inherit this tighter bound automatically.

Duplicate check

transformers 5.4.0–5.5.3 contain a bug in `Qwen2_5_VLForConditionalGeneration.
get_rope_index` where still images are incorrectly routed through the video
temporal-scaling path, causing `temporal_start` to be multiplied by
`tokens_per_second` rather than remaining zero-scaled. The result is that
image-based policies receive incorrect position encodings.

The upstream fix landed in transformers 5.5.4 (huggingface/transformers#45330).
The `eo1` policy calls `get_rope_index` directly via `vlm_backbone.model`
(modeling_eo1.py:566) and is therefore affected.

Raises the `transformers-dep` lower bound from 5.4.0 to 5.5.4 to exclude
all broken releases.

Closes huggingface#3648
@yushuosun

Copy link
Copy Markdown
Author

Closing because the dependency set on main has moved substantially, this branch now conflicts, and a version-bound-only change without a current regression test is no longer a good patch. I can revisit with a reproducer if the incompatibility returns.

@yushuosun yushuosun closed this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen2.5-VL-based policy performance degradation caused by RoPE index calculation bugs in Transformers v5.4.0

2 participants