Repository navigation
Conversation
transformers 5.4.0–5.5.3 contain a bug in `Qwen2_5_VLForConditionalGeneration. get_rope_index` where still images are incorrectly routed through the video temporal-scaling path, causing `temporal_start` to be multiplied by `tokens_per_second` rather than remaining zero-scaled. The result is that image-based policies receive incorrect position encodings. The upstream fix landed in transformers 5.5.4 (huggingface/transformers#45330). The `eo1` policy calls `get_rope_index` directly via `vlm_backbone.model` (modeling_eo1.py:566) and is therefore affected. Raises the `transformers-dep` lower bound from 5.4.0 to 5.5.4 to exclude all broken releases. Closes huggingface#3648
Author
|
Closing because the dependency set on main has moved substantially, this branch now conflicts, and a version-bound-only change without a current regression test is no longer a good patch. I can revisit with a reproducer if the incompatibility returns. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / Motivation
Closes #3648.
The
transformers-depextra currently allowstransformers>=5.4.0,<5.6.0.Releases 5.4.0–5.5.3 contain a bug in
Qwen2_5_VLForConditionalGeneration.get_rope_indexwhere still images areincorrectly routed through the video temporal-scaling path. Concretely,
the variable
second_per_grid_tis set to a non-zero value for imagetokens, causing
temporal_startto be multiplied bytokens_per_secondrather than remaining zero-scaled. The result is that every image token
receives a shifted temporal position encoding, degrading policy performance
on all Qwen2.5-VL-backed policies.
The bug was fixed in transformers 5.5.4 via
huggingface/transformers#45330 (tracking issue
huggingface/transformers#45325).
Root cause
lerobot/policies/eo1/modeling_eo1.py(line 566) callsvlm_backbone.model.get_rope_index(...)directly through the transformerspublic API. Any environment that resolves
transformers>=5.4.0to arelease between 5.4.0 and 5.5.3 (inclusive) will silently receive wrong
position IDs for still-image observations.
wall_xmaintains its own local copy ofget_rope_indexand already setssecond_per_grid_t = 0for images unconditionally, so it is not affectedby this dependency range.
What changed
The
transformers-depextra inpyproject.toml:The upper bound is unchanged; only the lower bound is raised to exclude the
broken 5.4.0–5.5.3 range. All extras that pull in
transformers-dep(
eo1,wallx,sarm,smolvla,pi,hilserl,xvla,peft,libero, etc.) inherit this tighter bound automatically.Duplicate check