Update GLM-4.1V MMRope implementation #41182
Conversation
zRzRzRzRzRzRzR
changed the title
|
cc @ArthurZucker for rope |
| cache_position: Optional[torch.LongTensor] = None, | ||
| **kwargs, | ||
| ) -> tuple[torch.FloatTensor, Optional[tuple[torch.FloatTensor, torch.FloatTensor]]]: | ||
| """ |
There was a problem hiding this comment.
lets use @auto_docstring instead
| outputs = (hidden_states,) | ||
|
|
||
| if output_attentions: | ||
| outputs += (self_attn_weights,) | ||
|
|
||
| return outputs |
There was a problem hiding this comment.
and keep @check_model_inputs below so we don't have to return these explicitly
There was a problem hiding this comment.
But this code inherits from Glm4MoeDecoderLayer, and this class doesn't seem to have @check_model_inputs. Where should it be added below?
| def forward( | ||
| self, | ||
| hidden_states: torch.Tensor, | ||
| position_embeddings: tuple[torch.Tensor, torch.Tensor], | ||
| attention_mask: Optional[torch.Tensor] = None, |
There was a problem hiding this comment.
Ig the overwriting is because of the @check_model_inputs, lets' bring the decorator back and remove unnecessary changes
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: glm4v, glm4v_moe |
zucchini-nlp
left a comment
zucchini-nlp
left a comment
There was a problem hiding this comment.
Some tests are failing, can you take a look?
src/transformers/models/glm4v_moe/convert_glm4v_moe_mgt_weights_to_hf.py
Outdated
Show resolved
Hide resolved
|
now ci pass |
zucchini-nlp
left a comment
zucchini-nlp
left a comment
There was a problem hiding this comment.
Thanks for iterating, merging
* update for 4D mask * update * Update modular_glm4v.py * 1 * Revert "1" This reverts commit d13a763. * update as glm4v logtic * update * 1 * update * Create convert_glm4v_moe_mgt_weights_to_hf.py * update * update
4 participants
What does this PR do?
This feature aims to enable the GLM-4.1V model to support 4D input mrope processing. It supports the same input format as Qwen3 in verl.
verl pr: volcengine/verl#3291