Repository navigation
fix(ernie4_5_vl_moe): resolve three config loading failures for ERNIE-4.5-VL MoE models #45275
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
1219f83
65b6cc2
a770099
e169dc9
c558547
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -145,6 +145,7 @@ | |
| ("ernie", "ErnieConfig"), | ||
| ("ernie4_5", "Ernie4_5Config"), | ||
| ("ernie4_5_moe", "Ernie4_5_MoeConfig"), | ||
| ("ernie4_5_moe_vl", "Ernie4_5_VLMoeConfig"), | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @vasqu for this, i remember you were changing model types recently
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yea this looks like the only valid change IF we change the config model type of the hub PRs - this would imply that we support 2 model types for one model which is not in our code base, i.e. a dirty workaround. Imo, we should sync with vLLM support first / change the model type there. But that needs v5 support first, so I'd like to withhold on this PR for now and potentially "fix" on vLLM side instead |
||
| ("ernie4_5_vl_moe", "Ernie4_5_VLMoeConfig"), | ||
| ("esm", "EsmConfig"), | ||
| ("eurobert", "EuroBertConfig"), | ||
|
|
@@ -661,6 +662,7 @@ | |
| ("ernie", "ERNIE"), | ||
| ("ernie4_5", "Ernie4_5"), | ||
| ("ernie4_5_moe", "Ernie4_5_MoE"), | ||
| ("ernie4_5_moe_vl", "Ernie4_5_VLMoE"), | ||
| ("ernie4_5_vl_moe", "Ernie4_5_VLMoE"), | ||
| ("esm", "ESM"), | ||
| ("eurobert", "EuroBERT"), | ||
|
|
@@ -1081,6 +1083,7 @@ | |
|
|
||
| SPECIAL_MODEL_TYPE_TO_MODULE_NAME = OrderedDict[str, str]( | ||
| [ | ||
| ("ernie4_5_moe_vl", "ernie4_5_vl_moe"), | ||
| ("audioflamingo3_encoder", "audioflamingo3"), | ||
| ("musicflamingo_encoder", "musicflamingo"), | ||
| ("openai-gpt", "openai"), | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -117,7 +117,7 @@ class Ernie4_5_VLMoeTextConfig(Ernie4_5_MoeConfig): | |
| Whether to use a bias in any of the projections including mlp and attention for example | ||
| moe_k (`int`, *optional*, defaults to 6): | ||
| Number of selected experts. | ||
| moe_num_experts (`int`, *optional*, defaults to 64): | ||
| moe_num_experts (`int` or `list[int]`, *optional*, defaults to 64): | ||
| Number of routed experts. | ||
| moe_num_shared_experts (`int`, *optional*, defaults to 2): | ||
| The number of experts that are shared for all MoE forwards. | ||
|
|
@@ -149,6 +149,7 @@ class Ernie4_5_VLMoeTextConfig(Ernie4_5_MoeConfig): | |
| pad_token_id: int | None = None | ||
| eos_token_id: int | list[int] | None = None | ||
| bos_token_id: int | None = None | ||
| moe_num_experts: int | list[int] | None = 64 | ||
|
Comment on lines
120
to
+152
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is still for remote only, no? The transformers version should not have a list for these as they are always the same size
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nope, this one is valid for strict-type validation 😓 update: actually no, hf-converted configs won't have list of ints (see https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT/discussions/11/files)
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. discussed internally, it's only the remote configs as well |
||
| moe_layer_end_index = AttributeError() | ||
| moe_layer_interval = AttributeError() | ||
| moe_layer_start_index = AttributeError() | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the rope is set to 50k even without this line. I see that the text config class has a
rope_parametersfield with defaultNoneso we will go by the firstif path