Skip to content
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/transformers/configuration_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,9 @@ def __post_init__(self, **kwargs):
# BC for rotary embeddings. We will pop out legacy keys from kwargs and rename to new format
if hasattr(self, "rope_parameters"):
kwargs = self.convert_rope_params_to_dict(**kwargs)
elif kwargs.get("rope_scaling") and kwargs.get("rope_theta"):
elif kwargs.get("rope_scaling") and (
kwargs.get("rope_theta") or getattr(self, "rope_theta", None) is not None
):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the rope is set to 50k even without this line. I see that the text config class has a rope_parameters field with default None so we will go by the first if path

logger.warning(
f"{self.__class__.__name__} got `key=rope_scaling` in kwargs but hasn't set it as attribute. "
"For RoPE standardization you need to set `self.rope_parameters` in model's config. "
Expand Down
3 changes: 3 additions & 0 deletions src/transformers/models/auto/configuration_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@
("ernie", "ErnieConfig"),
("ernie4_5", "Ernie4_5Config"),
("ernie4_5_moe", "Ernie4_5_MoeConfig"),
("ernie4_5_moe_vl", "Ernie4_5_VLMoeConfig"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vasqu for this, i remember you were changing model types recently

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea this looks like the only valid change IF we change the config model type of the hub PRs - this would imply that we support 2 model types for one model which is not in our code base, i.e. a dirty workaround.

Imo, we should sync with vLLM support first / change the model type there. But that needs v5 support first, so I'd like to withhold on this PR for now and potentially "fix" on vLLM side instead

("ernie4_5_vl_moe", "Ernie4_5_VLMoeConfig"),
("esm", "EsmConfig"),
("eurobert", "EuroBertConfig"),
Expand Down Expand Up @@ -661,6 +662,7 @@
("ernie", "ERNIE"),
("ernie4_5", "Ernie4_5"),
("ernie4_5_moe", "Ernie4_5_MoE"),
("ernie4_5_moe_vl", "Ernie4_5_VLMoE"),
("ernie4_5_vl_moe", "Ernie4_5_VLMoE"),
("esm", "ESM"),
("eurobert", "EuroBERT"),
Expand Down Expand Up @@ -1081,6 +1083,7 @@

SPECIAL_MODEL_TYPE_TO_MODULE_NAME = OrderedDict[str, str](
[
("ernie4_5_moe_vl", "ernie4_5_vl_moe"),
("audioflamingo3_encoder", "audioflamingo3"),
("musicflamingo_encoder", "musicflamingo"),
("openai-gpt", "openai"),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ class Ernie4_5_VLMoeTextConfig(PreTrainedConfig):
Whether to use a bias in any of the projections including mlp and attention for example
moe_k (`int`, *optional*, defaults to 6):
Number of selected experts.
moe_num_experts (`int`, *optional*, defaults to 64):
Number of routed experts.
moe_num_experts (`int` or `list[int]`, *optional*, defaults to 64):
Number of routed experts. Can be a list to specify per-layer expert counts.
moe_num_shared_experts (`int`, *optional*, defaults to 2):
The number of experts that are shared for all MoE forwards.
moe_norm_min (`float`, *optional*, defaults to 1e-12):
Expand Down Expand Up @@ -119,7 +119,7 @@ class Ernie4_5_VLMoeTextConfig(PreTrainedConfig):
use_bias: bool | None = False
moe_intermediate_size: list[int] | None = None
moe_k: int | None = 6
moe_num_experts: int | None = 64
moe_num_experts: int | list[int] | None = 64
moe_num_shared_experts: int | None = 2
moe_norm_min: float | None = 1e-12
output_router_logits: bool | None = False
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ class Ernie4_5_VLMoeTextConfig(Ernie4_5_MoeConfig):
Whether to use a bias in any of the projections including mlp and attention for example
moe_k (`int`, *optional*, defaults to 6):
Number of selected experts.
moe_num_experts (`int`, *optional*, defaults to 64):
moe_num_experts (`int` or `list[int]`, *optional*, defaults to 64):
Number of routed experts.
moe_num_shared_experts (`int`, *optional*, defaults to 2):
The number of experts that are shared for all MoE forwards.
Expand Down Expand Up @@ -149,6 +149,7 @@ class Ernie4_5_VLMoeTextConfig(Ernie4_5_MoeConfig):
pad_token_id: int | None = None
eos_token_id: int | list[int] | None = None
bos_token_id: int | None = None
moe_num_experts: int | list[int] | None = 64
Comment on lines 120 to +152

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still for remote only, no? The transformers version should not have a list for these as they are always the same size

@zucchini-nlp zucchini-nlp Apr 9, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, this one is valid for strict-type validation 😓

update: actually no, hf-converted configs won't have list of ints (see https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT/discussions/11/files)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

discussed internally, it's only the remote configs as well

moe_layer_end_index = AttributeError()
moe_layer_interval = AttributeError()
moe_layer_start_index = AttributeError()
Expand Down
Loading