The LoRA model trained with qwen3_moe on hf4.x cannot be used on the current main branch (hf5.x).

[linitra24]

System Info

• transformers version: 5.0.0.dev0
• Platform: Linux-5.15.0-122-generic-x86_64-with-glibc2.31
• Python version: 3.10.15
• Huggingface_hub version: 1.1.5
• Safetensors version: 0.4.5
• Accelerate version: 1.9.0
• Accelerate config: not found
• DeepSpeed version: 0.16.2
• PyTorch version (accelerator?): 2.9.0+cu128 (CUDA)
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA A800-SXM4-80GB

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

This is my test code:

import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel


model_name_or_path = "Qwen/Qwen3-30B-A3B"
qwen3moe_lora_files = "/path/to/lora_model"

model = AutoModelForCausalLM.from_pretrained(
    model_name_or_path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval()
print(model)
model = PeftModel.from_pretrained(model, qwen3moe_lora_files)
print(model)

The results printed by the LoRA model are as follows:

PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): Qwen3MoeForCausalLM(
      (model): Qwen3MoeModel(
        (embed_tokens): Embedding(151936, 2048)
        (layers): ModuleList(
          (0-47): 48 x Qwen3MoeDecoderLayer(
            (self_attn): Qwen3MoeAttention(
              (q_proj): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (k_proj): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=512, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=512, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (v_proj): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=512, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=512, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (o_proj): lora.Linear(
                (base_layer): Linear(in_features=4096, out_features=2048, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=2048, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (q_norm): Qwen3MoeRMSNorm((128,), eps=1e-06)
              (k_norm): Qwen3MoeRMSNorm((128,), eps=1e-06)
            )
            (mlp): Qwen3MoeSparseMoeBlock(
              (experts): Qwen3MoeExperts(
                (act_fn): SiLUActivation()
              )
              (router): Qwen3MoeTopKRouter()
            )
            (input_layernorm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
            (post_attention_layernorm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
          )
        )
        (norm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
        (rotary_emb): Qwen3MoeRotaryEmbedding()
      )
      (lm_head): Linear(in_features=2048, out_features=151936, bias=False)
    )
  )
)

The LoRA part of MoE has not been correctly loaded.

Expected behavior

This is the result of the same code on transformers 4.57.3:

PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): Qwen3MoeForCausalLM(
      (model): Qwen3MoeModel(
        (embed_tokens): Embedding(151936, 2048)
        (layers): ModuleList(
          (0-47): 48 x Qwen3MoeDecoderLayer(
            (self_attn): Qwen3MoeAttention(
              (q_proj): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=4096, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=4096, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (k_proj): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=512, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=512, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (v_proj): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=512, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=512, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (o_proj): lora.Linear(
                (base_layer): Linear(in_features=4096, out_features=2048, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=4096, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=2048, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (q_norm): Qwen3MoeRMSNorm((128,), eps=1e-06)
              (k_norm): Qwen3MoeRMSNorm((128,), eps=1e-06)
            )
            (mlp): Qwen3MoeSparseMoeBlock(
              (gate): lora.Linear(
                (base_layer): Linear(in_features=2048, out_features=128, bias=False)
                (lora_dropout): ModuleDict(
                  (default): Identity()
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=2048, out_features=8, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=8, out_features=128, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (lora_magnitude_vector): ModuleDict()
              )
              (experts): ModuleList(
                (0-127): 128 x Qwen3MoeMLP(
                  (gate_proj): lora.Linear(
                    (base_layer): Linear(in_features=2048, out_features=768, bias=False)
                    (lora_dropout): ModuleDict(
                      (default): Identity()
                    )
                    (lora_A): ModuleDict(
                      (default): Linear(in_features=2048, out_features=8, bias=False)
                    )
                    (lora_B): ModuleDict(
                      (default): Linear(in_features=8, out_features=768, bias=False)
                    )
                    (lora_embedding_A): ParameterDict()
                    (lora_embedding_B): ParameterDict()
                    (lora_magnitude_vector): ModuleDict()
                  )
                  (up_proj): lora.Linear(
                    (base_layer): Linear(in_features=2048, out_features=768, bias=False)
                    (lora_dropout): ModuleDict(
                      (default): Identity()
                    )
                    (lora_A): ModuleDict(
                      (default): Linear(in_features=2048, out_features=8, bias=False)
                    )
                    (lora_B): ModuleDict(
                      (default): Linear(in_features=8, out_features=768, bias=False)
                    )
                    (lora_embedding_A): ParameterDict()
                    (lora_embedding_B): ParameterDict()
                    (lora_magnitude_vector): ModuleDict()
                  )
                  (down_proj): lora.Linear(
                    (base_layer): Linear(in_features=768, out_features=2048, bias=False)
                    (lora_dropout): ModuleDict(
                      (default): Identity()
                    )
                    (lora_A): ModuleDict(
                      (default): Linear(in_features=768, out_features=8, bias=False)
                    )
                    (lora_B): ModuleDict(
                      (default): Linear(in_features=8, out_features=2048, bias=False)
                    )
                    (lora_embedding_A): ParameterDict()
                    (lora_embedding_B): ParameterDict()
                    (lora_magnitude_vector): ModuleDict()
                  )
                  (act_fn): SiLUActivation()
                )
              )
            )
            (input_layernorm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
            (post_attention_layernorm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
          )
        )
        (norm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
        (rotary_emb): Qwen3MoeRotaryEmbedding()
      )
      (lm_head): Linear(in_features=2048, out_features=151936, bias=False)
    )
  )
)

After upgrading to transformers 5.x, can the previously trained LoRA models still be used? Does this mean that old models need to be retrained according to transformers 5.x?

[linitra24]

Moreover, during training, the LoRA config in transformers 5.x is quite different from that in 4.x. For example, in https://huggingface.co/jeeejeee/qwen3-moe-text2sql-spider, the LoRA config is as follows:

...
"target_modules": [
  "k_proj",
  "up_proj",
  "o_proj",
  "v_proj",
  "q_proj",
  "down_proj",
  "gate",
  "gate_proj"
],
"target_parameters": null,
...

whereas training under transformers 5.x requires the following config instead:

...
"target_modules": [
  "k_proj",
  "o_proj",
  "v_proj",
  "q_proj",
],
"target_parameters": [
  "mlp.experts.gate_up_proj",
  "mlp.experts.down_proj"
]
...

Will this cause LoRA models trained with different versions of Transformers to be incompatible with each other? And will downstream applications such as vLLM also fail to load LoRA models due to these version differences?

[ArthurZucker]

We Will try to see if there is a possible fix but basically we merge module lists and fuse gate up on the fly now. This is not allow fast inference but needs to be adapted on peft side with a special conversion scheme / special replacement for experts.

V5 is not 100% ready yet so that’s expected and we are working on it!

[BenjaminBossan]

Note that we have the following situation: Say the v4 transformers model has layers with weights W0 and W1. These weights are targeted with LoRA (or other PEFT adapters for that matter). In transformers v5, these layers are fused into a single layer W01. It is mathematically impossible to fuse the separate LoRA layers into a single one that would result in the same output as with v4.

While we're internally discussing how best to solve this, there is a solution that may work for your use case. This would be to merge the LoRA weights with v4 and then load the merged weights with v5. This means that LoRA is baked into the base weights and then it can be loaded successfully.

In code, it would be something like:

# using v4
base_model = AutoModelForCausalLM.from_pretrained(<model-id>)
model = PeftModel.from_pretrained(base_model, <path-to-lora>)
model = model.merge_and_unload()
model.save_pretrained(<path-merged-model>)

# using v5
model = AutoModelForCausalLM.from_pretrained(<path-merged-model>)

This should yield you a model in v5 that generates the same outputs as the v4 model (of course, there is a bit of floating point imprecision from merging). There are disadvantages to this:

1. The size of the checkpoint of the merged model will be large, the same size as the original model, not small like a LoRA adapter.
2. You can only use this merged adapter. You cannot switch between multiple LoRA adapters or disable the LoRA adapter.

Depending on your use case, this may or may not work for you. Just putting this out there until we have a better solution.

[ArthurZucker]

It is mathematically impossible to fuse the separate LoRA layers into a single one that would result in the same output as with v4

By that @BenjaminBossan means the operation we are gonna apply is not a just torch.cat but more like torch.cat(B0, B1) + torch.block_diagonal(A0, A1). For the user, this should be quite invisible

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.