Skip to content

fix(qwen4): accept qwen_sparse_attention layer type alias - #36772

Merged
yhyang201 merged 1 commit into
sgl-project:qwen4-main-squashedfrom
huangzhilin-hzl:molou/qwen4-layer-type-alias
Aug 28, 2026
Merged

yhyang201 merged 1 commit into
sgl-project:qwen4-main-squashedfrom
huangzhilin-hzl:molou/qwen4-layer-type-alias

Conversation

@huangzhilin-hzl

@huangzhilin-hzl huangzhilin-hzl commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Transformers v5.16.1 normalizes Qwen4-Exp checkpoint layers from full_attention to qwen_sparse_attention. Checkpoints re-saved with that version can therefore contain qwen_sparse_attention.

Qwen4ExpTextConfig.layers_block_type currently normalizes only full_attention, leaving qwen_sparse_attention to reach the decoder layer dispatch and raise a KeyError.

Modifications

  • Normalize both full_attention and qwen_sparse_attention to the runtime attention layer type.

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #33137316255
Latest PR Test (Extra): ❌ Run #33137316077
Latest PR Test (AMD ROCm 7.2): ❌ Run #33137316183

@yhyang201
yhyang201 merged commit 639db13 into sgl-project:qwen4-main-squashed Aug 28, 2026
80 of 90 checks passed
@yhyang201

Copy link
Copy Markdown
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants