Repository navigation
[Fix] [XPU] rope: Fall back to native RoPE for head sizes the XPU kernel lacks - #46997
Closed
AgainstEntropy wants to merge 7 commits into
Closed
AgainstEntropy wants to merge 7 commits into
AgainstEntropy wants to merge 7 commits into
Conversation
AgainstEntropy
requested review from
BBuf,
Edwardf0t1,
Fridge003,
HaiShaw,
Ying1123,
ch-wan,
ispobock and
merrymercy
as code owners
October 28, 2026 07:16
This was referenced Oct 28, 2026
Closed
…nel lacks RotaryEmbedding.forward_xpu sent every full-rotary head size it had no fused path for to sgl_kernel.rotary_embedding, which only takes 32/64/96/128/256 and raises otherwise. Qwen3-32B's EAGLE3 draft (AngelSlim/Qwen3-32B_eagle3, head_dim 80) therefore crashed the server on its first draft forward. Route such head sizes to forward_native, as the partial-rotary case already does. Add an XPU regression test that fails with the kernel's RuntimeError on the previous code.
alexnails
force-pushed
the
xpu/qwen3-rope-native-fallback
branch
from
October 28, 2026 08:00
6952b69 to
e7fbf03
Compare
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Oct 28, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Oct 28, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 3, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 4, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 4, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 4, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 5, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 6, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 6, 2026
airMeng
pushed a commit
to airMeng/sglang
that referenced
this pull request
Nov 7, 2026
Collaborator
Author
|
Close as covered by the consolidation PR #47956 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
RotaryEmbedding.forward_xpusends every full-rotary head size that has no fused path tosgl_kernel.rotary_embedding. That kernel only accepts head sizes 32, 64, 96, 128 and 256 and raises aRuntimeErroron anything else.The EAGLE3 draft for Qwen3-32B (
AngelSlim/Qwen3-32B_eagle3) has head_dim 80. Serving Qwen3-32B with that draft on XPU therefore crashes the server on the first draft forward.Modifications
layers/rotary_embedding/base.py:_XPU_ROTARY_EMBEDDING_HEAD_SIZES, the head sizessgl_kernel.rotary_embeddingaccepts.forward_xpusends any other full-rotary head size toforward_native, as it already does for partial rotary.test/registered/xpu/test_rope_xpu.py: it checks head size 80 (neox and non-neox) againstforward_native. Without this fix it fails with the kernel'sRuntimeError; with it, it passes.Accuracy Tests
test_rope_xpu.pypasses on Arc Pro B70 with sgl-kernel-xpu 0.4.0 and 0.5.0. The head sizes the kernel supports did not change between those versions.Speed Tests and Profiling
Not a performance change: supported head sizes keep the kernel path. Only unsupported head sizes, which used to crash, take the native path. For the Qwen3-32B EAGLE3 draft that is a single layer.
The draft's attention needs a separate fix (#46998). With both, Qwen3-32B + EAGLE3 on 4x B70 reaches 42.4 tok/s at c=1, against 31.5 tok/s without speculative decoding. That run used greedy ShareGPT prompts and 3 draft steps with topk 1.
Checklist
CI States
Latest PR Test (Base): ✅ Run #42706803311
Latest PR Test (Extra): ❌ Run #42706803098
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: