Skip to content

[Fix] [XPU] rope: Fall back to native RoPE for head sizes the XPU kernel lacks - #46997

Closed
AgainstEntropy wants to merge 7 commits into
mainfrom
xpu/qwen3-rope-native-fallback
Closed

AgainstEntropy wants to merge 7 commits into
mainfrom
xpu/qwen3-rope-native-fallback

Conversation

@AgainstEntropy

@AgainstEntropy AgainstEntropy commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

RotaryEmbedding.forward_xpu sends every full-rotary head size that has no fused path to sgl_kernel.rotary_embedding. That kernel only accepts head sizes 32, 64, 96, 128 and 256 and raises a RuntimeError on anything else.

The EAGLE3 draft for Qwen3-32B (AngelSlim/Qwen3-32B_eagle3) has head_dim 80. Serving Qwen3-32B with that draft on XPU therefore crashes the server on the first draft forward.

Modifications

  • layers/rotary_embedding/base.py:
    • New _XPU_ROTARY_EMBEDDING_HEAD_SIZES, the head sizes sgl_kernel.rotary_embedding accepts.
    • forward_xpu sends any other full-rotary head size to forward_native, as it already does for partial rotary.
  • Supported head sizes still take the kernel, so existing models are unaffected.
  • New test/registered/xpu/test_rope_xpu.py: it checks head size 80 (neox and non-neox) against forward_native. Without this fix it fails with the kernel's RuntimeError; with it, it passes.

Accuracy Tests

  • test_rope_xpu.py passes on Arc Pro B70 with sgl-kernel-xpu 0.4.0 and 0.5.0. The head sizes the kernel supports did not change between those versions.
  • No behavior change for supported head sizes.

Speed Tests and Profiling

Not a performance change: supported head sizes keep the kernel path. Only unsupported head sizes, which used to crash, take the native path. For the Qwen3-32B EAGLE3 draft that is a single layer.

The draft's attention needs a separate fix (#46998). With both, Qwen3-32B + EAGLE3 on 4x B70 reaches 42.4 tok/s at c=1, against 31.5 tok/s without speculative decoding. That run used greedy ShareGPT prompts and 3 draft steps with topk 1.

Checklist


CI States

Latest PR Test (Base): ✅ Run #42706803311
Latest PR Test (Extra): ❌ Run #42706803098
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…nel lacks

RotaryEmbedding.forward_xpu sent every full-rotary head size it had no fused path
for to sgl_kernel.rotary_embedding, which only takes 32/64/96/128/256 and raises
otherwise. Qwen3-32B's EAGLE3 draft (AngelSlim/Qwen3-32B_eagle3, head_dim 80)
therefore crashed the server on its first draft forward. Route such head sizes
to forward_native, as the partial-rotary case already does.

Add an XPU regression test that fails with the kernel's RuntimeError on the
previous code.
@alexnails
alexnails force-pushed the xpu/qwen3-rope-native-fallback branch from 6952b69 to e7fbf03 Compare October 28, 2026 08:00
airMeng pushed a commit to airMeng/sglang that referenced this pull request Oct 28, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Oct 28, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 3, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 4, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 4, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 4, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 5, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 6, 2026
airMeng pushed a commit to airMeng/sglang that referenced this pull request Nov 6, 2026
@AgainstEntropy

Copy link
Copy Markdown
Collaborator Author

Close as covered by the consolidation PR #47956

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant