Repository navigation
[Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp - #40501
Merged
Merged
Conversation
5 tasks
(cherry picked from commit 9f3718b)
YAMY1234
force-pushed
the
pp-verify/qwen4-exp-pp-main
branch
from
September 20, 2026 23:46
1459225 to
2106ebe
Compare
(cherry picked from commit bc0f135)
…l embedding relay, gate allowlist The widened hidden state already carries the hyper-connection streams, so a PP boundary relays no separate residual. Multimodal embeddings are produced on the first stage only; the last stage needs them for the MTP draft prefill, so they ride along with the target hidden states. Add Qwen4ExpForConditionalGeneration to the PD-prefill PP + EAGLE architecture allowlist.
YAMY1234
force-pushed
the
pp-verify/qwen4-exp-pp-main
branch
from
September 20, 2026 23:50
2106ebe to
30e9750
Compare
YAMY1234
marked this pull request as ready for review
September 21, 2026 00:21
YAMY1234
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
Qiaolin-Yu,
ShangmingCai,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sogalin,
xiezhq-hermann and
yizhang2077
as code owners
September 21, 2026 00:21
3 of 4 tasks
Collaborator
Author
|
/tag-and-rerun-ci |
…7509-sync-f532ad1f
Collaborator
Author
YAMY1234
added a commit
to YAMY1234/sglang
that referenced
this pull request
Sep 22, 2026
Resolve the allowlist conflict from sgl-project#40501: this branch removes _PP_EAGLE_SUPPORTED_ARCHITECTURES entirely, so the Qwen4Exp entry and its allowlist test go with it.
detain
added a commit
to detain/sglang
that referenced
this pull request
Sep 28, 2026
Brings in 113 upstream commits (7ad55e4..d34f7b2). Upstream now contains three stack PRs as merged: sgl-project#40501, sgl-project#40175, sgl-project#33778; every line their squashes add is already present in the stack. Conflicts resolved: - arg_groups/fields/memory.py, managers/cache_controller.py: union of the stack's fast_file backend (sgl-project#39880) and upstream's tensorcast backend. - layers/attention/qsa/mqa.py: keep the stack's SM120 Triton imports and fp8 _scoring_dtype path; adopt upstream's ROCm 16-wide head alignment (sgl-project#38875) and its is_hip import. - models/qwen4_exp.py: all six hunks are stack-only additions (PLE host staging, CUDA-graph prewarm, replay prepare) against upstream's final sgl-project#40501; kept ours. The merged file equals the pre-merge stack version. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
10 of 21 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Qwen3.8-Flash-Next (
Qwen4ExpForConditionalGeneration) could not run with--pp-size > 1. #38698 by @zhangxiaolei123456 added pipeline-parallel serving and PD-prefill MTP for it, but that branch no longer merges after #30775, #39378 and #40045. This PR carries the two commits from #38698 (authorship preserved) rebased onto main, plus the follow-ups we needed to run it end to end.Modifications
feat(qwen4): support pipeline parallel servingandfeat(qwen4): support PP PD prefill with MTPfrom [Qwen3.8-Next] Add pipeline-parallel PD prefill with MTP #38698, cherry-picked with-x. Adapted to main: the PD-prefill gate they added is dropped in favour ofcheck_pipeline_parallel_compat(Fix disagg PP MTP for GLM-5.2 #39378), andget_embed_and_headreadsself.model.embed_tokens(main flattened the VL model).Qwen4ExpForConditionalGenerationto the PD-prefill PP + EAGLE architecture allowlist.The standalone
test_qwen4_exp_pipeline_parallel.pyfrom #38698 is not carried over; coverage stays in the existing disaggregation-wire and EAGLE-worker tests. The QSA PD state-transfer commits on the #38698 branch are already on main via #36651 and are not repeated here.Accuracy Tests
GB300, Qwen3.8-Flash-Next, GSM8K-200 via
sglang.test.run_eval, MTP steps/topk/draft = 3/1/4 (reference TP4 + MTP: 0.980, accept length 3.16):PYTHONOPTIMIZE, regular gate)SGLANG_ENABLE_PP_SPEC=1The aggregate path needs #39643 (draft-embedding loader; #30775's inline loader does not know Qwen4-Exp's embedding key) and #40001 (hybrid recurrent-state commit under PP + spec; Qwen4-Exp is a GDN hybrid). PD prefill depends on neither.
Speed Tests and Profiling
Prefill-only ISL 8192 / OSL 1, chunk 32768, MTP on, GB300, with #40499: TP1xPP4 38,310 tok/s/GPU vs DEP4 30,162 (1.27x); TP2xPP2 30,344 (1.01x).
Checklist
CI States
Latest PR Test (Base): ✅ Run #35656822499
Latest PR Test (Extra): ✅ Run #35694192167
Latest PR Test (AMD ROCm 10): ❌ Run #35656822433