Skip to content

[Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp - #40501

Merged
YAMY1234 merged 4 commits into
sgl-project:mainfrom
YAMY1234:pp-verify/qwen4-exp-pp-main
Sep 22, 2026
Merged

YAMY1234 merged 4 commits into
sgl-project:mainfrom
YAMY1234:pp-verify/qwen4-exp-pp-main

Conversation

@YAMY1234

@YAMY1234 YAMY1234 commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Qwen3.8-Flash-Next (Qwen4ExpForConditionalGeneration) could not run with --pp-size > 1. #38698 by @zhangxiaolei123456 added pipeline-parallel serving and PD-prefill MTP for it, but that branch no longer merges after #30775, #39378 and #40045. This PR carries the two commits from #38698 (authorship preserved) rebased onto main, plus the follow-ups we needed to run it end to end.

Modifications

  • feat(qwen4): support pipeline parallel serving and feat(qwen4): support PP PD prefill with MTP from [Qwen3.8-Next] Add pipeline-parallel PD prefill with MTP #38698, cherry-picked with -x. Adapted to main: the PD-prefill gate they added is dropped in favour of check_pipeline_parallel_compat (Fix disagg PP MTP for GLM-5.2 #39378), and get_embed_and_head reads self.model.embed_tokens (main flattened the VL model).
  • Fix the hyper-connection relay contract at PP boundaries (the widened hidden state carries the streams; no separate residual).
  • Relay multimodal embeddings to the last stage, which runs the MTP draft prefill.
  • Add Qwen4ExpForConditionalGeneration to the PD-prefill PP + EAGLE architecture allowlist.

The standalone test_qwen4_exp_pipeline_parallel.py from #38698 is not carried over; coverage stays in the existing disaggregation-wire and EAGLE-worker tests. The QSA PD state-transfer commits on the #38698 branch are already on main via #36651 and are not repeated here.

Accuracy Tests

GB300, Qwen3.8-Flash-Next, GSM8K-200 via sglang.test.run_eval, MTP steps/topk/draft = 3/1/4 (reference TP4 + MTP: 0.980, accept length 3.16):

arm source score accept length
PD prefill TP2xPP2 + MTP -> decode TP4 + MTP (Mooncake, no PYTHONOPTIMIZE, regular gate) this PR on main 0.980 3.16
aggregate TP2xPP2 + MTP, SGLANG_ENABLE_PP_SPEC=1 this PR + #39643 + #40001 0.975 3.13
aggregate, same, without #40001 this PR + #39643 0.955 3.02

The aggregate path needs #39643 (draft-embedding loader; #30775's inline loader does not know Qwen4-Exp's embedding key) and #40001 (hybrid recurrent-state commit under PP + spec; Qwen4-Exp is a GDN hybrid). PD prefill depends on neither.

Speed Tests and Profiling

Prefill-only ISL 8192 / OSL 1, chunk 32768, MTP on, GB300, with #40499: TP1xPP4 38,310 tok/s/GPU vs DEP4 30,162 (1.27x); TP2xPP2 30,344 (1.01x).

Checklist


CI States

Latest PR Test (Base): ✅ Run #35656822499
Latest PR Test (Extra): ✅ Run #35694192167
Latest PR Test (AMD ROCm 10): ❌ Run #35656822433

@YAMY1234
YAMY1234 force-pushed the pp-verify/qwen4-exp-pp-main branch from 1459225 to 2106ebe Compare September 20, 2026 23:46
zhangxiaolei123456 and others added 2 commits September 20, 2026 16:49
…l embedding relay, gate allowlist

The widened hidden state already carries the hyper-connection streams, so a
PP boundary relays no separate residual. Multimodal embeddings are produced on
the first stage only; the last stage needs them for the MTP draft prefill, so
they ride along with the target hidden states. Add Qwen4ExpForConditionalGeneration
to the PD-prefill PP + EAGLE architecture allowlist.
@YAMY1234

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 21, 2026
@YAMY1234 YAMY1234 added bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci-extra CI: also run the extra suite (requires run-ci) bypass-fastfail labels Sep 22, 2026
@YAMY1234

Copy link
Copy Markdown
Collaborator Author

@YAMY1234
YAMY1234 merged commit 6fd98c9 into sgl-project:main Sep 22, 2026
463 of 521 checks passed
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Sep 22, 2026
Resolve the allowlist conflict from sgl-project#40501: this branch removes
_PP_EAGLE_SUPPORTED_ARCHITECTURES entirely, so the Qwen4Exp entry and
its allowlist test go with it.
detain added a commit to detain/sglang that referenced this pull request Sep 28, 2026
Brings in 113 upstream commits (7ad55e4..d34f7b2). Upstream now
contains three stack PRs as merged: sgl-project#40501, sgl-project#40175, sgl-project#33778; every line
their squashes add is already present in the stack.

Conflicts resolved:
- arg_groups/fields/memory.py, managers/cache_controller.py: union of the
  stack's fast_file backend (sgl-project#39880) and upstream's tensorcast backend.
- layers/attention/qsa/mqa.py: keep the stack's SM120 Triton imports and
  fp8 _scoring_dtype path; adopt upstream's ROCm 16-wide head alignment
  (sgl-project#38875) and its is_hip import.
- models/qwen4_exp.py: all six hunks are stack-only additions (PLE host
  staging, CUDA-graph prewarm, replay prepare) against upstream's final
  sgl-project#40501; kept ours. The merged file equals the pre-merge stack version.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) bypass-fastfail run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants