Repository navigation
perf: split mixed GDN prefill and decode kernels - #36065
Closed
Destination2020 wants to merge 1 commit into
Closed
Destination2020 wants to merge 1 commit into
Destination2020 wants to merge 1 commit into
Conversation
Destination2020
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
Ying1123,
hebiao064,
hnyls2002,
ispobock,
merrymercy and
xiezhq-hermann
as code owners
August 23, 2026 10:06
Author
|
/tag-run-ci-label |
5 tasks done
4 of 5 tasks
Contributor
|
Thanks @Destination2020. Closing this because you have 6 PRs open and none updated in 7 days, over our soft cap of 5 for idle PRs. Reopen the two or three you most want reviewed - a short active set moves faster than a long idle one. Some directories moved recently, so an older branch may need retargeting: |
Author
|
This PR was automatically closed by the idle-PR quota rule. I’d like to prioritize this PR for review and continue working on it. Could a maintainer please reopen it? The source branch still exists. Thanks! |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With mixed chunk enabled,
ScheduleBatch.mix_with_running()lays a batch out as[prefill requests | running decode requests]and marks itMIXED. The hybridlinear-attention wrapper currently routes that whole batch through
GDNAttnBackend.forward_extend(). As a result, every one-token decode suffix isprocessed by causal-conv and FlashInfer prefill kernels in all 48 GDN layers of
Qwen3.8-27B.
This becomes material at high decode concurrency. The change is inspired by the
mixed GDN invariant in vLLM #44700, but is implemented using SGLang's existing
ScheduleBatch,ForwardBatch, GDN dispatcher, and state pools.This does not duplicate #34387: that WIP splits Ascend full-attention FIA calls
and explicitly leaves the linear/GDN side on its extend path. This PR targets
CUDA GDN causal-conv and recurrent decode. It uses the same proposed mixed
boundary field names to reduce future integration conflicts.
Modifications
appended to the mixed batch, and carry the two host scalars through
ForwardBatchand breakable-prefill-graph replay.causal-conv + FlashInfer extend kernels and the one-token decode suffix through
the existing causal-conv update + Triton packed recurrent decode kernels.
state tracking/checkpoints, non-contiguous/page-major state, ReplaySSM, invalid
boundaries, and backends without packed decode support.
The production change is 4 files and 177 insertions. Two focused unit-test files
add 89 lines.
Accuracy Tests
Deterministic serving A/B on Qwen3.8-27B, TP=2 (2x H20), with 128 already-running
decoders overlapping one 16,384-token prefill, followed by a pure-request
control:
Focused tests on the rebased commit:
Pure 64K prefill control, three samples per side, showed 0.080% median E2E and
0.083% median TTFT regression. Pure decode uses the unchanged
forward_decodepath.
Speed Tests and Profiling
Serving A/B used Qwen3.8-27B, TP=2 on 2x H20, chunked prefill size 8192,
speculation off, radix cache disabled, and breakable prefill graph enabled. Each
cell overlaps one 64K prefill with
Dalready-running decoders. Baseline andcandidate used identical model revision, seed, requests, backend choices, and
server arguments.
At D=953, baseline GDN mixed-window kernels spent 4,516.5 ms in causal-conv
prefill plus 1,327.2 ms in prefill delta. The split path spent 47.9 ms in prefill
conv, 24.6 ms in decode conv update, and 403.0 ms in recurrent decode. All 954
requests completed.
The serving measurements were collected on
bd3cc97e7ee49c8dcdae1679eca6b1f1cc36bbeb.This branch is rebased onto
e3a008a9db5e0575ab273a33ad5937cc40cacff6;the intervening changes in touched files are config-access refactors and do not
change mixed routing or GDN kernels.
Checklist
CI States
Latest PR Test (Base): ❌ Run #32632871184
Latest PR Test (Extra): ❌ Run #32632870907
Latest PR Test (AMD ROCm 7.2): ❌ Run #32632870994