Skip to content

perf: split mixed GDN prefill and decode kernels - #36065

Closed
Destination2020 wants to merge 1 commit into
sgl-project:mainfrom
Destination2020:feature/qwen38-mixed-gdn-split-v1
Closed

Destination2020 wants to merge 1 commit into
sgl-project:mainfrom
Destination2020:feature/qwen38-mixed-gdn-split-v1

Conversation

@Destination2020

@Destination2020 Destination2020 commented Aug 23, 2026 •

Copy link
Copy Markdown

Motivation

With mixed chunk enabled, ScheduleBatch.mix_with_running() lays a batch out as
[prefill requests | running decode requests] and marks it MIXED. The hybrid
linear-attention wrapper currently routes that whole batch through
GDNAttnBackend.forward_extend(). As a result, every one-token decode suffix is
processed by causal-conv and FlashInfer prefill kernels in all 48 GDN layers of
Qwen3.8-27B.

This becomes material at high decode concurrency. The change is inspired by the
mixed GDN invariant in vLLM #44700, but is implemented using SGLang's existing
ScheduleBatch, ForwardBatch, GDN dispatcher, and state pools.

This does not duplicate #34387: that WIP splits Ascend full-attention FIA calls
and explicitly leaves the linear/GDN side on its extend path. This PR targets
CUDA GDN causal-conv and recurrent decode. It uses the same proposed mixed
boundary field names to reduce future integration conflicts.

Modifications

  • Capture the prefill request/token boundary before running decode requests are
    appended to the mixed batch, and carry the two host scalars through
    ForwardBatch and breakable-prefill-graph replay.
  • In the guarded CUDA GDN path, run the prefill prefix through the existing
    causal-conv + FlashInfer extend kernels and the one-token decode suffix through
    the existing causal-conv update + Triton packed recurrent decode kernels.
  • Concatenate outputs in the scheduler's original prefill-first token order.
  • Keep the old unified extend path as fallback for speculation, DP padding, TBO,
    state tracking/checkpoints, non-contiguous/page-major state, ReplaySSM, invalid
    boundaries, and backends without packed decode support.
  • Keep breakable prefill CUDA graphs enabled; no graph mode is disabled.

The production change is 4 files and 177 insertions. Two focused unit-test files
add 89 lines.

Accuracy Tests

Deterministic serving A/B on Qwen3.8-27B, TP=2 (2x H20), with 128 already-running
decoders overlapping one 16,384-token prefill, followed by a pure-request
control:

  • 130 requests and 8,232 generated tokens compared.
  • Output token-ID mismatches: 0.
  • Output logprob token-ID mismatches: 0.
  • Maximum and mean output-logprob absolute difference: 0.0.
  • All requests completed.

Focused tests on the rebased commit:

PYTHONPATH="$PWD/python" python -m pytest -q \
  test/registered/unit/layers/attention/test_gdn_prefill_backend_policy.py \
  test/registered/unit/managers/test_schedule_batch_out_of_place.py
# 17 passed

Pure 64K prefill control, three samples per side, showed 0.080% median E2E and
0.083% median TTFT regression. Pure decode uses the unchanged forward_decode
path.

Speed Tests and Profiling

Serving A/B used Qwen3.8-27B, TP=2 on 2x H20, chunked prefill size 8192,
speculation off, radix cache disabled, and breakable prefill graph enabled. Each
cell overlaps one 64K prefill with D already-running decoders. Baseline and
candidate used identical model revision, seed, requests, backend choices, and
server arguments.

Decode concurrency Baseline prefill TTFT Candidate prefill TTFT TTFT gain Baseline mixed window Candidate mixed window Mixed-window gain
256 17.810 s 16.612 s 6.73% 17,297.6 ms 16,158.1 ms 6.59%
512 21.110 s 17.825 s 15.56% 19,857.7 ms 16,664.0 ms 16.08%
953 25.171 s 18.674 s 25.81% 24,268.5 ms 17,948.2 ms 26.04%

At D=953, baseline GDN mixed-window kernels spent 4,516.5 ms in causal-conv
prefill plus 1,327.2 ms in prefill delta. The split path spent 47.9 ms in prefill
conv, 24.6 ms in decode conv update, and 403.0 ms in recurrent decode. All 954
requests completed.

The serving measurements were collected on bd3cc97e7ee49c8dcdae1679eca6b1f1cc36bbeb.
This branch is rebased onto e3a008a9db5e0575ab273a33ad5937cc40cacff6;
the intervening changes in touched files are config-access refactors and do not
change mixed routing or GDN kernels.

Checklist

  • Format changed Python files with Black.
  • Add focused unit tests (17 passed).
  • Provide deterministic accuracy results.
  • Provide serving speed and profiler results.
  • Keep unsupported configurations on the existing fallback path.
  • Documentation is not needed for this internal kernel routing change.

CI States

Latest PR Test (Base): ❌ Run #32632871184
Latest PR Test (Extra): ❌ Run #32632870907
Latest PR Test (AMD ROCm 7.2): ❌ Run #32632870994

Copilot AI lite review requested due to automatic review settings August 23, 2026 10:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Destination2020

Copy link
Copy Markdown
Author

/tag-run-ci-label

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Thanks @Destination2020. Closing this because you have 6 PRs open and none updated in 7 days, over our soft cap of 5 for idle PRs.

Reopen the two or three you most want reviewed - a short active set moves faster than a long idle one.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Oct 2, 2026

Copy link
Copy Markdown
Author

This PR was automatically closed by the idle-PR quota rule. I’d like to prioritize this PR for review and continue working on it. Could a maintainer please reopen it? The source branch still exists. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants