Skip to content

[Feat] [XPU] GDN: support speculative decoding - #44017

Merged
mingfeima merged 13 commits into
sgl-project:mainfrom
Xia-Weiwen:gdn-spec-decode-xpu
Nov 12, 2026
Merged

mingfeima merged 13 commits into
sgl-project:mainfrom
Xia-Weiwen:gdn-spec-decode-xpu

Conversation

@Xia-Weiwen

@Xia-Weiwen Xia-Weiwen commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Support speculative decoding for GDN on XPU.

Modifications

It replaces the Qwen3.5-specific fused XPU GDN route with SGLang’s shared GDN backend, using standalone SYCL kernels for conv1d and supported prefill, decode, and speculative verification phases.

It also adds XPU dispatch tests and a Qwen3.5-9B nightly NEXTN end-to-end test, and updates the existing nightly test’s descriptions.

Accuracy Tests

Verified with Qwen3.5-9B. The outputs are the same with and without speculative decoding, configs:

  • NEXTN/top-k=1
  • NEXTN/top-k=2, with triton backend for full attention
  • NGRAM/breadth=1

Speed Tests and Profiling

Qwen3.5-9B, single Intel Arc B60 GPU, input length 1024 (real natural text), output length 512, batch size 1.

Mode Mean TTFT Mean TPOT Mean E2E
No speculation 0.267 s 40.97 ms/token 21.20 s
NEXTN/top-k=1 0.281 s 33.09 ms/token 17.19 s
NGRAM/breadth=1 0.268 s 37.95 ms/token 19.66 s

If larger inputs won't fit a single B60 GPU with NEXTN speculative decoding (due to draft weights).

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #42970350582
Latest PR Test (Extra): ❌ Run #42970350665
Latest PR Test (AMD ROCm 10): ✅ Run #42970350809

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@Xia-Weiwen Xia-Weiwen changed the title [XPU] support speculative decoding for GDN [Feat] [XPU] support speculative decoding for GDN Oct 15, 2026
@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 29, 2026
@Xia-Weiwen
Xia-Weiwen marked this pull request as ready for review October 29, 2026 07:48

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@github-actions

This comment has been minimized.

@Xia-Weiwen Xia-Weiwen changed the title [Feat] [XPU] support speculative decoding for GDN [Feat] [XPU] GDN: support speculative decoding Oct 29, 2026
@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

Hi @mingfeima @airMeng Please review. Thanks.

@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@mingfeima mingfeima added intel xpu intel gpu with device `torch.xpu` labels Nov 5, 2026
@Xia-Weiwen
Xia-Weiwen requested a review from mingfeima November 5, 2026 06:10
@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@mingfeima

Copy link
Copy Markdown
Collaborator

previous ci gated by base too old.

@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Xia-Weiwen and others added 13 commits November 11, 2026 14:01
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…he zero initial state

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…trees

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…st fixture

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Xia-Weiwen

Copy link
Copy Markdown
Contributor Author

Rebased onto upstream main 8703b47e0a23a799c9493f2620eb04620c96e1a0 and synced the latest GDN fixes:

  • Clamp padded decode and verification seed-state indices without mutating caller metadata.
  • Disable strided target-verification QKV until the SYCL kernel supports it directly.
  • Skip redundant prefill gating on the SYCL path, computing gates only when falling back to Triton.
  • Compute the recurrence initial-state mask only inside the SYCL path, avoiding an extra predicate in shared CUDA prefill.
  • Add branching-tree numerical parity and nonzero post-prefix checkpoint regression coverage.

The rebase preserves newer upstream dispatcher, state-pool, argument-location and sgl-eval changes.

Local validation on Intel Arc Pro B60: 24 tests and 20 subtests passed across the GDN dispatch, tracked-state and branching-tree parity test files. All changed-file pre-commit hooks passed.

A focused original-versus-synced wrapper benchmark used identical native kernels, BF16 Q/K/V with 16 query heads, 32 value heads, head dimension 128 and FP32 recurrent state. Medians of 60 alternating XPU-event measurements, with a 128 MiB cache flush before each measurement:

Phase Batch Tokens/request Original (us) Synced (us)
Decode 1 1 44.687 48.438
Decode 8 1 72.161 75.547
Tree verification 1 4 46.432 51.198
Tree verification 8 4 161.719 165.287
Plain prefill 1 64 228.594 206.458
Plain prefill 8 64 1317.213 1294.818

The safety clamps add roughly 3-5 us to decode/verification; skipping redundant gating saves roughly 22 us in these prefill cases. Outputs matched within BF16 tolerance. This isolates wrapper changes rather than measuring full serving performance. The earlier end-to-end results in the PR description were not rerun; no CUDA numerical or performance validation was performed for this update.

@mingfeima
mingfeima merged commit 46dc697 into sgl-project:main Nov 12, 2026
164 of 169 checks passed
vshekhawat-hlab added a commit to vshekhawat-hlab/sglang that referenced this pull request Nov 12, 2026
The previous guard checked torch.ops.sgl_kernel.gdn_attention, which
was specific to the old per-model XpuGDNAttnBackend fused path removed
by sgl-project#44017. The current generic GDNKernelDispatcher path imports
XpuGDNKernel from kernels/gdn_xpu.py instead; guard on that import
succeeding so the override still degrades silently to triton when the
kernel isn't present in a given sgl-kernel build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

intel run-ci CI: run the baseline test suite on this PR xpu intel gpu with device `torch.xpu`

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants