Repository navigation
[Feat] [XPU] GDN: support speculative decoding - #44017
Conversation
|
/tag-and-rerun-ci |
|
/rerun-failed-ci |
This comment has been minimized.
This comment has been minimized.
|
/rerun-failed-ci |
|
Hi @mingfeima @airMeng Please review. Thanks. |
|
/rerun-failed-ci |
|
/rerun-failed-ci |
|
previous ci gated by base too old. |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…he zero initial state Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…trees Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…st fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
9380d9a to
65ef86a
Compare
|
Rebased onto upstream main
The rebase preserves newer upstream dispatcher, state-pool, argument-location and sgl-eval changes. Local validation on Intel Arc Pro B60: 24 tests and 20 subtests passed across the GDN dispatch, tracked-state and branching-tree parity test files. All changed-file pre-commit hooks passed. A focused original-versus-synced wrapper benchmark used identical native kernels, BF16 Q/K/V with 16 query heads, 32 value heads, head dimension 128 and FP32 recurrent state. Medians of 60 alternating XPU-event measurements, with a 128 MiB cache flush before each measurement:
The safety clamps add roughly 3-5 us to decode/verification; skipping redundant gating saves roughly 22 us in these prefill cases. Outputs matched within BF16 tolerance. This isolates wrapper changes rather than measuring full serving performance. The earlier end-to-end results in the PR description were not rerun; no CUDA numerical or performance validation was performed for this update. |
The previous guard checked torch.ops.sgl_kernel.gdn_attention, which was specific to the old per-model XpuGDNAttnBackend fused path removed by sgl-project#44017. The current generic GDNKernelDispatcher path imports XpuGDNKernel from kernels/gdn_xpu.py instead; guard on that import succeeding so the override still degrades silently to triton when the kernel isn't present in a given sgl-kernel build.
Motivation
Support speculative decoding for GDN on XPU.
Modifications
It replaces the Qwen3.5-specific fused XPU GDN route with SGLang’s shared GDN backend, using standalone SYCL kernels for conv1d and supported prefill, decode, and speculative verification phases.
It also adds XPU dispatch tests and a Qwen3.5-9B nightly NEXTN end-to-end test, and updates the existing nightly test’s descriptions.
Accuracy Tests
Verified with Qwen3.5-9B. The outputs are the same with and without speculative decoding, configs:
Speed Tests and Profiling
Qwen3.5-9B, single Intel Arc B60 GPU, input length 1024 (real natural text), output length 512, batch size 1.
If larger inputs won't fit a single B60 GPU with NEXTN speculative decoding (due to draft weights).
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #42970350582
Latest PR Test (Extra): ❌ Run #42970350665
Latest PR Test (AMD ROCm 10): ✅ Run #42970350809
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: