Skip to content

[Fix] [XPU] spec: Start NEXTN/MTP drafts on XPU and stop silent greedy verify - #46740

Open
siju-samuel wants to merge 1 commit into
sgl-project:mainfrom
siju-samuel:siju/xpu-mtp-spec-fixes
Open

siju-samuel wants to merge 1 commit into
sgl-project:mainfrom
siju-samuel:siju/xpu-mtp-spec-fixes

Conversation

@siju-samuel

@siju-samuel siju-samuel commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

On Intel XPU, EAGLE-family speculative decoding (EAGLE / EAGLE3 / NEXTN, i.e. MTP) has two problems:

  • Most NextN/MTP drafts crash at startup. set_embed_and_head / set_embed call torch.cuda.synchronize(), which raises Torch not compiled with CUDA enabled on torch+xpu. Affected: DeepSeek-V3/R1, GLM-4.x/5, Qwen3-Next, MiMo-V2, Nemotron-H, LongCat, Kimi-K2.5 EAGLE3 and others.
  • Sampling is silently dropped. XPU verify is always greedy, with no warning, and --speculative-use-rejection-sampling is accepted but does nothing.

Modifications

  • models: use current_platform.empty_cache() / synchronize() in set_embed_and_head / set_embed. This is mechanical: 75 sites in 33 files. CUDA/ROCm/MUSA behaviour is unchanged.
  • eagle_utils.py: warn once when EAGLE-path verify falls back to greedy on a sampled batch (XPU, CPU, and ROCm without rejection sampling), as DFLASH already does. It uses the same branch as Add opt-in error for greedy fallback in speculative verification #42828.
  • speculative_hook.py: reject --speculative-use-rejection-sampling on XPU for every algorithm, next to the block-verification device check.
  • docs:
    • mark XPU spec topk=1 as supported in the attention-backend matrix;
    • add an XPU spec section with its limits to xpu.mdx;
    • note the XPU limit in the block-verification paragraph of speculative_decoding.mdx.
  • tests (CPU CI):
    • XPU rejection-sampling cases (EAGLE, NGRAM) in test_block_verification_args.py;
    • test_set_embed_device_neutral.py scans models/ for direct torch.cuda.*() calls in set_embed*;
    • test_spec_registry.py: add speculative_use_rejection_sampling to the hand-built args stub, which the hook now reads.

Accuracy Tests

Arc Pro B60, main vs this PR:

Scenario main This PR
NEXTN, glm-moe-dsa-tiny-random crash at draft init draft initializes, server starts
EAGLE Llama-2-7B, intel_xpu, 5/1/6, greedy + temperature=0.7 requests accept length 1.78 identical output and accept length; warning logged once

Unit tests: test/registered/unit/spec + unit/server_args give 753 passed with this PR vs 752 on main (+1 test, +2 subtests), with identical pre-existing failures.

DSA models (V3.2 / GLM-5) still hit a separate CUDA-only assert in DSA verify (pad.py). That is a follow-up.

Speed Tests and Profiling

Not needed: the changes run at load time, at argument parsing, or as one bool check per verify.

Checklist


CI States

Latest PR Test (Base): ✅ Run #43804630832
Latest PR Test (Extra): ❌ Run #43804630492
Latest PR Test (AMD ROCm 10): ❌ Run #43804630800

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@mintlify

mintlify Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 071dca5.

Approve preview

@siju-samuel

Copy link
Copy Markdown
Contributor Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 3, 2026
@mingfeima

Copy link
Copy Markdown
Collaborator

@siju-samuel Switching the embed/head setup to current_platform makes sense. Those cache/sync calls are initialization work, not per-token overhead.

Can you add a focused, CI-registered XPU regression for draft initialization and the sampling-verification behavior? The new device-neutral embed test checks source on CPU, so it won't catch an actual NEXTN/MTP startup failure on XPU. The test should also check that the greedy fallback is surfaced and unsupported rejection-sampling settings are rejected.

The CUDA port-collision/timeout failure doesn't look like evidence against this patch, but we still need the interrupted required coverage completed.

@siju-samuel
siju-samuel force-pushed the siju/xpu-mtp-spec-fixes branch 2 times, most recently from 3f80f89 to f408935 Compare November 10, 2026 05:59
@siju-samuel

Copy link
Copy Markdown
Contributor Author

Added the XPU hardware coverage:

  • test/registered/xpu/test_spec_xpu_guards.py (per-commit XPU lane): calls the real set_embed_and_head / set_embed with XPU tensors, so it runs the torch.cuda.synchronize() that killed draft init. Fails on main with "Torch not compiled with CUDA enabled", passes here — 27s, no model load. It also checks the launcher refuses --speculative-use-rejection-sampling.
  • test_spec_eagle_intel_xpu.py (nightly): asserts the greedy-verify warning is logged once on a live EAGLE server. Also fails on main. This needed a small opt-in server_log_path hook on the spec fixture.

One correction: the fallback applies to verification only, so a sampled request does not come back greedy — the first token still goes through the normal prefill sampler, which honours the sampling params. The test checks the warning rather than output equality.

Also swept ernie55_nextn.py, new in #48547, so the rewrite is 76 sites in 34 files. Rebased onto main. The earlier MI35x failure was a MIOpen kernel-cache error on the runner, in a test that doesn't touch this patch.

…y verify

- models: set_embed_and_head / set_embed call current_platform.empty_cache()
  and current_platform.synchronize() instead of torch.cuda.*. On torch+xpu,
  torch.cuda.synchronize() raised "Torch not compiled with CUDA enabled" at
  draft init, so NEXTN/MTP crashed for DeepSeek-V3/R1, GLM-4.x/5, Qwen3-Next,
  MiMo-V2, Nemotron-H, LongCat and others. Mechanical: 76 sites in 34 files.
- eagle_utils: warn once when EAGLE-path verify falls back to greedy on a
  sampled batch (XPU, CPU, and ROCm without rejection sampling).
- speculative_hook: reject --speculative-use-rejection-sampling on XPU for
  every algorithm, next to the block-verification device check.
- docs: XPU spec topk=1 supported in the attention matrix; XPU spec section
  in xpu.mdx with its limits.
- tests: test_spec_xpu_guards.py runs the real set_embed_and_head / set_embed
  bodies with XPU tensors and asserts the launcher refuses rejection sampling,
  both on the per-commit XPU lane; test_spec_eagle_intel_xpu.py asserts the
  greedy-verify warning is surfaced once on a live EAGLE server. A source scan
  on CPU cannot see either, so the XPU-only paths now have hardware coverage.
  spec_eagle_fixture gains an opt-in server_log_path for the log assertion.
@siju-samuel
siju-samuel force-pushed the siju/xpu-mtp-spec-fixes branch from f408935 to 071dca5 Compare November 16, 2026 04:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek documentation Improvements or additions to documentation run-ci CI: run the baseline test suite on this PR speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants