Repository navigation
[Fix] [XPU] spec: Start NEXTN/MTP drafts on XPU and stop silent greedy verify - #46740
siju-samuel wants to merge 1 commit into
Conversation
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
e40c0a4 to
539d252
Compare
539d252 to
0ceb31b
Compare
|
/tag-and-rerun-ci |
|
@siju-samuel Switching the embed/head setup to Can you add a focused, CI-registered XPU regression for draft initialization and the sampling-verification behavior? The new device-neutral embed test checks source on CPU, so it won't catch an actual NEXTN/MTP startup failure on XPU. The test should also check that the greedy fallback is surfaced and unsupported rejection-sampling settings are rejected. The CUDA port-collision/timeout failure doesn't look like evidence against this patch, but we still need the interrupted required coverage completed. |
3f80f89 to
f408935
Compare
|
Added the XPU hardware coverage:
One correction: the fallback applies to verification only, so a sampled request does not come back greedy — the first token still goes through the normal prefill sampler, which honours the sampling params. The test checks the warning rather than output equality. Also swept ernie55_nextn.py, new in #48547, so the rewrite is 76 sites in 34 files. Rebased onto main. The earlier MI35x failure was a MIOpen kernel-cache error on the runner, in a test that doesn't touch this patch. |
…y verify - models: set_embed_and_head / set_embed call current_platform.empty_cache() and current_platform.synchronize() instead of torch.cuda.*. On torch+xpu, torch.cuda.synchronize() raised "Torch not compiled with CUDA enabled" at draft init, so NEXTN/MTP crashed for DeepSeek-V3/R1, GLM-4.x/5, Qwen3-Next, MiMo-V2, Nemotron-H, LongCat and others. Mechanical: 76 sites in 34 files. - eagle_utils: warn once when EAGLE-path verify falls back to greedy on a sampled batch (XPU, CPU, and ROCm without rejection sampling). - speculative_hook: reject --speculative-use-rejection-sampling on XPU for every algorithm, next to the block-verification device check. - docs: XPU spec topk=1 supported in the attention matrix; XPU spec section in xpu.mdx with its limits. - tests: test_spec_xpu_guards.py runs the real set_embed_and_head / set_embed bodies with XPU tensors and asserts the launcher refuses rejection sampling, both on the per-commit XPU lane; test_spec_eagle_intel_xpu.py asserts the greedy-verify warning is surfaced once on a live EAGLE server. A source scan on CPU cannot see either, so the XPU-only paths now have hardware coverage. spec_eagle_fixture gains an opt-in server_log_path for the log assertion.
f408935 to
071dca5
Compare
Motivation
On Intel XPU, EAGLE-family speculative decoding (EAGLE / EAGLE3 / NEXTN, i.e. MTP) has two problems:
set_embed_and_head/set_embedcalltorch.cuda.synchronize(), which raisesTorch not compiled with CUDA enabledon torch+xpu. Affected: DeepSeek-V3/R1, GLM-4.x/5, Qwen3-Next, MiMo-V2, Nemotron-H, LongCat, Kimi-K2.5 EAGLE3 and others.--speculative-use-rejection-samplingis accepted but does nothing.Modifications
current_platform.empty_cache()/synchronize()inset_embed_and_head/set_embed. This is mechanical: 75 sites in 33 files. CUDA/ROCm/MUSA behaviour is unchanged.eagle_utils.py: warn once when EAGLE-path verify falls back to greedy on a sampled batch (XPU, CPU, and ROCm without rejection sampling), as DFLASH already does. It uses the same branch as Add opt-in error for greedy fallback in speculative verification #42828.speculative_hook.py: reject--speculative-use-rejection-samplingon XPU for every algorithm, next to the block-verification device check.xpu.mdx;speculative_decoding.mdx.test_block_verification_args.py;test_set_embed_device_neutral.pyscansmodels/for directtorch.cuda.*()calls inset_embed*;test_spec_registry.py: addspeculative_use_rejection_samplingto the hand-built args stub, which the hook now reads.Accuracy Tests
Arc Pro B60,
mainvs this PR:mainglm-moe-dsa-tiny-randomintel_xpu, 5/1/6, greedy +temperature=0.7requestsUnit tests:
test/registered/unit/spec+unit/server_argsgive 753 passed with this PR vs 752 onmain(+1 test, +2 subtests), with identical pre-existing failures.DSA models (V3.2 / GLM-5) still hit a separate CUDA-only assert in DSA verify (
pad.py). That is a follow-up.Speed Tests and Profiling
Not needed: the changes run at load time, at argument parsing, or as one bool check per verify.
Checklist
CI States
Latest PR Test (Base): ✅ Run #43804630832
Latest PR Test (Extra): ❌ Run #43804630492
Latest PR Test (AMD ROCm 10): ❌ Run #43804630800
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: