Repository navigation
[NPU] Adapt DFlash2 speculative decoding to Ascend NPUs - #35629
Merged
Merged
Conversation
…Qwen3.8-27B page Signed-off-by: syd520zy <529477025@qq.com>
syd520zy
requested review from
JustinTong0323,
sogalin,
wisclmy0611 and
zijiexia
as code owners
August 20, 2026 03:58
…8-27B page Signed-off-by: syd520zy <529477025@qq.com>
chain_speculative_sampling_torch mirrors the Triton kernel buffer-for-buffer (acceptance rule coin*q < p, residual max(p - q, 0) final sampling with the NaN draft guard, predicts/accept_index/ accept_token_num layouts and dtypes) and runs on devices without Triton. _accept_sampling_core dispatches by input device (CUDA -> Triton, otherwise -> torch reference), following the existing SoftmaxTemp/ AcceptSampling dispatch convention. This unblocks the DFlash2 selector verify path (which never falls back to greedy) on Ascend NPUs. triton_compat provides a tolerant triton import shared by the DFlash kernel modules so NPU/CPU environments without triton can still import them; launching a Triton kernel there raises a clear error. CPU unit tests cover analytic accept/reject/bonus cases, mixed-batch independence, the statistical losslessness of the first committed token, and a CUDA parity check against the Triton kernel. Signed-off-by: syd520zy <529477025@qq.com>
Signed-off-by: syd520zy <529477025@qq.com>
Signed-off-by: syd520zy <529477025@qq.com>
Signed-off-by: syd520zy <529477025@qq.com>
Signed-off-by: syd520zy <529477025@qq.com>
Signed-off-by: syd520zy <529477025@qq.com>
iforgetmyname
approved these changes
Sep 4, 2026
Collaborator
|
/tag-run-ci-label |
5 tasks done
Jiminator
added a commit
that referenced
this pull request
Sep 11, 2026
The page described three eras at once: cells "validated on 1cf2b8c to a weaker standard", a docker tag that no longer exists on it, and the v0.5.19 sweep. Everything now cites the release the Install section pins. - The deployment Note and the SM121 paragraph said boot-and-serve coverage only on 1cf2b8c, for 48 configurations. Both now state what was actually run: 202 cells on v0.5.19, each served and scored on the full 1319-question GSM8K, 93.18-95.15%; GB10 covers 80 of those across two boxes. - Ten config comments citing "measured at commit 1cf2b8c" now cite v0.5.19. - The install accordion and the image map use the plain `v0.5.19` tag. It is the same image as `v0.5.19-cu130` (identical digest) and is the clearer thing to publish; the comment records which name the sweep ran under. The #35629 Ascend comparison keeps its own accuracy protocol (128 examples, concurrency 1-16) - that describes someone else's experiment, not this sweep, and is now attributed so the two are not read as one methodology. No launch command changes: all 42 RTX 5090 cells still render exactly the command they were measured on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
DFlash2 (#35371) landed on main as the successor to DFlash. It uses a block-diffusion drafter with local grouped convolutions and a candidate selector.
First-generation DFlash drafts already run on Ascend NPUs (#23122, #31739), but DFlash2 checkpoints could not serve on NPU because the target verification path assumed CUDA behavior.
This PR adapts the DFlash2 draft and verification paths for Ascend NPUs while preserving the existing CUDA behavior.
The validated Ascend environment includes
triton-ascend; this PR assumes Triton is available in the runtime environment.Modifications
Fix target verification metadata handling on NPU.
skip_attn_backend_init=Truebehavior becauseprepare_for_verifyhas already planned the attention metadata.None, allowing the target forward path to initialize the required attention metadata.raw_num_token.Align DFlash2 selector verification with the existing NPU behavior of EAGLE and first-generation DFlash.
[batch_size, gamma, vocab_size]draft-probability buffer and performs vocabulary-wide operations whose cost does not shrink with the verification window.triton-ascendenvironment, but it added about 6 ms of TPOT on the tested 910C setup and is unnecessary for lossless greedy requests.Keep draft proposal and target verification consistent on NPU.
_selector_sampleis staged only when selector sampling is enabled, and acceptance dispatches from that staged state rather than rechecking the hardware.Handle non-greedy requests on NPU explicitly.
temperature=0andtop_k=1.Keep DSpark NPU sampling support outside this PR.
Update the Qwen3.8-27B cookbook with the DFlash2 NPU behavior, validated configuration, greedy-only losslessness boundary, and benchmark setup.
Why the remaining NPU guards are necessary
The runtime changes contain two direct NPU decisions, each representing a separate backend constraint:
[batch_size, gamma, vocab_size]probability buffer and performs full-vocabulary work. All downstream selector paths use the resulting capability state.prepare_for_verify()pre-plans graph or attention metadata, while the NPU path must defer planning until DP/EP padding finalizes runtime shapes. Passingskip_attn_backend_init=Trueon NPU would incorrectly mark uninitialized metadata as ready and can makeNPUGraphRunneraccess uninitialized state such asraw_num_token.The existing
is_dflash_sampling_verify_available()helper cannot replace the selector capability decision: it describes a different target-only verification implementation and using it here would change existing behavior on other backends such as HIP.Runtime Validation
temperature=0,top_k=1).--disable-cuda-graph) execution paths.batch_invariant_opsextension required by--enable-deterministic-inference.Reviewer question: non-greedy requests on Ascend
DFlash2 currently cannot preserve the requested non-greedy sampling distribution on Ascend because selector verification falls back to argmax.
The current implementation keeps the request valid, emits a warning once, and forces both the draft proposal and target verification to greedy so their behavior remains consistent.
Another option is to reject non-greedy requests explicitly instead of allowing this fallback.
Which behavior would be preferred for this PR?
Accuracy Tests
The accuracy setup follows #23122 and #16818, except for the hardware and models.
--tp-size 2Qwen/Qwen3.8-27Bwith BF16 weightsz-lab/Qwen3.8-27B-DFlash2--speculative-num-draft-tokens 8--attention-backend ascend--mamba-ssm-dtype bfloat16 --mamba-scheduler-strategy extra_buffertemperature=0.0,top_p=1.0, andtop_k=12048Baseline accuracy
DFlash2 accuracy
DFlash2 acceptance length
The configured block size is 8. The following values are the windowed mean acceptance lengths reported by scheduler decode logs.
The log-reported mean acceptance length was approximately 5.8, with an acceptance rate of approximately 0.68.
End-to-end validation on Ascend 910C confirmed that:
Focused unit-test coverage includes:
Local static validation completed:
python -m compileallgit diff --check upstream/mainFocused Python unit tests were executed in the Ascend validation environment and passed.
Speed Tests and Profiling
The throughput benchmark used the same hardware, model configuration, workload, and RadixCache setting as the accuracy test.
Baseline output throughput
DFlash2 output throughput
Speedup
The end-to-end speedup decreases from 3.05x at concurrency 1 to 1.75x at concurrency 16 as target-model batching and speculative-decoding overhead become more prominent at higher concurrency.
Local profiling showed:
Further performance work, including acceptance-length tuning and reducing verify-preparation host overhead, is outside the scope of this PR.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #33854879909
Latest PR Test (Extra): ❌ Run #33854879221
Latest PR Test (AMD ROCm 7.2): ❌ Run #33854879668