Skip to content

[NPU] Adapt DFlash2 speculative decoding to Ascend NPUs - #35629

Merged
iforgetmyname merged 80 commits into
sgl-project:mainfrom
syd520zy:npu-dflash2-adapt
Sep 7, 2026
Merged

iforgetmyname merged 80 commits into
sgl-project:mainfrom
syd520zy:npu-dflash2-adapt

Conversation

@syd520zy

@syd520zy syd520zy commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

DFlash2 (#35371) landed on main as the successor to DFlash. It uses a block-diffusion drafter with local grouped convolutions and a candidate selector.

First-generation DFlash drafts already run on Ascend NPUs (#23122, #31739), but DFlash2 checkpoints could not serve on NPU because the target verification path assumed CUDA behavior.

This PR adapts the DFlash2 draft and verification paths for Ascend NPUs while preserving the existing CUDA behavior.

The validated Ascend environment includes triton-ascend; this PR assumes Triton is available in the runtime environment.

Modifications

  • Fix target verification metadata handling on NPU.

    • Non-NPU devices preserve the existing skip_attn_backend_init=True behavior because prepare_for_verify has already planned the attention metadata.
    • NPU passes None, allowing the target forward path to initialize the required attention metadata.
    • This prevents the NPU graph runner from skipping initialization and accessing an uninitialized raw_num_token.
  • Align DFlash2 selector verification with the existing NPU behavior of EAGLE and first-generation DFlash.

    • On CUDA, selector sampling verification remains unchanged.
    • On NPU, the full-vocabulary selector sampling verification is skipped and target verification uses argmax.
    • The full-vocabulary path scatters a dense [batch_size, gamma, vocab_size] draft-probability buffer and performs vocabulary-wide operations whose cost does not shrink with the verification window.
    • This is an NPU-specific performance policy, not a limitation of the selector algorithm itself: the probability-based path can execute in the validated triton-ascend environment, but it added about 6 ms of TPOT on the tested 910C setup and is unnecessary for lossless greedy requests.
  • Keep draft proposal and target verification consistent on NPU.

    • Selector sampling support is resolved once as a worker-level capability; graph-captured and eager paths consume that semantic state instead of repeatedly checking the device.
    • The capability is disabled on NPU, forcing both selector draft paths to greedy proposal.
    • _selector_sample is staged only when selector sampling is enabled, and acceptance dispatches from that staged state rather than rechecking the hardware.
    • This avoids stochastic draft proposals being verified by a greedy target, which would unnecessarily reduce the acceptance length.
    • CUDA and other non-NPU devices continue to use the original request sampling parameters.
  • Handle non-greedy requests on NPU explicitly.

    • The request is currently accepted.
    • Both draft proposal and target verification fall back to greedy.
    • A warning is emitted once to state that the requested sampling distribution is not preserved.
    • Lossless behavior on NPU is currently guaranteed only for greedy requests such as temperature=0 and top_k=1.
  • Keep DSpark NPU sampling support outside this PR.

    • This PR does not add a torch chain speculative-sampling implementation or a DSpark NPU sampling fallback.
    • The scope is limited to DFlash2 serving and greedy verification on Ascend NPUs.
  • Update the Qwen3.8-27B cookbook with the DFlash2 NPU behavior, validated configuration, greedy-only losslessness boundary, and benchmark setup.

Why the remaining NPU guards are necessary

The runtime changes contain two direct NPU decisions, each representing a separate backend constraint:

  1. Selector sampling policy is decided once when the worker is initialized. Ascend currently uses greedy proposal and verification because selector rejection sampling materializes a dense [batch_size, gamma, vocab_size] probability buffer and performs full-vocabulary work. All downstream selector paths use the resulting capability state.
  2. Target verification metadata is handled differently on Ascend. Non-NPU prepare_for_verify() pre-plans graph or attention metadata, while the NPU path must defer planning until DP/EP padding finalizes runtime shapes. Passing skip_attn_backend_init=True on NPU would incorrectly mark uninitialized metadata as ready and can make NPUGraphRunner access uninitialized state such as raw_num_token.

The existing is_dflash_sampling_verify_available() helper cannot replace the selector capability decision: it describes a different target-only verification implementation and using it here would change existing behavior on other backends such as HIP.

Runtime Validation

  • Verified DFlash2 on Ascend NPU with greedy decoding (temperature=0, top_k=1).
  • Three repeated requests with the same prompt produced identical output hashes.
  • Verified both NPU graph-enabled and eager (--disable-cuda-graph) execution paths.
  • Non-greedy requests complete successfully and emit the expected one-time warning before falling back to greedy DFlash2 proposal/verification.
  • Full deterministic non-greedy validation was not performed because the test image does not include the NPU batch_invariant_ops extension required by --enable-deterministic-inference.

Reviewer question: non-greedy requests on Ascend

DFlash2 currently cannot preserve the requested non-greedy sampling distribution on Ascend because selector verification falls back to argmax.

The current implementation keeps the request valid, emits a warning once, and forces both the draft proposal and target verification to greedy so their behavior remains consistent.

Another option is to reject non-greedy requests explicitly instead of allowing this fallback.

Which behavior would be preferred for this PR?

  1. Accept the request, warn once, and run both draft proposal and verification greedily.
  2. Reject non-greedy requests on Ascend with an explicit error.

Accuracy Tests

The accuracy setup follows #23122 and #16818, except for the hardware and models.

  • Hardware: Ascend 910C
  • Tensor parallelism: --tp-size 2
  • Target model: Qwen/Qwen3.8-27B with BF16 weights
  • Draft model: z-lab/Qwen3.8-27B-DFlash2
  • Draft tokens: --speculative-num-draft-tokens 8
  • Attention backend: --attention-backend ascend
  • Mamba configuration: --mamba-ssm-dtype bfloat16 --mamba-scheduler-strategy extra_buffer
  • Dataset: GSM8K, zero-shot
  • Sampling: greedy, with temperature=0.0, top_p=1.0, and top_k=1
  • Maximum new tokens: 2048
  • Samples: 128 per concurrency level
  • RadixCache: disabled for both baseline and DFlash2 to keep the comparison independent of cache warm-up and prefix reuse

Baseline accuracy

Concurrency 1 2 4 8 16
Accuracy 0.820 0.836 0.836 0.828 0.820

DFlash2 accuracy

Concurrency 1 2 4 8 16
Accuracy 0.820 0.836 0.836 0.828 0.820

DFlash2 acceptance length

The configured block size is 8. The following values are the windowed mean acceptance lengths reported by scheduler decode logs.

Concurrency 1 2 4 8 16
Mean acceptance length 5.8 5.8 5.8 5.8 5.8

The log-reported mean acceptance length was approximately 5.8, with an acceptance rate of approximately 0.68.

End-to-end validation on Ascend 910C confirmed that:

  • The target and DFlash2 draft models load successfully.
  • The selector draft initializes successfully.
  • Both NPU graph captures complete successfully.
  • Greedy requests complete successfully.
  • Baseline and DFlash2 produced identical aggregate GSM8K accuracy at every tested concurrency level.

Focused unit-test coverage includes:

  • Warning once when selector sampling is disabled and a non-greedy request falls back.
  • Propagating the worker-level selector sampling capability into the graph sampler.
  • Forcing greedy selector draft parameters when sampling is disabled.
  • Preserving selector sampling parameters when the capability is enabled.
  • Verifying that acceptance uses staged selector state and otherwise takes the greedy fallback.
  • Keeping the generic/CUDA Mamba verify-hook ordering test isolated from NPU-specific DSV4 bundle setup.

Local static validation completed:

  • python -m compileall
  • git diff --check upstream/main

Focused Python unit tests were executed in the Ascend validation environment and passed.

Speed Tests and Profiling

The throughput benchmark used the same hardware, model configuration, workload, and RadixCache setting as the accuracy test.

Baseline output throughput

Concurrency 1 2 4 8 16
Output tok/s 33.182 60.925 109.544 176.193 247.068

DFlash2 output throughput

Concurrency 1 2 4 8 16
Output tok/s 101.249 162.244 259.899 329.448 431.172

Speedup

Concurrency 1 2 4 8 16
DFlash2 / baseline 3.05x 2.66x 2.37x 1.87x 1.75x

The end-to-end speedup decreases from 3.05x at concurrency 1 to 1.75x at concurrency 16 as target-model batching and speculative-decoding overhead become more prominent at higher concurrency.

Local profiling showed:

  • Selector top-k costs approximately 0.04 ms per decode step.
  • DFlash2 and NEXTN have matching kernel-level draft behavior.
  • The target verification step is primarily bounded by the TP2 per-layer all-reduce floor, with approximately 43 collectives per step.
  • The original full-vocabulary selector sampling verification added approximately 6 ms of TPOT on the tested setup: about 51 ms with selector sampling verification versus 45 ms with greedy verification.

Further performance work, including acceptance-length tuning and reducing verify-preparation host overhead, is outside the scope of this PR.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #33854879909
Latest PR Test (Extra): ❌ Run #33854879221
Latest PR Test (AMD ROCm 7.2): ❌ Run #33854879668

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 20, 2026
…8-27B page

Signed-off-by: syd520zy <529477025@qq.com>
chain_speculative_sampling_torch mirrors the Triton kernel
buffer-for-buffer (acceptance rule coin*q < p, residual max(p - q, 0)
final sampling with the NaN draft guard, predicts/accept_index/
accept_token_num layouts and dtypes) and runs on devices without Triton.
_accept_sampling_core dispatches by input device (CUDA -> Triton,
otherwise -> torch reference), following the existing SoftmaxTemp/
AcceptSampling dispatch convention. This unblocks the DFlash2 selector
verify path (which never falls back to greedy) on Ascend NPUs.

triton_compat provides a tolerant triton import shared by the DFlash
kernel modules so NPU/CPU environments without triton can still import
them; launching a Triton kernel there raises a clear error.

CPU unit tests cover analytic accept/reject/bonus cases, mixed-batch
independence, the statistical losslessness of the first committed token,
and a CUDA parity check against the Triton kernel.

Signed-off-by: syd520zy <529477025@qq.com>
@iforgetmyname

Copy link
Copy Markdown
Collaborator

/tag-run-ci-label

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 4, 2026
@iforgetmyname
iforgetmyname merged commit 30d0eb2 into sgl-project:main Sep 7, 2026
251 of 282 checks passed
Jiminator added a commit that referenced this pull request Sep 11, 2026
The page described three eras at once: cells "validated on 1cf2b8c to a
weaker standard", a docker tag that no longer exists on it, and the v0.5.19
sweep. Everything now cites the release the Install section pins.

- The deployment Note and the SM121 paragraph said boot-and-serve coverage
  only on 1cf2b8c, for 48 configurations. Both now state what was actually
  run: 202 cells on v0.5.19, each served and scored on the full
  1319-question GSM8K, 93.18-95.15%; GB10 covers 80 of those across two
  boxes.
- Ten config comments citing "measured at commit 1cf2b8c" now cite v0.5.19.
- The install accordion and the image map use the plain `v0.5.19` tag. It is
  the same image as `v0.5.19-cu130` (identical digest) and is the clearer
  thing to publish; the comment records which name the sweep ran under.

The #35629 Ascend comparison keeps its own accuracy protocol (128 examples,
concurrency 1-16) - that describes someone else's experiment, not this
sweep, and is now attributed so the two are not read as one methodology.

No launch command changes: all 42 RTX 5090 cells still render exactly the
command they were measured on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail documentation Improvements or additions to documentation jit-kernel npu run-ci CI: run the baseline test suite on this PR speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants