Repository navigation
[Feat] [NPU] Qwen3.8-Flash-Next: Support graph replay, MTP, and Triton kernels - #37570
Conversation
- Route NPU QSA through Torch operators and shared indexer orchestration - Preserve fixed tensor shapes for graph capture and replay - Carry the live request-to-token mapping in graph metadata - Adapt NPU KV cache layouts and zero masked values before attention - Add coverage for dispatch, padding, slot mapping, and graph replay
Decompose depthwise convolution into slice, multiply and sum operations for NPU decode and target verify, avoiding the uncapturable aclop Conv2D. Adapt the convolution implementation from the qwen3d8fn branch of fishAndShrimp/sglang.
24adaec to
64b0532
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@fishAndShrimp Could you provide an A2 image that includes this PR? The Huawei team is very slow to respond; it’s likely they won’t end up merging your PR. |
@chenminghua8 Thanks for your interest! I don’t currently have a prebuilt A2 image to share. My validation so far has primarily been on A3. If your A2 SGLang image includes the sglang and sgl-kernel-npu repos, you could locate them and apply the changes from this PR and sgl-project/sgl-kernel-npu#807. Please make sure the updated code is actually being loaded. The new kernels in sgl-project/sgl-kernel-npu#807 are Python/Triton code, so these additions alone do not require rebuilding the C++ extensions. However, I haven’t tested this setup on A2, so compatibility still needs to be checked. |
@fishAndShrimp @chenminghua8 We're working on A2 validation in #43102 and are building on this PR and sgl-kernel-npu#807 for QSA integration. We don't have a fully validated QSA-enabled A2 image to share yet, but would be happy to coordinate on A2-specific fixes and reproducible setup instructions. |
Route NPU QSA through the model-specific kernel package and shared indexer. Preserve live request mappings during graph replay and retain FIA updates for the explicit dense fallback. Adapted from sgl-project#37570 at 68dff35. Requires the QSA modules from sgl-project/sgl-kernel-npu#807; full-model NPU validation is pending. Co-authored-by: Estrella-xx <820167017@qq.com> Co-authored-by: fishAndShrimp <ruiyuanqian@foxmail.com> Signed-off-by: IcyDesert <icy_desert@outlook.com> Co-authored-by: xuchen-w <adrian.xc.wei@gmail.com>
This comment has been minimized.
This comment has been minimized.
Route NPU QSA through the model-specific kernel package and shared indexer. Preserve live request mappings during graph replay and retain FIA updates for the explicit dense fallback. Adapted from sgl-project#37570 at 68dff35. Requires the QSA modules from sgl-project/sgl-kernel-npu#807; full-model NPU validation is pending. Co-authored-by: Estrella-xx <820167017@qq.com> Co-authored-by: fishAndShrimp <ruiyuanqian@foxmail.com> Signed-off-by: IcyDesert <icy_desert@outlook.com> Co-authored-by: xuchen-w <adrian.xc.wei@gmail.com>
Merge affb3eb. Git merged qwen4_exp.py without textual conflicts; no manual source resolutions were applied. Keep upstream PLE state-slot snapshots and CUDA variable-length prefill, alongside the NPU short-convolution dispatch. Keep the upstream device-neutral graph backend migration. Any test fixture updates are recorded separately.
Populate query_start_loc for prefill and leave it unset for decode and verify, matching _prepare_ple_batch. The merged upstream struct requires this field. Verified the missing-field failure before this change. After the fix, all 25 selected CPU PLE cases pass. Production code is unchanged.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
/tag-and-rerun-ci |
Merge main at cabb43a. Resolve the only conflict in qsa_indexer.py by dropping the removed get_is_capture_mode import and retaining is_npu and the _is_npu initialization. Keep upstream's RoPE cache cleanup and preserve the existing NPU dispatch. The PR's additions and deletions relative to main are unchanged. Validation: all applicable pre-commit hooks passed for the 13 PR files; 71 CPU QSA/CP regression tests passed. No hardware inference was run for this merge.
Keep upstream get_forward and PR is_npu imports in qwen4_exp. Adopt the upstream post5 kernel release for NPU CI. Keep candidate performance changes in separate worktrees.
|
/rerun-failed-ci |
Motivation
Enable Qwen3.8-Flash-Next inference on NPU with Qwen Sparse Attention (QSA), graph replay, and NEXTN/MTP speculative decoding, building on the mainline model support from #37500.
Dependency: sgl-project/sgl-kernel-npu#807 provides the production kernels. Before merging, an installable kernel package containing these modules and the corresponding SGLang dependency must be verified together; local source-tree validation does not establish release-package availability.
Modifications
sgl_kernel_npu.qwen3_8_flash_next: PLE short convolution, packed/paged MQA scoring, Top-K, block expansion, sparse attention, and hyperconnection norm/mix/combine. Wrappers own their model input contracts and internal Triton/native paths; unsupported NPU metadata is not silently redirected to a general reference.Accuracy Tests
Launch script: TP8 BF16 with NEXTN/MTP
Replace the placeholder paths with your runtime installation, source checkouts, and model directory. Use this SGLang branch with the matching kernels from sgl-kernel-npu #807.
GPQA Diamond
GPQA evaluation script
The latest evaluation completed on October 12, 2026, after updating the companion kernel repository to
b01807d(QSA Top-K and sparse attention dispatch adjustments). The kernel checkout was confirmed on the evaluation machine before testing. A post-run check of the same machine showed SGLang7841e9bf1de41721c0ade41caad6738ac913909dwith a clean working tree; the exported results do not independently record the runtime source Git SHAs.64b0532544b9f784a5731b1c894d25347acc950ba02fa50abf08e3ba27d0051ac47b340b37a701a8e2f90ac5f4616cd7d521ef7bc19124dd86ed3b95e58dd411b006b55c62feb37f10c782a64f801ad57841e9bf1de41721c0ade41caad6738ac913909d7841e9bf1de41721c0ade41caad6738ac913909d(post-run checkout check)270b9f8a61d906c0c958c7ebdb8b62ee6884d3bf270b9f8a61d906c0c958c7ebdb8b62ee6884d3bfEvaluation settings: EvalScope 1.11.1,
temperature=0.9, generationseed=3407,n=1, thinking enabled,max_tokens=250000, streaming, and client concurrency 32. Dataset seed is 42, with question and answer-choice shuffling enabled. The latest run took approximately 8 h 23 min and did not load a prior prediction cache.All 198 questions completed, with 183 correct and zero final request errors. One request timed out and was retried before completion. The earlier October 8 run used
max_tokens=260000and counted one context-length rejection as incorrect; that rejection did not recur with the latest 250000-token output budget.The latest score is above the model-card reference in this individual run, but this is not a matched evaluation proving superiority or exact accuracy parity. The output-token limit also differs from the earlier runs, so the score change cannot be attributed solely to the kernel dispatch update. Historical revision attribution follows the previously documented run records; the early QSA Triton revision was reconstructed from a pre-run source snapshot. These results do not replace the separately recorded operator-level numerical limitations.
Local regression and smoke checks
These checks do not establish all TP configurations, full-context coverage, multimodal accuracy, or long-run stability. Remote CI and reviewer approval remain separate merge requirements.
Speed Tests and Profiling
Performance benchmark script
October 12 serving benchmark
TP8 BF16 on NPU, with graph replay and NEXTN/MTP: 3 speculative steps, top-k 1, and 4 draft tokens. Server configuration uses a 4096-token prefill chunk, at most 8 running requests, static memory fraction 0.85, and disabled radix caching.
Workload:
sglang-oai-chat, 80 requests, concurrency 8, unlimited offered request rate, one warmup request, and request-bodytemperature=0.01. Therandomdataset mode uses ShareGPT text repeated/truncated to randomized input lengths; configured input/output lengths are 1024/1024 with range ratio 0.5 and benchmark seed 10130. This is not the original ShareGPT length distribution. EOS is ignored to reach the sampled output budgets. Thinking is not explicitly overridden in the benchmark request.Each run completed 80/80 requests, with 62,912 actual input tokens and 62,023 generated output tokens. All four collected runs are shown in execution order:
The screenshot below shows run 4, not an average. Run 1 is substantially slower despite matching recorded benchmark/server settings; its cause has not been established, so it is retained rather than silently discarded. These are absolute serving measurements, not a controlled speedup over the earlier PR benchmark. Benchmark exports include installed package version metadata, not a verified runtime source Git SHA.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #43872117790
Latest PR Test (Extra): ❌ Run #43872117422
Latest PR Test (AMD ROCm 10): ✅ Run #43872117974
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: