Skip to content

[Feat] [NPU] Qwen3.8-Flash-Next: Support graph replay, MTP, and Triton kernels - #37570

Merged
sglang-npu-bot merged 54 commits into
sgl-project:mainfrom
fishAndShrimp:pr-36601
Nov 17, 2026
Merged

sglang-npu-bot merged 54 commits into
sgl-project:mainfrom
fishAndShrimp:pr-36601

Conversation

@fishAndShrimp

@fishAndShrimp fishAndShrimp commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Enable Qwen3.8-Flash-Next inference on NPU with Qwen Sparse Attention (QSA), graph replay, and NEXTN/MTP speculative decoding, building on the mainline model support from #37500.

Dependency: sgl-project/sgl-kernel-npu#807 provides the production kernels. Before merging, an installable kernel package containing these modules and the corresponding SGLang dependency must be verified together; local source-tree validation does not establish release-package availability.

Modifications

  • Reuse shared QSA indexer orchestration on NPU, preserving per-query bounds, padding, MTP index reuse, and logical-to-physical KV mapping.
  • Integrate six operator groups from sgl_kernel_npu.qwen3_8_flash_next: PLE short convolution, packed/paged MQA scoring, Top-K, block expansion, sparse attention, and hyperconnection norm/mix/combine. Wrappers own their model input contracts and internal Triton/native paths; unsupported NPU metadata is not silently redirected to a general reference.
  • Adapt rank-4 NPU KV pools through zero-copy views, so incompatible layouts fail rather than silently copying cache storage.
  • Keep NPU dispatch beside GPU entrypoints and preserve other platforms' behavior. Support sigmoid-gated normalization and retain the NPU hyperconnection compilation workaround for cached restarts.
  • Use graph-capturable PLE convolution for decode/target verify, preserving state updates, tracking, MTP, and ordinary prefill. Guard the newly merged CUDA verify-fusion path on NPU.
  • Maintain live graph metadata and draft-extend bounds; bypass CPU attention-input updates for Qwen QSA paths that consume device metadata. Avoid the CPU-fallback assertion in NPU QSA write planning.
  • Add and update regression tests for dispatch, indexing, cache layout, normalization, PLE, operator chains, and graph replay. Adapt tests to upstream runtime-context and graph-worker changes, including direct execution through registered CI entrypoints.

Accuracy Tests

Launch script: TP8 BF16 with NEXTN/MTP

Replace the placeholder paths with your runtime installation, source checkouts, and model directory. Use this SGLang branch with the matching kernels from sgl-kernel-npu #807.

#!/usr/bin/env bash

set -o pipefail

# Host-wide tuning; requires appropriate privileges.
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0

export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy http_proxy all_proxy
unset HTTPS_PROXY HTTP_PROXY ALL_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /path/to/toolkit/set_env.sh
source /path/to/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export ASCEND_USE_FIA=1

export PYTHONPATH="/path/to/sgl-kernel-npu/python/sgl_kernel_npu:/path/to/sglang/python${PYTHONPATH:+:${PYTHONPATH}}"

MODEL_PATH="/path/to/Qwen3.8-Flash-Next"

LOG_DIR="logs_serve"
mkdir -p -- "${LOG_DIR}" || exit 1
RUN_TIME="$(date +%Y%m%d_%H%M%S)"

python3 -m sglang.launch_server \
    --model-path "${MODEL_PATH}" \
    --host 127.0.0.1 \
    --port 30000 \
    --tp-size 8 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 4096 \
    --max-prefill-tokens 52000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 8 \
    --mem-fraction-static 0.85 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --mm-enable-dp-encoder \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    2>&1 | tee "${LOG_DIR}/${RUN_TIME}.log"

GPQA Diamond

GPQA evaluation script
# Set --model to the model ID served by your server (model path or served name).
# Set local_path in --dataset-args to your GPQA Diamond directory.
# Set --api-url and --api-key to match your server (default: localhost:30000, no auth).
RUN_TIME="$(date +%Y%m%d_%H%M%S)"

evalscope eval \
  --model "/path/to/Qwen3.8-Flash-Next/" \
  --api-url http://127.0.0.1:30000/v1 \
  --api-key EMPTY \
  --eval-type openai_api \
  --generation-config '{
        "temperature": 0.9,
        "max_tokens": 250000,
        "seed": 3407,
        "n": 1,
        "timeout": 3600,
        "stream": true,
        "extra_body": {"chat_template_kwargs": {"enable_thinking": true}}}' \
  --datasets gpqa_diamond \
  --dataset-hub Local \
  --dataset-args '{"gpqa_diamond": {"local_path": "/path/to/gpqa_diamond", "shuffle": true, "shuffle_choices": true}}' \
  --eval-batch-size 32 \
  --ignore-errors \
  2>&1 | tee "EVAL_${RUN_TIME}.log"

The latest evaluation completed on October 12, 2026, after updating the companion kernel repository to b01807d (QSA Top-K and sparse attention dispatch adjustments). The kernel checkout was confirmed on the evaluation machine before testing. A post-run check of the same machine showed SGLang 7841e9bf1de41721c0ade41caad6738ac913909d with a clean working tree; the exported results do not independently record the runtime source Git SHAs.

Implementation SGLang revision GPQA Diamond
Official model-card reference External reference 91.7%
Earlier Torch baseline 64b0532544b9f784a5731b1c894d25347acc950b 181/198, 91.41%
Early QSA Triton stage a02fa50abf08e3ba27d0051ac47b340b37a701a8 187/198, 94.44%
September 17 integration e2f90ac5f4616cd7d521ef7bc19124dd86ed3b95 180/198, 90.91%
September 22 integration e58dd411b006b55c62feb37f10c782a64f801ad5 181/198, 91.41%
October 8 post-merge integration 7841e9bf1de41721c0ade41caad6738ac913909d 181/198, 91.41%
October 12 kernel-dispatch update 7841e9bf1de41721c0ade41caad6738ac913909d (post-run checkout check) 183/198, 92.42%
November 9 integration (EvalScope 1.12.0, seed 3407) 270b9f8a61d906c0c958c7ebdb8b62ee6884d3bf 178/198, 89.90%
November 10 repeat (EvalScope 1.12.0, seed 7893; seed change only) 270b9f8a61d906c0c958c7ebdb8b62ee6884d3bf 184/198, 92.93%

Evaluation settings: EvalScope 1.11.1, temperature=0.9, generation seed=3407, n=1, thinking enabled, max_tokens=250000, streaming, and client concurrency 32. Dataset seed is 42, with question and answer-choice shuffling enabled. The latest run took approximately 8 h 23 min and did not load a prior prediction cache.

All 198 questions completed, with 183 correct and zero final request errors. One request timed out and was retried before completion. The earlier October 8 run used max_tokens=260000 and counted one context-length rejection as incorrect; that rejection did not recur with the latest 250000-token output budget.

The latest score is above the model-card reference in this individual run, but this is not a matched evaluation proving superiority or exact accuracy parity. The output-token limit also differs from the earlier runs, so the score change cannot be attributed solely to the kernel dispatch update. Historical revision attribution follows the previously documented run records; the early QSA Triton revision was reconstructed from a pre-run source snapshot. These results do not replace the separately recorded operator-level numerical limitations.

fig_gpqa

Local regression and smoke checks

  • Focused post-merge SGLang regression: 451 passed, 7 skipped, 0 failed; 23 CUDA-specific cases were excluded. This is not the full repository test suite.
  • After adding executable test entrypoints, the five affected files passed direct CI-style execution: 194 passed, 2 skipped, 0 failed. These overlap the regression coverage above and are not an additional independent test count.
  • TP8 BF16 service startup, target-verify/draft-decode/draft-extend graph capture, short requests, and a batched long-prefill request passed with NEXTN/MTP. The long prompt exercised chunked prefill followed by decode.
  • PR-changed files in both repositories passed all applicable local pre-commit hooks. Final formatting and test-entrypoint changes do not alter the production computation.

These checks do not establish all TP configurations, full-context coverage, multimodal accuracy, or long-run stability. Remote CI and reviewer approval remain separate merge requirements.

Speed Tests and Profiling

Performance benchmark script
#!/usr/bin/env bash
set -euo pipefail

# Edit DATASET_PATH and PYTHONPATH before running.
# Set --host and --port to match your server (default: 127.0.0.1:30000).
# Usage: bash bench_sharegpt_public.sh
# ShareGPT text sampled to 512–1024 tokens.
DATASET_PATH='/path/to/ShareGPT_V3_unfiltered_cleaned_split.json'

export PYTHONPATH="/path/to/sgl-kernel-npu/python/sgl_kernel_npu:/path/to/sglang/python${PYTHONPATH:+:${PYTHONPATH}}"

RUN_TIME="$(date +%Y%m%d_%H%M%S)"

python3 -m sglang.benchmark.serving \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 30000 \
    --dataset-name random \
    --dataset-path "$DATASET_PATH" \
    --max-concurrency 8 \
    --random-input-len 1024 \
    --random-output-len 1024 \
    --num-prompts 80 \
    --random-range-ratio 0.5 \
    --seed 10130 \
    --extra-request-body '{"temperature": 0.01}' \
    --output-file RESULT_bench_sharegpt.jsonl \
    2>&1 | tee "BENCH_${RUN_TIME}.log"

October 12 serving benchmark

TP8 BF16 on NPU, with graph replay and NEXTN/MTP: 3 speculative steps, top-k 1, and 4 draft tokens. Server configuration uses a 4096-token prefill chunk, at most 8 running requests, static memory fraction 0.85, and disabled radix caching.

Workload: sglang-oai-chat, 80 requests, concurrency 8, unlimited offered request rate, one warmup request, and request-body temperature=0.01. The random dataset mode uses ShareGPT text repeated/truncated to randomized input lengths; configured input/output lengths are 1024/1024 with range ratio 0.5 and benchmark seed 10130. This is not the original ShareGPT length distribution. EOS is ignored to reach the sampled output budgets. Thinking is not explicitly overridden in the benchmark request.

Each run completed 80/80 requests, with 62,912 actual input tokens and 62,023 generated output tokens. All four collected runs are shown in execution order:

Run Duration (s) Output throughput (tokens/s) Total token throughput (tokens/s) Mean TTFT (ms) Mean TPOT (ms) Mean acceptance length
1 687.01 90.28 181.85 3853.91 82.58 2.44
2 563.60 110.05 221.67 1557.55 68.91 2.44
3 577.74 107.35 216.25 1684.31 69.84 2.44
4 559.15 110.92 223.44 1493.87 68.48 2.45

The screenshot below shows run 4, not an average. Run 1 is substantially slower despite matching recorded benchmark/server settings; its cause has not been established, so it is retained rather than silently discarded. These are absolute serving measurements, not a controlled speedup over the earlier PR benchmark. Benchmark exports include installed package version metadata, not a verified runtime source Git SHA.

fig_prof

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #43872117790
Latest PR Test (Extra): ❌ Run #43872117422
Latest PR Test (AMD ROCm 10): ✅ Run #43872117974

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

fishAndShrimp and others added 11 commits September 10, 2026 03:48
- Route NPU QSA through Torch operators and shared indexer orchestration
- Preserve fixed tensor shapes for graph capture and replay
- Carry the live request-to-token mapping in graph metadata
- Adapt NPU KV cache layouts and zero masked values before attention
- Add coverage for dispatch, padding, slot mapping, and graph replay
Decompose depthwise convolution into slice, multiply and sum operations
for NPU decode and target verify, avoiding the uncapturable aclop Conv2D.

Adapt the convolution implementation from the qwen3d8fn branch of
fishAndShrimp/sglang.
@fishAndShrimp fishAndShrimp changed the title [NPU] Support Qwen3.8-Flash-Next graph mode and MTP [NPU] Support Qwen3.8-Flash-Next with graph replay, MTP, and Triton kernels Sep 17, 2026
@fishAndShrimp
fishAndShrimp marked this pull request as ready for review September 17, 2026 03:17
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@chenminghua8

Copy link
Copy Markdown

@fishAndShrimp Could you provide an A2 image that includes this PR? The Huawei team is very slow to respond; it’s likely they won’t end up merging your PR.

@fishAndShrimp

Copy link
Copy Markdown
Contributor Author

@fishAndShrimp Could you provide an A2 image that includes this PR?

@chenminghua8 Thanks for your interest! I don’t currently have a prebuilt A2 image to share. My validation so far has primarily been on A3.

If your A2 SGLang image includes the sglang and sgl-kernel-npu repos, you could locate them and apply the changes from this PR and sgl-project/sgl-kernel-npu#807. Please make sure the updated code is actually being loaded.

The new kernels in sgl-project/sgl-kernel-npu#807 are Python/Triton code, so these additions alone do not require rebuilding the C++ extensions. However, I haven’t tested this setup on A2, so compatibility still needs to be checked.

@IcyDesert

Copy link
Copy Markdown

@fishAndShrimp Could you provide an A2 image that includes this PR? The Huawei team is very slow to respond; it’s likely they won’t end up merging your PR.

@fishAndShrimp @chenminghua8 We're working on A2 validation in #43102 and are building on this PR and sgl-kernel-npu#807 for QSA integration. We don't have a fully validated QSA-enabled A2 image to share yet, but would be happy to coordinate on A2-specific fixes and reproducible setup instructions.

IcyDesert added a commit to IcyDesert/sglang that referenced this pull request Oct 26, 2026
Route NPU QSA through the model-specific kernel package and shared indexer.
Preserve live request mappings during graph replay and retain FIA updates
for the explicit dense fallback.

Adapted from sgl-project#37570 at 68dff35. Requires the QSA
modules from sgl-project/sgl-kernel-npu#807; full-model NPU validation is pending.

Co-authored-by: Estrella-xx <820167017@qq.com>
Co-authored-by: fishAndShrimp <ruiyuanqian@foxmail.com>
Signed-off-by: IcyDesert <icy_desert@outlook.com>
Co-authored-by: xuchen-w <adrian.xc.wei@gmail.com>
@github-actions

This comment has been minimized.

IcyDesert added a commit to IcyDesert/sglang that referenced this pull request Oct 30, 2026
Route NPU QSA through the model-specific kernel package and shared indexer.
Preserve live request mappings during graph replay and retain FIA updates
for the explicit dense fallback.

Adapted from sgl-project#37570 at 68dff35. Requires the QSA
modules from sgl-project/sgl-kernel-npu#807; full-model NPU validation is pending.

Co-authored-by: Estrella-xx <820167017@qq.com>
Co-authored-by: fishAndShrimp <ruiyuanqian@foxmail.com>
Signed-off-by: IcyDesert <icy_desert@outlook.com>
Co-authored-by: xuchen-w <adrian.xc.wei@gmail.com>
Merge affb3eb. Git merged qwen4_exp.py without textual conflicts; no manual source resolutions were applied.

Keep upstream PLE state-slot snapshots and CUDA variable-length prefill, alongside the NPU short-convolution dispatch. Keep the upstream device-neutral graph backend migration. Any test fixture updates are recorded separately.
Populate query_start_loc for prefill and leave it unset for decode and verify, matching _prepare_ple_batch. The merged upstream struct requires this field.

Verified the missing-field failure before this change. After the fix, all 25 selected CPU PLE cases pass. Production code is unchanged.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@fishAndShrimp fishAndShrimp changed the title [NPU] Support Qwen3.8-Flash-Next with graph replay, MTP, and Triton kernels [Feat] [NPU] Qwen3.8-Flash-Next: Support graph replay, MTP, and Triton kernels Nov 11, 2026
@sglang-npu-bot

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 11, 2026
Merge main at cabb43a. Resolve the only conflict in qsa_indexer.py
by dropping the removed get_is_capture_mode import and retaining
is_npu and the _is_npu initialization.

Keep upstream's RoPE cache cleanup and preserve the existing NPU dispatch.
The PR's additions and deletions relative to main are unchanged.

Validation: all applicable pre-commit hooks passed for the 13 PR files;
71 CPU QSA/CP regression tests passed. No hardware inference was run
for this merge.
Keep upstream get_forward and PR is_npu imports in qwen4_exp. Adopt the upstream post5 kernel release for NPU CI. Keep candidate performance changes in separate worktrees.
@fishAndShrimp

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@sglang-npu-bot
sglang-npu-bot merged commit eef4271 into sgl-project:main Nov 17, 2026
270 of 289 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants