Skip to content

[Feat] spec/dflash: Per-request draft KV ring for all-sliding-window MHA drafts (trtllm_mha, fa4) - #50740

Merged
Fridge003 merged 3 commits into
sgl-project:mainfrom
shyeh25:dflash-mha-draft-kv-ring
Nov 17, 2026
Merged

Fridge003 merged 3 commits into
sgl-project:mainfrom
shyeh25:dflash-mha-draft-kv-ring

Conversation

@shyeh25

@shyeh25 shyeh25 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

#49664 keeps an all-sliding-window MLA DFLASH draft's KV in a per-request ring of window + block rows instead of one draft KV row per target token, and #49665 sends that ring over Mooncake PD. MHA drafts (for example Muse-Spark-Open + DFlash2, five sliding_attention layers with a 2048-token window) still allocate the per-token draft pool, so draft KV grows with the target's context although the draft only ever reads the last window.

This PR extends the same ring to all-sliding-window MHA drafts on the trtllm_mha and fa4 draft backends. For Muse-Spark-Open DFlash2 at TP4 the draft KV becomes 2176 rows per request (34 pages of 64), 0.17 GB per rank for 32 requests, and the freed memory goes to the target KV pool.

Modifications

  • speculative/dflash_draft_ring.py: a paged ring layout for MHA drafts. A page table has no per-request start, so the ring is page aligned (sink_rows = page_size, rows_per_req a whole number of pages) and positions are taken relative to the request's prefix-hit floor min_pos. The draft attends seq_len - min_pos keys; logical page j maps to ring page 1 + req * pages_per_req + j % pages_per_req. The page table and relative lengths are built on the GPU, so they are CUDA-graph safe with no host sync.
  • trtllm_mha_backend.py, flashattention_backend.py (fa4): set_draft_kv_ring; for TARGET_VERIFY the draft reads through the ring page table and relative sequence lengths (eager, CUDA graph, and ragged verify paths). Nothing changes when no ring is set.
  • Ring resolution (SGLANG_ENABLE_DFLASH_DRAFT_KV_RING, unchanged semantics: default on when eligible, =1 errors if not eligible, =0 off). The MHA ring is used only on CUDA, every draft layer sliding_attention, non-MLA, draft backend trtllm_mha/fa4, no DCP/attention CP, no DP attention, no --speculative-draft-window-size, no unified memory; PD only with Mooncake. Otherwise it resolves off with a logged reason and the per-token pool is used as before. The MLA ring path is unchanged.
  • PD (Mooncake): prefill sends the request's whole ring block in ring order plus its floor; the ring rows hold the rank's head shard unsliced, so registration rejects unequal attention TP between prefill and decode.
  • KV budget: the ring is charged instead of the per-token draft pool.
  • Startup: the ring's block-locs Triton kernel is compiled for every decode batch size when the ring is created, instead of on the first requests (it otherwise added about 6 s to the first batch after each start).
  • Tests: ring layout and resolution unit tests, backend metadata tests (graph and eager, ragged verify, encoder-only expansion), Mooncake registration tests, test_dflash_ring_locs.

Behavior change

The ring is now on by default for eligible MHA drafts, the same as for MLA drafts in #49664.

  • What changes: the ring only keeps draft KV written after a request starts. After a prefix-cache hit (radix or HiCache), the draft cannot see the hit part until new tokens refill its window.
  • Cost: lower acceptance right after a hit. In a bs=1 test (about 6000-token hit, then 2400 generated tokens), acceptance over those 2400 tokens was 7.5%-17.7% lower. Later turns of the same session were unaffected (identical outputs, acceptance within 0.3%), and so was the shared-prefix benchmark below.
  • Gain: +19.7% target KV capacity on Muse-Spark-Open (below).

Accuracy Tests

Muse-Spark-Open NVFP4 + DFlash2, TP4 GB300, GSM8K (all 1319 examples) with the eval settings of test_muse_spark_12_b300_pd.py (thinking, reasoning effort xhigh, max_tokens 32768, temperature 1.0, top_p 0.95):

GSM8K accept length
main 0b098e1 97.65% 4.96
this PR 97.57% 4.93

Upstream test/registered/e2e/models/test_muse_spark_12_b300_pd.py (8xB300, PD 1P1D, fa4 draft), ring on for both roles with this PR:

GSM8K MMMU-Pro
main 0b098e1 0.9773 0.90
this PR 0.9765 0.88

Speed Tests and Profiling

Server (both arms identical):

python -m sglang.launch_server --model-path <Muse-Spark-Open-816B-A42B-NVFP4> --trust-remote-code \
  --tensor-parallel-size 4 --expert-parallel-size 4 --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm \
  --kv-cache-dtype fp8_e4m3 --enable-layernorm-sp --enable-megamoe-for-layernorm-sp \
  --speculative-algorithm DFLASH --speculative-draft-model-path <Muse-Spark-Open-DFlash2> \
  --speculative-num-draft-tokens 8 --speculative-draft-attention-backend trtllm_mha \
  --mem-fraction-static 0.95 --swa-full-tokens-ratio 0.2 --max-running-requests 32 --cuda-graph-max-bs-decode 32 \
  --chunked-prefill-size 16384 --max-prefill-tokens 16384 --enable-hierarchical-cache --hicache-size 140

Capacity (from the server log): max_total_num_tokens 7,826,368 (main) vs 9,370,304 (this PR), +19.7%; the draft KV ring takes 71,872 rows, 0.17 GB per rank.

python -m sglang.benchmark.serving --backend sglang --dataset-name random-ids \
  --random-input-len 4096 --random-output-len 256 --random-range-ratio 1.0 --num-prompts 512 --max-concurrency 32
output tok/s accept length median TPOT
main 2002.99 4.63 7.54 ms
this PR 2044.37 4.68 7.33 ms

No measurable change when the extra capacity is not needed.

python -m sglang.benchmark.serving --backend sglang --dataset-name generated-shared-prefix \
  --gsp-num-groups 384 --gsp-prompts-per-group 3 --gsp-system-prompt-len 60000 --gsp-question-len 1024 \
  --gsp-output-len 128 --max-concurrency 32 --cache-report
total tok/s median TTFT cached tokens accept length
main 188.64 19.96 s 32.96M 5.67
this PR 198.62 18.58 s 34.99M 5.73

With about 23M tokens of live context, the larger target pool keeps more of the shared prefix cached: +5.3% throughput and -7% median TTFT.

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests.
  • Update documentation according to Write documentations.
  • Provide accuracy and speed benchmark results.
  • Follow the SGLang code style guidance.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #44090615042
Latest PR Test (Extra): ✅ Run #44090614949
Latest PR Test (AMD ROCm 10): ❌ Run #44090614935

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…MHA drafts (trtllm_mha, fa4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@shyeh25

shyeh25 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

AgentX results: the ring frees host memory, so HiCache can be larger and throughput improves

We ran Muse-Spark-Open-816B-A42B-NVFP4 with DFlash2 on GB300 under the AgentX long multi-turn workload (aggregated TP4, HiCache enabled). On main, the MHA draft keeps a per-token draft KV pool, and HiCache adds a per-token draft host pool of about 0.2 × the HiCache size on every rank. With the HiCache sizes used for this PR (--hicache-size 195 and 170), main fails to start with Not enough host memory available; the largest size it starts with is about 156, and only with a few GB to spare. With this PR the draft KV is a small per-request ring and needs no draft host pool, so HiCache can be larger, and the GPU target KV pool is about 19% larger as well. Apart from the HiCache size, the two configurations are identical.

Point Configuration HiCache size Total tokens per $1 TCO (millions) P90 interactivity (tokens/s/user) P90 TTFT (s)
TP4, concurrency 76 main 156 96.77 47.90 36.24
this PR 195 124.37 49.54 10.34
improvement +28.5% +3.4% −71%
TP4, concurrency 56 main 156 113.71 85.67 7.28
this PR 170 115.33 88.39 5.77
improvement +1.4% +3.2% −21%

@nvpohanh

Copy link
Copy Markdown
Collaborator

@yhyang201 @Fridge003 Could you review this? Thanks!

@nvpohanh

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 17, 2026
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, fix the PR title if the title check fails, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] All NV pipelines have passed.

Resolve conflicts with sgl-project#48414: use its draft_window_size parameter in
draft_kv_ring_unsupported_reason for the MHA ring check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, fix the PR title if the title check fails, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] All NV pipelines have passed.

@Fridge003
Fridge003 merged commit 7aaebb1 into sgl-project:main Nov 17, 2026
165 of 170 checks passed
kevin-mii added a commit to kevin-mii/sglang that referenced this pull request Nov 18, 2026
Conflicts:
- dflash_draft_ring.py: main (sgl-project#50740) added the MHA ring path ahead of the MLA
  backend check; keep it and keep this PR's dflash_draft_reads_rows_by_position
  in place of the DFLASH_MLA_ONLY_DRAFT_BACKENDS membership test.
- test_dflash_draft_kv_ring.py: keep both main's current_platform import and
  this PR's override_platform import.
- Registration lines in test_verify_shared_kv.py and
  test_fused_kda_conv_recurrent_verify.py: take main's renamed suites (sgl-project#50874)
  and keep this PR's est_time. The other AMD registrations this PR touches
  move to the new suite names too, since a stale name is never dispatched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 mooncake run-ci CI: run the baseline test suite on this PR speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants