Repository navigation
[Feat] spec/dflash: Per-request draft KV ring for all-sliding-window MHA drafts (trtllm_mha, fa4) - #50740
Conversation
…MHA drafts (trtllm_mha, fa4) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
AgentX results: the ring frees host memory, so HiCache can be larger and throughput improves We ran Muse-Spark-Open-816B-A42B-NVFP4 with DFlash2 on GB300 under the AgentX long multi-turn workload (aggregated TP4, HiCache enabled). On main, the MHA draft keeps a per-token draft KV pool, and HiCache adds a per-token draft host pool of about 0.2 × the HiCache size on every rank. With the HiCache sizes used for this PR (
|
|
@yhyang201 @Fridge003 Could you review this? Thanks! |
|
/tag-and-rerun-ci |
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
|
[by Claude Code] All NV pipelines have passed. |
Resolve conflicts with sgl-project#48414: use its draft_window_size parameter in draft_kv_ring_unsupported_reason for the MHA ring check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
|
[by Claude Code] All NV pipelines have passed. |
Conflicts: - dflash_draft_ring.py: main (sgl-project#50740) added the MHA ring path ahead of the MLA backend check; keep it and keep this PR's dflash_draft_reads_rows_by_position in place of the DFLASH_MLA_ONLY_DRAFT_BACKENDS membership test. - test_dflash_draft_kv_ring.py: keep both main's current_platform import and this PR's override_platform import. - Registration lines in test_verify_shared_kv.py and test_fused_kda_conv_recurrent_verify.py: take main's renamed suites (sgl-project#50874) and keep this PR's est_time. The other AMD registrations this PR touches move to the new suite names too, since a stale name is never dispatched.
Motivation
#49664 keeps an all-sliding-window MLA DFLASH draft's KV in a per-request ring of
window + blockrows instead of one draft KV row per target token, and #49665 sends that ring over Mooncake PD. MHA drafts (for example Muse-Spark-Open + DFlash2, fivesliding_attentionlayers with a 2048-token window) still allocate the per-token draft pool, so draft KV grows with the target's context although the draft only ever reads the last window.This PR extends the same ring to all-sliding-window MHA drafts on the
trtllm_mhaandfa4draft backends. For Muse-Spark-Open DFlash2 at TP4 the draft KV becomes 2176 rows per request (34 pages of 64), 0.17 GB per rank for 32 requests, and the freed memory goes to the target KV pool.Modifications
speculative/dflash_draft_ring.py: a paged ring layout for MHA drafts. A page table has no per-request start, so the ring is page aligned (sink_rows = page_size,rows_per_reqa whole number of pages) and positions are taken relative to the request's prefix-hit floormin_pos. The draft attendsseq_len - min_poskeys; logical pagejmaps to ring page1 + req * pages_per_req + j % pages_per_req. The page table and relative lengths are built on the GPU, so they are CUDA-graph safe with no host sync.trtllm_mha_backend.py,flashattention_backend.py(fa4):set_draft_kv_ring; for TARGET_VERIFY the draft reads through the ring page table and relative sequence lengths (eager, CUDA graph, and ragged verify paths). Nothing changes when no ring is set.SGLANG_ENABLE_DFLASH_DRAFT_KV_RING, unchanged semantics: default on when eligible,=1errors if not eligible,=0off). The MHA ring is used only on CUDA, every draft layersliding_attention, non-MLA, draft backendtrtllm_mha/fa4, no DCP/attention CP, no DP attention, no--speculative-draft-window-size, no unified memory; PD only with Mooncake. Otherwise it resolves off with a logged reason and the per-token pool is used as before. The MLA ring path is unchanged.test_dflash_ring_locs.Behavior change
The ring is now on by default for eligible MHA drafts, the same as for MLA drafts in #49664.
Accuracy Tests
Muse-Spark-Open NVFP4 + DFlash2, TP4 GB300, GSM8K (all 1319 examples) with the eval settings of
test_muse_spark_12_b300_pd.py(thinking, reasoning effort xhigh, max_tokens 32768, temperature 1.0, top_p 0.95):Upstream
test/registered/e2e/models/test_muse_spark_12_b300_pd.py(8xB300, PD 1P1D, fa4 draft), ring on for both roles with this PR:Speed Tests and Profiling
Server (both arms identical):
Capacity (from the server log):
max_total_num_tokens7,826,368 (main) vs 9,370,304 (this PR), +19.7%; the draft KV ring takes 71,872 rows, 0.17 GB per rank.No measurable change when the extra capacity is not needed.
With about 23M tokens of live context, the larger target pool keeps more of the shared prefix cached: +5.3% throughput and -7% median TTFT.
Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #44090615042
Latest PR Test (Extra): ✅ Run #44090614949
Latest PR Test (AMD ROCm 10): ❌ Run #44090614935
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: