Repository navigation
[Refactor] mem_cache: Plan draft KV bands by retention lifetime - #48005
Conversation
2275870 to
fb674d6
Compare
6a47923 to
15c3550
Compare
fb674d6 to
f3b7529
Compare
15c3550 to
d08c7e3
Compare
f3b7529 to
27b9360
Compare
d08c7e3 to
05acb65
Compare
27b9360 to
a2ef0b6
Compare
12fce20 to
b7d7126
Compare
a2ef0b6 to
a7ce649
Compare
… by retention lifetime
…endent windows for cleanup and admission
…coordinates from attention execution
b7d7126 to
325bd5f
Compare
|
@ch-wan
This PR exposed an older bug rather than adding one. Before it, FlashInfer MLA draft steps attended only to the prefix (eager did not pass Fix in #48734: the per-step rows are made contiguous, and |
…_mla An EAGLE chain draft writes step i's token at the committed length plus i, and `draft_forward` passes every step the same committed `seq_lens`. So each step of a multi-step draft backend must read the committed prefix plus i + 1 draft tokens, as fa3 (`speculative_step_id`) and triton do, and as FlashInfer MLA's multi-step backend has since sgl-project#48005. FlashMLA and TRT-LLM MLA (and cutedsl_mla, which inherits from it) still give the kernel the committed length, and FlashMLA's block table can also stop a page short. So every draft step leaves out its own token and the earlier steps' drafts. The target still verifies every draft token; this corrects the draft's attention window. Reading the right keys does not raise acceptance. Measured on SM90 before sgl-project#48005, so as history rather than for this exact tree: the median accept length in the scheduler logs falls on Ling-3.1-flash's NEXTN head (TP 8 + EP 8), from 2.69 to 2.64 tokens per step on FlashMLA. That model ran at about 9 running requests. On FlashMLA, Ling-2.6-flash (about 62) and Ling-3.0-flash-fp8 (about 47) show no clear change. Why Ling-3.1-flash falls is not known, and these runs do not separate the model from the load. GSM8K moved no more than between two runs of the same code. The step id lives on the FlashInfer MLA backend, which the others subclass, and their multi-step backends give each step backend its id. A draft step's decode adds its tokens to the key lengths, the scheduler metadata and the block-table width. This covers the eager path and CUDA graph replay, including the DCP replay of the TRT-LLM family. Other decodes are unchanged. FlashInfer MLA's own multi-step backend leaves the id unset, because it already passes each step lengths that hold its tokens. Sail's multi-step draft backend is not changed: it creates its step backends without an id, so they behave as on main. Two places would otherwise undo or repeat that count on the TRT-LLM family: - Its eager multi-step backend plans every step on one batch before any step decodes, and decode prefers the metadata the batch holds, which by then is the last step's. Earlier steps would attend over draft rows not written yet. A draft step now reads its own metadata; other decodes keep the batch's copy, since one backend can plan several batches. - Capture reaches its steps through FlashInfer MLA's wrapper, which adds the tokens itself. That wrapper now leaves them to a step backend that has an id, so capture and replay count them once. A captured graph fixes its bounds at capture, so they have to reach a step's tokens too. A request can commit up to two tokens short of the context, and its draft chain then reads past it: with a 4096 context, 8 steps and 4090 committed, the last step reads 4097 tokens. The TRT-LLM family's page table and scheduling bound, including the rank-local DCP one, stopped at the context, as did FlashMLA's table on a context just short of a page, whose reserved page does not cover the tail. A step's graph now reserves its own tokens. The same tables were also short, on main already, for target verify and draft extend, whose keys reach the drafts past the committed length: 4090 committed plus 8 drafts needs 65 pages of a table that holds 64. Their scheduling bounds already add the drafts. The table these modes share with the decode graph now holds the longest of these tails, and each mode's graph takes the slice its own tail needs. Without speculation, every bound stays at the context. Tests: - test_mla_draft_step_lengths.py (CPU): enters through each family's multi-step backend, with three requests around a page boundary. FlashMLA and the TRT-LLM family run their real decode on one batch and check the lengths and block table the kernel receives, eagerly, under DCP, after DP padding replans the batch, and with and without the host length mirror these backends opt out of; capture then replay count each step once. FlashInfer MLA's eager plan must use its prepared CSR, and its replay must count each step once, so a backend that adds them on top of its wrapper fails too. Before this change the FlashMLA and TRT-LLM steps read the committed lengths. Near the context limit, a long chain runs the real graph allocators, then the decode, and the lengths the kernel receives must fit its table and scheduling bound, with and without DCP. A verify and a draft extend near the limit run their real forward from graph metadata, on both TRT-LLM targets with and without DCP and on FlashMLA on contexts at and just short of a page, and their kernel lengths must fit the same way. A backend without a step keeps the context as its decode bound, and without speculation its table too. Stood in for: the kernels, scheduler metadata, KV views, quantization, the parallel context and the step constructors, so these cases say nothing about physical page ids, translated pools, quantization arithmetic or cross-rank numerics. - FlashMLA (CUDA, SM90): draft steps 0-2 match the reference attention over the committed prefix plus their own tokens, both eagerly and from graph metadata, with prefixes around a page boundary. On a context just short of a page, a draft step and a verify of 8 drafts that read past it match the reference from graph metadata. These cases run the metadata a graph is captured with, not a captured graph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR is part of a stack (oldest at bottom):
Motivation
A speculative draft fused into the unified pool used to be placed by its layer kind; draft layers whose retention differed from the target's either got a private draft pool or were forced into a band that does not match their lifetime. This PR places draft KV by retention lifetime: a draft layer shares the target sub-pool with the same lifetime, and gets its own band otherwise. It also makes window reclamation independent per window and derives the draft's KV coordinates from its attention execution (DCP width, backend).
Modifications
mem_cache/layout/fused_draft.py,kv_cache_configurator.py,kv/plan.py,kv/alloc/runner_ids.py,kv/alloc/id_space.py,unified_draft_pool.py: draft bands are planned by retention lifetime; fused draft views share the target's storage arena.kv/alloc/{chain,band,pricing,interim}.py,kv/storage.py,prefill_budget.py: independent windows are reclaimed for cleanup and admission; physical reservation scopes and takes cover every chain band.configs/attention_execution.py,layers/attention/{flashinfer_mla_backend,triton_backend}.py,kv/layout/{mha,mla,page}.py,layout/token_major.py,models/dflash.py,speculative/multi_layer_eagle_worker_v2.py: draft KV coordinates come from the attention execution (replicated vs DCP-sharded).managers/cache_controller.py,hybrid_cache/hybrid_cache_controller.py,kv/memory/host_backing.py,hicache_storage.py,storage/umbp/umbp_store.py: storage rank replication is decided per view; the"file"storage backend uses the page-envelope host pool (it reads and writes whole pages); a single-pool envelope host is sized from that pool's capacity under--hicache-ratiorather than from the whole region.test/registered/unit/.Behavior changes:
--enable-unified-memory, a draft whose fusion into the unified pool is declined now fails at startup with the reason (invalid unified draft storage declaration: ...) instead of falling back to a private draft pool. Examples: a Triton draft attention backend with--dcp-size > 1(select a replicated draft backend or disable unified memory); recurrent draft layers without a target recurrent-state band; draft prefill and decode backends with different DCP widths. Some DSPARK/DFLASH configurations that started onmainthis way are refused; a fix is planned in a follow-up.--hicache-storage-backend fileuses the page-envelope host layout under unified memory.Accuracy Tests
Serving validation was run at the top of this 26-PR stack (the cumulative change), not at this PR's head. Software: torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10. GSM8K is few-shot, 500 or 1319 questions as noted;
mainis the same commit the stack is based on, same flags.--enable-unified-memory(rebased stack)--enable-unified-memory --max-total-tokens 120000(rebased stack)trtllm_mha(1319 q)On B200 and B300, 40 greedy GSM8K generations were compared with
main: 35–40 of 40 are identical, and the rest first differ after at least 27 characters (near ties). The H200 runs above other than the two marked "rebased stack" were run at the stack top before it was rebased onto the current base; the code they cover is unchanged by the rebase except for conflict resolutions.The MTP and DFlash rows exercise draft placement by lifetime; the HiCache row exercises the host changes.
Speed Tests and Profiling
Measured at the stack top, same flags as
main(bench_serving, random 4096-in / 512-out unless noted):--max-total-tokens 120000Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): 🚫 Run #41907607446
Latest PR Test (Extra): 🚫 Run #41907607194
Latest PR Test (AMD ROCm 10): ⏳ Run #41907607161
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: