Skip to content

[Refactor] mem_cache: Plan draft KV bands by retention lifetime - #48005

Merged
ch-wan merged 3 commits into
mainfrom
cheng/refactor/unified-kv-draft-lifetimes
Nov 4, 2026
Merged

ch-wan merged 3 commits into
mainfrom
cheng/refactor/unified-kv-draft-lifetimes

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

A speculative draft fused into the unified pool used to be placed by its layer kind; draft layers whose retention differed from the target's either got a private draft pool or were forced into a band that does not match their lifetime. This PR places draft KV by retention lifetime: a draft layer shares the target sub-pool with the same lifetime, and gets its own band otherwise. It also makes window reclamation independent per window and derives the draft's KV coordinates from its attention execution (DCP width, backend).

Modifications

  • Placement: mem_cache/layout/fused_draft.py, kv_cache_configurator.py, kv/plan.py, kv/alloc/runner_ids.py, kv/alloc/id_space.py, unified_draft_pool.py: draft bands are planned by retention lifetime; fused draft views share the target's storage arena.
  • Allocation: kv/alloc/{chain,band,pricing,interim}.py, kv/storage.py, prefill_budget.py: independent windows are reclaimed for cleanup and admission; physical reservation scopes and takes cover every chain band.
  • Execution: configs/attention_execution.py, layers/attention/{flashinfer_mla_backend,triton_backend}.py, kv/layout/{mha,mla,page}.py, layout/token_major.py, models/dflash.py, speculative/multi_layer_eagle_worker_v2.py: draft KV coordinates come from the attention execution (replicated vs DCP-sharded).
  • HiCache: managers/cache_controller.py, hybrid_cache/hybrid_cache_controller.py, kv/memory/host_backing.py, hicache_storage.py, storage/umbp/umbp_store.py: storage rank replication is decided per view; the "file" storage backend uses the page-envelope host pool (it reads and writes whole pages); a single-pool envelope host is sized from that pool's capacity under --hicache-ratio rather than from the whole region.
  • Tests: draft placement, DCP capacity, HiCache DCP/identity, multiwindow placement, tri-pool and radix-cache tests under test/registered/unit/.

Behavior changes:

  • Under --enable-unified-memory, a draft whose fusion into the unified pool is declined now fails at startup with the reason (invalid unified draft storage declaration: ...) instead of falling back to a private draft pool. Examples: a Triton draft attention backend with --dcp-size > 1 (select a replicated draft backend or disable unified memory); recurrent draft layers without a target recurrent-state band; draft prefill and decode backends with different DCP widths. Some DSPARK/DFLASH configurations that started on main this way are refused; a fix is planned in a follow-up.
  • HiCache with --hicache-storage-backend file uses the page-envelope host layout under unified memory.

Accuracy Tests

Serving validation was run at the top of this 26-PR stack (the cumulative change), not at this PR's head. Software: torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10. GSM8K is few-shot, 500 or 1319 questions as noted; main is the same commit the stack is based on, same flags.

Hardware Model Configuration Stack top main
2x H200 Qwen/Qwen3.8-27B TP2, --enable-unified-memory (rebased stack) 0.954 (1319 q) 0.950–0.954
1x H200 openai/gpt-oss-120b TP1, --enable-unified-memory --max-total-tokens 120000 (rebased stack) 0.842 (1319 q) 0.839
2x H200 Qwen/Qwen3.8-27B TP2, unified, MTP (EAGLE, 3 steps, topk 1, 4 draft tokens) 0.951, accept length 3.56 0.953, 3.56
2x H200 Qwen/Qwen3.8-27B TP2, unified, DFlash draft 0.951, accept length 5.42 0.952, 5.44
2x H200 Qwen/Qwen3.8-27B PD (prefill TP1 + decode TP1, mooncake), unified, repeated runs 0.952–0.960, no hang 0.954–0.958
2x H200 openai/gpt-oss-120b PD (TP1 + TP1, mooncake), unified 0.810 0.808
2x H200 Qwen/Qwen3.8-27B TP2, unified, HiCache 0.954 —
2x H200 Qwen/Qwen3.8-27B PP2, unified 0.953 0.955 (static pools)
2x B200 Qwen/Qwen3.8-27B TP2, unified (500 q) 0.958 0.960
2x B200 openai/gpt-oss-120b TP2, unified, trtllm_mha (1319 q) 0.849 0.863
2x B300 Qwen/Qwen3.8-27B / openai/gpt-oss-120b TP2, unified 0.950 / 0.849 0.954 / 0.849

On B200 and B300, 40 greedy GSM8K generations were compared with main: 35–40 of 40 are identical, and the rest first differ after at least 27 characters (near ties). The H200 runs above other than the two marked "rebased stack" were run at the stack top before it was rebased onto the current base; the code they cover is unchanged by the rebase except for conflict resolutions.

The MTP and DFlash rows exercise draft placement by lifetime; the HiCache row exercises the host changes.

Speed Tests and Profiling

Measured at the stack top, same flags as main (bench_serving, random 4096-in / 512-out unless noted):

Hardware Model / configuration Stack top main
1x H200 openai/gpt-oss-120b, unified, --max-total-tokens 120000 3.93–3.95 req/s 3.95 req/s
2x H200 Qwen/Qwen3.8-27B PD (TP1 + TP1), 2048-in / 256-out 5.65–6.28 req/s 6.21–6.29 req/s
2x H200 openai/gpt-oss-120b PD (TP1 + TP1) 9.39–9.45 req/s 9.21 req/s
2x H200 Qwen/Qwen3.8-27B MTP TP2 / DFlash TP2 4.26 / 4.36 req/s 4.29 / 4.35 req/s
2x B200 Qwen/Qwen3.8-27B TP2 / openai/gpt-oss-120b TP2 6.35 / 19.31 req/s 6.34 / 19.28 req/s

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #41907607446
Latest PR Test (Extra): 🚫 Run #41907607194
Latest PR Test (AMD ROCm 10): ⏳ Run #41907607161

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-declared-payloads branch from 2275870 to fb674d6 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from 6a47923 to 15c3550 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-declared-payloads branch from fb674d6 to f3b7529 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from 15c3550 to d08c7e3 Compare November 4, 2026 07:42
@ch-wan ch-wan added bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) parallel-stages CI: stages dispatch together instead of waiting on each other max-concurrency CI: a suite fans out to every shard; apply before triggering the run run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) labels Nov 4, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-declared-payloads branch from f3b7529 to 27b9360 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from d08c7e3 to 05acb65 Compare November 4, 2026 09:31
@ch-wan
ch-wan marked this pull request as ready for review November 4, 2026 09:34
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from b7d7126 to 325bd5f Compare November 4, 2026 20:32
@ch-wan
ch-wan merged commit f09b88e into main Nov 4, 2026
37 of 92 checks passed
@ch-wan
ch-wan deleted the cheng/refactor/unified-kv-draft-lifetimes branch November 4, 2026 20:32
@hnyls2002

Copy link
Copy Markdown
Collaborator

@ch-wan test/registered/attention/unittests/mla/test_flashinfer.py has failed on main since this PR: test_runner_mode_eagle_draft_cuda_graph_runner_cases, case runner_eagle_draft_decode_mla_cuda_graph_chain, max abs diff 43 at index (1, 2).

This PR exposed an older bug rather than adding one. Before it, FlashInfer MLA draft steps attended only to the prefix (eager did not pass spec_info, and graph replay planned prefix-only lengths), so the draft tokens' KV was never read. Those writes were misplaced: per_step_draft_out_cache_loc returns strided rows, and set_mla_kv_buffer_triton reads loc by flat offset, so with bs * topk > 1 a draft step wrote request 1's KV into request 0's next-step slot and never wrote its own.

Fix in #48734: the per-step rows are made contiguous, and KVWriteLoc refuses a strided loc.

Yuxingwang-intel pushed a commit to Yuxingwang-intel/sglang that referenced this pull request Nov 6, 2026
caihuali95 pushed a commit to caihuali95/sglang that referenced this pull request Nov 11, 2026
…_mla

An EAGLE chain draft writes step i's token at the committed length plus
i, and `draft_forward` passes every step the same committed `seq_lens`.
So each step of a multi-step draft backend must read the committed prefix
plus i + 1 draft tokens, as fa3 (`speculative_step_id`) and triton do, and
as FlashInfer MLA's multi-step backend has since sgl-project#48005. FlashMLA and
TRT-LLM MLA (and cutedsl_mla, which inherits from it) still give the
kernel the committed length, and FlashMLA's block table can also stop a
page short.

So every draft step leaves out its own token and the earlier steps'
drafts. The target still verifies every draft token; this corrects the
draft's attention window.

Reading the right keys does not raise acceptance. Measured on SM90 before
sgl-project#48005, so as history rather than for this exact tree: the median accept
length in the scheduler logs falls on Ling-3.1-flash's NEXTN head (TP 8 +
EP 8), from 2.69 to 2.64 tokens per step on FlashMLA. That model ran at
about 9 running requests. On FlashMLA, Ling-2.6-flash (about 62) and
Ling-3.0-flash-fp8 (about 47) show no clear change. Why Ling-3.1-flash
falls is not known, and these runs do not separate the model from the
load. GSM8K moved no more than between two runs of the same code.

The step id lives on the FlashInfer MLA backend, which the others
subclass, and their multi-step backends give each step backend its id. A
draft step's decode adds its tokens to the key lengths, the scheduler
metadata and the block-table width. This covers the eager path and CUDA
graph replay, including the DCP replay of the TRT-LLM family. Other
decodes are unchanged. FlashInfer MLA's own multi-step backend leaves the
id unset, because it already passes each step lengths that hold its
tokens. Sail's multi-step draft backend is not changed: it creates its
step backends without an id, so they behave as on main.

Two places would otherwise undo or repeat that count on the TRT-LLM
family:

- Its eager multi-step backend plans every step on one batch before any
  step decodes, and decode prefers the metadata the batch holds, which by
  then is the last step's. Earlier steps would attend over draft rows not
  written yet. A draft step now reads its own metadata; other decodes keep
  the batch's copy, since one backend can plan several batches.
- Capture reaches its steps through FlashInfer MLA's wrapper, which adds
  the tokens itself. That wrapper now leaves them to a step backend that
  has an id, so capture and replay count them once.

A captured graph fixes its bounds at capture, so they have to reach a step's
tokens too. A request can commit up to two tokens short of the context, and
its draft chain then reads past it: with a 4096 context, 8 steps and 4090
committed, the last step reads 4097 tokens. The TRT-LLM family's page table
and scheduling bound, including the rank-local DCP one, stopped at the
context, as did FlashMLA's table on a context just short of a page, whose
reserved page does not cover the tail. A step's graph now reserves its own
tokens.

The same tables were also short, on main already, for target verify and
draft extend, whose keys reach the drafts past the committed length:
4090 committed plus 8 drafts needs 65 pages of a table that holds 64.
Their scheduling bounds already add the drafts. The table these modes
share with the decode graph now holds the longest of these tails, and each
mode's graph takes the slice its own tail needs. Without speculation, every
bound stays at the context.

Tests:
- test_mla_draft_step_lengths.py (CPU): enters through each family's
  multi-step backend, with three requests around a page boundary. FlashMLA
  and the TRT-LLM family run their real decode on one batch and check the
  lengths and block table the kernel receives, eagerly, under DCP, after
  DP padding replans the batch, and with and without the host length
  mirror these backends opt out of; capture then replay count each step
  once. FlashInfer MLA's eager plan must use its prepared CSR, and its
  replay must count each step once, so a backend that adds them on top of
  its wrapper fails too. Before this change the FlashMLA and TRT-LLM steps
  read the committed lengths. Near the context limit, a long chain runs the
  real graph allocators, then the decode, and the lengths the kernel receives
  must fit its table and scheduling bound, with and without DCP. A verify
  and a draft extend near the limit run their real forward from graph
  metadata, on both TRT-LLM targets with and without DCP and on FlashMLA on
  contexts at and just short of a page, and their kernel lengths must fit
  the same way. A backend without a step keeps the context as its decode
  bound, and without speculation its table too. Stood in for: the kernels, scheduler
  metadata, KV views, quantization, the parallel context and the step
  constructors, so these cases say nothing about physical page ids,
  translated pools, quantization arithmetic or cross-rank numerics.
- FlashMLA (CUDA, SM90): draft steps 0-2 match the reference attention
  over the committed prefix plus their own tokens, both eagerly and from
  graph metadata, with prefixes around a page boundary. On a context just
  short of a page, a draft step and a verify of 8 drafts that read past it
  match the reference from graph metadata. These cases run the metadata a
  graph is captured with, not a captured graph.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) deepseek hicache Hierarchical Caching for SGLang jit-kernel max-concurrency CI: a suite fans out to every shard; apply before triggering the run memory-pool mooncake parallel-stages CI: stages dispatch together instead of waiting on each other run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants