Skip to content

[Refactor] mem_cache: Evict and take for a demand that names every window - #48017

Closed
ch-wan wants to merge 4 commits into
cheng/refactor/unified-kv-window-metricsfrom
cheng/refactor/unified-kv-evict-for-demand
Closed

ch-wan wants to merge 4 commits into
cheng/refactor/unified-kv-window-metricsfrom
cheng/refactor/unified-kv-evict-for-demand

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

Eviction for an allocation took each component's shortfall: callers read a component's capacity, subtracted it from what they meant to take, and the cache added the same read back to recover a target; under unified memory the cache then turned those targets into one demand for the allocator. Allocation demands also had several shapes: the owner's tokens plus the target window's (swa_tokens), with further windows charged the owner's tokens unless a side map named them, so a window a caller forgot was silently charged a default.

Modifications

  • mem_cache/base_prefix_cache.py, unified_radix_cache.py, pure_swa_radix_cache.py: evict_for_alloc(demand) takes what the allocation will take. Pools sharing memory reclaim the whole demand at once (ChainAllocator.reclaim_for_demand, replacing reclaim_to_availability); pools of their own evict each component's shortfall, computed in the cache. The joint walk an allocator drives while reclaiming is its own method, evict_until(params, satisfied).
  • EvictParams (what to evict) and the new AllocDemand (what an allocation takes) are separate types; both and EvictResult keep every window in one map by component id (window_num_tokens); swa_num_tokens and swa_num_tokens_evicted are removed. The all-SWA cache names its one pool as the window (all_windows).
  • mem_cache/allocator/base.py: TokenDemand (owner tokens, every window's tokens by window pool, state slots) is the one shape of prealloc_fits, prealloc_ceiling_fits, reclaim_for_prealloc, reclaim_for_load_back, ensure_capacity, page_demands and scoped_physical_takes, for the chain and static pools alike; a demand that omits a window the allocator keeps, or names one it does not, is an error.
  • disaggregation/decode.py: every preallocation demand is built by one helper (_prealloc_demand); with tail preallocation each window takes the tail it keeps; ChainAllocator.alloc_extend_swa_tail and KVStorage.take_extend take one map naming every window's tail.
  • Callers updated: token and window allocators, state-slot checks, host-load and device-alloc evictors, buffer pipeline, load-back, decode radix eviction, linker loads.
  • The int8 state checkpoint slot: a full checkpoint pool evicts one cached state by count, since no allocation demand reads that pool.
  • Tests: python/sglang/test/token_demand.py (test helper keeping the two-number form explicit), a full checkpoint pool with free active slots evicts a cached checkpoint and takes its slot, the default cache evicts a demand's shortfall, a window demand on the all-SWA cache evicts from its single pool, plus updates across unified allocator and radix-cache tests.

No intended behavior change apart from the int8 checkpoint slot case above; this is an internal API change of the prefix caches and allocators.

Accuracy Tests

Serving validation was run at the top of this 26-PR stack (the cumulative change), not at this PR's head. Software: torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10. GSM8K is few-shot, 500 or 1319 questions as noted; main is the same commit the stack is based on, same flags.

Hardware Model Configuration Stack top main
2x H200 Qwen/Qwen3.8-27B TP2, --enable-unified-memory (rebased stack) 0.954 (1319 q) 0.950–0.954
1x H200 openai/gpt-oss-120b TP1, --enable-unified-memory --max-total-tokens 120000 (rebased stack) 0.842 (1319 q) 0.839
2x H200 Qwen/Qwen3.8-27B TP2, unified, MTP (EAGLE, 3 steps, topk 1, 4 draft tokens) 0.951, accept length 3.56 0.953, 3.56
2x H200 Qwen/Qwen3.8-27B TP2, unified, DFlash draft 0.951, accept length 5.42 0.952, 5.44
2x H200 Qwen/Qwen3.8-27B PD (prefill TP1 + decode TP1, mooncake), unified, repeated runs 0.952–0.960, no hang 0.954–0.958
2x H200 openai/gpt-oss-120b PD (TP1 + TP1, mooncake), unified 0.810 0.808
2x H200 Qwen/Qwen3.8-27B TP2, unified, HiCache 0.954 —
2x H200 Qwen/Qwen3.8-27B PP2, unified 0.953 0.955 (static pools)
2x B200 Qwen/Qwen3.8-27B TP2, unified (500 q) 0.958 0.960
2x B200 openai/gpt-oss-120b TP2, unified, trtllm_mha (1319 q) 0.849 0.863
2x B300 Qwen/Qwen3.8-27B / openai/gpt-oss-120b TP2, unified 0.950 / 0.849 0.954 / 0.849

On B200 and B300, 40 greedy GSM8K generations were compared with main: 35–40 of 40 are identical, and the rest first differ after at least 27 characters (near ties). The H200 runs above other than the two marked "rebased stack" were run at the stack top before it was rebased onto the current base; the code they cover is unchanged by the rebase except for conflict resolutions.

Every serving row above allocates and evicts through these demand shapes.

Speed Tests and Profiling

The demand is built once per allocation; the availability round-trip it replaces is removed. Measured at the stack top, same flags as main (bench_serving, random 4096-in / 512-out unless noted):

Hardware Model / configuration Stack top main
1x H200 openai/gpt-oss-120b, unified, --max-total-tokens 120000 3.93–3.95 req/s 3.95 req/s
2x H200 Qwen/Qwen3.8-27B PD (TP1 + TP1), 2048-in / 256-out 5.65–6.28 req/s 6.21–6.29 req/s
2x H200 openai/gpt-oss-120b PD (TP1 + TP1) 9.39–9.45 req/s 9.21 req/s
2x H200 Qwen/Qwen3.8-27B MTP TP2 / DFlash TP2 4.26 / 4.36 req/s 4.29 / 4.35 req/s
2x B200 Qwen/Qwen3.8-27B TP2 / openai/gpt-oss-120b TP2 6.35 / 19.31 req/s 6.34 / 19.28 req/s

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #41837981509
Latest PR Test (Extra): ❌ Run #41837981493
Latest PR Test (AMD ROCm 10): ❌ Run #41837981508

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang apple-silicon unified-radix-cache memory-pool labels Nov 3, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 7dea9cd to b0e0bee Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch 2 times, most recently from d8c98e4 to 87d6e80 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from b0e0bee to 8bd568a Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from 87d6e80 to b794f0d Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 8bd568a to 806d2b6 Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from b794f0d to 70ef215 Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 806d2b6 to 072b15c Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from 70ef215 to 384cb84 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 072b15c to 0b23e70 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from 384cb84 to e63bbb8 Compare November 4, 2026 10:50
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 0b23e70 to b07586d Compare November 4, 2026 10:51
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from e63bbb8 to 212d6e1 Compare November 4, 2026 12:23
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from b07586d to bb44f84 Compare November 4, 2026 12:23
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from 212d6e1 to eb3d723 Compare November 4, 2026 12:36
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from bb44f84 to 0b2788c Compare November 4, 2026 12:36
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from eb3d723 to 114d4cc Compare November 4, 2026 12:45
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 0b2788c to 51b5411 Compare November 4, 2026 12:45
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from 114d4cc to e71f63c Compare November 4, 2026 13:05
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 51b5411 to e785d13 Compare November 4, 2026 13:05
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from e71f63c to 42221b7 Compare November 4, 2026 13:11
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from e785d13 to 9c23582 Compare November 4, 2026 13:11
…he to evict for an allocation's demand

`evict_for_alloc` took each component's shortfall: callers read the
component's capacity, subtracted it from what they meant to take, and the
cache added the same read back to recover an availability target. Under
unified memory the cache then turned those targets into one demand for the
allocator. The shortfall was a detour, and every caller carried the read.

`evict_for_alloc(demand)` now takes what the allocation will take. Pools
that share memory reclaim the whole demand at once (the allocator's
`reclaim_for_demand`, which replaces `reclaim_to_availability`); pools of
their own evict each component's shortfall, computed in the cache from the
demand, and a walk stops once its pool can serve its demand, as before. The
default for other caches evicts the shortfall of the pools its allocator
reads. The joint walk an allocator drives while it reclaims one demand is
its own method, `evict_until(params, satisfied)`, instead of a mode of
`evict_for_alloc`.

Callers state demands: the token and window allocators, the state slot
check before binding requests, the host-load and device-alloc evictors,
the buffer pipeline, load-back, the decode queue's radix-cache eviction
before a preallocation, and the linker loads of the full, window and state
components (the window component now names its own slot, not the first
window's). The int8 state checkpoint slot is the exception: the tree's
states live in the checkpoint pool, which no allocation demand reads, so a
full checkpoint pool evicts one cached state by count. The availability-target bookkeeping in the walk is
replaced by the demand itself.

Tests: a full checkpoint pool with free active slots evicts a cached
checkpoint and takes its slot.

Test: the default cache evicts a demand's shortfall.
…counts and allocation demands their own types

`EvictParams` carried two meanings: what to evict (`evict`, `evict_until`)
and what an allocation is about to take (`evict_for_alloc`). The second now
has its own type, `AllocDemand`, with the same fields.

Both, and `EvictResult`, keep a tree's windows in one map by component id,
`window_num_tokens`, the first window included: `swa_num_tokens` and
`swa_num_tokens_evicted` are gone, so no window is asked for, or reported,
apart from the others. A cache keeping a single pool (the default radix
cache, the all-SWA cache) serves a window demand from that pool, as before:
the all-SWA cache names its one pool as the window (`all_windows`), so a
decode preallocation's window demand evicts from it;
callers that only know "the window pool" of a static allocator name every
window slot of the tree (`all_windows`), which for a static pool is its one
window. `EvictResult.add` replaces the hand-summed accumulations.

Tests: a window demand on the all-SWA cache evicts from its single pool.
… in an allocation's token demand

The chain priced a preallocation, a host load and a capacity check from
two numbers: the owner's tokens and the target window's (`swa_tokens`),
with any further window taking the owner's tokens unless a side map named
it. A window the caller forgot was silently charged a default.

`TokenDemand` (owner tokens, every window's tokens by window pool, state
slots) is now the one shape of `prealloc_fits`, `prealloc_ceiling_fits`,
`reclaim_for_prealloc`, `reclaim_for_load_back`, `ensure_capacity`,
`page_demands` and `scoped_physical_takes`, for the chain and the static
pools alike (a static pool's one window goes by its window pool's name).
A demand that leaves out a window the allocator keeps, or names one it
does not, is an error; so is an `admits` window map that leaves one out.
`window_pool_names` lists the windows a demand must name;
`TokenDemand.same_tokens` is the extend/decode case where every window
keeps the new tokens. `evict_to_free_tokens` loses its per-window keywords.

The decode node builds every preallocation demand with one helper
(`_prealloc_demand`): with tail preallocation each window takes the tail it
keeps of the request (`_window_tails`, its own length widened by a reader
reaching further back), plus the decode reservation when admitting;
without it a window keeps the whole sequence and takes what the owner
takes. Only the target window's admission share is still replaced by the
figure the scheduler's window budget prices. The ceiling check now charges
a further window its own tail rather than the whole demand. The prefill
budget's load-back demand and empty-demand checks name every window.

Tests use `sglang.test.token_demand`, which keeps their two-number form
(the target window's tokens, every other window the owner's) explicit.
…dow's decode tail by name

A decode node's tail preallocation passed the target window's tail
separately from the further windows' (`swa_tail_len` beside
`window_tail_lens`), and a window neither named kept the whole sequence.
`ChainAllocator.alloc_extend_swa_tail` and `KVStorage.take_extend` now take
one map naming every window's tail (an omitted window is an error); a
static pool's `take_extend` reads its one window's entry. The decode
preallocation passes `_window_tails` and sets each window's eviction
cursor from its own entry, mapping a window pool to its prefix-tree slot.
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-metrics branch from 42221b7 to 45d0b89 Compare November 4, 2026 13:25
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-evict-for-demand branch from 9c23582 to 78714f5 Compare November 4, 2026 13:25
@ch-wan

ch-wan commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by the restructured stack #48006-#48014, which now carries this change. Closing; the branch is kept.

@ch-wan ch-wan closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apple-silicon documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang memory-pool unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant