Repository navigation
Conversation
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 06:18
7dea9cd to
b0e0bee
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
2 times, most recently
from
November 4, 2026 07:42
d8c98e4 to
87d6e80
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 07:42
b0e0bee to
8bd568a
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 08:02
87d6e80 to
b794f0d
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 08:02
8bd568a to
806d2b6
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 09:14
b794f0d to
70ef215
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 09:14
806d2b6 to
072b15c
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 09:31
70ef215 to
384cb84
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 09:31
072b15c to
0b23e70
Compare
This was referenced Nov 4, 2026
Merged
Merged
Merged
Merged
This was referenced Nov 4, 2026
Closed
Closed
[Refactor] mem_cache: Choose reclaim in one place, keep one static storage, joint LRU reclaim
#48025
Closed
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 10:50
384cb84 to
e63bbb8
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 10:51
0b23e70 to
b07586d
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 12:23
e63bbb8 to
212d6e1
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 12:23
b07586d to
bb44f84
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 12:36
212d6e1 to
eb3d723
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 12:36
bb44f84 to
0b2788c
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 12:45
eb3d723 to
114d4cc
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 12:45
0b2788c to
51b5411
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 13:05
114d4cc to
e71f63c
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 13:05
51b5411 to
e785d13
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 13:11
e71f63c to
42221b7
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 13:11
e785d13 to
9c23582
Compare
…he to evict for an allocation's demand `evict_for_alloc` took each component's shortfall: callers read the component's capacity, subtracted it from what they meant to take, and the cache added the same read back to recover an availability target. Under unified memory the cache then turned those targets into one demand for the allocator. The shortfall was a detour, and every caller carried the read. `evict_for_alloc(demand)` now takes what the allocation will take. Pools that share memory reclaim the whole demand at once (the allocator's `reclaim_for_demand`, which replaces `reclaim_to_availability`); pools of their own evict each component's shortfall, computed in the cache from the demand, and a walk stops once its pool can serve its demand, as before. The default for other caches evicts the shortfall of the pools its allocator reads. The joint walk an allocator drives while it reclaims one demand is its own method, `evict_until(params, satisfied)`, instead of a mode of `evict_for_alloc`. Callers state demands: the token and window allocators, the state slot check before binding requests, the host-load and device-alloc evictors, the buffer pipeline, load-back, the decode queue's radix-cache eviction before a preallocation, and the linker loads of the full, window and state components (the window component now names its own slot, not the first window's). The int8 state checkpoint slot is the exception: the tree's states live in the checkpoint pool, which no allocation demand reads, so a full checkpoint pool evicts one cached state by count. The availability-target bookkeeping in the walk is replaced by the demand itself. Tests: a full checkpoint pool with free active slots evicts a cached checkpoint and takes its slot. Test: the default cache evicts a demand's shortfall.
…counts and allocation demands their own types `EvictParams` carried two meanings: what to evict (`evict`, `evict_until`) and what an allocation is about to take (`evict_for_alloc`). The second now has its own type, `AllocDemand`, with the same fields. Both, and `EvictResult`, keep a tree's windows in one map by component id, `window_num_tokens`, the first window included: `swa_num_tokens` and `swa_num_tokens_evicted` are gone, so no window is asked for, or reported, apart from the others. A cache keeping a single pool (the default radix cache, the all-SWA cache) serves a window demand from that pool, as before: the all-SWA cache names its one pool as the window (`all_windows`), so a decode preallocation's window demand evicts from it; callers that only know "the window pool" of a static allocator name every window slot of the tree (`all_windows`), which for a static pool is its one window. `EvictResult.add` replaces the hand-summed accumulations. Tests: a window demand on the all-SWA cache evicts from its single pool.
… in an allocation's token demand The chain priced a preallocation, a host load and a capacity check from two numbers: the owner's tokens and the target window's (`swa_tokens`), with any further window taking the owner's tokens unless a side map named it. A window the caller forgot was silently charged a default. `TokenDemand` (owner tokens, every window's tokens by window pool, state slots) is now the one shape of `prealloc_fits`, `prealloc_ceiling_fits`, `reclaim_for_prealloc`, `reclaim_for_load_back`, `ensure_capacity`, `page_demands` and `scoped_physical_takes`, for the chain and the static pools alike (a static pool's one window goes by its window pool's name). A demand that leaves out a window the allocator keeps, or names one it does not, is an error; so is an `admits` window map that leaves one out. `window_pool_names` lists the windows a demand must name; `TokenDemand.same_tokens` is the extend/decode case where every window keeps the new tokens. `evict_to_free_tokens` loses its per-window keywords. The decode node builds every preallocation demand with one helper (`_prealloc_demand`): with tail preallocation each window takes the tail it keeps of the request (`_window_tails`, its own length widened by a reader reaching further back), plus the decode reservation when admitting; without it a window keeps the whole sequence and takes what the owner takes. Only the target window's admission share is still replaced by the figure the scheduler's window budget prices. The ceiling check now charges a further window its own tail rather than the whole demand. The prefill budget's load-back demand and empty-demand checks name every window. Tests use `sglang.test.token_demand`, which keeps their two-number form (the target window's tokens, every other window the owner's) explicit.
…dow's decode tail by name A decode node's tail preallocation passed the target window's tail separately from the further windows' (`swa_tail_len` beside `window_tail_lens`), and a window neither named kept the whole sequence. `ChainAllocator.alloc_extend_swa_tail` and `KVStorage.take_extend` now take one map naming every window's tail (an omitted window is an error); a static pool's `take_extend` reads its one window's entry. The decode preallocation passes `_window_tails` and sets each window's eviction cursor from its own entry, mapping a window pool to its prefix-tree slot.
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-metrics
branch
from
November 4, 2026 13:25
42221b7 to
45d0b89
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-evict-for-demand
branch
from
November 4, 2026 13:25
9c23582 to
78714f5
Compare
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
Eviction for an allocation took each component's shortfall: callers read a component's capacity, subtracted it from what they meant to take, and the cache added the same read back to recover a target; under unified memory the cache then turned those targets into one demand for the allocator. Allocation demands also had several shapes: the owner's tokens plus the target window's (
swa_tokens), with further windows charged the owner's tokens unless a side map named them, so a window a caller forgot was silently charged a default.Modifications
mem_cache/base_prefix_cache.py,unified_radix_cache.py,pure_swa_radix_cache.py:evict_for_alloc(demand)takes what the allocation will take. Pools sharing memory reclaim the whole demand at once (ChainAllocator.reclaim_for_demand, replacingreclaim_to_availability); pools of their own evict each component's shortfall, computed in the cache. The joint walk an allocator drives while reclaiming is its own method,evict_until(params, satisfied).EvictParams(what to evict) and the newAllocDemand(what an allocation takes) are separate types; both andEvictResultkeep every window in one map by component id (window_num_tokens);swa_num_tokensandswa_num_tokens_evictedare removed. The all-SWA cache names its one pool as the window (all_windows).mem_cache/allocator/base.py:TokenDemand(owner tokens, every window's tokens by window pool, state slots) is the one shape ofprealloc_fits,prealloc_ceiling_fits,reclaim_for_prealloc,reclaim_for_load_back,ensure_capacity,page_demandsandscoped_physical_takes, for the chain and static pools alike; a demand that omits a window the allocator keeps, or names one it does not, is an error.disaggregation/decode.py: every preallocation demand is built by one helper (_prealloc_demand); with tail preallocation each window takes the tail it keeps;ChainAllocator.alloc_extend_swa_tailandKVStorage.take_extendtake one map naming every window's tail.python/sglang/test/token_demand.py(test helper keeping the two-number form explicit), a full checkpoint pool with free active slots evicts a cached checkpoint and takes its slot, the default cache evicts a demand's shortfall, a window demand on the all-SWA cache evicts from its single pool, plus updates across unified allocator and radix-cache tests.No intended behavior change apart from the int8 checkpoint slot case above; this is an internal API change of the prefix caches and allocators.
Accuracy Tests
Serving validation was run at the top of this 26-PR stack (the cumulative change), not at this PR's head. Software: torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10. GSM8K is few-shot, 500 or 1319 questions as noted;
mainis the same commit the stack is based on, same flags.--enable-unified-memory(rebased stack)--enable-unified-memory --max-total-tokens 120000(rebased stack)trtllm_mha(1319 q)On B200 and B300, 40 greedy GSM8K generations were compared with
main: 35–40 of 40 are identical, and the rest first differ after at least 27 characters (near ties). The H200 runs above other than the two marked "rebased stack" were run at the stack top before it was rebased onto the current base; the code they cover is unchanged by the rebase except for conflict resolutions.Every serving row above allocates and evicts through these demand shapes.
Speed Tests and Profiling
The demand is built once per allocation; the availability round-trip it replaces is removed. Measured at the stack top, same flags as
main(bench_serving, random 4096-in / 512-out unless noted):--max-total-tokens 120000Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #41837981509
Latest PR Test (Extra): ❌ Run #41837981493
Latest PR Test (AMD ROCm 10): ❌ Run #41837981508
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: