Repository navigation
[Refactor] mem_cache: Address prefix-tree window slots, cursors and evictions by sub-pool name - #48007
Merged
Conversation
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 4, 2026 06:18
d31e0a2 to
b9c1dbc
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 06:18
2d10448 to
75012d2
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 4, 2026 07:42
b9c1dbc to
87fc647
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
3 times, most recently
from
November 4, 2026 09:14
860b862 to
e5b52e0
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 4, 2026 09:31
87fc647 to
1f97321
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 09:31
e5b52e0 to
d171f97
Compare
This was referenced Nov 4, 2026
Merged
Merged
Merged
Merged
Closed
Closed
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 7, 2026 13:11
0d7429c to
713f9db
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 7, 2026 22:46
de0d281 to
ca0e40d
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
2 times, most recently
from
November 8, 2026 03:58
6c87f47 to
1cca01f
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 8, 2026 08:09
ca0e40d to
9fded2f
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 8, 2026 08:09
1cca01f to
1d4c05c
Compare
Contributor
…victions by sub-pool name The prefix tree addressed windows by fixed roles: `ComponentType` was a closed enum with `SWA` and `AUXILIARY_SWA`, a request's window cursors were keyed by them, eviction counts and match results carried a field per role (`swa_num_tokens`, `auxiliary_swa_host_hit_length`, ...) and the cache answered `swa_evictable_size()` beside `auxiliary_swa_evictable_size()`. The tail a request keeps of a window was computed from a draft's reaches in several places. - `ComponentType` is an int id. FULL, SWA, MAMBA and C128 are fixed; any other slot name gets the next free id (`ComponentType.of`), and a tree numbers its own slots (`tree_slot_ids`). A unified window's slot is named by its sub-pool; its host pool and PD state type are unchanged. - Outside the tree a window is named by its sub-pool: `EvictParams` / `EvictResult.window_num_tokens*`, `MatchResult.window_host_hit_lengths`, `ReqKvInfo.window_evicted_seqlens`, `Req.window_host_hit_lengths`, and the cache's `window_slots`, `window_slot_of`, `window_sub_pool_of_slot`, `window_lengths`, `window_evictable_size`, `window_protected_size` and `window_available_size`. The per-role reads are removed. - The storage owns each window's kept tail: the cache builder binds the readers' reaches once (`bind_window_retention`), and tree slots and the cache's retained window read `retained_window_lengths`. - Window retirement moves into the KV package (`kv/advance.py`) and takes the window's sub-pool. - In a tree with several window slots every window follows the same rules for its cursor, its prefix-lock release and its inline backup; the locks below the windows go once every window has released its own. - The native tree core adapter, the external cache linker and LMCache map the tree's one window by family. - The chain allocator owns its window-segment invariants (`kv/alloc/invariants.py`) and imports its allocation kernels from the kernels package, so the KV package imports no static allocator. - Allocators answer `window_pool_names()` and `window_available_size(name)`; the admission, budget, PD decode, pool stats and invariant-check callers read windows by name.
This was referenced Nov 8, 2026
5 tasks
This was referenced Nov 9, 2026
This was referenced Nov 10, 2026
3 of 5 tasks
This was referenced Nov 11, 2026
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
The previous PR names every unified window sub-pool by its own window length (
window_<len>). The prefix tree and its callers still address windows by fixed roles:ComponentTypeis a closed enum with aSWAand anAUXILIARY_SWAmember, a request's window cursors are keyed by those members, eviction counts carryswa_num_tokensandauxiliary_swa_num_tokens, match results carryswa_host_hit_lengthandauxiliary_swa_host_hit_length, and the cache answersswa_evictable_size()/auxiliary_swa_evictable_size(). Every new window needs another field, and code that wants "a window" has to know which role it plays. The tail a running request keeps of a window (its length widened by a reader that reaches further back, such as a fused draft) is also computed from the draft's reaches in several places.This PR makes the tree number its own slots and lets everything outside it name a window by its sub-pool.
Modifications
unified_cache/component_type.py,components/base.py,unified_tree_core.py,tree_core_registry.py):ComponentTypeis an int id.FULL,SWA,MAMBAandC128are fixed; any other slot name gets the next free id (ComponentType.of), kept for the life of the process.tree_slot_idsnumbers a tree's slots from their names. Lock-uuid ranges size themselves to the ids the tree holds (slot_width). A node owns aComponentDatarecord only for the token component and its tree's own components (node_component_records); every other id holds one sharedABSENT_COMPONENT_DATA, which reads as an empty record and refuses writes. Its LRU pointers are one device and one host pointer per tree component, indexed by the component's position in the tree (lru_pointer_slot), not by its id.kv/plan.py): a unified window's prefix-tree slot is named by its sub-pool (window_tree_component); its host pool and PD state type are unchanged. Tree slots list the target's window first, and carry their host pool.base_prefix_cache.py,unified_radix_cache.py,pure_swa_radix_cache.py, LMCache):EvictParams.window_num_tokens/EvictResult.window_num_tokens_evicted(withEvictResult.add) andMatchResult.window_host_hit_lengths, keyed by sub-pool name outside the tree; the tree's slot-keyed host hits are converted once, byMatchResult.with_window_hits_by_sub_pool;hybrid_pool_assembler._evict_swa_for_device_alloc) and LMCache's external slots (_allocate_external_slots) name only the window they allocate in, so a fused draft's window is not evicted for a shortfall of the target window (test_hybrid_pool_assembler.py::TestHostLoadEvictsTheTargetWindowOnly);window_slots,window_slot_of,window_sub_pool_of_slot,window_sub_pools,window_lengths,window_evictable_size(name),window_protected_size(name),window_available_size(name),window_reprefill_tail_tokens,insert_window_cursors;swa_evictable_size,auxiliary_swa_evictable_size,swa_protected_size,swa_transient_sizeandsupports_auxiliary_swaare removed.managers/schedule_batch.py):ReqKvInfo.window_evicted_seqlenskeys each window's cursor by sub-pool name (window_dead_lo,window_evict_floor,shield_window_prefill);Req.window_host_hit_lengthsgives each window's host hit. Decode-time retirement checks every window against its own cursor; the HIP MegaMoE periodic eviction keeps its every-forward cadence.kv/advance.py):coalesce_rangesand the window retirement (retire_window_rows, by sub-pool name) move frommem_cache/common.pyinto the KV package;KVStorage.advanceandStaticKVStorage.advancetake the window's sub-pool.kv/storage.py,static_kv_storage.py,kv_cache_builder.py,cache_init_params.py,registry.py): the cache builder binds, once at startup, the readers' reaches and the length of a window the model does not state (bind_window_retention).retained_window_lengthsanswers each window's kept tail, and tree slots and the cache's retained window read it; caches no longer carrydraft_window_reaches. A static pool keeps its window's tail on its allocator.components/swa.py,unified_radix_cache.py): every window follows the same rules for its cursor, its prefix-lock release and its inline backup; the locks below the windows (for example Mamba's) go once every window has released its own.rust_tree_core/adapter.py,NATIVE_FAMILY_IDS), the external cache linker and LMCache.allocator/base.py,allocator/swa.py,allocator/hisparse.py,kv/alloc/chain.py):window_pool_names()andwindow_available_size(name); the chain answers its target window by its existing policy and another window from its own band.PrefillAdder(host hits as one map), HiCache host-stack assembly, buffer-mode pipeline, storage prefetch, PD decode preallocation and its pre-allocated token counts, pool stats and the invariant checker, the Mooncake L3 object layout (Scheduler._hicache_l3_layoutfinds the window by family, so a window named by its sub-pool keeps its trailing-page hit), and the unified allocator's current admission code.KV_PART,STATE_PARTandRegionlive inkv/decl.py, and the chain allocator owns its window-segment invariants (kv/alloc/invariants.py) instead of borrowing them from the static sliding-window allocator.test/registered/unit/mem_cache/kv/test_import_order.py;test_component_split_contract.py::TestNodeRecordsOfTreeComponents(records only for the tree's components, the absent record reads empty and refuses writes, device and host LRU pointers never collide);test_hicache_l3_object_layout.py::test_window_named_by_its_sub_pool_reports_its_trailing_pages;python/sglang/test/tree_slots.pybuilds a cache's slots from its storage;Behaviour changes
sliding_window_sizestill answers the first (target) window's retained length, as onmain, because the prefill budgets at this PR price the target window from it; each window's own retained length iswindow_retain_size(name).main; a write to the record of a component the tree does not hold now raises.--enable-unified-memorybeyond the renamed internal API.Accuracy Tests
This stack is validated at its top, on
main3e38c5b77a: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed againstmain, all reported in #48014.Checklist
pre-commiton the changed files.CI States
Latest PR Test (Base): 🚫 Run #42476327470
Latest PR Test (Extra): 🚫 Run #42476327518
Latest PR Test (AMD ROCm 10): ❌ Run #42476327407
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: