Skip to content

[Refactor] mem_cache: Address prefix-tree window slots, cursors and evictions by sub-pool name - #48007

Merged
ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-move-safety
Nov 8, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-move-safety

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

The previous PR names every unified window sub-pool by its own window length (window_<len>). The prefix tree and its callers still address windows by fixed roles: ComponentType is a closed enum with a SWA and an AUXILIARY_SWA member, a request's window cursors are keyed by those members, eviction counts carry swa_num_tokens and auxiliary_swa_num_tokens, match results carry swa_host_hit_length and auxiliary_swa_host_hit_length, and the cache answers swa_evictable_size() / auxiliary_swa_evictable_size(). Every new window needs another field, and code that wants "a window" has to know which role it plays. The tail a running request keeps of a window (its length widened by a reader that reaches further back, such as a fused draft) is also computed from the draft's reaches in several places.

This PR makes the tree number its own slots and lets everything outside it name a window by its sub-pool.

Modifications

  • Slot ids (unified_cache/component_type.py, components/base.py, unified_tree_core.py, tree_core_registry.py): ComponentType is an int id. FULL, SWA, MAMBA and C128 are fixed; any other slot name gets the next free id (ComponentType.of), kept for the life of the process. tree_slot_ids numbers a tree's slots from their names. Lock-uuid ranges size themselves to the ids the tree holds (slot_width). A node owns a ComponentData record only for the token component and its tree's own components (node_component_records); every other id holds one shared ABSENT_COMPONENT_DATA, which reads as an empty record and refuses writes. Its LRU pointers are one device and one host pointer per tree component, indexed by the component's position in the tree (lru_pointer_slot), not by its id.
  • Slot names (kv/plan.py): a unified window's prefix-tree slot is named by its sub-pool (window_tree_component); its host pool and PD state type are unchanged. Tree slots list the target's window first, and carry their host pool.
  • By-name tree API (base_prefix_cache.py, unified_radix_cache.py, pure_swa_radix_cache.py, LMCache):
    • EvictParams.window_num_tokens / EvictResult.window_num_tokens_evicted (with EvictResult.add) and MatchResult.window_host_hit_lengths, keyed by sub-pool name outside the tree; the tree's slot-keyed host hits are converted once, by MatchResult.with_window_hits_by_sub_pool;
    • a host load's window reclaim (hybrid_pool_assembler._evict_swa_for_device_alloc) and LMCache's external slots (_allocate_external_slots) name only the window they allocate in, so a fused draft's window is not evicted for a shortfall of the target window (test_hybrid_pool_assembler.py::TestHostLoadEvictsTheTargetWindowOnly);
    • window_slots, window_slot_of, window_sub_pool_of_slot, window_sub_pools, window_lengths, window_evictable_size(name), window_protected_size(name), window_available_size(name), window_reprefill_tail_tokens, insert_window_cursors;
    • swa_evictable_size, auxiliary_swa_evictable_size, swa_protected_size, swa_transient_size and supports_auxiliary_swa are removed.
  • Cursors (managers/schedule_batch.py): ReqKvInfo.window_evicted_seqlens keys each window's cursor by sub-pool name (window_dead_lo, window_evict_floor, shield_window_prefill); Req.window_host_hit_lengths gives each window's host hit. Decode-time retirement checks every window against its own cursor; the HIP MegaMoE periodic eviction keeps its every-forward cadence.
  • Retirement (kv/advance.py): coalesce_ranges and the window retirement (retire_window_rows, by sub-pool name) move from mem_cache/common.py into the KV package; KVStorage.advance and StaticKVStorage.advance take the window's sub-pool.
  • Retention owned by the storage (kv/storage.py, static_kv_storage.py, kv_cache_builder.py, cache_init_params.py, registry.py): the cache builder binds, once at startup, the readers' reaches and the length of a window the model does not state (bind_window_retention). retained_window_lengths answers each window's kept tail, and tree slots and the cache's retained window read it; caches no longer carry draft_window_reaches. A static pool keeps its window's tail on its allocator.
  • Windows alike in a tree with several window slots (components/swa.py, unified_radix_cache.py): every window follows the same rules for its cursor, its prefix-lock release and its inline backup; the locks below the windows (for example Mamba's) go once every window has released its own.
  • Boundaries that keep one window of their own map it by family: the native tree core adapter (rust_tree_core/adapter.py, NATIVE_FAMILY_IDS), the external cache linker and LMCache.
  • Allocators (allocator/base.py, allocator/swa.py, allocator/hisparse.py, kv/alloc/chain.py): window_pool_names() and window_available_size(name); the chain answers its target window by its existing policy and another window from its own band.
  • Callers adapted to the by-name API: the prefill budgets and PrefillAdder (host hits as one map), HiCache host-stack assembly, buffer-mode pipeline, storage prefetch, PD decode preallocation and its pre-allocated token counts, pool stats and the invariant checker, the Mooncake L3 object layout (Scheduler._hicache_l3_layout finds the window by family, so a window named by its sub-pool keeps its trailing-page hit), and the unified allocator's current admission code.
  • Imports inside the KV package go downward: KV_PART, STATE_PART and Region live in kv/decl.py, and the chain allocator owns its window-segment invariants (kv/alloc/invariants.py) instead of borrowing them from the static sliding-window allocator.
  • Tests:
    • new test/registered/unit/mem_cache/kv/test_import_order.py;
    • test_component_split_contract.py::TestNodeRecordsOfTreeComponents (records only for the tree's components, the absent record reads empty and refuses writes, device and host LRU pointers never collide);
    • test_hicache_l3_object_layout.py::test_window_named_by_its_sub_pool_reports_its_trailing_pages;
    • python/sglang/test/tree_slots.py builds a cache's slots from its storage;
    • the release-after-forward test addresses window slots and evictions by sub-pool name;
    • API updates in the unified radix cache kits and tests, tree-core registry, native tree core, cache linker, LMCache cursors, storage prefetch, streaming session, PD wire, request lengths, prefill adder and budgets, declared parts and the allocator tests.

Behaviour changes

  • A tree with several window slots (a fused draft's window of another length beside the target's) treats every window alike: each window's retirement and prefix lock release follow its own cursor, and a window's rows below its prefix floor are freed by the same rule as the target's. Before, the second window used a different floor and different lock rules.
  • In such a tree sliding_window_size still answers the first (target) window's retained length, as on main, because the prefill budgets at this PR price the target window from it; each window's own retained length is window_retain_size(name).
  • A Python tree-core node allocates records and LRU pointers only for its tree's components, so a target plus draft-window node is smaller than on main; a write to the record of a component the tree does not hold now raises.
  • No change without --enable-unified-memory beyond the renamed internal API.

Accuracy Tests

This stack is validated at its top, on main 3e38c5b77a: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed against main, all reported in #48014.

Checklist

  • Format with pre-commit on the changed files.
  • Unit tests added and updated.

CI States

Latest PR Test (Base): 🚫 Run #42476327470
Latest PR Test (Extra): 🚫 Run #42476327518
Latest PR Test (AMD ROCm 10): ❌ Run #42476327407

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang unified-radix-cache labels Nov 3, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from d31e0a2 to b9c1dbc Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 2d10448 to 75012d2 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from b9c1dbc to 87fc647 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch 3 times, most recently from 860b862 to e5b52e0 Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 87fc647 to 1f97321 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from e5b52e0 to d171f97 Compare November 4, 2026 09:31
This was referenced Nov 4, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 0d7429c to 713f9db Compare November 7, 2026 13:11
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from de0d281 to ca0e40d Compare November 7, 2026 22:46
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch 2 times, most recently from 6c87f47 to 1cca01f Compare November 8, 2026 03:58
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from ca0e40d to 9fded2f Compare November 8, 2026 08:09
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-move-safety branch from 1cca01f to 1d4c05c Compare November 8, 2026 08:09
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Required checks failed on 6888e92; this PR cannot be merged until they pass.

Lint (run)
Run pre-commit run --all-files locally and push the result. The run log shows any other failing step.

…victions by sub-pool name

The prefix tree addressed windows by fixed roles: `ComponentType` was a
closed enum with `SWA` and `AUXILIARY_SWA`, a request's window cursors were
keyed by them, eviction counts and match results carried a field per role
(`swa_num_tokens`, `auxiliary_swa_host_hit_length`, ...) and the cache
answered `swa_evictable_size()` beside `auxiliary_swa_evictable_size()`.
The tail a request keeps of a window was computed from a draft's reaches
in several places.

- `ComponentType` is an int id. FULL, SWA, MAMBA and C128 are fixed; any
  other slot name gets the next free id (`ComponentType.of`), and a tree
  numbers its own slots (`tree_slot_ids`). A unified window's slot is named
  by its sub-pool; its host pool and PD state type are unchanged.
- Outside the tree a window is named by its sub-pool: `EvictParams` /
  `EvictResult.window_num_tokens*`, `MatchResult.window_host_hit_lengths`,
  `ReqKvInfo.window_evicted_seqlens`, `Req.window_host_hit_lengths`, and
  the cache's `window_slots`, `window_slot_of`, `window_sub_pool_of_slot`,
  `window_lengths`, `window_evictable_size`, `window_protected_size` and
  `window_available_size`. The per-role reads are removed.
- The storage owns each window's kept tail: the cache builder binds the
  readers' reaches once (`bind_window_retention`), and tree slots and the
  cache's retained window read `retained_window_lengths`.
- Window retirement moves into the KV package (`kv/advance.py`) and takes
  the window's sub-pool.
- In a tree with several window slots every window follows the same rules
  for its cursor, its prefix-lock release and its inline backup; the
  locks below the windows go once every window has released its own.
- The native tree core adapter, the external cache linker and LMCache map
  the tree's one window by family.
- The chain allocator owns its window-segment invariants
  (`kv/alloc/invariants.py`) and imports its allocation kernels from the
  kernels package, so the KV package imports no static allocator.
- Allocators answer `window_pool_names()` and `window_available_size(name)`;
  the admission, budget, PD decode, pool stats and invariant-check callers
  read windows by name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang memory-pool unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant