Repository navigation
[Refactor] mem_cache: Give each window sub-pool its own host pool, PD entry and metrics - #48013
Merged
Merged
Conversation
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 4, 2026 06:18
39809f6 to
164edbf
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 06:18
f82cc57 to
ceff909
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 4, 2026 07:42
164edbf to
87d1e52
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 07:42
ceff909 to
a91493b
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 4, 2026 08:02
87d1e52 to
d5f0cd4
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 08:02
a91493b to
f0221ef
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 4, 2026 09:14
d5f0cd4 to
1e87663
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 09:14
f0221ef to
18c97dd
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 4, 2026 09:31
1e87663 to
b1ed043
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 4, 2026 09:31
18c97dd to
ceb2bef
Compare
This was referenced Nov 4, 2026
Merged
Merged
Merged
Closed
Closed
ch-wan
requested review from
ByronHsu,
ShangmingCai,
billishyahao and
sogalin
as code owners
November 5, 2026 10:16
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 5, 2026 11:25
23cbaff to
b45d6a7
Compare
ch-wan
requested review from
Fridge003,
Qiaolin-Yu,
hebiao064,
yeahdongcn and
yuan-luo
as code owners
November 5, 2026 11:25
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 5, 2026 11:25
4591cce to
13d9991
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 5, 2026 22:11
b45d6a7 to
8f8ce87
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 5, 2026 22:15
13d9991 to
673c3c4
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 6, 2026 00:00
8f8ce87 to
37356ef
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 6, 2026 00:00
673c3c4 to
9a5a3a0
Compare
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 6, 2026 01:16
37356ef to
11bafd4
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 6, 2026 01:16
9a5a3a0 to
8b20905
Compare
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-names
branch
from
November 6, 2026 03:37
11bafd4 to
2fe3264
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-pd-window-entries
branch
from
November 6, 2026 03:37
8b20905 to
f15189b
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
4 of 5 tasks
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
… entry and metrics A unified pool may hold two window sub-pools: the target's window and a fused draft's window of another length. Past the prefix tree, the target's window was still "the" window and the draft's a second, special one: host pools and PD state types went by window slot (`swa`, then `auxiliary_swa`), the pool stats, leak checks and metrics saw one window, PD host receive allocated one window host pool, and the KV checksum digested one window. PD peers paired state entries by position without comparing them. Host pools: - A window sub-pool's host pool is named by the sub-pool (`window_<len>`), so its L3 objects carry that suffix. Host pool names are well-known `PoolName`s or window sub-pool names; `PoolName.AUXILIARY_SWA` is removed. - The host stack builds one entry per window sub-pool and refuses a window sub-pool named like a well-known host pool. Window components, the cache linker, state-eviction host reservations and the retraction backup go by each window's own host pool. Mooncake keys each window pool's objects by name, with its own trailing-page hit; the Rust tree core maps its single window pool name at its boundary. PD state entries: - Every window travels as the `swa` state type, told apart by its sub-pool (`KVArgs.state_sub_pools`), and each entry carries its own sub-pool's pages. `StateType.AUXILIARY_SWA` and `wire_sub_pool` are removed. - Each entry declares its key and kept window tail; the prefill reports its entries to the bootstrap server, and a decode whose entries differ refuses to connect (`check_state_wire`). A prefill keeping a longer tail is accepted and its sends cut each window entry to the decode's newest pages. - Decode host receive takes one host pool per window entry, sized and reserved by that window's kept tail; the KV checksum digests each window entry's own rows. Accounting and metrics: - `PoolStats.windows` holds one entry per window sub-pool; the leak checks conserve every window. - Window pools are published as `sglang:window_*` with a `window` label, and `/server_info` reports `memory_usage.token_capacity_window`, replacing the `swa` series and `token_capacity_swa`; the Rust server follows.
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
With
--enable-unified-memory, a pool may hold two window sub-pools: the target's window and a fused draft's window of another length. Below this PR, everything past the prefix tree still treats the target's window as "the" window and the draft's as a second, special one. Host pools and PD state types go by window slot (swa, thenauxiliary_swa). The pool stats, the leak checks and the metrics see one window (PoolStats.swa_*,sglang:swa_*,token_capacity_swa). PD host receive allocates one window host pool, the KV checksum digests one window, and the PD state entries do not say which sub-pool they carry. Two PD peers also pair state entries by position without comparing them, so peers that hold different window sub-pools or keep different window tails pair the wrong pages without an error.This PR gives every window sub-pool its own host pool, PD entry, accounting and metrics, named by the sub-pool. PD peers now check that their state entries agree before they transfer.
Modifications
mem_cache/kv/plan.py,hicache_storage.py,pool_host/group.py,hybrid_cache/{hybrid_pool_assembler,hybrid_cache_controller}.py,unified_cache/{unified_cache_linker,components/swa}.py,unified_radix_cache.py,rust_tree_core/adapter.py,buffer_mode/pipeline.py,storage/mooncake_store/mooncake_store.py,layout/fused_draft.py,managers/scheduler.py):window_<len>), so its L3 objects carry that suffix; host pool names are well-knownPoolNames or window sub-pool names, read withstr();PoolName.AUXILIARY_SWAis removed;PoolNamevalues and every window sub-pool any rank of the groups holds, agreed once per set of groups (HybridCacheController._host_pool_schema), so pipeline stages holding different window pools reduce tensors of one length, and each named window keeps its own hit boundary. In a hit query a rank without a pool gives no verdict (the pool heals from the folded cut); in an ACK it contributes 0;target_window_host_pool:swafor static pools, the window sub-pool's name for unified ones), for its lookup and for the indices it reuses;storage/nixl/nixl_cleaner.py) strips awindoworwindow_<n>component suffix, with its K/V or index part, before the rank suffix, so a page's window objects on every TP rank fall in the group of its KV and are evicted with it;window.disaggregation/base/conn.py,utils.py,common/conn.py,mooncake/conn.py,nixl/conn.py,mori/conn.py,prefill.py,decode.py,checksum.py,kv/storage.py,static_kv_storage.py):swastate type, told apart by its sub-pool (KVArgs.state_sub_pools,KVStorage.entry_sub_pool), and each entry carries its own sub-pool's pages (state_indices_per_entry);StateType.AUXILIARY_SWAandwire_sub_poolare removed; entries are ordered by state type, then sub-pool name, on both peers;SubPoolExternal.wire_key,KVArgs.state_wire_keys) and the window tail it keeps (KVArgs.state_window_tails);check_state_wire); a decode with fewer pipeline stages than its prefill checks that every prefill stage it receives from registers the decode's leading entries, in order, and that the stages together register all of them (check_stage_state_wires): a stage that owns no layer of a trailing state leaves it out, as static DeepSeek-V4.0's first stage does with its C128 state. A prefill that keeps a longer tail than its decode is accepted, and its send cuts each window entry to the decode's newest pages (align_window_tails) in the Mooncake, NIXL and Mori send paths. A prefill that keeps a shorter tail is refused, and so is a unified pool paired with a non-unified one. An older peer reports no entries and is not checked;KVStorage.state_unit_ids), not the window's rows.scheduler_components/{pool_stats_observer,invariant_checker,metrics_reporter,memory_usage,internal_state}.py,managers/scheduler.py,observability/metrics_collector.py,rust/sglang-server/src/{message,api_server/http}/info.rs):PoolStats.windowslists oneWindowPoolStatsper window sub-pool in the tree's window order; each window's evictable, protected and session-held tokens and its pending PD rows are read by its own name;sglang:window_token_usage,sglang:window_available_tokens,sglang:window_evictable_tokens,sglang:window_used_tokens,sglang:prealloc_window_token_usageandsglang:max_total_num_tokens_window, each with awindowlabel (SubPoolUsage.label,SubPoolExternal.label); a customwindowlabel from--extra-metric-labelsis kept on every other series and replaced by the window's name on these;PoolStats.target_window_stats):sglang:swa_token_usage,sglang:swa_available_tokens,sglang:swa_evictable_tokens,sglang:swa_used_tokens,sglang:prealloc_swa_token_usageandsglang:max_total_num_tokens_swa, with the same labels and values as before;/server_inforeportsmemory_usage.token_capacity_window, a map from window name to capacity, every window the storage holds included (a fused draft's window beside a target without windows), besidememory_usage.token_capacity_swa, which keeps its value; the Rust server's contract carries both.observability/test_window_pool_metrics.py(including a draft-onlywindow_64capacity, and startup emission with a customwindowlabel),disaggregation/test_unified_window_wire.py(including a dense prefill's bootstrap answer, which carries only the fields a decode from before this change knows, and a two-stage static DeepSeek-V4.0 prefill paired with a one-stage decode) anddisaggregation/test_unified_window_lengths.py(including a non-fused window draft's sidecar on a unified pool with named windows);test_pp_prefetch_ticket.py::TestPrefetchAckSchema: two simulated pipeline stages, one holdingwindow_4096and one without it, reduce an empty failure ACK in tensors of one length;test_buffer_mode_sidecar.py::test_a_named_window_already_stored_writes_nothing: two named windows with different hit boundaries; the window whose pages are already stored keeps them and produces no write intent, the shorter one writes its missing page; hit-query fakes reduce over the agreed schema;test_hicache_nixl_cleaner.py: window suffixes parse to the page's group, and a page's KV and named-window objects across four TP ranks are removed together;test_window_pool_metrics.py: the single-window series report the target's window beside three named ones; a static SWA pool reports under them, and intoken_capacity_swa, what it reported before; without windows they stay 0 andNone;windowin the fused-draft placement and geometry tests;state_sub_poolsin the PD test fakes.Behaviour changes
sglang:window_*{window=...}andsglang:max_total_num_tokens_windoware added, one series per window, a fused draft's window included. Thesglang:swa_*series andsglang:max_total_num_tokens_swaare kept and report the target's own window, as before./server_info:memory_usage.token_capacity_windowis added;memory_usage.token_capacity_swais kept with its value.--enable-unified-memory, except the added window metrics and/server_infofield for static hybrid SWA pools.Accuracy Tests
This stack is validated at its top, on
maina4b8801197: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed againstmain, all reported in #48014.Checklist
pre-commiton the changed files.CI States
Latest PR Test (Base): 🚫 Run #42608946623
Latest PR Test (Extra): 🚫 Run #42608946702
Latest PR Test (AMD ROCm 10): 🚫 Run #42608946673
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: