Skip to content

[Refactor] mem_cache: Give each window sub-pool its own host pool, PD entry and metrics - #48013

Merged
ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-pd-window-entries
Nov 9, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-pd-window-entries

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

With --enable-unified-memory, a pool may hold two window sub-pools: the target's window and a fused draft's window of another length. Below this PR, everything past the prefix tree still treats the target's window as "the" window and the draft's as a second, special one. Host pools and PD state types go by window slot (swa, then auxiliary_swa). The pool stats, the leak checks and the metrics see one window (PoolStats.swa_*, sglang:swa_*, token_capacity_swa). PD host receive allocates one window host pool, the KV checksum digests one window, and the PD state entries do not say which sub-pool they carry. Two PD peers also pair state entries by position without comparing them, so peers that hold different window sub-pools or keep different window tails pair the wrong pages without an error.

This PR gives every window sub-pool its own host pool, PD entry, accounting and metrics, named by the sub-pool. PD peers now check that their state entries agree before they transfer.

Modifications

  • Host pools (mem_cache/kv/plan.py, hicache_storage.py, pool_host/group.py, hybrid_cache/{hybrid_pool_assembler,hybrid_cache_controller}.py, unified_cache/{unified_cache_linker,components/swa}.py, unified_radix_cache.py, rust_tree_core/adapter.py, buffer_mode/pipeline.py, storage/mooncake_store/mooncake_store.py, layout/fused_draft.py, managers/scheduler.py):
    • a window sub-pool's host pool is named by the sub-pool (window_<len>), so its L3 objects carry that suffix; host pool names are well-known PoolNames or window sub-pool names, read with str(); PoolName.AUXILIARY_SWA is removed;
    • a prefetch hit query and a prefetch ACK are each reduced over one ordered list of pool names on every rank of their sync groups: the PoolName values and every window sub-pool any rank of the groups holds, agreed once per set of groups (HybridCacheController._host_pool_schema), so pipeline stages holding different window pools reduce tensors of one length, and each named window keeps its own hit boundary. In a hit query a rank without a pool gives no verdict (the pool heals from the folded cut); in an ACK it contributes 0;
    • the host stack builds one entry per window sub-pool, adds the draft-window sidecar that addresses the owner's ids once, and refuses a window sub-pool whose name is a well-known host pool name; a non-fused window draft's sidecar follows the target window's own host pool (target_window_host_pool: swa for static pools, the window sub-pool's name for unified ones), for its lookup and for the indices it reuses;
    • window components, the external cache linker, state-eviction host reservations and the retraction-backup check go by each window's own host pool; Mooncake keys each window pool's objects by name and gives each its own trailing-page hit; the Rust tree core maps its single window pool name at its boundary;
    • the NIXL FILE cleaner (storage/nixl/nixl_cleaner.py) strips a window or window_<n> component suffix, with its K/V or index part, before the rank suffix, so a page's window objects on every TP rank fall in the group of its KV and are evicted with it;
    • the fused-draft window host kind is named window.
  • PD state entries (disaggregation/base/conn.py, utils.py, common/conn.py, mooncake/conn.py, nixl/conn.py, mori/conn.py, prefill.py, decode.py, checksum.py, kv/storage.py, static_kv_storage.py):
    • every window travels as the swa state type, told apart by its sub-pool (KVArgs.state_sub_pools, KVStorage.entry_sub_pool), and each entry carries its own sub-pool's pages (state_indices_per_entry); StateType.AUXILIARY_SWA and wire_sub_pool are removed; entries are ordered by state type, then sub-pool name, on both peers;
    • each entry declares the key peers tell entries apart by (SubPoolExternal.wire_key, KVArgs.state_wire_keys) and the window tail it keeps (KVArgs.state_window_tails);
    • the prefill reports its state entries to the bootstrap server, per pipeline stage (a prefill without state entries leaves them out of its registration and its bootstrap answer, as before this change), and a decode whose entries differ refuses to connect, naming both lists (check_state_wire); a decode with fewer pipeline stages than its prefill checks that every prefill stage it receives from registers the decode's leading entries, in order, and that the stages together register all of them (check_stage_state_wires): a stage that owns no layer of a trailing state leaves it out, as static DeepSeek-V4.0's first stage does with its C128 state. A prefill that keeps a longer tail than its decode is accepted, and its send cuts each window entry to the decode's newest pages (align_window_tails) in the Mooncake, NIXL and Mori send paths. A prefill that keeps a shorter tail is refused, and so is a unified pool paired with a non-unified one. An older peer reports no entries and is not checked;
    • decode host receive takes one host pool per window state entry, sized and reserved by that window's own kept tail. It serves the windows whose host pool shares the full KV pool's host backing, the chain's end windows; a pool with any other window is refused at startup with a message naming those windows (a FIXME records what receiving into them needs: an envelope host layout per window and a peer-write gate on each distinct backing);
    • the KV checksum digests each window entry's own rows, and each state entry the slots its transfer sends (KVStorage.state_unit_ids), not the window's rows.
  • Accounting and metrics (scheduler_components/{pool_stats_observer,invariant_checker,metrics_reporter,memory_usage,internal_state}.py, managers/scheduler.py, observability/metrics_collector.py, rust/sglang-server/src/{message,api_server/http}/info.rs):
    • PoolStats.windows lists one WindowPoolStats per window sub-pool in the tree's window order; each window's evictable, protected and session-held tokens and its pending PD rows are read by its own name;
    • the busy and idle leak checks conserve every window;
    • every window pool is published under its sub-pool's name, as sglang:window_token_usage, sglang:window_available_tokens, sglang:window_evictable_tokens, sglang:window_used_tokens, sglang:prealloc_window_token_usage and sglang:max_total_num_tokens_window, each with a window label (SubPoolUsage.label, SubPoolExternal.label); a custom window label from --extra-metric-labels is kept on every other series and replaced by the window's name on these;
    • the single-window series stay, for the target's own window (PoolStats.target_window_stats): sglang:swa_token_usage, sglang:swa_available_tokens, sglang:swa_evictable_tokens, sglang:swa_used_tokens, sglang:prealloc_swa_token_usage and sglang:max_total_num_tokens_swa, with the same labels and values as before;
    • /server_info reports memory_usage.token_capacity_window, a map from window name to capacity, every window the storage holds included (a fused draft's window beside a target without windows), beside memory_usage.token_capacity_swa, which keeps its value; the Rust server's contract carries both.
  • Tests:
    • new observability/test_window_pool_metrics.py (including a draft-only window_64 capacity, and startup emission with a custom window label), disaggregation/test_unified_window_wire.py (including a dense prefill's bootstrap answer, which carries only the fields a decode from before this change knows, and a two-stage static DeepSeek-V4.0 prefill paired with a one-stage decode) and disaggregation/test_unified_window_lengths.py (including a non-fused window draft's sidecar on a unified pool with named windows);
    • test_pp_prefetch_ticket.py::TestPrefetchAckSchema: two simulated pipeline stages, one holding window_4096 and one without it, reduce an empty failure ACK in tensors of one length;
    • test_buffer_mode_sidecar.py::test_a_named_window_already_stored_writes_nothing: two named windows with different hit boundaries; the window whose pages are already stored keeps them and produces no write intent, the shorter one writes its missing page; hit-query fakes reduce over the agreed schema;
    • test_hicache_nixl_cleaner.py: window suffixes parse to the page's group, and a page's KV and named-window objects across four TP ranks are removed together;
    • test_window_pool_metrics.py: the single-window series report the target's window beside three named ones; a static SWA pool reports under them, and in token_capacity_swa, what it reported before; without windows they stay 0 and None;
    • per-window cases in the invariant checker, prealloc usage, forward-pass metrics, KV checksum, retraction backup (including a window tail widened by a fused draft, received into host memory and restored), linker, buffer tail and L3 layout tests;
    • host pool, state type and metric names in the HiCache, post-capture sizing, storage, assembler, draft-gate, layout golden, window sub-pool and endpoint tests; the fused-draft window host kind window in the fused-draft placement and geometry tests; state_sub_pools in the PD test fakes.

Behaviour changes

  • Metrics: sglang:window_*{window=...} and sglang:max_total_num_tokens_window are added, one series per window, a fused draft's window included. The sglang:swa_* series and sglang:max_total_num_tokens_swa are kept and report the target's own window, as before.
  • /server_info: memory_usage.token_capacity_window is added; memory_usage.token_capacity_swa is kept with its value.
  • HiCache L3: window pools' objects are keyed by the window sub-pool's name, so objects written before this change are not found (a cold cache once).
  • PD: peers whose state entries or window tails disagree are refused at bootstrap instead of transferring; a peer from before this change is not checked. A prefill without state entries (a dense model) answers bootstrap with the same fields as before. PD requires the same version on both sides. A decode node receiving into host memory keeps each window's own tail in its own host pool; a unified pool with a window outside the chain's ends refuses host receive at startup.
  • No change without --enable-unified-memory, except the added window metrics and /server_info field for static hybrid SWA pools.

Accuracy Tests

This stack is validated at its top, on main a4b8801197: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed against main, all reported in #48014.

Checklist

  • Format with pre-commit on the changed files.
  • Unit tests added and updated.

CI States

Latest PR Test (Base): 🚫 Run #42608946623
Latest PR Test (Extra): 🚫 Run #42608946702
Latest PR Test (AMD ROCm 10): 🚫 Run #42608946673

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 39809f6 to 164edbf Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from f82cc57 to ceff909 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 164edbf to 87d1e52 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from ceff909 to a91493b Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 87d1e52 to d5f0cd4 Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from a91493b to f0221ef Compare November 4, 2026 08:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from d5f0cd4 to 1e87663 Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from f0221ef to 18c97dd Compare November 4, 2026 09:14
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 1e87663 to b1ed043 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 18c97dd to ceb2bef Compare November 4, 2026 09:31
This was referenced Nov 4, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 23cbaff to b45d6a7 Compare November 5, 2026 11:25
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 4591cce to 13d9991 Compare November 5, 2026 11:25
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from b45d6a7 to 8f8ce87 Compare November 5, 2026 22:11
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 13d9991 to 673c3c4 Compare November 5, 2026 22:15
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 8f8ce87 to 37356ef Compare November 6, 2026 00:00
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 673c3c4 to 9a5a3a0 Compare November 6, 2026 00:00
@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 37356ef to 11bafd4 Compare November 6, 2026 01:16
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 9a5a3a0 to 8b20905 Compare November 6, 2026 01:16
@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-window-names branch from 11bafd4 to 2fe3264 Compare November 6, 2026 03:37
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-pd-window-entries branch from 8b20905 to f15189b Compare November 6, 2026 03:37
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

… entry and metrics

A unified pool may hold two window sub-pools: the target's window and a fused
draft's window of another length. Past the prefix tree, the target's window
was still "the" window and the draft's a second, special one: host pools and
PD state types went by window slot (`swa`, then `auxiliary_swa`), the pool
stats, leak checks and metrics saw one window, PD host receive allocated one
window host pool, and the KV checksum digested one window. PD peers paired
state entries by position without comparing them.

Host pools:
- A window sub-pool's host pool is named by the sub-pool (`window_<len>`),
  so its L3 objects carry that suffix. Host pool names are well-known
  `PoolName`s or window sub-pool names; `PoolName.AUXILIARY_SWA` is removed.
- The host stack builds one entry per window sub-pool and refuses a window
  sub-pool named like a well-known host pool. Window components, the cache
  linker, state-eviction host reservations and the retraction backup go by
  each window's own host pool. Mooncake keys each window pool's objects by
  name, with its own trailing-page hit; the Rust tree core maps its single
  window pool name at its boundary.

PD state entries:
- Every window travels as the `swa` state type, told apart by its sub-pool
  (`KVArgs.state_sub_pools`), and each entry carries its own sub-pool's
  pages. `StateType.AUXILIARY_SWA` and `wire_sub_pool` are removed.
- Each entry declares its key and kept window tail; the prefill reports its
  entries to the bootstrap server, and a decode whose entries differ refuses
  to connect (`check_state_wire`). A prefill keeping a longer tail is
  accepted and its sends cut each window entry to the decode's newest pages.
- Decode host receive takes one host pool per window entry, sized and
  reserved by that window's kept tail; the KV checksum digests each window
  entry's own rows.

Accounting and metrics:
- `PoolStats.windows` holds one entry per window sub-pool; the leak checks
  conserve every window.
- Window pools are published as `sglang:window_*` with a `window` label, and
  `/server_info` reports `memory_usage.token_capacity_window`, replacing the
  `swa` series and `token_capacity_swa`; the Rust server follows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant