Skip to content

[Refactor] mem_cache: Name each unified window sub-pool by its own window length - #48006

Merged
ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-layout-planner
Nov 8, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/refactor/unified-kv-layout-planner

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

With --enable-unified-memory, the unified KV pool names its window sub-pools by the role they play: the target's window is swa and a draft window of another length is draft_swa. The name says which model the window serves, not which tokens it keeps, so a sub-pool's name depends on what else the plan holds, and the consumers that look for "the window" compare a layer's sub-pool name with the sliding-window role (IdSpaceKind.SLIDING_WINDOW, whose value is also "swa"). The two meanings only coincide while the target's window happens to be named swa.

This PR names every window sub-pool by its own window length, and makes the role a lookup.

Modifications

  • Names (mem_cache/kv/decl.py, kv/plan.py, layout/fused_draft.py): window_sub_pool_name(window) names a window sub-pool window_<len>, or window when the model does not state the length; the target's window and a fused draft's band of another length both take it. KVStoragePlan.primary_window() picks the target's window by rule (the longest window the target declares), and of_lifetime(WindowLifetime) is refused. A draft band that would take a target sub-pool's name is refused. A plan with several target windows is still refused, as before.
  • External names are unchanged: the target's window keeps the SWA prefix-tree component, the swa host pool, PD state type and metrics family; a draft's own window band beside it keeps the AUXILIARY_SWA / auxiliary_swa names.
  • Fused-draft placement (layout/fused_draft.py, kv_cache_configurator.py): a draft window layer that rides in the target's window is hosted by the target's window sub-pool under its name (FusedDraftPlacement.hosted maps it to its host kind). The placement context carries that sub-pool's name, taken from the target's layer declarations, so a draft whose configured window equals the target's rides in it even when the runner resolves the target's window to another length.
  • Roles (kv/router.py, kv/routed_cache.py, kv/build.py, kv/alloc/runner_ids.py, unified_draft_pool.py): the unified pool's layer router maps the sliding-window role to the target's window sub-pool (sub_pool, view, storage_name), and KVLayerRouter.plays(layer_id, role) answers whether a layer lives in the sub-pool playing a role. A fused draft's router routes each layer by its host sub-pool's name and maps the role to its window host. The allocator's id spaces are keyed by sub-pool name only; RunnerIds takes the role from its pool's router, the same way for the target and a fused draft, and RunnerIds.sub_pool_name resolves a role to a sub-pool name for space, ids_in and every plan read.
  • Consumers:
    • Triton, TRT-LLM MHA and Intel AMX backends ask whether a layer plays the sliding-window role instead of comparing its sub-pool name;
    • resolve_window_id_space takes a fused pool's window id space from the runner, so a fused draft whose window has a band of its own reads through that band's ids;
    • the PD KV checksum and the decode KV offload manager take the window id space their pool's router names for the role;
    • PD prefill and decode send the window payloads of the target's window and of a draft's own window by their sub-pool names (window_sub_pool_of, wire_sub_pool);
    • draft readers' window reaches are looked up by the target's window sub-pool name;
    • the hybrid SWA pool configurator prices the fused window entry of the target's window sub-pool.
  • Imports (mem_cache/kv/__init__.py): the package no longer imports its layouts when it is initialized; HostPoolOptions is still exported, loaded on first access. Before this, importing mem_cache/layout/fused_draft.py first failed with an import cycle once it imported the naming helper (kv package → kv.layout.base → the half-loaded fused_draft).
  • Tests:
    • new test/registered/unit/mem_cache/test_unified_window_sub_pools.py (names, external names, draft bands, role lookups, a fused draft's router naming its window host by role);
    • new test/registered/unit/mem_cache/kv/test_fresh_process_imports.py (weekly CPU suite: each KV storage module as the first import of a fresh interpreter);
    • new test/registered/unit/layers/attention/test_trtllm_mha_unified_window.py (TRT-LLM MHA reads and writes a unified window layer through its window ids; SM100);
    • name updates in the fused-draft placement, declared-parts, layout golden (unified_layout_golden.json), byte-budget sizing, pool configurator, tri-pool, post-capture sizing, HiCache, free and recovery tests; layer_router on test fakes of unified pools.

Behaviour changes

  • HiCache L3 objects written for a plan with a fused draft before this change are not found after it: the storage format tag hashes sub-pool names. The cache starts cold once.
  • A fused draft whose window layers have a band of their own reads its window id space from its runner, which names that band; before, a backend resolving the window id space through the allocator got the target's window.
  • No change without --enable-unified-memory.

Accuracy Tests

This stack is validated at its top, on main 3e38c5b77a: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed against main, all reported in #48014.

Checklist

  • Format with pre-commit on the changed files.
  • Unit tests added and updated.

CI States

Latest PR Test (Base): 🚫 Run #42476266246
Latest PR Test (Extra): 🚫 Run #42476266082
Latest PR Test (AMD ROCm 10): ❌ Run #42476266022

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from 6a47923 to 15c3550 Compare November 4, 2026 06:18
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from d31e0a2 to b9c1dbc Compare November 4, 2026 06:18
@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Nov 4, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from 15c3550 to d08c7e3 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from b9c1dbc to 87fc647 Compare November 4, 2026 07:42
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-draft-lifetimes branch from d08c7e3 to 05acb65 Compare November 4, 2026 09:31
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 87fc647 to 1f97321 Compare November 4, 2026 09:31
This was referenced Nov 4, 2026
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 047f333 to d30941a Compare November 6, 2026 10:43
@ch-wan
ch-wan changed the base branch from main to cheng/hot-fix/unified-kv-release-after-forward November 6, 2026 10:43
@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 7d025c6 to 8233a22 Compare November 6, 2026 12:26
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from d30941a to 17e2c20 Compare November 6, 2026 12:31
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 8233a22 to 69d8596 Compare November 6, 2026 13:40
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 17e2c20 to 1082cd7 Compare November 6, 2026 13:40
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 69d8596 to f479939 Compare November 6, 2026 21:21
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 1082cd7 to 4c55cfa Compare November 6, 2026 21:21
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch 5 times, most recently from 8fef929 to 08bff8b Compare November 7, 2026 09:02
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 4c55cfa to 9083b45 Compare November 7, 2026 09:02
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 08bff8b to b440dcd Compare November 7, 2026 11:22
@ch-wan
ch-wan requested a review from yuan-luo as a code owner November 7, 2026 11:22
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 9083b45 to 5cbf54c Compare November 7, 2026 11:22
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from b440dcd to 2be848c Compare November 7, 2026 13:11
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 5cbf54c to de0d281 Compare November 7, 2026 13:11
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 2be848c to cae4a69 Compare November 7, 2026 22:46
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from de0d281 to ca0e40d Compare November 7, 2026 22:46
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from cae4a69 to 7c7eeaa Compare November 8, 2026 08:09
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from ca0e40d to 9fded2f Compare November 8, 2026 08:09
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 7c7eeaa to 13cadd6 Compare November 8, 2026 09:06
@ch-wan
ch-wan force-pushed the cheng/refactor/unified-kv-layout-planner branch from 9fded2f to 040aab3 Compare November 8, 2026 09:06
…ndow length

A unified KV pool named its target window sub-pool "swa" and a draft's
window of another length "draft_swa": names by the role a window plays,
not by the tokens it keeps. Consumers that look for "the window"
compared a layer's sub-pool name with the sliding-window role
(IdSpaceKind.SLIDING_WINDOW, whose value is also "swa"), which held only
while the target's window happened to carry that name.

Every window sub-pool is now named by its own window length,
"window_<len>", or "window" when the model does not state the length
(kv/decl.py::window_sub_pool_name). KVStoragePlan.primary_window picks
the target's window by rule; of_lifetime no longer answers for windows.
A draft band that would take a target sub-pool's name is refused. A plan
with several target windows is still refused.

The names a window goes by outside this process are unchanged: the
target's window keeps the SWA tree component, the swa host pool, PD
state type and metrics family, and a draft's own window beside it keeps
the auxiliary ones.

The sliding-window role becomes a lookup. The unified pool's layer
router maps it to the target's window (sub_pool, view, storage_name),
and KVLayerRouter.plays answers whether a layer lives in the sub-pool
playing a role; the Triton, TRT-LLM MHA and Intel AMX backends ask that
instead of comparing names. The allocator's id spaces are keyed by
sub-pool name only, and RunnerIds.sub_pool_name resolves a role for
space, ids_in and every plan read. The PD KV checksum, the decode KV
offload manager and resolve_window_id_space take the window id space by
the name their pool or runner gives the role.

A fused draft's window layer that rides in the target's window is
hosted by that sub-pool under its name (FusedDraftPlacement.hosted).
The placement context carries the name from the target's layer
declarations, so a draft whose configured window equals the target's
rides in it even when the runner resolves the target's window to
another length. PD payloads, draft readers' window reaches and the
fused window entry the hybrid SWA pool configurator prices are looked
up by the window sub-pool's actual name.

HiCache L3 objects written for a plan with a fused draft are keyed by a
format tag that hashes sub-pool names, so they start cold once.

The KV storage package imports nothing when it is initialized and loads
HostPoolOptions on first access: importing it ran kv.layout.base, which
imports layout/fused_draft.py, so a declaration helper fused_draft.py takes
from the package closed an import cycle whenever fused_draft.py was the
first import of a process. A test imports each KV storage module first in a
fresh interpreter, where a process that already loaded the package cannot
mask the order.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant