Repository navigation
[Refactor] mem_cache: Name each unified window sub-pool by its own window length - #48006
Merged
Merged
Conversation
ch-wan
force-pushed
the
cheng/refactor/unified-kv-draft-lifetimes
branch
from
November 4, 2026 06:18
6a47923 to
15c3550
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 4, 2026 06:18
d31e0a2 to
b9c1dbc
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-draft-lifetimes
branch
from
November 4, 2026 07:42
15c3550 to
d08c7e3
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 4, 2026 07:42
b9c1dbc to
87fc647
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-draft-lifetimes
branch
from
November 4, 2026 09:31
d08c7e3 to
05acb65
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 4, 2026 09:31
87fc647 to
1f97321
Compare
This was referenced Nov 4, 2026
Merged
Merged
Merged
Merged
Closed
Closed
[Refactor] mem_cache: Choose reclaim in one place, keep one static storage, joint LRU reclaim
#48025
Closed
4 of 5 tasks
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 6, 2026 10:43
047f333 to
d30941a
Compare
ch-wan
changed the base branch from
main
to
cheng/hot-fix/unified-kv-release-after-forward
November 6, 2026 10:43
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 6, 2026 12:26
7d025c6 to
8233a22
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 6, 2026 12:31
d30941a to
17e2c20
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 6, 2026 13:40
8233a22 to
69d8596
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 6, 2026 13:40
17e2c20 to
1082cd7
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 6, 2026 21:21
69d8596 to
f479939
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 6, 2026 21:21
1082cd7 to
4c55cfa
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
5 times, most recently
from
November 7, 2026 09:02
8fef929 to
08bff8b
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 7, 2026 09:02
4c55cfa to
9083b45
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 7, 2026 11:22
08bff8b to
b440dcd
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 7, 2026 11:22
9083b45 to
5cbf54c
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 7, 2026 13:11
b440dcd to
2be848c
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 7, 2026 13:11
5cbf54c to
de0d281
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 7, 2026 22:46
2be848c to
cae4a69
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 7, 2026 22:46
de0d281 to
ca0e40d
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 8, 2026 08:09
cae4a69 to
7c7eeaa
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 8, 2026 08:09
ca0e40d to
9fded2f
Compare
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 8, 2026 09:06
7c7eeaa to
13cadd6
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-layout-planner
branch
from
November 8, 2026 09:06
9fded2f to
040aab3
Compare
…ndow length A unified KV pool named its target window sub-pool "swa" and a draft's window of another length "draft_swa": names by the role a window plays, not by the tokens it keeps. Consumers that look for "the window" compared a layer's sub-pool name with the sliding-window role (IdSpaceKind.SLIDING_WINDOW, whose value is also "swa"), which held only while the target's window happened to carry that name. Every window sub-pool is now named by its own window length, "window_<len>", or "window" when the model does not state the length (kv/decl.py::window_sub_pool_name). KVStoragePlan.primary_window picks the target's window by rule; of_lifetime no longer answers for windows. A draft band that would take a target sub-pool's name is refused. A plan with several target windows is still refused. The names a window goes by outside this process are unchanged: the target's window keeps the SWA tree component, the swa host pool, PD state type and metrics family, and a draft's own window beside it keeps the auxiliary ones. The sliding-window role becomes a lookup. The unified pool's layer router maps it to the target's window (sub_pool, view, storage_name), and KVLayerRouter.plays answers whether a layer lives in the sub-pool playing a role; the Triton, TRT-LLM MHA and Intel AMX backends ask that instead of comparing names. The allocator's id spaces are keyed by sub-pool name only, and RunnerIds.sub_pool_name resolves a role for space, ids_in and every plan read. The PD KV checksum, the decode KV offload manager and resolve_window_id_space take the window id space by the name their pool or runner gives the role. A fused draft's window layer that rides in the target's window is hosted by that sub-pool under its name (FusedDraftPlacement.hosted). The placement context carries the name from the target's layer declarations, so a draft whose configured window equals the target's rides in it even when the runner resolves the target's window to another length. PD payloads, draft readers' window reaches and the fused window entry the hybrid SWA pool configurator prices are looked up by the window sub-pool's actual name. HiCache L3 objects written for a plan with a fused draft are keyed by a format tag that hashes sub-pool names, so they start cold once. The KV storage package imports nothing when it is initialized and loads HostPoolOptions on first access: importing it ran kv.layout.base, which imports layout/fused_draft.py, so a declaration helper fused_draft.py takes from the package closed an import cycle whenever fused_draft.py was the first import of a process. A test imports each KV storage module first in a fresh interpreter, where a process that already loaded the package cannot mask the order.
This was referenced Nov 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
With
--enable-unified-memory, the unified KV pool names its window sub-pools by the role they play: the target's window isswaand a draft window of another length isdraft_swa. The name says which model the window serves, not which tokens it keeps, so a sub-pool's name depends on what else the plan holds, and the consumers that look for "the window" compare a layer's sub-pool name with the sliding-window role (IdSpaceKind.SLIDING_WINDOW, whose value is also"swa"). The two meanings only coincide while the target's window happens to be namedswa.This PR names every window sub-pool by its own window length, and makes the role a lookup.
Modifications
mem_cache/kv/decl.py,kv/plan.py,layout/fused_draft.py):window_sub_pool_name(window)names a window sub-poolwindow_<len>, orwindowwhen the model does not state the length; the target's window and a fused draft's band of another length both take it.KVStoragePlan.primary_window()picks the target's window by rule (the longest window the target declares), andof_lifetime(WindowLifetime)is refused. A draft band that would take a target sub-pool's name is refused. A plan with several target windows is still refused, as before.SWAprefix-tree component, theswahost pool, PD state type and metrics family; a draft's own window band beside it keeps theAUXILIARY_SWA/auxiliary_swanames.layout/fused_draft.py,kv_cache_configurator.py): a draft window layer that rides in the target's window is hosted by the target's window sub-pool under its name (FusedDraftPlacement.hostedmaps it to its host kind). The placement context carries that sub-pool's name, taken from the target's layer declarations, so a draft whose configured window equals the target's rides in it even when the runner resolves the target's window to another length.kv/router.py,kv/routed_cache.py,kv/build.py,kv/alloc/runner_ids.py,unified_draft_pool.py): the unified pool's layer router maps the sliding-window role to the target's window sub-pool (sub_pool,view,storage_name), andKVLayerRouter.plays(layer_id, role)answers whether a layer lives in the sub-pool playing a role. A fused draft's router routes each layer by its host sub-pool's name and maps the role to its window host. The allocator's id spaces are keyed by sub-pool name only;RunnerIdstakes the role from its pool's router, the same way for the target and a fused draft, andRunnerIds.sub_pool_nameresolves a role to a sub-pool name forspace,ids_inand every plan read.resolve_window_id_spacetakes a fused pool's window id space from the runner, so a fused draft whose window has a band of its own reads through that band's ids;window_sub_pool_of,wire_sub_pool);mem_cache/kv/__init__.py): the package no longer imports its layouts when it is initialized;HostPoolOptionsis still exported, loaded on first access. Before this, importingmem_cache/layout/fused_draft.pyfirst failed with an import cycle once it imported the naming helper (kvpackage →kv.layout.base→ the half-loadedfused_draft).test/registered/unit/mem_cache/test_unified_window_sub_pools.py(names, external names, draft bands, role lookups, a fused draft's router naming its window host by role);test/registered/unit/mem_cache/kv/test_fresh_process_imports.py(weekly CPU suite: each KV storage module as the first import of a fresh interpreter);test/registered/unit/layers/attention/test_trtllm_mha_unified_window.py(TRT-LLM MHA reads and writes a unified window layer through its window ids; SM100);unified_layout_golden.json), byte-budget sizing, pool configurator, tri-pool, post-capture sizing, HiCache, free and recovery tests;layer_routeron test fakes of unified pools.Behaviour changes
--enable-unified-memory.Accuracy Tests
This stack is validated at its top, on
main3e38c5b77a: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed againstmain, all reported in #48014.Checklist
pre-commiton the changed files.CI States
Latest PR Test (Base): 🚫 Run #42476266246
Latest PR Test (Extra): 🚫 Run #42476266082
Latest PR Test (AMD ROCm 10): ❌ Run #42476266022
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: