Repository navigation
[Refactor] mem_cache: Gate unified page reuse and moves on one per-band page ledger - #48008
Merged
Merged
Conversation
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 06:18
2d10448 to
75012d2
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 4, 2026 06:18
0e9aae3 to
8df6f60
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 07:42
75012d2 to
6388235
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 4, 2026 07:42
8df6f60 to
4af2ce8
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 08:02
6388235 to
860b862
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
2 times, most recently
from
November 4, 2026 09:14
7e1cf2d to
11c2779
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 09:14
860b862 to
e5b52e0
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 4, 2026 09:31
11c2779 to
8262d14
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 4, 2026 09:31
e5b52e0 to
d171f97
Compare
This was referenced Nov 4, 2026
Merged
Merged
Merged
Merged
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
2 times, most recently
from
November 5, 2026 22:15
e6dadd0 to
0b14d62
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 5, 2026 23:59
ebe3e51 to
a54b0e9
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
3 times, most recently
from
November 6, 2026 03:37
9277df7 to
72f4463
Compare
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 6, 2026 06:33
a54b0e9 to
ce3ae69
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 6, 2026 06:33
72f4463 to
dd8bc46
Compare
This comment has been minimized.
This comment has been minimized.
4 of 5 tasks
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 6, 2026 10:43
ce3ae69 to
551c591
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 6, 2026 10:43
dd8bc46 to
8e4c606
Compare
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 6, 2026 12:31
551c591 to
0b37510
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 6, 2026 12:31
8e4c606 to
6b06dfb
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 6, 2026 13:40
0b37510 to
6c2996f
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 6, 2026 13:40
6b06dfb to
5b1b6fa
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 6, 2026 21:21
6c2996f to
cc7cf65
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-window-rails
branch
from
November 6, 2026 21:21
5b1b6fa to
22dc693
Compare
ch-wan
force-pushed
the
cheng/refactor/unified-kv-move-safety
branch
from
November 7, 2026 09:02
cc7cf65 to
3812bbc
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…nd page ledger Under `--enable-unified-memory`, when a physical page may be written, and by whom, was decided separately by each take, free and move path. Several paths checked nothing: a hole freed while an overlapped forward still wrote it could be handed to a schedule-stream copy or a compaction, a float handed recent holes back to the gap unguarded, and eager compaction ignored pages reserved for a host load. HiCache kept every band's pages in place while any host copy was queued or unacknowledged. Ledger: - Each band keeps one `PageUsage` record (`kv/alloc/page_usage.py`) of who may still touch its pages: unfinished forwards with the ids they write (unknown for overlapped speculative and pipeline verify writes), host copies, pages a move vacated while a reader could still read them (pending pages and fences), and fixed pages. - The scheduler records every launched forward through the storage and retires it when its result is processed; decisions read retirement, never whether an event fired, so every rank makes the same ones. - A move waits only the forwards and copies that write its sources; vacated bytes are fenced for their next writer. Pages that must stay (a host load's reservation) are fixed per page, and compaction moves around them instead of stopping the band; no frontier or float span shrinks past a fixed page. - HiCache window copies name their rows by the owner's token ids and translate them at submission, and submitted copies are recorded on the rows they touch, so the band-wide HiCache pin is removed. PD transfers keep their band-wide pin sources. One acquire and one release path: - `Band.acquire(n, consumer)` is the only path that hands out writable pages and `Band.release` the only one that returns them. What a consumer (a forward, a schedule-stream copy, page zeroing, a host load, a network receive) must wait for each hazard is one table, `page_usage.CLEARANCE`. Holes no hazard makes the consumer wait for are taken first, without a host read when none is hazarded. - New state slots are acquired as schedule-stream copies: a state copy may write a slot an overlapped forward still writes in place. - A float's on-demand moves (`make_room`, `compact_holes`) clear their destinations, wait only writers of their sources, and fence what they vacate. - A band's or a chain's take, a flush and a recovery each run as one allocator operation (the outermost call). Before its first stream wait it reads ahead on the host what it reads after: everything, for an operation that moves pages; for a take, only the free and pending pages of bands a fence may make choose holes on the host, so an ordinary reuse of pending pages reads nothing. A device read on the host after that wait fails at invariant level WARN and up. An unfenced take outside such a read keeps no host copy of the free list. - A take chooses holes from the byte ranges unretired forwards hold, found in one walk of each ledger and swept once over the holes; copy fences order nothing, so they leave the device's choice and only its clearance is read. The clearance walks each ledger once for all the chosen runs, asking each fence's signals once; fences whose readers retired and whose signals completed are dropped at retirement. Lifecycle: - Unified memory requires lazy compaction: startup refuses `SGLANG_DISABLE_LAZY_COMPACTION=1` with `--enable-unified-memory` (before, only PD refused it), and eager end-band compaction is removed. Static pools: - `kv_storage_of` keeps one `StaticKVStorage` per pool, created on first use and rebuilt (keeping its startup bindings) when the request table changes. `shape` and the PD draft pool are bound to it (`bind_shape`, `bind_transfer_draft`) rather than passed to a new instance, and a static window's retention is held by the storage instead of the allocator. - `take_extend` / `take_decode` take `tree_cache` and `batch` per call (a unified pool ignores them); the decode preallocation passes its tree cache through. The static storage answers the ledger's recording calls as no-ops.
This was referenced Nov 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
Under
--enable-unified-memory, the bands of the shared KV arena reuse freed pages and move live pages (compaction of an end, a float making room for its neighbour) while forwards and host copies may still read or write them. Whether a page may be written, and by whom, was decided separately on each take, free and move path, and several paths checked nothing:SGLANG_DISABLE_LAZY_COMPACTION=1) never read the pages reserved for a host load, and could move one under its queued copy;This PR gives each band one record of who may still touch its pages, and routes every hand-out and every return of a page through one path that knows who will write it.
Modifications
static_kv_storage.py,allocation.py,disaggregation/decode.py):kv_storage_ofkeeps oneStaticKVStorageper pool instead of building one per call, so the static pools' startup bindings (window retention, shape, the PD draft pool) live on it;take_extend/take_decodetake the step'stree_cacheandbatchper call. The allocator'swindow_retentionattribute is removed. Each caller ofbind_shapeknows part of the shape (the prefix tree its window and checkpoint interval, the scheduler the capacities), so a binding merges into the one held: a field it leavesNonekeeps its earlier value, and the tree's slots do not depend on the order of the bindings.kv/alloc/page_usage.py,kv/alloc/signals.py,kv/alloc/band.py):PageUsagelists, per band, unfinished forwards with the ids they write, host copies with the rows they touch, pages a move vacated while a reader may still read them (vacated pages that return to the free list once their readers retire, and fences over bytes that left the band), and fixed pages.Hazard.FORWARD: it may still write the page, or read a vacated page's old bytes. A page its owner frees while a forward runs waits in its band's pending batches under that forward ([Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free #48852) and reaches the free list quiet.FREE_OUTSIDE_RESERVATION).PageUsage.forward_retired(event)answers for a forward's event only, from its logical retirement, and refuses any other key;transfer_ended(key)answers for a transfer, and a host copy holds no decision, since whoever writes its pages waits its events once the choice is made. A host copy's key is aStreamTransferaround its events (signals.py), so it is never looked up as a forward; a release key holding forwards and copies is asked withended, member by member.page_usage._owned: forwards, transfers, vacated batches), so a request-table row reused after a free cannot rewrite the ids of a recorded write. Records are made on the stream page moves run on, where that copy is made (stated onBand.record_transferandBand.set_inflight_forward).band.py,page_usage.CLEARANCE):Band.acquire(n, consumer)is the only path that hands out writable pages;Band.releasethe only one that returns them. The consumer is a forward, a schedule-stream copy, page zeroing, a host load or a network receive; one table says what each must wait for each hazard (a forward's late write, a forward's read of vacated bytes, a host copy's read or write, a network transfer, a fixed page). Zeroing waits a forward's late write as a schedule-stream copy does: freed bytes can pass to another band, and a finite FP32 state word read as BF16 attention padding can be a NaN. A wait that two consumers of one take name is issued once.acquirekeeps [Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free #48852's take order: holes; when they are short, the oldest whole pending batches that cover the rest, settled (Band.settle_pending); then the frontier. Every check runs before a settle, so a refused take changes nothing. A planned take of a float extends only into its planned block.allocator_operation): a band take, a chain allocation, a planned or scoped take, a host load's reservation, a flush and the urgent recovery. Before the operation's first stream wait, the bands it touches read ahead once on the host what it will read (_before_wait): an operation that moves pages reads everything (Band._read_ahead: free lists, pending pages, the pages unfinished forwards write, transfer pages); a take in a chain where a fence may meet a span reads only what host hole selection needs; an ordinary take with no fence, a pending settle included, reads nothing. After that wait, a host read of the device is an invariant violation (HOST_READ_BEFORE_WAIT, throughutils/invariants.py::expect_host: logged at invariant level WARN, raised at STRICT). A take keeps the free list's host copy only inside an operation that read it ahead, or while a fence may make it choose holes on the host.PageUsage.take_clearance_runs,clearance): every use is bisected into the take's sorted runs, and each signal is asked once per call. Fences whose readers retired and whose signals completed are dropped at retirement.Band._take_holes_on_host) marks hazarded holes in one sweep of the sorted holes against the sorted fence byte spans, instead of asking the ledger per hole.Band._take_holes): holes no unretired forward makes the consumer wait for come first, then the rest, each class in order from the take's end. The device path and the host path choose the same pages in the same order, so every rank takes the same pages whatever its host copies have done; a fired event only skips a wait on the pages chosen. With no hazarded hole the take reads nothing on the host.settle_pending_for_recovery) also reads ahead, after its one read of the holes and pending pages and before any wait, what each band's moves read on the host: the pages unfinished forwards write and live transfers touch. Inside it, a host read of the free list after a wait raises. An end a pin source holds reclaims nothing on an urgent flush, so it is left out of the read and the settle. The operation carries one read plan (_AllocatorOperation.reads_all) that the recovery and every later wait of the operation follow, so a band the recovery leaves out is never read in full; it reads its geometry only where a fence needs it.kv/alloc/host_free_list.py::HostFreeList: one ordered backing list consumed from both ends by offsets, so a take of a few pages touches only those pages and a free appends; once the consumed ends together pass the pages held, the backing list compacts, so it stays within about twice the pages held. A band keeps it only while a fence it may meet is outstanding (_fence_may_order_takes: an unfinished fence over bytes the band may span,PageUsage.fence_outstanding); a completed fence, or one over bytes the band cannot reach, keeps none. A hole take under such a fence walks from its end until it has enough quiet pages, checking each against the sorted fence spans, and walks the whole list only when too few holes are quiet.Band.virtual_pages_of(token_ids)turns the ids the allocator hands out (widened bydcp_sizeunder decode context parallelism) into virtual pages, for_forward_written_pages,_free_lazy,free_segment,free_group_end,FloatBand.freeand the chain's window and full-segment frees._physical_pages_ofkeeps translating through the v2p table, whose unit is the physical page.ChainAllocator.alloc_state_slots).make_room,compact_holes) clear every destination, wait only writers of their sources, and fence what they vacate; an interior shift, whose destinations include sources vacated earlier in the same pass, waits every running forward.Band.alloc_physical) reads the free list ahead. When holes do not cover it, the band settles its pending pages first, so the reservation takes holes and growth only. If the band's holes and pending pages together still fall short, a relief follows, so the operation counts as moving and every chain band reads ahead before that settle waits; the reserved pages are named exactly, as the span's growth plus the holes that left the free list, from host copies the take kept.managers/scheduler.py,scheduler_pp_mixin.py,managers/utils.py,kv/storage.py,kv/alloc/chain.py): the scheduler records each launched forward through the storage (record_forward, with token and state write ids; unknown for overlapped speculative and pipeline verify writes) and retires it at the processed hook (Scheduler._on_step_processed→ForwardSteps.forward_processed), after the result processor's synchronize, together with the pages pending under it._record_unified_forwardrecords the forward's event with the chain's forward steps too; a pipeline stage carries it in its microbatch metadata (unified_forward_done), and its skipped output records its completion behind it. A speculative forward's draft and verify slots are reserved before the step is launched: no managed page is allocated inside a forward. A static pool's storage answers these as no-ops. Without overlap scheduling, a forward is recorded only under pipeline parallelism: a single stage runs it on the schedule stream and processes its result before the next allocation.cache_controller.py,l2_transfer.py,hybrid_cache/*,unified_radix_cache.py,unified_cache/components/swa.py,rust_tree_core/adapter.py,buffer_mode/pipeline.py,disaggregation/decode_kvcache_offload_manager.py):KVStorage.host_ids), so a move before submission is seen; after it, a row a load-back writes is copied only after that copy, and a row a backup only reads moves at once, its old bytes reused only after the read;KVStorage.record_host_transfer), and a load's window reservation is fixed only until its rows are named by owner ids;KVStorage.record_host_transfer).UnifiedRadixCache.start_restore_kv_cache(..., copy_stream=None)builds the ids and records the copy on the current (scheduler) stream; withcopy_stream, only the submission runs there, aftercopy_stream.wait_stream(current). PD decode's_start_host_restores(host receive and retraction restore) passes the forward stream for the kernel I/O backend instead of entering that stream around the whole restore (mem_cache/common.py,disaggregation/decode.py);set_hicache_transfer_done_event/has_inflight_device_transfersare removed.arg_groups/kv_cache_hook.py,band.py,chain.py,kv/build.py): every unified launch asserts lazy compaction; the eager end-band compaction and every eager branch of the bands are removed, and ends compact lazily by default.test/registered/unit/mem_cache/test_unified_page_writability.py(CPU): the clearance table, each consumer × hazard pair through the ledger, a transfer's key never answered as a forward (test_a_transfer_key_is_never_a_forward), hole takes under an overlapped forward, a compaction into a slot a running forward writes, float moves and hand-back, a state slot reused under a running forward, and the startup refusal;test_unified_page_writability.py::TestTakesReadNoEventCompletion(whether a host copy has finished changes no take: the pages, their order and the owner'savailable_sizeagree),TestFreedPagesAreQuiet(a free during a recorded forward stays out of the holes; a free of a reserved page trips the check) andTestHostLoadReservations(a reservation from pending pages stays fixed through relief; drained holes are reserved exactly, every host read before any wait; a load cancelled after submission keeps its copy hazard);test_unified_transfer_ordering.py(CUDA): float and end moves against delayed host copies, each transfer recorded on the stream the moves run on;test_unified_handout_zeroing.py::TestZeroingWaitsALateStateWrite(CUDA): a forward on its own stream writes FP32 words, whose low halves read as BF16 NaNs, into a state slot after the slot is freed; the attention band grows into the slot's bytes and zeroes its pages; after both streams finish, the pages read zero;test_unified_kv_storage.py::test_a_later_shape_keeps_the_tree_fields_it_leaves_out: a capacity binding after the tree's binding leavestree_slots()unchanged;test_unified_page_writability.py::TestWidenedPagesUnderDcp(CPU, DCP 2): an owner band growing down and one growing up free a page while a forward writes it, and compaction into it must wait that forward;test_unified_fixed_free_pages.py,test_unified_host_transfer_tracking.py,test_unified_scheduling_no_host_sync.py(CUDA, sync-debug mode; each case registers its forward and asserts that the wait it guards ran);managers/test_pp_skip_output_retirement.pycarries the storage's forward event;test_unified_scheduler_hooks.py: the processed step retires the forward;test_unified_handout_zeroing.py: a hand-out takes a pending page, its zeroing queued behind the forward;test_unified_take_clearance_runs.py: one pass over the ledger gives the same waits and remaining fences as run-by-run on randomized takes;test_unified_release_after_forward.py: a two-band planned take reads every band before its first wait; a take's settle reads ahead and waits each forward once with no host read after; whether the forward fired decides only whether a wait is queued; a drained move source keeps the free list's host copy through an urgent flush, at invariant levels WARN and STRICT; an owner shortfall leaves a pinned window's pending pages alone and reads nothing of it;TestAHostLoadThatNeedsRelief: a window-only host load on a full, window and state chain, with pending pages, too few local pages and room a relief recovers from a neighbour, reserves its pages with no host read after the first wait and keeps byte accounting, at WARN and STRICT;test_unified_host_transfer_tracking.py::TestRestoreRecordedWhereMovesReadIt(CPU, stand-in streams): through the real_start_host_restores, the ids and the record are on the scheduler stream and the copy on the forward stream after a forward-waits-scheduler wait (kernel backend), or all on the scheduler stream (direct backend);test_unified_mamba_compaction.py(CUDA) covers moves under delayed forwards and host copies through the ledger, and that the ledger holds a copy of the write ids it is given; HiCache tests cover queued and unacknowledged copies with moving rows;test_unified_release_after_forward.py::test_pinned_ends_are_not_read_when_the_float_settles: with every band pinned, a float's settle under owner pressure reads neither end;TestHostFreeListStorage: prefix and suffix takes against a list oracle, and the storage bound;test_unified_take_host_work.py::TestSmallTakesUnderAFenceScaleWithTheTakes: 256 one-page takes under an outstanding copy fence or an unretired forward touch at most a few pages each, the same at 4096 and 16384 holes, with no device read of the free list; a completed fence and an unreachable one keep no host copy;python/sglang/test/unified_allocator_trace.py,test/manual/mem_cache/unified_trace_compare.py) reads pending pages from the ledger and records lazy compaction only (--lazyis removed); newTestBuilderAdapterRoundTrip(CPU) records and replays seeded traces through the real adapter on the window, Mamba and three-band chains.Behaviour changes
--enable-unified-memorywithSGLANG_DISABLE_LAZY_COMPACTION=1now fails at startup with "--enable-unified-memory requires lazy compaction; unset SGLANG_DISABLE_LAZY_COMPACTION." Before, only PD disaggregation refused it.GEOMETRY_READY): logged at invariant level WARN, raised at STRICT.--enable-unified-memorybeyond the static storage being kept per pool.Accuracy Tests
This stack is validated at its top, on
main3e38c5b77a: the affected unit tests at the default and strict invariant levels, the CPU unit suite, the page-safety stress run and sweep, and serving accuracy and speed againstmain, all reported in #48014.Checklist
pre-commiton the changed files.CI States
Latest PR Test (Base): 🚫 Run #42476407492
Latest PR Test (Extra): 🚫 Run #42476407402
Latest PR Test (AMD ROCm 10): 🚫 Run #42476407337
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: