Repository navigation
[Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id - #44592
Merged
Merged
Conversation
metamergebot
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
Ying1123,
alphabetc1,
celve,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann,
yizhang2077 and
yuan-luo
as code owners
October 15, 2026 22:23
Collaborator
Author
|
This port clears our bar on gate✓ lint✓ build✓ A✓ B✓ C 0/14⏳ other✓ · accel ✗amd,npu @fei-xx @merrymercy @ZYHowell — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side. |
metamergebot
force-pushed
the
oss-sync-44dc253d26
branch
from
October 16, 2026 07:29
f5717b3 to
74662a5
Compare
…y whole page envelopes by page id The first compaction of the unified memory pool used to pay its first-use costs while serving: the move kernel compiled and loaded, the commit's `.all()` reduction and `_assert_async` ran for the first time, and the kernel's first launch could need device memory outside every pool (the context's local-memory reservation for a spilling kernel). When that first compaction was an urgent one under a full pool, the launch failed with `Triton Error [CUDA]: out of memory` and took the scheduler down. `MultiEndedAllocator.prewarm_compaction()` now runs that work once at startup on the reserved sink page 0 and synchronizes; the scheduler calls it right after `init_memory_pools()` under `--enable-unified-memory`, before attention backend init, graph capture and post-capture KV sizing, so what it allocates is accounted like any other startup allocation. CPU pools return immediately; the hybrid-SWA and Mamba composites fan out to every member that compacts. Compaction also moves whole page envelopes by physical page id now. `UnifiedMHATokenToKVPool`, `UnifiedMLATokenToKVPool` and `UnifiedMambaPool` gain `move_physical_pages(tgt_pages, src_pages, page_size)`; the MHA and MLA versions go straight to the existing `copy_pages` kernel, which addresses each envelope in-kernel and allocates nothing, and the Mamba version moves state slots. The token-id `move_kv_cache` methods stay and delegate to it, and the per-token index expansion in both compaction paths is gone. `MultiEndedAllocator._move_pages` issues a move as consecutive launches of at most `SGLANG_COMPACTION_MAX_PAGES_PER_MOVE` pages (default 4096, clamped to the pool) within one call, ahead of the v2p/p2v remap on the same stream. `copy_pages` no longer specializes on the alignment of its page-id pointers: each program loads one scalar id, and a chunk slice at an odd element offset would otherwise compile a new variant mid-serving. The kernel thus specializes on neither the move size nor the page-id alignment, so the startup launch selects the variant every later move uses. Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
…rtup weight-load test `init_model_worker` now reads `self.enable_unified_memory` to decide whether to prewarm the unified pool's compaction; the startup weight-load test builds its Scheduler with `__new__` and sets only the attributes that step reads, so it sets this one too.
metamergebot
force-pushed
the
oss-sync-44dc253d26
branch
from
October 16, 2026 20:49
74662a5 to
9e6585c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The first compaction of the unified memory pool pays its first-use costs while serving: the move kernel compiles and loads, the commit's
.all()reduction and_assert_asyncrun for the first time, and the kernel's first launch can need device memory outside every pool (the context's local-memory reservation for a spilling kernel). When that first compaction is an urgent one under a full pool, the launch fails withTriton Error [CUDA]: out of memoryand the scheduler goes down with it. Even when it does not fail, the first-use work stalls the scheduler loop.Modifications
MultiEndedAllocator.prewarm_compaction()runs the compaction commit's first-use work once at startup, on the reserved sink page 0 (a two-entry self-copy, so the.all()reduction kernel a real commit uses actually launches), and synchronizes. The scheduler calls it right afterinit_memory_pools()under--enable-unified-memory, before attention backend init, graph capture and post-capture KV sizing, so whatever it allocates is accounted like any other startup allocation. CPU pools return immediately. The hybrid-SWA and Mamba composites fan out to every member that compacts.UnifiedMHATokenToKVPool,UnifiedMLATokenToKVPoolandUnifiedMambaPoolgainmove_physical_pages(tgt_pages, src_pages, page_size). The MHA and MLA versions go straight to the existingcopy_pageskernel, which addresses each page envelope in-kernel and allocates nothing; the Mamba version moves state slots (one slot per page).page_sizemust equal the pool's own page size (asserted). The token-idmove_kv_cachemethods stay and delegate to it. The per-token index expansion (pages[:, None] * ps + offsets) in both compaction paths is gone.MultiEndedAllocator._move_pagesissues a move as consecutive launches of at mostSGLANG_COMPACTION_MAX_PAGES_PER_MOVEpages (new env, default 4096, clamped to the pool's page count) within the same call, so an urgent compaction above the cap still completes before it returns. Copies are issued before the v2p/p2v remap on the same stream, so work ordered after the remap sees the moved pages.copy_pagesno longer specializes on the alignment of its page-id pointers (do_not_specialize_on_alignment): each program loads one scalar id, and a chunk slice at an odd element offset would otherwise compile a new variant mid-serving. The kernel now specializes on neither the move size (a grid dimension) nor the page-id alignment, so the startup launch selects the variant every serving-time move of that sub-pool uses.move_physical_pages).Accuracy Tests
No numerics change: each moved page is the same envelope the previous path copied row by row, now copied once as a whole.
test/registered/unit/mem_cache/test_unified*.py,test_multi_ended_allocator.py,test_kv_index_translator_draft_disposition.py,test_dense_fused_draft_geometry.py): all pass with this change applied.prewarm_compaction()runs; a chunked_move_pages(cap 3 pages per launch, 7 pages moved, 3 launches) and the delegating token-idmove_kv_cacheboth reproduce the torch reference bytes exactly, untouched pages are unchanged, and a page-size mismatch is rejected.Speed Tests and Profiling
The move step is now one
copy_pageslaunch per chunk of page ids and no index temporaries, instead of a per-move token-id expansion plus a per-(layer, token) row copy. The first compaction while serving no longer compiles, loads or allocates.Checklist
CI States
Latest PR Test (Base): ❌ Run #38965078081
Latest PR Test (Extra): ❌ Run #38965077866
Latest PR Test (AMD ROCm 10): 🚫 Run #38965078006