Skip to content

[Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id - #44592

Merged
merrymercy merged 2 commits into
mainfrom
oss-sync-44dc253d26
Oct 16, 2026
Merged

merrymercy merged 2 commits into
mainfrom
oss-sync-44dc253d26

Conversation

@metamergebot

@metamergebot metamergebot commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The first compaction of the unified memory pool pays its first-use costs while serving: the move kernel compiles and loads, the commit's .all() reduction and _assert_async run for the first time, and the kernel's first launch can need device memory outside every pool (the context's local-memory reservation for a spilling kernel). When that first compaction is an urgent one under a full pool, the launch fails with Triton Error [CUDA]: out of memory and the scheduler goes down with it. Even when it does not fail, the first-use work stalls the scheduler loop.

Modifications

  • Startup prewarm. MultiEndedAllocator.prewarm_compaction() runs the compaction commit's first-use work once at startup, on the reserved sink page 0 (a two-entry self-copy, so the .all() reduction kernel a real commit uses actually launches), and synchronizes. The scheduler calls it right after init_memory_pools() under --enable-unified-memory, before attention backend init, graph capture and post-capture KV sizing, so whatever it allocates is accounted like any other startup allocation. CPU pools return immediately. The hybrid-SWA and Mamba composites fan out to every member that compacts.
  • Whole-envelope page moves. UnifiedMHATokenToKVPool, UnifiedMLATokenToKVPool and UnifiedMambaPool gain move_physical_pages(tgt_pages, src_pages, page_size). The MHA and MLA versions go straight to the existing copy_pages kernel, which addresses each page envelope in-kernel and allocates nothing; the Mamba version moves state slots (one slot per page). page_size must equal the pool's own page size (asserted). The token-id move_kv_cache methods stay and delegate to it. The per-token index expansion (pages[:, None] * ps + offsets) in both compaction paths is gone.
  • Bounded launches. MultiEndedAllocator._move_pages issues a move as consecutive launches of at most SGLANG_COMPACTION_MAX_PAGES_PER_MOVE pages (new env, default 4096, clamped to the pool's page count) within the same call, so an urgent compaction above the cap still completes before it returns. Copies are issued before the v2p/p2v remap on the same stream, so work ordered after the remap sees the moved pages.
  • One kernel variant. copy_pages no longer specializes on the alignment of its page-id pointers (do_not_specialize_on_alignment): each program loads one scalar id, and a chunk slice at an odd element offset would otherwise compile a new variant mid-serving. The kernel now specializes on neither the move size (a grid dimension) nor the page-id alignment, so the startup launch selects the variant every serving-time move of that sub-pool uses.
  • Test fakes and patch targets follow the renamed contract (move_physical_pages).

Accuracy Tests

No numerics change: each moved page is the same envelope the previous path copied row by row, now copied once as a whole.

  • CPU-only (test/registered/unit/mem_cache/test_unified*.py, test_multi_ended_allocator.py, test_kv_index_translator_draft_disposition.py, test_dense_fused_draft_geometry.py): all pass with this change applied.
  • GPU smoke on one CUDA device with real unified full+SWA pools at page sizes 1, 16 and 128: prewarm_compaction() runs; a chunked _move_pages (cap 3 pages per launch, 7 pages moved, 3 launches) and the delegating token-id move_kv_cache both reproduce the torch reference bytes exactly, untouched pages are unchanged, and a page-size mismatch is rejected.

Speed Tests and Profiling

The move step is now one copy_pages launch per chunk of page ids and no index temporaries, instead of a per-move token-id expansion plus a per-(layer, token) row copy. The first compaction while serving no longer compiles, loads or allocates.

Checklist


CI States

Latest PR Test (Base): ❌ Run #38965078081
Latest PR Test (Extra): ❌ Run #38965077866
Latest PR Test (AMD ROCm 10): 🚫 Run #38965078006

@metamergebot metamergebot added the run-ci CI: run the baseline test suite on this PR label Oct 15, 2026
@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang jit-kernel memory-pool labels Oct 15, 2026
@metamergebot metamergebot added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 16, 2026
@metamergebot

Copy link
Copy Markdown
Collaborator Author

This port clears our bar on f5717b3171ca6adb0665af81be61281224c9a97a: every base-a and base-b job is green.
base-c and the accelerator suites are reported but not gated, because they run
on scarce hardware and upstream merges through them routinely.

gate✓ lint✓ build✓ A✓ B✓ C 0/14⏳ other✓ · accel ✗amd,npu

@fei-xx @merrymercy @ZYHowell — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side.

metamergebot and others added 2 commits October 16, 2026 13:49
…y whole page envelopes by page id

The first compaction of the unified memory pool used to pay its first-use
costs while serving: the move kernel compiled and loaded, the commit's
`.all()` reduction and `_assert_async` ran for the first time, and the
kernel's first launch could need device memory outside every pool (the
context's local-memory reservation for a spilling kernel). When that first
compaction was an urgent one under a full pool, the launch failed with
`Triton Error [CUDA]: out of memory` and took the scheduler down.

`MultiEndedAllocator.prewarm_compaction()` now runs that work once at
startup on the reserved sink page 0 and synchronizes; the scheduler calls it
right after `init_memory_pools()` under `--enable-unified-memory`, before
attention backend init, graph capture and post-capture KV sizing, so what it
allocates is accounted like any other startup allocation. CPU pools return
immediately; the hybrid-SWA and Mamba composites fan out to every member
that compacts.

Compaction also moves whole page envelopes by physical page id now.
`UnifiedMHATokenToKVPool`, `UnifiedMLATokenToKVPool` and `UnifiedMambaPool`
gain `move_physical_pages(tgt_pages, src_pages, page_size)`; the MHA and MLA
versions go straight to the existing `copy_pages` kernel, which addresses
each envelope in-kernel and allocates nothing, and the Mamba version moves
state slots. The token-id `move_kv_cache` methods stay and delegate to it,
and the per-token index expansion in both compaction paths is gone.
`MultiEndedAllocator._move_pages` issues a move as consecutive launches of
at most `SGLANG_COMPACTION_MAX_PAGES_PER_MOVE` pages (default 4096, clamped
to the pool) within one call, ahead of the v2p/p2v remap on the same stream.

`copy_pages` no longer specializes on the alignment of its page-id
pointers: each program loads one scalar id, and a chunk slice at an odd
element offset would otherwise compile a new variant mid-serving. The
kernel thus specializes on neither the move size nor the page-id alignment,
so the startup launch selects the variant every later move uses.

Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
…rtup weight-load test

`init_model_worker` now reads `self.enable_unified_memory` to decide whether
to prewarm the unified pool's compaction; the startup weight-load test
builds its Scheduler with `__new__` and sets only the attributes that step
reads, so it sets this one too.
@merrymercy
merrymercy merged commit 36cf028 into main Oct 16, 2026
164 of 183 checks passed
@merrymercy
merrymercy deleted the oss-sync-44dc253d26 branch October 16, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang jit-kernel memory-pool run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants