Repository navigation
[Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free - #48852
Merged
Merged
Conversation
ch-wan
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
ShangmingCai,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sogalin,
xiezhq-hermann and
yizhang2077
as code owners
November 6, 2026 08:17
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 6, 2026 10:43
8084a16 to
7d025c6
Compare
This was referenced Nov 6, 2026
Merged
Merged
This comment has been minimized.
This comment has been minimized.
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
4 times, most recently
from
November 7, 2026 01:00
f479939 to
f2da7be
Compare
ch-wan
requested review from
Qiaolin-Yu,
hebiao064 and
yuan-luo
as code owners
November 7, 2026 01:00
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
9 times, most recently
from
November 8, 2026 09:06
7c7eeaa to
13cadd6
Compare
Contributor
ch-wan
force-pushed
the
cheng/hot-fix/unified-kv-release-after-forward
branch
from
November 8, 2026 11:27
13cadd6 to
3300612
Compare
This was referenced Nov 8, 2026
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack (oldest at bottom):
Motivation
With
--enable-unified-memoryand the overlap scheduler, the forward launched before a result is processed (N+1) can still use the pages of a request that this result ends. It reads them through its prepared page table, and it writes the request's next slot. A unified pool writes a freed page from the schedule stream when it hands the page out (zeroing) or compacts into it. The schedule stream is not ordered after N+1.On
mainthis is handled by #48496 (which replaced #48450's wait on every lazy free). Freed physical pages are held, keyed by a forward event, and settled once that event has fired (_settle_freedreadsevent.query()). A take that needs them orders its writers after the forward through a reuse fence; zeroing moves to the forward stream; PD publication waits on reuse fences. Which pages are free therefore depends on rank-local event timing, so TP ranks can choose different pages.This PR replaces that mechanism. It frees at once and delays only the physical reuse, deciding from logical progress only:
Scheduler._apply_war_barrier) already orders these schedule-stream writes after N+1's shared reads.Modifications
mem_cache/kv/alloc/forward_steps.py(new):ForwardSteps, one per lazily compacting unified chain:ForwardLaunch), which its result carries to processing;forward_doneevent becomes the key for pages freed while it is in flight;Scheduler._on_step_processedruns after the processor synchronized a completion covering the step's forward. It advances the processed prefix (recorded forwards only, in order; asserted) and returns that step's pending pages to the free lists.PendingPages: a band's pending pages, one batch per forward, kept as the device tensors the free produced. A free adds no host read.copy_donebefore it advances, prebuilt batches that carry an inner idle forward included. A pipeline stage that skips its output relay records the completion it synchronizes behind its own forward.[Perf] mem_cache: Order recycled unified-page writers after the forward instead of the current stream #48496's mechanism is removed:
_unsettled_free/_settle_freed,_order_reuseand the reuse fence, zeroing on the forward stream,reuse_fences()and the PD publication waits on them, and_unsettled_page_count. PD publication stays ordered: the publish stream waits on the allocation'sallocatedevent, recorded on the schedule stream after any wait a take queued for pending pages.Band/FloatBand:Band.settle_pending): the stream waits those batches' forwards and the pages join the free list; then growing the frontier. This is the same order in which capacity counts them, so one band never grows into room a neighbour was priced on. Every check runs before anything is settled, and a take's settle never parks or moves a band. A planned take never grows beyond its planned fresh range. Choices depend on counts only, never on whether an event has fired, so ranks decide identically.Band._reusable_pages, holes plus pending) feeds every capacity count, a float's credit for a neighbour's drainable pages included.ChainAllocator.settle_pending_for_recovery, at the entry of capacity recovery, relief, urgent flush, relocation and the last-resort layout):The allocator trace harness (
python/sglang/test/unified_allocator_trace.py) drives forward launches, recordings and processing, so its frees go pending as in serving.Static pools and the eager chain are unchanged.
FIXME comments mark the places where a forward reads or resets request-scoped state after its declared shared-read end, which the barrier does not cover:
qwen4_exp.py::_prepare_ple_batch;spec_utils.py,hybrid_linear_attn_backend.pyandinkling_sconv_backend.py;StateStorage.acquire;kv_cache_configurator.py.FIXMEs also mark:
managers/utils.py);disaggregation/decode.py).Tests
test/registered/unit/mem_cache/test_unified_release_after_forward.py:Also: the prebuilt and pipeline-skip completion ordering (
test_pp_skip_output_retirement), andtest_unified_take_host_work.py(weekly CPU suite: draining many holes one page at a time scales with the takes).Accuracy Tests and Speed Tests
Unit tests, NVIDIA B300, on
main3e38c5b77a: the unified-memory unit tests at the top of this stack, at the default and strict invariant levels, are reported in #48014.Serving on NVIDIA H200 (driver 595.71.05; torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10), side by side with
main5e1774d137. The GPUs are swapped between trees across runs.openai/gpt-oss-120b, TP1, overlap,--enable-unified-memory --attention-backend fa3:--max-total-tokens 120000, random 4096/512, 200 prompts, concurrency 64: output tok/s / TPOT ms / GSM8K (200)main--max-total-tokens 30000, GSM8K (1319): accuracymainChecklist
CI States
Latest PR Test (Base): ❌ Run #42475825494
Latest PR Test (Extra): ❌ Run #42475825452
Latest PR Test (AMD ROCm 10): ❌ Run #42475825535
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: