Skip to content

[Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free - #48852

Merged
ch-wan merged 1 commit into
mainfrom
cheng/hot-fix/unified-kv-release-after-forward
Nov 8, 2026
Merged

ch-wan merged 1 commit into
mainfrom
cheng/hot-fix/unified-kv-release-after-forward

Conversation

@ch-wan

@ch-wan ch-wan commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Motivation

With --enable-unified-memory and the overlap scheduler, the forward launched before a result is processed (N+1) can still use the pages of a request that this result ends. It reads them through its prepared page table, and it writes the request's next slot. A unified pool writes a freed page from the schedule stream when it hands the page out (zeroing) or compacts into it. The schedule stream is not ordered after N+1.

On main this is handled by #48496 (which replaced #48450's wait on every lazy free). Freed physical pages are held, keyed by a forward event, and settled once that event has fired (_settle_freed reads event.query()). A take that needs them orders its writers after the forward through a reuse fence; zeroing moves to the forward stream; PD publication waits on reuse fences. Which pages are free therefore depends on rank-local event timing, so TP ranks can choose different pages.

This PR replaces that mechanism. It frees at once and delays only the physical reuse, deciding from logical progress only:

  • The ids, the virtual-to-physical entries and the request rows are released immediately. The scheduler's write-after-read barrier (Scheduler._apply_war_barrier) already orders these schedule-stream writes after N+1's shared reads.
  • The freed physical pages stay pending until N+1's result is processed. They count as free in every capacity check. A take that needs them before then settles them first: its stream waits for the forward they were freed under.

Modifications

  • mem_cache/kv/alloc/forward_steps.py (new):

    • ForwardSteps, one per lazily compacting unified chain:
      • every launched step gets an immutable launch record (ForwardLaunch), which its result carries to processing;
      • a forward that records its forward_done event becomes the key for pages freed while it is in flight;
      • a step without a forward (a prebuilt batch) is never a key.
    • Scheduler._on_step_processed runs after the processor synchronized a completion covering the step's forward. It advances the processed prefix (recorded forwards only, in order; asserted) and returns that step's pending pages to the free lists.
    • No managed page is allocated or freed between a step's launch and the recording of its event (asserted), so every pending page has an event to wait on.
    • PendingPages: a band's pending pages, one batch per forward, kept as the device tensors the free produced. A free adds no host read.
    • Result processing synchronizes copy_done before it advances, prebuilt batches that carry an inner idle forward included. A pipeline stage that skips its output relay records the completion it synchronizes behind its own forward.
  • [Perf] mem_cache: Order recycled unified-page writers after the forward instead of the current stream #48496's mechanism is removed: _unsettled_free / _settle_freed, _order_reuse and the reuse fence, zeroing on the forward stream, reuse_fences() and the PD publication waits on them, and _unsettled_page_count. PD publication stays ordered: the publish stream waits on the allocation's allocated event, recorded on the schedule stream after any wait a take queued for pending pages.

  • Band / FloatBand:

    • Free: ids return at once; with a forward in flight, the physical pages go to pending instead of the free list.
    • Take: holes first; when the holes are short, settle whole pending batches, oldest first, until they suffice (Band.settle_pending): the stream waits those batches' forwards and the pages join the free list; then growing the frontier. This is the same order in which capacity counts them, so one band never grows into room a neighbour was priced on. Every check runs before anything is settled, and a take's settle never parks or moves a band. A planned take never grows beyond its planned fresh range. Choices depend on counts only, never on whether an event has fired, so ranks decide identically.
    • Reusable pages: one helper (Band._reusable_pages, holes plus pending) feeds every capacity count, a float's credit for a neighbour's drainable pages included.
    • Urgent recovery (ChainAllocator.settle_pending_for_recovery, at the entry of capacity recovery, relief, urgent flush, relocation and the last-resort layout):
      • first reads the holes and pending pages of every band that can take part in one host read;
      • then settles them;
      • then installs that read as each band's host view, and runs the existing compaction and relocation unchanged, with no host read after the wait.
      • A pinned end band, which cannot move, is left out.
      • An empty float parks after its pages settle.
    • Assertions: with invariant checks on, layout-changing compaction and relocation assert that their bands hold no pending pages and that each band's host view matches its free list (an empty free list is a known empty view).
    • Non-urgent compaction never settles; it stops at pending pages and never moves them.
    • Host copy of the free list: an ordinary take outside urgent recovery drops it instead of re-sorting and copying it, so draining many holes with small takes stays linear; urgent recovery prepares its own view at entry and keeps it by slicing.
    • Floats: a float with pending pages keeps its extent (no transparency, no parking) until they settle.
    • Capacity: pending pages count as free in availability, pricing snapshots and the accounting checks. Snapshots carry live and pending pages separately: reservations are debited by live pages, and a float is transparent only with neither.
  • The allocator trace harness (python/sglang/test/unified_allocator_trace.py) drives forward launches, recordings and processing, so its frees go pending as in serving.

  • Static pools and the eager chain are unchanged.

  • FIXME comments mark the places where a forward reads or resets request-scoped state after its declared shared-read end, which the barrier does not cover:

    • qwen4_exp.py::_prepare_ple_batch;
    • the post-verify Mamba commits in spec_utils.py, hybrid_linear_attn_backend.py and inkling_sconv_backend.py;
    • the ReplaySSM cursor reset in StateStorage.acquire;
    • the request-ring reset in kv_cache_configurator.py.

    FIXMEs also mark:

    • the EAGLE post-verify graphs replaying that commit on a side stream joined after draft extend;
    • the barrier being off by default off CUDA (managers/utils.py);
    • the PP loop not applying it;
    • the PD failed-receive release (disaggregation/decode.py).

Tests

test/registered/unit/mem_cache/test_unified_release_after_forward.py:

  • a freed page is unbound at once, counted free and pending; a free adds no host sync;
  • a take uses pending pages before growing, settles only the oldest batches it needs and waits only their forwards (one wait per event, no host read); an owner band does not grow into the room a window was priced on; a rejected take settles nothing;
  • processing in order releases only its prefix; a settled step is never returned twice; a free after a settle joins the same step once;
  • pricing snapshots are renewed on drains and settles;
  • urgent recovery settles first: a live float with a later pending-only float, two pending-only floats read once before any wait, a live float with pending boundary pages, a pinned end donor left out, a drained move source keeping the host view; geometry that sees pending pages, and a host read inside recovery, raise;
  • non-urgent compaction never moves or picks pending pages;
  • floats: an all-pending float stays opaque and recovery parks it; planned takes served by pending pages, or with no fresh range, do not grow; a scoped take debits its reservation; a float counts its neighbours' pending pages; relocation settles first and keeps live pages bound;
  • page choice does not depend on whether the forward finished;
  • regression scenarios: rehome onto ids whose window free is pending, state slots with pending frees, an early window release followed by window eviction.

Also: the prebuilt and pipeline-skip completion ordering (test_pp_skip_output_retirement), and test_unified_take_host_work.py (weekly CPU suite: draining many holes one page at a time scales with the takes).

Accuracy Tests and Speed Tests

Unit tests, NVIDIA B300, on main 3e38c5b77a: the unified-memory unit tests at the top of this stack, at the default and strict invariant levels, are reported in #48014.

Serving on NVIDIA H200 (driver 595.71.05; torch 2.14.1+cu130, flashinfer 0.7.2, sglang-kernel 0.4.10), side by side with main 5e1774d137. The GPUs are swapped between trees across runs.

openai/gpt-oss-120b, TP1, overlap, --enable-unified-memory --attention-backend fa3:

Workload Tree Run 1 Run 2
--max-total-tokens 120000, random 4096/512, 200 prompts, concurrency 64: output tok/s / TPOT ms / GSM8K (200) this PR 1464.2 / 32.84 / 0.820 1463.3 / 32.86 / 0.830
main 1468.4 / 32.74 / 0.810 1465.0 / 32.80 / 0.820
--max-total-tokens 30000, GSM8K (1319): accuracy this PR 0.8446 0.8431
main 0.8469 0.8431
  • At 120k tokens every run reaches full KV usage 1.00 and retracts one request.
  • At 30k tokens every run is at full usage with dozens of retractions.
  • Per-question paired comparisons show no accuracy difference (all p ≥ 0.56).
  • No run logs a traceback, assertion, CUDA error or NaN.

Checklist


CI States

Latest PR Test (Base): ❌ Run #42475825494
Latest PR Test (Extra): ❌ Run #42475825452
Latest PR Test (AMD ROCm 10): ❌ Run #42475825535

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch 4 times, most recently from f479939 to f2da7be Compare November 7, 2026 01:00
@ch-wan ch-wan changed the title [Fix] mem_cache: Release a request's unified KV only after its last forward retires [Perf] mem_cache: Defer reuse of freed unified pages instead of waiting on every free Nov 7, 2026
@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch 9 times, most recently from 7c7eeaa to 13cadd6 Compare November 8, 2026 09:06
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Required checks failed on 13cadd6; this PR cannot be merged until they pass.

Lint (run)
Run pre-commit run --all-files locally and push the result. The run log shows any other failing step.

@ch-wan
ch-wan force-pushed the cheng/hot-fix/unified-kv-release-after-forward branch from 13cadd6 to 3300612 Compare November 8, 2026 11:27
@ch-wan
ch-wan merged commit de36361 into main Nov 8, 2026
108 of 127 checks passed
@ch-wan
ch-wan deleted the cheng/hot-fix/unified-kv-release-after-forward branch November 8, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang memory-pool unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant