Skip to content

[unified-memory] Stop eviction when shared allocation capacity is sufficient - #33091

Merged
ch-wan merged 8 commits into
sgl-project:mainfrom
seokwoosong:unified-memory-peer-aware-eviction
Aug 26, 2026
Merged

ch-wan merged 8 commits into
sgl-project:mainfrom
seokwoosong:unified-memory-peer-aware-eviction

Conversation

@seokwoosong

@seokwoosong seokwoosong commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Important

This optimization applies only when --enable-unified-memory selects
UnifiedRadixCache. Non-unified caches retain their existing behavior.

Unified radix-cache components share a physical memory pool. Evicting one leaf
can therefore release capacity for multiple components; for example, evicting a
FULL leaf may also release a Mamba state whose compaction makes additional FULL
capacity schedulable.

Allocation-driven eviction previously used evict() with a component-local
shortfall. Because evict() intentionally satisfies that component count, it
could continue removing reusable prefixes even after a peer release had already
made the pending allocation feasible.

This PR separates the two semantics:

  • explicit evict() still satisfies the requested per-component count;
  • allocation-triggered evict_for_alloc() stops once the shared allocator can
    satisfy the pending allocation, including capacity from peer compaction.

This avoids unnecessary prefix eviction without changing explicit eviction or
allocation safety.

Modifications

  • Add BasePrefixCache.evict_for_alloc() with a backward-compatible
    evict() fallback for non-unified caches.
  • Override it in UnifiedRadixCache to convert allocation shortfalls into
    available-capacity targets and stop when the relevant FULL, SWA, or Mamba
    allocator has sufficient schedulable capacity.
  • Route allocation-driven KV, Mamba, disaggregated decode, MLX auxiliary-state,
    HiCache load-back, and streaming-session paths through the new entry point.
  • Preserve the original component-count semantics of explicit evict().
  • Add CPU regression tests and a reproducible GPU benchmark at
    benchmark/unified_memory/bench_peer_aware_eviction.py.

Accuracy Tests

Unit tests

PYTHONPATH=python python -m pytest -q \
  test/registered/unit/mem_cache/test_mamba_donated_alloc_ratio.py \
  test/registered/unit/mem_cache/test_unified_radix_allocation_eviction.py
10 passed

The tests cover peer-capacity early termination, unchanged explicit evict()
semantics, collateral FULL/Mamba capacity, and allocation call-site routing.

A broader mem-cache regression run passed 552 tests and 121 subtests
(2083 hardware/configuration-dependent tests skipped). The optional NIXL
storage test was excluded because NIXL was not installed; this PR does not
modify that backend.

GPU output parity

This change does not alter model math. The relevant accuracy risk is incorrect
cache reuse or allocation, so output-token parity was checked directly under
cache pressure on Qwen/Qwen3.5-0.8B and Qwen/Qwen3.5-4B:

  • warm-prefix, pressure-request, and sequential-replay token IDs matched;
  • deterministic concurrent A/B token IDs matched for all 30 requests;
  • all requests completed with zero retractions or allocation failures.

Speed Tests and Profiling

Environment and methodology

  • GPU: NVIDIA GeForce RTX 5090 32 GB
  • Driver / PyTorch: 610.43.02 / 2.11.0+cu130
  • Attention backend: Triton
  • Models: Qwen/Qwen3.5-0.8B, Qwen/Qwen3.5-4B
  • Unified pool: 8,192 FULL tokens and 32 Mamba states
  • Baseline: pre-change commit 690de097c
  • Proposed: the same functional patch as this PR, measured before its final
    rebase onto current upstream main

The benchmark uses an explicit warm -> pressure -> replay sequence because the
generic bench_serving workload cannot observe per-prefix retention. The
concurrent case uses 30 prompts at concurrency 6, satisfying the benchmark
guide's num_prompts >= 5 * max_concurrency recommendation.

Server settings
--enable-unified-memory
--attention-backend triton
--max-total-tokens 8192
--max-mamba-cache-size 32
--max-running-requests 8
--chunked-prefill-size 8192
--max-prefill-tokens 8192
--disable-prefill-cuda-graph

The 4B runs used --max-running-requests 6 for both variants.

Controlled cache pressure

Each run warms 28 distinct ~390-token prefixes, submits one unrelated
7,000-token prefill to pressure the shared pool, and then replays all prefixes.
Both models produced the same deterministic retention result in every run:

Metric Baseline Proposed Delta
Prefixes retained 10 / 28 19 / 28 +90%
Cache survival rate 35.7% 67.9% +32.2 pp
Cached tokens reused 3,840 7,296 +90%
Request retractions 0 0 no change

Six repetitions on Qwen/Qwen3.5-4B showed:

Metric (median) Baseline Proposed Delta
Total E2E for 28 replays 4.105 s 2.548 s -37.9%
Mean replay prefill latency 105.65 ms 59.54 ms -43.6%

Concurrent shared-prefix fan-out

After pressure, 30 unique-suffix requests reused a prefix retained only by the
proposed path. Five server-restart-paired repetitions alternated A/B order:

Metric (median of 5) Baseline Proposed Delta
Requests hitting shared prefix 24 / 30 30 / 30 +25.0%
p95 request E2E 1.445 s 1.208 s -16.4%
Mean prefill latency 167.3 ms 101.6 ms -39.3%
p95 prefill latency 516.4 ms 251.6 ms -51.3%
Request retractions 0 0 no change

The retention pattern was deterministic in all pairs, and tail latency improved
in four of five. The result supports better prefix retention and less re-prefill
work under unified-memory pressure. Aggregate throughput showed no stable
change and was not a target of this cache-pressure experiment.

Control case

With the default 2,048-token chunked-prefill size, both variants retained 17 of
28 prefixes and reused 6,528 cached tokens. This confirms that eviction behavior
remains unchanged when collateral peer capacity does not satisfy the allocation
shortfall.

Formatting checks

pre-commit on all changed files: passed
git diff --check: passed

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #32913356967
Latest PR Test (Extra): ❌ Run #32913356694
Latest PR Test (AMD ROCm 7.2): ⏳ Run #32913356828

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@ch-wan ch-wan self-assigned this Aug 1, 2026
@ch-wan

ch-wan commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Aug 8, 2026
@ch-wan

ch-wan commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

@seokwoosong
seokwoosong force-pushed the unified-memory-peer-aware-eviction branch from 3242cfe to 4a656dd Compare August 22, 2026 08:21
@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Aug 22, 2026
@seokwoosong

Copy link
Copy Markdown
Contributor Author

Could you check this CI failure? https://github.com/sgl-project/sglang/actions/runs/31776530197/job/94693145470?pr=33091

Fixed the CI failure by updating the Mamba load-back test to verify evict_for_alloc() instead of evict().

I also rebased onto the latest upstream and updated the new disaggregated-decode and buffer-mode allocation paths (additional commit) to preserve the PR’s allocation-aware eviction semantics.

The relevant tests pass locally.

@seokwoosong

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@seokwoosong
seokwoosong force-pushed the unified-memory-peer-aware-eviction branch from 4a656dd to 9ac4474 Compare August 22, 2026 22:55
if component_type == ComponentType.SWA:
return self.token_to_kv_pool_allocator.swa_available_size()
if component_type == ComponentType.MAMBA:
allocator = getattr(self.req_to_token_pool, "mamba_allocator", None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] New _component_available_size() uses getattr/hasattr-style defensive fallbacks for mamba_allocator and schedulable_available_size. Workspace rule is not to hide missing fields this way. alloc_req_slots() and available_and_evictable_str() already call req_to_token_pool.mamba_allocator.schedulable_available_size() / available_size() directly. The fallbacks exist for MLX (MlxAuxiliaryStateReqToTokenPool has mamba_pool but no mamba_allocator), so a typo or rename on the CUDA path would silently use mamba_pool / available_size() (the leak-check slot count on UnifiedMambaSlotAllocator, not the byte-coordinated planner view).

Suggestion: Call self.req_to_token_pool.mamba_allocator.schedulable_available_size() directly. If MLX must share this helper, give MlxAuxiliaryStateReqToTokenPool a real mamba_allocator (the pool can implement schedulable_available_size as identity, matching MambaSlotAllocator) and branch with isinstance rather than getattr.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_component_available_size() now calls mamba_allocator.schedulable_available_size() directly, without fallbacks.

Rather than adding an MLX-specific branch, I normalized MLX to the same allocator contract: it now exposes mamba_allocator, implements schedulable_available_size() as an identity to available_size(), and supports group allocation while returning unused slots. This keeps missing CUDA fields fail-fast.

Validation:

  • Focused contract tests and pre-commit passed.
  • CUDA unified-memory serving on an RTX 5090 with Qwen/Qwen3.5-0.8B and Qwen/Qwen3.5-4B completed pressure/replay without allocator errors or retractions.
  • The MLX path was also validated on Apple Silicon with Qwen/Qwen3.5-0.8B.


def evict(self, params: EvictParams):
def evict_for_alloc(self, params: EvictParams):
# Reclaim up to mamba_num evictable (unlocked) prefix snapshots, mirroring

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The mock’s comment restates what the loop does and narrates why the PR added alloc_evict_params (“call-site contract introduced alongside allocation-aware eviction”). Comments should only record a non-obvious constraint.

Suggestion: Drop the block comment. If the params assertion needs a docstring, put a one-liner on the test method that states the failure mode (MambaComponent calling evict() or a different EvictParams shape).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the redundant block comment.

@seokwoosong
seokwoosong force-pushed the unified-memory-peer-aware-eviction branch from 9ac4474 to 216cc74 Compare August 24, 2026 16:05
while (
node_id := self._evict_device_next_node(ct, tracker)
) is not None:
while not target_reached(ct):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] target_reached() is only consulted between _evict_device_next_node() calls. FULL next_node returns one heap leaf at a time, so the FULL/unified-memory path matches the PR. SWA and Mamba next_node, however, tombstone non-leaf LRU entries and _cascade_evict them inside a single call until tracker[ct] >= request_cnt or a device leaf is found (swa_component.py / mamba_component.py). Frees are applied only when that call returns, so a capacity target met by the first internal (or by peer holes from its cascade) cannot stop the rest of the count walk. HiCache extra-pool SWA/Mamba evict_for_alloc(shortfall) can therefore still wipe FULL prefixes the outer FULL-side early stop just retained. Allocation still succeeds (over-evict, not under-evict). Unit tests mock _evict_device_next_node as one leaf per call and do not exercise this.

Suggestion: After each internal tombstone (or at the start of each next_node iteration), stop when the component's _component_available_size already meets available_size_targets. Alternatively, have next_node return after a single tombstone so the outer while not target_reached(ct) loop can observe the free.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — good catch.

I addressed this using the second approach you suggested: SWA and Mamba device-eviction walks now yield after each internal tombstone/cascade step, allowing the outer controller loop to observe the resulting frees before continuing.

I chose this approach because the controller already owns capacity-target evaluation, pending-free draining, and tracker updates. Keeping target_reached() there avoids duplicating allocator-specific target logic inside the SWA and Mamba components. It also preserves the existing distinction between allocation-driven eviction and explicit count-based eviction.

Since node_id=None can now mean either “an internal tombstone made progress” or “the walk is exhausted,” I added an explicit made_progress result flag. The controller drains the pending frees, updates the tracker, and rechecks target_reached() before requesting another step.

This allows evict_for_alloc() to stop immediately once the allocator-visible capacity target is satisfied, without continuing through additional LRU entries or cascading into retained FULL prefixes. Count-based evict() behavior remains unchanged: it continues taking individual steps until the requested eviction count is met.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added real SWA/Mamba regression tests for both normal and session-radix-cache modes. They verify that allocation-driven eviction stops after the first capacity-satisfying tombstone without removing later entries or the FULL prefix, while explicit count-based eviction continues as before.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also compared the change against the pre-fix revision on an RTX 5090 with CUDA 13.0.

In a targeted benchmark using real SWA/Mamba trees with 32 internal LRU entries, I ran three independent processes with 60 measured samples per configuration:

  • Mamba allocation-driven early-stop latency improved by 78–79%.
  • SWA allocation-driven early-stop latency improved by 94–95%.
  • Explicit count-based Mamba eviction was unchanged to slightly faster.
  • In the forced full-walk explicit-count case, SWA eviction showed a small 3–4% median overhead because control returns to the controller after every internal step.

Broader UnifiedRadixCache insert, evict, and cache_finished benchmarks remained within ±1.4%.

End-to-end validation also passed:

  • Qwen/Qwen3.5-0.8B with --enable-unified-memory: median throughput improved by 0.8%, p95 latency improved by 2.45%, cache survival was identical, generated token IDs matched, and there were zero request retractions.

  • openai/gpt-oss-20b with hybrid-SWA HiCache: sequential replay latency changed by +0.94%, concurrent throughput by -0.26%, and pressure-request latency by -0.64%. Median eviction duration improved by 19.7%. One sequential and six concurrent host-cache replays produced matching token IDs with zero request retractions.

Focused validation:

  • allocation-eviction tests: 5/5 passed;
  • real SWA/Mamba component and result-drain tests: 8/8 passed;
  • changed-file pre-commit checks: passed.

@Jialin Jialin Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, this condition might lead to under eviction, for example, target_reached(FULL) and target_reached(SWA) might both be true individually, but jointly there might not be sufficient room for both FULL and SWA.

CC @ZYHowell

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, i'm going to check the logic.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jialin

Thanks for pointing this out. you’re right.

I found the problem in a FULL+SWA model: eviction stopped when both individual availability targets were met, but the joint capacity was still insufficient, causing the allocation to fail.

I’m planning to implement joint feasibility checks for both entering and stopping eviction.

I’ll validate the fix across different two-pool and three-pool configurations and follow up with a patch PR.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jialin

Thanks for pointing this out. you’re right.

I found the problem in a FULL+SWA model: eviction stopped when both individual availability targets were met, but the joint capacity was still insufficient, causing the allocation to fail.

I’m planning to implement joint feasibility checks for both entering and stopping eviction.

I’ll validate the fix across different two-pool and three-pool configurations and follow up with a patch PR.

Thanks ahead for driving the followup and fix forward.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jialin
I've verified that the joint allocation bug you mentioned is fixed by PR #36729. I also found an additional related bug and am working on a follow-up fix in PR #39982.

Thanks.

@seokwoosong
seokwoosong force-pushed the unified-memory-peer-aware-eviction branch from 216cc74 to d94e717 Compare August 25, 2026 23:14
@seokwoosong
seokwoosong force-pushed the unified-memory-peer-aware-eviction branch from d94e717 to 238c135 Compare August 26, 2026 00:01
@seokwoosong

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@ch-wan
ch-wan merged commit 2511743 into sgl-project:main Aug 26, 2026
242 of 292 checks passed
amd-danli103 added a commit to amd-danli103/sglang that referenced this pull request Aug 26, 2026
One textual conflict: sgl-project#33091 swapped the device_swa_evict_fn lambda for the
_evict_swa_for_device_alloc helper and dropped the function-local EvictParams
import this branch appended its write_through guard next to. Guard stays,
import goes.

Signed-off-by: amd-danli103 <danli103@amd.com>
Jialin added a commit to Jialin/sglang that referenced this pull request Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apple-silicon bypass-fastfail hicache Hierarchical Caching for SGLang memory-pool run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants