Repository navigation
[unified-memory] Stop eviction when shared allocation capacity is sufficient - #33091
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
/tag-and-rerun-ci |
|
Could you check this CI failure? https://github.com/sgl-project/sglang/actions/runs/31776530197/job/94693145470?pr=33091 |
3242cfe to
4a656dd
Compare
Fixed the CI failure by updating the Mamba load-back test to verify evict_for_alloc() instead of evict(). I also rebased onto the latest upstream and updated the new disaggregated-decode and buffer-mode allocation paths (additional commit) to preserve the PR’s allocation-aware eviction semantics. The relevant tests pass locally. |
|
/rerun-failed-ci |
4a656dd to
9ac4474
Compare
| if component_type == ComponentType.SWA: | ||
| return self.token_to_kv_pool_allocator.swa_available_size() | ||
| if component_type == ComponentType.MAMBA: | ||
| allocator = getattr(self.req_to_token_pool, "mamba_allocator", None) |
There was a problem hiding this comment.
[suggestion] New _component_available_size() uses getattr/hasattr-style defensive fallbacks for mamba_allocator and schedulable_available_size. Workspace rule is not to hide missing fields this way. alloc_req_slots() and available_and_evictable_str() already call req_to_token_pool.mamba_allocator.schedulable_available_size() / available_size() directly. The fallbacks exist for MLX (MlxAuxiliaryStateReqToTokenPool has mamba_pool but no mamba_allocator), so a typo or rename on the CUDA path would silently use mamba_pool / available_size() (the leak-check slot count on UnifiedMambaSlotAllocator, not the byte-coordinated planner view).
Suggestion: Call self.req_to_token_pool.mamba_allocator.schedulable_available_size() directly. If MLX must share this helper, give MlxAuxiliaryStateReqToTokenPool a real mamba_allocator (the pool can implement schedulable_available_size as identity, matching MambaSlotAllocator) and branch with isinstance rather than getattr.
There was a problem hiding this comment.
_component_available_size() now calls mamba_allocator.schedulable_available_size() directly, without fallbacks.
Rather than adding an MLX-specific branch, I normalized MLX to the same allocator contract: it now exposes mamba_allocator, implements schedulable_available_size() as an identity to available_size(), and supports group allocation while returning unused slots. This keeps missing CUDA fields fail-fast.
Validation:
- Focused contract tests and pre-commit passed.
- CUDA unified-memory serving on an RTX 5090 with Qwen/Qwen3.5-0.8B and Qwen/Qwen3.5-4B completed pressure/replay without allocator errors or retractions.
- The MLX path was also validated on Apple Silicon with Qwen/Qwen3.5-0.8B.
|
|
||
| def evict(self, params: EvictParams): | ||
| def evict_for_alloc(self, params: EvictParams): | ||
| # Reclaim up to mamba_num evictable (unlocked) prefix snapshots, mirroring |
There was a problem hiding this comment.
[suggestion] The mock’s comment restates what the loop does and narrates why the PR added alloc_evict_params (“call-site contract introduced alongside allocation-aware eviction”). Comments should only record a non-obvious constraint.
Suggestion: Drop the block comment. If the params assertion needs a docstring, put a one-liner on the test method that states the failure mode (MambaComponent calling evict() or a different EvictParams shape).
There was a problem hiding this comment.
Removed the redundant block comment.
9ac4474 to
216cc74
Compare
| while ( | ||
| node_id := self._evict_device_next_node(ct, tracker) | ||
| ) is not None: | ||
| while not target_reached(ct): |
There was a problem hiding this comment.
[suggestion] target_reached() is only consulted between _evict_device_next_node() calls. FULL next_node returns one heap leaf at a time, so the FULL/unified-memory path matches the PR. SWA and Mamba next_node, however, tombstone non-leaf LRU entries and _cascade_evict them inside a single call until tracker[ct] >= request_cnt or a device leaf is found (swa_component.py / mamba_component.py). Frees are applied only when that call returns, so a capacity target met by the first internal (or by peer holes from its cascade) cannot stop the rest of the count walk. HiCache extra-pool SWA/Mamba evict_for_alloc(shortfall) can therefore still wipe FULL prefixes the outer FULL-side early stop just retained. Allocation still succeeds (over-evict, not under-evict). Unit tests mock _evict_device_next_node as one leaf per call and do not exercise this.
Suggestion: After each internal tombstone (or at the start of each next_node iteration), stop when the component's _component_available_size already meets available_size_targets. Alternatively, have next_node return after a single tombstone so the outer while not target_reached(ct) loop can observe the free.
There was a problem hiding this comment.
Thanks — good catch.
I addressed this using the second approach you suggested: SWA and Mamba device-eviction walks now yield after each internal tombstone/cascade step, allowing the outer controller loop to observe the resulting frees before continuing.
I chose this approach because the controller already owns capacity-target evaluation, pending-free draining, and tracker updates. Keeping target_reached() there avoids duplicating allocator-specific target logic inside the SWA and Mamba components. It also preserves the existing distinction between allocation-driven eviction and explicit count-based eviction.
Since node_id=None can now mean either “an internal tombstone made progress” or “the walk is exhausted,” I added an explicit made_progress result flag. The controller drains the pending frees, updates the tracker, and rechecks target_reached() before requesting another step.
This allows evict_for_alloc() to stop immediately once the allocator-visible capacity target is satisfied, without continuing through additional LRU entries or cascading into retained FULL prefixes. Count-based evict() behavior remains unchanged: it continues taking individual steps until the requested eviction count is met.
There was a problem hiding this comment.
I added real SWA/Mamba regression tests for both normal and session-radix-cache modes. They verify that allocation-driven eviction stops after the first capacity-satisfying tombstone without removing later entries or the FULL prefix, while explicit count-based eviction continues as before.
There was a problem hiding this comment.
I also compared the change against the pre-fix revision on an RTX 5090 with CUDA 13.0.
In a targeted benchmark using real SWA/Mamba trees with 32 internal LRU entries, I ran three independent processes with 60 measured samples per configuration:
- Mamba allocation-driven early-stop latency improved by 78–79%.
- SWA allocation-driven early-stop latency improved by 94–95%.
- Explicit count-based Mamba eviction was unchanged to slightly faster.
- In the forced full-walk explicit-count case, SWA eviction showed a small 3–4% median overhead because control returns to the controller after every internal step.
Broader UnifiedRadixCache insert, evict, and cache_finished benchmarks remained within ±1.4%.
End-to-end validation also passed:
-
Qwen/Qwen3.5-0.8B with --enable-unified-memory: median throughput improved by 0.8%, p95 latency improved by 2.45%, cache survival was identical, generated token IDs matched, and there were zero request retractions.
-
openai/gpt-oss-20b with hybrid-SWA HiCache: sequential replay latency changed by +0.94%, concurrent throughput by -0.26%, and pressure-request latency by -0.64%. Median eviction duration improved by 19.7%. One sequential and six concurrent host-cache replays produced matching token IDs with zero request retractions.
Focused validation:
- allocation-eviction tests: 5/5 passed;
- real SWA/Mamba component and result-drain tests: 8/8 passed;
- changed-file pre-commit checks: passed.
There was a problem hiding this comment.
IIUC, this condition might lead to under eviction, for example, target_reached(FULL) and target_reached(SWA) might both be true individually, but jointly there might not be sufficient room for both FULL and SWA.
CC @ZYHowell
There was a problem hiding this comment.
Thanks, i'm going to check the logic.
There was a problem hiding this comment.
Thanks for pointing this out. you’re right.
I found the problem in a FULL+SWA model: eviction stopped when both individual availability targets were met, but the joint capacity was still insufficient, causing the allocation to fail.
I’m planning to implement joint feasibility checks for both entering and stopping eviction.
I’ll validate the fix across different two-pool and three-pool configurations and follow up with a patch PR.
There was a problem hiding this comment.
Thanks for pointing this out. you’re right.
I found the problem in a FULL+SWA model: eviction stopped when both individual availability targets were met, but the joint capacity was still insufficient, causing the allocation to fail.
I’m planning to implement joint feasibility checks for both entering and stopping eviction.
I’ll validate the fix across different two-pool and three-pool configurations and follow up with a patch PR.
Thanks ahead for driving the followup and fix forward.
216cc74 to
d94e717
Compare
d94e717 to
238c135
Compare
|
/rerun-failed-ci |
One textual conflict: sgl-project#33091 swapped the device_swa_evict_fn lambda for the _evict_swa_for_device_alloc helper and dropped the function-local EvictParams import this branch appended its write_through guard next to. Guard stays, import goes. Signed-off-by: amd-danli103 <danli103@amd.com>
Port the Rust TreeCore parity changes from sgl-project#33091.
Motivation
Important
This optimization applies only when
--enable-unified-memoryselectsUnifiedRadixCache. Non-unified caches retain their existing behavior.Unified radix-cache components share a physical memory pool. Evicting one leaf
can therefore release capacity for multiple components; for example, evicting a
FULL leaf may also release a Mamba state whose compaction makes additional FULL
capacity schedulable.
Allocation-driven eviction previously used
evict()with a component-localshortfall. Because
evict()intentionally satisfies that component count, itcould continue removing reusable prefixes even after a peer release had already
made the pending allocation feasible.
This PR separates the two semantics:
evict()still satisfies the requested per-component count;evict_for_alloc()stops once the shared allocator cansatisfy the pending allocation, including capacity from peer compaction.
This avoids unnecessary prefix eviction without changing explicit eviction or
allocation safety.
Modifications
BasePrefixCache.evict_for_alloc()with a backward-compatibleevict()fallback for non-unified caches.UnifiedRadixCacheto convert allocation shortfalls intoavailable-capacity targets and stop when the relevant FULL, SWA, or Mamba
allocator has sufficient schedulable capacity.
HiCache load-back, and streaming-session paths through the new entry point.
evict().benchmark/unified_memory/bench_peer_aware_eviction.py.Accuracy Tests
Unit tests
The tests cover peer-capacity early termination, unchanged explicit
evict()semantics, collateral FULL/Mamba capacity, and allocation call-site routing.
A broader mem-cache regression run passed
552tests and121subtests(
2083hardware/configuration-dependent tests skipped). The optional NIXLstorage test was excluded because NIXL was not installed; this PR does not
modify that backend.
GPU output parity
This change does not alter model math. The relevant accuracy risk is incorrect
cache reuse or allocation, so output-token parity was checked directly under
cache pressure on
Qwen/Qwen3.5-0.8BandQwen/Qwen3.5-4B:Speed Tests and Profiling
Environment and methodology
Qwen/Qwen3.5-0.8B,Qwen/Qwen3.5-4B690de097crebase onto current upstream
mainThe benchmark uses an explicit warm -> pressure -> replay sequence because the
generic
bench_servingworkload cannot observe per-prefix retention. Theconcurrent case uses 30 prompts at concurrency 6, satisfying the benchmark
guide's
num_prompts >= 5 * max_concurrencyrecommendation.Server settings
The 4B runs used
--max-running-requests 6for both variants.Controlled cache pressure
Each run warms 28 distinct ~390-token prefixes, submits one unrelated
7,000-token prefill to pressure the shared pool, and then replays all prefixes.
Both models produced the same deterministic retention result in every run:
Six repetitions on
Qwen/Qwen3.5-4Bshowed:Concurrent shared-prefix fan-out
After pressure, 30 unique-suffix requests reused a prefix retained only by the
proposed path. Five server-restart-paired repetitions alternated A/B order:
The retention pattern was deterministic in all pairs, and tail latency improved
in four of five. The result supports better prefix retention and less re-prefill
work under unified-memory pressure. Aggregate throughput showed no stable
change and was not a target of this cache-pressure experiment.
Control case
With the default 2,048-token chunked-prefill size, both variants retained 17 of
28 prefixes and reused 6,528 cached tokens. This confirms that eviction behavior
remains unchanged when collateral peer capacity does not satisfy the allocation
shortfall.
Formatting checks
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #32913356967
Latest PR Test (Extra): ❌ Run #32913356694
Latest PR Test (AMD ROCm 7.2): ⏳ Run #32913356828