Repository navigation
[Fix] mem_cache: SWA prefix lock pins one window, not a whole long ancestor - #41580
bill-h-lin wants to merge 1 commit into
Conversation
|
@ispobock @xiezhq-hermann (KV Cache merge oncalls): could one of you take a look at this PR and at #41263 when you have a moment? Both are small hybrid-SWA fixes in |
A lock walk that overshoots the sliding window pins the whole crossing node, so a match ending just short of the window lets a long live-SWA ancestor (e.g. a finished request's untrimmed last chunk) be pinned in full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the native Rust TreeCore: before locking a node, split off everything but the page-aligned tail the trailing window still needs. Only live-on-device, unlocked, non-write-through-pending nodes are split (SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool); the original node keeps its id as the in-window tail, so a lock receipt anchored on it stays valid. sgl-project#40283 is complementary (it removes the known trigger) rather than a replacement for this lock-walk split. Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window` locks a 7-token endpoint under an 8-token window with a 32-token live ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_ crossing_node` is updated for the split semantics. Closes the Rust gap reported in sgl-project#41579. Signed-off-by: apex-mochen <2756823972@qq.com>
|
Rust TreeCore companion: #42745 (@apex-mochen), the same split in |
65ab9f0 to
f6bd8ab
Compare
|
Rebased onto main ( |
f6bd8ab to
4dad756
Compare
|
Rebased onto With that, the whole file gives the same results with the flag on and off on GPU, and the buffer test errors again if the check is removed. On CPU, the new test, upstream's leaf-cap test and the four mutations behave as before. Description updated. @hnyls2002, could you take a look, and add |
A lock walk that overshoots the sliding window pins the whole crossing node, so a match ending just short of the window lets a long live-SWA ancestor (e.g. a finished request's untrimmed last chunk) be pinned in full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the native Rust TreeCore: before locking a node, split off everything but the page-aligned tail the trailing window still needs. Only live-on-device, unlocked, non-write-through-pending nodes are split (SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool); the original node keeps its id as the in-window tail, so a lock receipt anchored on it stays valid. sgl-project#40283 is complementary (it removes the known trigger) rather than a replacement for this lock-walk split. Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window` locks a 7-token endpoint under an 8-token window with a 32-token live ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_ crossing_node` is updated for the split semantics. Closes the Rust gap reported in sgl-project#41579. Signed-off-by: apex-mochen <2756823972@qq.com>
A lock walk that overshoots the sliding window pins the whole crossing node, so a match ending just short of the window lets a long live-SWA ancestor (e.g. a finished request's untrimmed last chunk) be pinned in full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the native Rust TreeCore: before locking a node, split off everything but the page-aligned tail the trailing window still needs. Only live-on-device, unlocked, non-write-through-pending nodes are split (SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool); the original node keeps its id as the in-window tail, so a lock receipt anchored on it stays valid. sgl-project#40283 is complementary (it removes the known trigger) rather than a replacement for this lock-walk split. Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window` locks a 7-token endpoint under an 8-token window with a 32-token live ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_ crossing_node` is updated for the split semantics. Closes the Rust gap reported in sgl-project#41579. Signed-off-by: apex-mochen <2756823972@qq.com>
xiezhq-hermann
left a comment
There was a problem hiding this comment.
@bill-h-lin thanks for the fix and the careful write-up, and sorry this sat so long. #42745 is the Rust counterpart of this PR (see my review there). I re-checked this PR on current main (b5a7900ae4) with your test ported to main's split suite (unit fixtures, Python core unless noted):
- The over-pin is still there. When a match diverges on the last page of the capped window leaf, the lock walk reaches the long live parent and pins the whole matched prefix. This happens on both the Python and the Rust core, in all 8 SWA configs the test runs on (ps1/sw4: 7 tokens for a 4-token window; ps4/sw4: 16 for 4; ps128/sw128: 512 for 128).
- The
swa.pyandenviron.pyhunks apply cleanly to main. With them:- the Python core pins one page-aligned window in all 8 configs;
SGLANG_SWA_LOCK_WINDOW_ONLY=0restores the old pin;sanity_checkpasses.
- The core, insert-walk and kv-events topic modules give the same results with and without the patch. So do the hicache and storage modules on the SWA configs with page size <= 16.
Requesting changes for the following:
-
Blocker: buffer_only HiCache trips the new assert. Since #45242 (merged after your last push),
UnifiedTreeCore._split_nodereturnsUpdatePendingBackupOnNodeSplit(ack_id=None)for every split whenis_host_memory_buffer_only. Soassert action is Nonein_maybe_split_for_window_lockfails on the first window split.- Repro: build with
_build_buffer_hicache_fixture, insert one page-aligned window + 4 pages, drain the backup pipeline, match a key that diverges on the last page, then callinc_lock_ref. It raisesAssertionError: a node without write-through cannot return an actionin all 8 SWA configs. - None of the existing tests I ran reach this path.
The fix options are the same as on #42745. Please pick the same one in both PRs (cc @apex-mochen) so the two cores split the same nodes:
- Skip the window split when
tree_core.is_host_memory_buffer_only. Thebuffer_backup_pendingcheck can then go. - Or keep the split, assert only
action.ack_id is None, and apply the action, the way_maybe_split_leaf_for_swa_lockforwards its own. I tried this locally withself.cache._apply_cache_actions([action]). The buffer_only flow above then pins one window in all 8 configs and passessanity_check.
Either way, please add the buffer_only case as a regression test.
- Repro: build with
-
Rebase. Only the test conflicts. #45236 split the unit test file into topic modules, so the new test now goes in
test/registered/unit/mem_cache/unified_radix_cache/test_unified_radix_cache_core.py. Two helpers changed on main:swa_protected_size()is gone; usewindow_protected(cache)fromsglang.test.tree_slots.MatchResulthasdevice_prefix_leninstead ofdevice_indices.
-
Land together with #42745.
test_unified_radix_cache_rust.pyreruns the core module on the Rust core. Main's Rust core still pins the whole prefix, so the new test fails there until #42745 lands. As you said, the two should go in together, or #42745 first. Please also keep a singleSGLANG_SWA_LOCK_WINDOW_ONLYdeclaration inenviron.pythat both cores read. On #42745 I asked for the Rust core to get it through the init params. -
Heads-up: #42703 (SWA cache-window margin) caps the insert-time leaf at window + margin. A lock-walk split at exactly one window would put the margin back outside the lock. Whichever PR lands second should split at the cached window size.
Once it's rebased and 1 is fixed, I'm happy to take another look and add run-ci.
…cestor acquire_component_lock pins whole tree nodes until it has covered one sliding window. _maybe_split_leaf_for_swa_lock caps a fresh leaf at one window, but a match that ends inside that capped leaf leaves a node shorter than the window, and the walk then reaches the leaf's parent. When that parent is a finished request's last chunk, its SWA is still live (cleanup_after_caching_req trims out-of-window SWA only for unfinished requests), so the lock pins the whole chunk. With a tight SWA pool this deadlocks admission: on MiMo-V2.6-Flash (window 128, --chunked-prefill-size 8192, SWA pool 8,451, overlap on, one running request), a 32,768-token request that stops at its first token followed by a request that reuses those 32,768 tokens plus 5,000 new ones: the lock pins 8,192 slots, 259 remain against 5,001 needed, and since that fits an empty pool it is not treated as a permanent shortfall, so the request waits forever with nothing running. In the lock walk, split a live, unlocked device-SWA node that is longer than what the window still needs, with the tree's own _split_node, so only the in-window tail is pinned. The node keeps its id (lock receipts anchored on it stay valid); locked, tombstoned, write-through-pending and buffer-backup-pending nodes and host locks are left alone. SGLANG_SWA_LOCK_WINDOW_ONLY=0 restores the old walk. Buffer-only HiCache has every node split return a pending-backup action with no tree ack (sgl-project#45242); the lock walk's split applies it, as the leaf cap forwards its own, instead of asserting that none comes back. The buffer-only case is a regression test in the storage topic module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4dad756 to
302922a
Compare
|
Thanks for the review and for porting the test. Rebased onto 1, the blocker: fixed the second way. The window split keeps going in buffer-only mode, asserts only Regression test: Rebase: the PR's test moved to Runs (RTX PRO 6000 Blackwell, Python core, the six topic modules, Landing with #42745: proposed there that the Rust core take the same option (keep the split, forward the action), with the knob passed through the init params from this declaration. I'll re-pair the two branches once #42745 is updated. On #42703: if it lands first, I'll switch this split to the cached window size. |
The Rust window-only lock walk still asserted that split_node_ returns no action, while the Python core in sgl-project#41580 keeps the split and applies the action immediately. That made the two cores split different nodes. Carry the action on IncLockRefResult, expose it through the PyO3 binding, and apply it in UnifiedRadixCache.inc_lock_ref before callers use the receipt. The Python core keeps applying directly in the SWA component, so its receipt carries no actions. Adds a CPU test for the receipt plumbing. Signed-off-by: apex-mochen <2756823972@qq.com>
Motivation
Fixes #41579: an admission livelock with a hybrid-SWA model, the radix cache and a tight SWA pool.
acquire_component_lockpins whole nodes until it has covered one sliding window._maybe_split_leaf_for_swa_lockcaps a fresh leaf at one window, but a match that ends inside that capped leaf leaves a node shorter than the window, and the walk then reaches the leaf's parent. When that parent is a finished request's last chunk, its SWA is still live (cleanup_after_caching_reqtrims only unfinished requests), so the lock pins the whole chunk: 8,192 SWA slots for a 128-token window on MiMo-V2.6-Flash. At a pool of one chunk + window + slack, the next request then waits forever with nothing running.Modifications
SWAComponent.acquire_component_lock: before counting each node,_maybe_split_for_window_lock(cur, window - covered)splits a live, unlocked device-SWA node that is longer than what the window still needs (rounded up to a page), with the tree's own_split_node. Only the in-window tail is then pinned.redistribute_on_node_split, as for any split.value is None), write-through-pending nodes, nodes with a buffer-mode write pending (buffer_backup_pending: the pipeline's pin at D2H launch is itself a lock walk, and the write reads the whole node) and host locks (lock_host)._apply_cache_action, as the leaf cap forwards its own, instead of asserting that none comes back.SGLANG_SWA_LOCK_WINDOW_ONLY(default on) is a kill switch:0restores the old walk. Happy to drop it._split_node, so the new older half lands next to the tail in each LRU list and the tail's access time is refreshed (it is being locked for compute anyway). Full-attention locking is unchanged: Full still protects the whole matched prefix.Accuracy Tests
The change is lock bookkeeping plus a topology-only split: no KV moves and no chunk schedule changes. End to end (on a 542a043 build, MiMo-V2.6-Flash, chunk 8,192, SWA pool 8,451, greedy): the request that deadlocked is admitted as one 5,000-token extend on 32,768 cached tokens, and every output (20 requests up to 127,852 tokens, ids and all returned logprobs) is bit-identical to the same build with the flag off at a 16,643-token SWA pool, where the old walk does not deadlock.
Unit tests
test_swa_lock_pins_one_window_when_match_ends_inside_capped_leaf(test_unified_radix_cache_core.py) inserts a live parent + the capped window leaf, matches a key that diverges on the last page, then locks, unlocks, re-matches and re-locks withsanity_checkat every step. With the flag off it asserts the old whole-parent pin; with it on, exactly one page-aligned window. It runs on all 8 SWA configurations of the suite (page 1/4/16/64/128, window 2–128); the Mamba and non-SWA ones skip it.test_swa_lock_window_split_in_buffer_only_mode(test_unified_radix_cache_storage.py) is the buffer-only HiCache case, through_build_buffer_hicache_fixture: the same shape with its backups drained pins one window (the split's pending-backup action is applied, not asserted away); then a parent longer than the window with its backup queued and not yet written stays whole. Without the action forwarding it errors with the old assert on 8/8 configurations; without the pending-backup guard it fails on 8/8 (the pending parent is split to one window).maine0187268f6vs this branch, Python core: 1,319 → 1,335 tests, all OK; the FAIL/ERROR lists are identical (empty).test_unified_radix_cache_rust.pyreruns the modules on the Rust TreeCore: 1,319 OK onmain; on this branch only the 16 runs of the two new tests fail (the Rust walk has no window split yet), which [Fix] radix_cache: Cap SWA lock-walk pinning at one window in the Rust TreeCore #42745 supplies; nothing else changes.Not covered
Several running requests (the end-to-end runs used
--max-running-requests 1), PD disaggregation, and page size > 1 end to end (unit tests only).Checklist
CI States
Latest PR Test (Base): ❌ Run #42863051441
Latest PR Test (Extra): ❌ Run #42863051157
Latest PR Test (AMD ROCm 10): ❌ Run #42863051618
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: