Repository navigation
Conversation
…efill A request that finishes directly after prefill cached its whole final chunk as live sliding-window KV: cache_finished_req never called the component free_out_of_window_slots hook that cache_unfinished_req already calls, so an 8192-token single-output-token request retained 8192 SWA slots instead of a window, and the node it inserted was recorded with live SWA for tokens whose read window had long passed. Reuse the existing hook under the existing flag, measured against the page-aligned radix key the way the unfinished path measures it: an EAGLE bigram key spans one token more than its length, so its insert stops one page lower. The native helper keeps the tree-owned prefix, page alignment, prior eviction and the retention floor intact. No allocator or scheduling algorithm changes. Test: TestSWAFinishedPrefill in test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py -- the unmodified source retains the whole aligned key on the flag-on arm, the fix retains two pages of live SWA and preserves the full aligned prefix match.
- test_protected_prefix_is_not_trimmed now stages the request the way the scheduler admits a prefill: real cache.match_prefix, matched tree-owned indices borrowed into the request row, only the unmatched tail allocated, real lock receipt, and the independent live-request lock kept. It asserts exact swa_evicted_seqlen and FULL/SWA allocator accounting, that the borrowed prefix and the finished span stay reusable, and that a full evict after unlock reclaims every owned slot. - Comments only in the source: describe the page-aligned insert frontier and the live window the created leaf retains instead of claiming the removed slots are universally unreadable (AST-identical to the measured revision).
apex-mochen
added a commit
to apex-mochen/sglang
that referenced
this pull request
Oct 6, 2026
A lock walk that overshoots the sliding window pins the whole crossing node, so a match ending just short of the window lets a long live-SWA ancestor (e.g. a finished request's untrimmed last chunk) be pinned in full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the native Rust TreeCore: before locking a node, split off everything but the page-aligned tail the trailing window still needs. Only live-on-device, unlocked, non-write-through-pending nodes are split (SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool); the original node keeps its id as the in-window tail, so a lock receipt anchored on it stays valid. sgl-project#40283 is complementary (it removes the known trigger) rather than a replacement for this lock-walk split. Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window` locks a 7-token endpoint under an 8-token window with a 32-token live ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_ crossing_node` is updated for the split semantics. Closes the Rust gap reported in sgl-project#41579. Signed-off-by: apex-mochen <2756823972@qq.com>
5 tasks done
apex-mochen
added a commit
to apex-mochen/sglang
that referenced
this pull request
Oct 7, 2026
A lock walk that overshoots the sliding window pins the whole crossing node, so a match ending just short of the window lets a long live-SWA ancestor (e.g. a finished request's untrimmed last chunk) be pinned in full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the native Rust TreeCore: before locking a node, split off everything but the page-aligned tail the trailing window still needs. Only live-on-device, unlocked, non-write-through-pending nodes are split (SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool); the original node keeps its id as the in-window tail, so a lock receipt anchored on it stays valid. sgl-project#40283 is complementary (it removes the known trigger) rather than a replacement for this lock-walk split. Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window` locks a 7-token endpoint under an 8-token window with a 32-token live ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_ crossing_node` is updated for the split semantics. Closes the Rust gap reported in sgl-project#41579. Signed-off-by: apex-mochen <2756823972@qq.com>
Contributor
|
Thanks @Stelath. Closing this because it is still a draft and has not been updated in 29 days. Reopen it if the work is still relevant. Some directories moved recently, so an older branch may need retargeting: |
apex-mochen
added a commit
to apex-mochen/sglang
that referenced
this pull request
Oct 20, 2026
A lock walk that overshoots the sliding window pins the whole crossing node, so a match ending just short of the window lets a long live-SWA ancestor (e.g. a finished request's untrimmed last chunk) be pinned in full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the native Rust TreeCore: before locking a node, split off everything but the page-aligned tail the trailing window still needs. Only live-on-device, unlocked, non-write-through-pending nodes are split (SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool); the original node keeps its id as the in-window tail, so a lock receipt anchored on it stays valid. sgl-project#40283 is complementary (it removes the known trigger) rather than a replacement for this lock-walk split. Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window` locks a 7-token endpoint under an 8-token window with a 32-token live ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_ crossing_node` is updated for the split semantics. Closes the Rust gap reported in sgl-project#41579. Signed-off-by: apex-mochen <2756823972@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Call the existing component
free_out_of_window_slotshook before inserting a finished request into the unified radix cache.cache_unfinished_reqalready invokes this hook; a request finishing directly after prefill previously retained its entire final chunk as live SWA KV.The frontier is measured from the page-aligned insertion key, preserving a live SWA window at the resulting leaf, including for EAGLE bigram keys. The existing default-on
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTSguard is reused. No new flag, allocator or pressure heuristic.Draft: cache-operation benchmarks show both benefits and regressions. End-to-end serving performance remains an open validation gate.
Performance: cache operations, not serving throughput
Unmodified
81421b91e9ad02c526bb3025f66f42e2a7c52059versus patched source, with the existing flag enabled in both. GB300, Python 3.12.3, torch 2.13.0+cu130, Python tree core, page size 256, SWA window 512. Five rounds per workload; fresh fixtures; four warmup requests excluded from timing. Entries are medians of round statistics; brackets give the round min–max, not confidence intervals.Measured requests per round: 156 for either pressure workload, 44 for the fixed set and 1024-token workload, and 20 for either 8192-token workload. Synchronized host latency around
cache_finished_reqincreases by 0.56 ms for nonbinding 8192-token prompts, 0.51 ms for 1024-token prompts, and 0.78 ms for bigram prompts. The added work is SWA page release and tombstone insertion, not just the hook dispatch.Supporting capacity result: in the nonbinding 8192-token workload, retained live SWA across all 24 requests decreases from 196608 to 18432 slots (10.7×). These are pool slots, not reduced CUDA reservation.
Measurement limits
input_len - 1and page-ceils the extension, so 3.62× must not be interpreted as a serving speedup or full-prefill elimination.Correctness and checks
Test file:
test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py(existing CUDA/AMD CI registration).TestSWAFinishedPrefill: 6 passed / 10 subtests, on both Python and Rust tree cores. Unmodified source reproduces 5 failures with the same tests.Performance/full-file evidence used source at
ad09abeefb67404a0ba5c0047b28c26833d354ad; final source at2beaba852a8afb3107616e4d608acf5a377571c5differs only in comments, with independently verified whole-module AST identity. The final test revision was used for the targeted reruns above.HiCache, multi-rank/TP, SWA+Mamba integration and million-token contexts were not exercised. Keep draft pending end-to-end serving A/B and review of the default-on cleanup-cost versus cache-capacity tradeoff.
CI States
Latest PR Test (Base): ❌ Run #35412957728
Latest PR Test (Extra): ❌ Run #35412957632
Latest PR Test (AMD ROCm 10): ❌ Run #35412957743