Skip to content

[mem_cache] Trim out-of-window SWA slots when caching finished requests - #40283

Closed
Stelath wants to merge 2 commits into
sgl-project:mainfrom
Stelath:korte/swa-finished-prefill-main
Closed

Stelath wants to merge 2 commits into
sgl-project:mainfrom
Stelath:korte/swa-finished-prefill-main

Conversation

@Stelath

@Stelath Stelath commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Call the existing component free_out_of_window_slots hook before inserting a finished request into the unified radix cache. cache_unfinished_req already invokes this hook; a request finishing directly after prefill previously retained its entire final chunk as live SWA KV.

The frontier is measured from the page-aligned insertion key, preserving a live SWA window at the resulting leaf, including for EAGLE bigram keys. The existing default-on SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS guard is reused. No new flag, allocator or pressure heuristic.

Draft: cache-operation benchmarks show both benefits and regressions. End-to-end serving performance remains an open validation gate.

Performance: cache operations, not serving throughput

Unmodified 81421b91e9ad02c526bb3025f66f42e2a7c52059 versus patched source, with the existing flag enabled in both. GB300, Python 3.12.3, torch 2.13.0+cu130, Python tree core, page size 256, SWA window 512. Five rounds per workload; fresh fixtures; four warmup requests excluded from timing. Entries are medians of round statistics; brackets give the round min–max, not confidence intervals.

Workload FULL/SWA pool slots Baseline cache ops/s [range] Patched cache ops/s [range] Change
SWA-bound: 8 shared 4096-token prefixes + unique 256-token tails 262144 / 16384 386.05 [315.09–437.31] 553.95 [461.36–586.19] +43.5%
Fixed set: 4 identical 6144-token prompts 65536 / 16384 394.87 [393.85–412.75] 1430.98 [1361.48–1476.17] 3.62×
Unique 8192-token prompts, SWA not binding 262144 / 262144 995.17 [972.15–1038.56] 644.98 [636.03–680.22] −35.2%
Unique 1024-token prompts, SWA not binding 262144 / 262144 1349.14 [1319.94–1402.09] 782.39 [711.88–839.90] −42.0%
EAGLE bigram, unique 8192-token prompts 262144 / 262144 457.03 [438.13–460.18] 328.04 [268.71–338.28] −28.2%
FULL-bound: 4 shared 4096-token prefixes + unique 2048-token tails 65536 / 16384 298.35 [291.05–306.10] 270.13 [251.76–274.80] −9.5%

Measured requests per round: 156 for either pressure workload, 44 for the fixed set and 1024-token workload, and 20 for either 8192-token workload. Synchronized host latency around cache_finished_req increases by 0.56 ms for nonbinding 8192-token prompts, 0.51 ms for 1024-token prompts, and 0.78 ms for bigram prompts. The added work is SWA page release and tombstone insertion, not just the hook dispatch.

Supporting capacity result: in the nonbinding 8192-token workload, retained live SWA across all 24 requests decreases from 196608 to 18432 slots (10.7×). These are pool slots, not reduced CUDA reservation.

Measurement limits

  • These synthetic cache fixtures do not execute model inference. The percentages are not serving-throughput changes or quality measurements; no TPM/TTFT A/B is available.
  • The fixed-set fixture allows whole-prompt matches with zero unmatched tail. Real scheduling caps the match at input_len - 1 and page-ceils the extension, so 3.62× must not be interpreted as a serving speedup or full-prefill elimination.
  • Source/flag-off controls were also measured. Within-round ordering was fixed, and the late bigram/SWA-pressure batches had same-node co-tenancy. In particular, the bigram baseline-off and patched-off ranges do not overlap; controls do not establish that all timing drift canceled.
  • Original eviction counters include the four warmup requests, while timing excludes them. No measured-window eviction-rate claim is made here.
  • Earlier SWA trimming can affect shorter/branching-prefix reuse; it is not a claim that every removed slot is useless to every possible future request.

Correctness and checks

Test file: test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py (existing CUDA/AMD CI registration).

  • Final TestSWAFinishedPrefill: 6 passed / 10 subtests, on both Python and Rust tree cores. Unmodified source reproduces 5 failures with the same tests.
  • Covers flag on/off, normal/bigram keys, short/page-boundary prompts, real borrowed shared-prefix KV and locks with exact allocator accounting and full reclamation, prior eviction, retention floor, and non-inserting release.
  • Full-file run before tightening the shared-prefix fixture: Python 1264 passed, 1435 fixture skips, 0 failures; Rust 1242 passed, 1456 skips, 1 pre-existing registry-override failure, also present in the unmodified tree. The final affected class was rerun on both cores after the fixture repair; the full file was not rerun.
  • Final changed-file pre-commit: 17 hooks passed, including formatting, AST and CI-registration checks.

Performance/full-file evidence used source at ad09abeefb67404a0ba5c0047b28c26833d354ad; final source at 2beaba852a8afb3107616e4d608acf5a377571c5 differs only in comments, with independently verified whole-module AST identity. The final test revision was used for the targeted reruns above.

HiCache, multi-rank/TP, SWA+Mamba integration and million-token contexts were not exercised. Keep draft pending end-to-end serving A/B and review of the default-on cleanup-cost versus cache-capacity tradeoff.


CI States

Latest PR Test (Base): ❌ Run #35412957728
Latest PR Test (Extra): ❌ Run #35412957632
Latest PR Test (AMD ROCm 10): ❌ Run #35412957743

…efill

A request that finishes directly after prefill cached its whole final chunk as
live sliding-window KV: cache_finished_req never called the component
free_out_of_window_slots hook that cache_unfinished_req already calls, so an
8192-token single-output-token request retained 8192 SWA slots instead of a
window, and the node it inserted was recorded with live SWA for tokens whose
read window had long passed.

Reuse the existing hook under the existing flag, measured against the
page-aligned radix key the way the unfinished path measures it: an EAGLE bigram
key spans one token more than its length, so its insert stops one page lower.
The native helper keeps the tree-owned prefix, page alignment, prior eviction
and the retention floor intact. No allocator or scheduling algorithm changes.

Test: TestSWAFinishedPrefill in
test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py -- the
unmodified source retains the whole aligned key on the flag-on arm, the fix
retains two pages of live SWA and preserves the full aligned prefix match.
- test_protected_prefix_is_not_trimmed now stages the request the way the scheduler
  admits a prefill: real cache.match_prefix, matched tree-owned indices borrowed into
  the request row, only the unmatched tail allocated, real lock receipt, and the
  independent live-request lock kept. It asserts exact swa_evicted_seqlen and
  FULL/SWA allocator accounting, that the borrowed prefix and the finished span stay
  reusable, and that a full evict after unlock reclaims every owned slot.
- Comments only in the source: describe the page-aligned insert frontier and the live
  window the created leaf retains instead of claiming the removed slots are
  universally unreadable (AST-identical to the measured revision).
apex-mochen added a commit to apex-mochen/sglang that referenced this pull request Oct 6, 2026
A lock walk that overshoots the sliding window pins the whole crossing
node, so a match ending just short of the window lets a long live-SWA
ancestor (e.g. a finished request's untrimmed last chunk) be pinned in
full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the
native Rust TreeCore: before locking a node, split off everything but the
page-aligned tail the trailing window still needs.

Only live-on-device, unlocked, non-write-through-pending nodes are split
(SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool);
the original node keeps its id as the in-window tail, so a lock receipt
anchored on it stays valid. sgl-project#40283 is complementary (it removes the
known trigger) rather than a replacement for this lock-walk split.

Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window`
locks a 7-token endpoint under an 8-token window with a 32-token live
ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock
round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_
crossing_node` is updated for the split semantics.

Closes the Rust gap reported in sgl-project#41579.

Signed-off-by: apex-mochen <2756823972@qq.com>
apex-mochen added a commit to apex-mochen/sglang that referenced this pull request Oct 7, 2026
A lock walk that overshoots the sliding window pins the whole crossing
node, so a match ending just short of the window lets a long live-SWA
ancestor (e.g. a finished request's untrimmed last chunk) be pinned in
full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the
native Rust TreeCore: before locking a node, split off everything but the
page-aligned tail the trailing window still needs.

Only live-on-device, unlocked, non-write-through-pending nodes are split
(SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool);
the original node keeps its id as the in-window tail, so a lock receipt
anchored on it stays valid. sgl-project#40283 is complementary (it removes the
known trigger) rather than a replacement for this lock-walk split.

Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window`
locks a 7-token endpoint under an 8-token window with a 32-token live
ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock
round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_
crossing_node` is updated for the split semantics.

Closes the Rust gap reported in sgl-project#41579.

Signed-off-by: apex-mochen <2756823972@qq.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks @Stelath. Closing this because it is still a draft and has not been updated in 29 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Oct 19, 2026
apex-mochen added a commit to apex-mochen/sglang that referenced this pull request Oct 20, 2026
A lock walk that overshoots the sliding window pins the whole crossing
node, so a match ending just short of the window lets a long live-SWA
ancestor (e.g. a finished request's untrimmed last chunk) be pinned in
full. Mirrors Python `_maybe_split_for_window_lock` from sgl-project#41580 in the
native Rust TreeCore: before locking a node, split off everything but the
page-aligned tail the trailing window still needs.

Only live-on-device, unlocked, non-write-through-pending nodes are split
(SGLANG_SWA_LOCK_WINDOW_ONLY, default on, matching the Python EnvBool);
the original node keeps its id as the in-window tail, so a lock receipt
anchored on it stays valid. sgl-project#40283 is complementary (it removes the
known trigger) rather than a replacement for this lock-walk split.

Regression: `swa_lock_walk_splits_a_long_live_ancestor_to_one_window`
locks a 7-token endpoint under an 8-token window with a 32-token live
ancestor and asserts 8 pinned tokens instead of 39, plus a clean unlock
round-trip. Existing `acquire_lock_overshooting_the_window_stops_at_the_
crossing_node` is updated for the split semantics.

Closes the Rust gap reported in sgl-project#41579.

Signed-off-by: apex-mochen <2756823972@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant