Skip to content

[Fix] scheduler: Free a chunked prefill's out-of-window SWA slots before sizing its next chunk - #41263

Open
bill-h-lin wants to merge 1 commit into
sgl-project:mainfrom
bill-h-lin:swa-evict-before-chunk-plan
Open

bill-h-lin wants to merge 1 commit into
sgl-project:mainfrom
bill-h-lin:swa-evict-before-chunk-plan

Conversation

@bill-h-lin

@bill-h-lin bill-h-lin commented Sep 25, 2026 •

Copy link
Copy Markdown

Motivation

On a hybrid-SWA model served with --disable-radix-cache, when the SWA pool is sized by SWARequestCapPoolConfigurator, a long prompt's chunks shrink after the second one. On MiMo-V2.6-Flash (window 128, --chunked-prefill-size 4096, --max-running-requests 1, overlap on) the SWA pool is 8,451 tokens, and a 127,852-token prompt ran as 2 × 4,096 and then 258 and 3,968 alternating: 60 forwards instead of 32. On current main, gpt-oss-20b (window 128, page size 64, SWA pool 8,640) runs a 33,644-token prompt as 4096, 4096, then 384 and 3968 alternating, ending with 3308: 14 forwards instead of 9 (the reviewer's run on one B200).

The cause is ordering. PrefillAdder.add_chunked_req sizes a continuing chunk from SWAPrefillBudget.available_chunk_tokens, which is the SWA pool's free count. At that point the previous chunks' out-of-window rows are still held: ScheduleBatch.maybe_evict_swa retires them only later, in alloc_for_extend, once the chunk is placed.

compute_swa_request_cap already assumes those rows are gone by then. Under overlap it budgets two chunks plus one window (chunks_in_flight = 2), which is what is live when chunk N+1 is placed while chunk N runs, once everything before chunk N's window has been retired. Without overlap it budgets one chunk, and the old order shrinks chunks there too (4096, 258, 3968, …; see the unit test).

With a pool below two chunks plus a window, the same order stops the request for good: under overlap the first two chunks are both held when the third is sized, nothing fits, add_chunked_req returns the request unscheduled, and maybe_evict_swa never runs because no chunk is placed. At 6,200 tokens the unit test's old order plans nothing after [4096, 2103] (the hang in #36458 has the same mechanism; I have not run its DeepSeek-V4 repro).

Modifications

  • mem_cache/allocation.py:
    • prefill_window_frontier(): maybe_evict_swa's extend branch as one function. No frontier for a cache that shares prefixes; without overlap, the chunk's prefix_len; under overlap, nothing before the third chunk and one chunk back after it, because the chunk in flight still reads the window before its start.
    • retire_window_before_chunk(): for a chunked request on an SWA cache, retires through retire_window to that frontier. It skips a per-request SWA ring, whose chunks are not sized from the free count (SWAPrefillBudget.available_chunk_tokens skips the window for a ring), and a request that holds no KV.
  • managers/schedule_batch.py: maybe_evict_swa's extend branch calls prefill_window_frontier, with no change in behaviour.
  • managers/scheduler.py: _get_new_batch_prefill_raw calls retire_window_before_chunk before building the adder whenever a chunked request is in progress, PD prefill included (its sender reads window rows only on the last chunk, from the retained tail). dLLM is skipped: _add_chunked_req sizes its blocks from _get_dllm_remain_tokens, not from the window's free rows.
  • test_scheduler_held_row_admission.py: the scheduler stub's tree_cache gets supports_swa, which the new call reads.
  • test_request_demand.py: patches get_schedule in mem_cache.allocation, where maybe_evict_swa now reads the chunk size.

Both callers take the same frontier, so this retires exactly the rows maybe_evict_swa would retire later in the same step, and that pass then finds nothing left for this request. After a chunk cut short (e.g. by shortest_prefill_chunk_limit), the overlap frontier trails the chunk in flight, and what it retires is still enough: the rows held are then one chunk plus one window, which is what compute_swa_request_cap budgets. The unit test checks both.

Accuracy Tests

The change moves chunk boundaries, so prompts longer than two chunks no longer get bit-identical output: different tokens share a forward. To check that the schedule is the only thing that changes, I ran a control: the old order with the SWA pool enlarged by one chunk, so the old planner produces the same full chunks. On the 127,852-token MiMo prompt the fix and the control gave identical output token ids and identical logprobs for all 64 output tokens, on each of two runs. Prompts that fit in two chunks were identical to the unpatched build. On gpt-oss-20b (current main, page size 64) the reviewer's run gave the same 10 greedy output tokens before and after.

The MiMo runs were on a local build at 542a043 that also carries unrelated out-of-tree changes (a MoE expert offload path). Since then main has moved under the branch several times; the current version is written on 9fdd39206d after review (the helper moved to allocation.py and shares its frontier with maybe_evict_swa, the kill switch is gone, PD prefill is no longer skipped). The MiMo run took the frontier from the start of the chunk in flight; with full chunks that equals the frontier above, which is what that run had at every step.

Speed Tests and Profiling

MiMo-V2.6-Flash, one RTX PRO 6000 Blackwell, one request, 127,852-token prompt, two runs each:

before after
prefill forwards 60 32
prefill throughput (tok/s) 508 / 541 722 / 741
end to end (s) 253.9 / 238.7 179.5 / 174.9
decode, short prompts (tok/s) 32.74 32.73

This box streams MoE experts from host memory and SSD (out of tree), so the absolute throughput is not representative. The forward count is the part that carries over; gpt-oss-20b on a B200 (above) goes from 14 to 9.

Unit test

test/registered/unit/mem_cache/test_retire_window_before_chunk.py runs on CPU. It drives one 33,644-token request through the real SWATokenToKVPoolAllocator, UnifiedRadixCache with disable=True (what --disable-radix-cache builds for a hybrid SWA model), PrefillAdder.add_chunked_req and ScheduleBatch.maybe_evict_swa, at the pool size compute_swa_request_cap gives (chunk 4096, window 128, one running request), at page sizes 1 and 64:

  • old order, overlap: page 1, 4096, 4096, then 258 and 3968 alternating (the sequence we saw on MiMo); page 64, 4096, 4096, then 384 and 3968 alternating, ending with 3308 (the reviewer's gpt-oss-20b sequence);
  • this change, with and without overlap, at both page sizes: 8 × 4096 + 876; with the third chunk cut to 1,000 tokens (page 1), 4096, 4096, 1000, 5 × 4096, 3972;
  • a 6,200-token pool: the old order stalls after [4096, 2103] under overlap; with the change it completes (each later chunk 1,975 tokens under overlap, full chunks without);
  • after each early retirement, every row the in-flight chunk reads is still mapped in the SWA pool, and the full pool is untouched;
  • the maybe_evict_swa in alloc_for_extend then finds nothing left to retire;
  • a cache without SWA, a prefix-sharing cache, an SWA ring, a request without KV, and the first chunk in flight under overlap are each left alone (each case starts from a continuing request, so only the check under test can stop it).

The test fails for each of nine mutations: no early retirement; overlap ignored by the early retirement; an early frontier one chunk behind maybe_evict_swa's; the shared frontier one more chunk back; and each of the five checks above removed. Upstream's unit tests that touch these paths (56 files, 1,990 tests, run with a GPU visible) give the same results as on main.

Not covered

Several concurrent chunked or decoding requests, dLLM, and the non-overlap path end to end (it is covered by the unit test only). The radix-cache path is unchanged, because the frontier is None for prefix-sharing caches; the scripted SWA chunked-prefill tests (test_scripted_swa_1gpu.py) run with the radix cache on, so they don't reach it.

Checklist

  • Format your code according to the Format code with pre-commit (isort 7.0.0, ruff 0.15.1, ruff-format, codespell 2.4.1 on the changed files).
  • Add unit tests.
  • Update documentation: nothing user-facing changes (no flag).
  • Provide accuracy and speed benchmark results (above).
  • Follow the SGLang code style guidance.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #42939266721
Latest PR Test (Extra): ❌ Run #42939266562
Latest PR Test (AMD ROCm 10): ❌ Run #42939267602

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@bill-h-lin

Copy link
Copy Markdown
Author

Rebased onto main 0b635266d4 and pushed (1c6b653a3e). main had moved under it: the eviction now goes through tree_cache.evict_sliding_windows, chunk caches are told apart by supports_prefix_sharing(), and per-request SWA rings are new. So free_chunked_swa_before_plan now calls evict_sliding_windows (the path maybe_evict_swa takes) and returns early for prefix-sharing caches, as before, and for an SWA ring, whose chunks aren't sized from the free count.

The bug is still on main: the test's old-order control gives 4096, 4096, then 258 and 3968 alternating; with the change, 8 × 4096 + 876, with and without overlap. While re-checking I found that the radix-cache case returned at the extend_range is None guard before it reached the cache check, so it didn't test that check; it and the new ring case now start from a continuing request. The test fails for each of five mutations: no early free, the in-flight chunk's end as the bound, overlap ignored, and the ring or the prefix-sharing check removed. Upstream's unit tests that touch these paths (15 CPU files, 3 CUDA) give the same results as on main. Description updated.

@hnyls2002, you've merged most of the recent mem_cache changes; could you take a look, and add run-ci if it looks right? I'm not in CI_PERMISSIONS.json.

@bill-h-lin
bill-h-lin force-pushed the swa-evict-before-chunk-plan branch from 1c6b653 to ffe840c Compare October 12, 2026 17:54
@bill-h-lin

Copy link
Copy Markdown
Author

Ported onto main f92199b569 and pushed (ffe840c9ea). main had moved under it again: your #42822, #42824 and #42825 replaced extend_range and Req.prefix_indices with prefix_len / extend_end, and #43201 deleted ChunkCache, which the test was built on.

  • The bound is now maybe_evict_swa's own: prefix_len, and under overlap prefix_len − chunked_prefill_size, with nothing freed before the third chunk. The previous version took the start of the chunk in flight, which isn't kept once that chunk is checkpointed. With full chunks the two are equal; after a chunk cut short this one frees less, and still enough (a new test case cuts the third chunk to 1,000 tokens, and the rest stay full). So the early free is the free alloc_for_extend would do in the same step, moved before the chunk is sized, and the test checks that maybe_evict_swa then finds nothing left.
  • The test now runs on UnifiedRadixCache(disable=True), which is what --disable-radix-cache builds for a hybrid SWA model.

The bug is still on main: the old-order control gives 4096, 4096, then 258 and 3968 alternating; with the change, 8 × 4096 + 876, with and without overlap. Each of eight mutations fails the test it targets. Upstream's unit tests that touch these paths (28 files, 3 of them on CUDA) give the same results as on main. Description updated.

@hnyls2002, since this now sits on your prefix_len refactor, could you take a look, and add run-ci if it looks right? I'm not in CI_PERMISSIONS.json.

@bill-h-lin
bill-h-lin force-pushed the swa-evict-before-chunk-plan branch from ffe840c to 7463f1e Compare October 14, 2026 04:45

@xiezhq-hermann xiezhq-hermann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bill-h-lin, and sorry for the slow reply to your 09-28 ping on #41580 (I reviewed #41580 today). This PR is independent of #41580/#42745. Those change the SWA lock walk in the prefix-sharing tree, and this one returns early for any cache that shares prefixes, so they can land in either order.

The bug is real and still on current main (1be97a0a631):

  • End to end: gpt-oss-20b (window 128) on one B200, --disable-radix-cache --max-running-requests 1 --chunked-prefill-size 4096, overlap on, page size 64 (trtllm_mha), SWA pool 8,640 tokens. One 33,644-token prompt runs as 4096, 4096, then 384 and 3968 alternating, and ends with 3308: 14 prefill forwards. With this PR ported onto main it runs as 8 × 4096 + 876 (9 forwards), with the same 10 greedy output tokens. So the fix also holds at page size > 1, which your tests don't cover.
  • Your unit test, ported to main: the old-order control still gives 4096, 4096, then 258 and 3968 alternating. With the change, all 9 tests pass.
  • A pool below two chunks plus a window: at 6,200 tokens in the same harness with overlap on, the old order plans nothing after [4096, 2103]. add_chunked_req returns the request unscheduled, and maybe_evict_swa only runs once a chunk is allocated. That looks like the hang reported in #36458. With the early free it keeps going, so this would be a good test case to add.

Before we can merge:

  1. Rebase. It conflicts in scheduler.py and mem_cache/common.py, and the APIs underneath have moved:

    • evict_sliding_windows is gone since #46991. maybe_evict_swa now goes through retire_window(req, frontier, tree_cache) in mem_cache/allocation.py. Since allocation.py imports common.py, the helper has to live in allocation.py.
    • In the test:
      • _swa_eviction_trigger is now _window_eviction_trigger.
      • Window cursors are keyed by sub-pool name since #48007 (req.kv.get_evicted_seqlen(BaseSWAKVPool.WINDOW_SUB_POOL)).
      • The cache is built with tree_slots=tree_slots_of((ComponentType.FULL, ComponentType.SWA), window=WINDOW).
      • The storage needs kv_storage_of(req_to_token_pool, allocator).bind_window_retention(WINDOW, None).
    • test/registered/unit/managers/test_scheduler_held_row_admission.py (added after your base) stubs tree_cache without supports_swa. Once the new call is in _get_new_batch_prefill_raw, its NULL-mode case fails with an AttributeError. Adding supports_swa=lambda: False to that stub fixes it.

    With exactly these changes on main + this PR, your test passes, and the 17 unit files that cover these paths (that one included) give the same results as main: 747 passed, 59 skipped.

  2. Share the bound with maybe_evict_swa. The new helper copies its extend branch (skip before the third chunk, one chunk back under overlap). Please move that into one function that both call, so the two can't drift and the alloc-time pass stays a no-op for this request.

  3. Drop SGLANG_SWA_EVICT_BEFORE_CHUNK_PLAN. The change frees the same rows in the same scheduling step, just before the chunk is sized. I don't think we need a kill switch for that.

  4. PD prefill: is there a reason to skip it? SWARequestCapPoolConfigurator also applies to a chunked prefill server with --disable-radix-cache, and maybe_evict_swa already frees these rows there in the same step. If you keep the exclusion, please say why in the code comment.

  5. Title: the "Check PR title" job needs the [Type] scope: title form, e.g. [Fix] scheduler: Free a chunked prefill's out-of-window SWA slots before sizing its next chunk.

CI hasn't run on this PR yet. Happy to start it once it's rebased.

cc @ch-wan: this moves window retirement for a chunked prefill ahead of admission, which relates to your TODO in kv/alloc/demand.py about moving retirement into the memory layer.

…ore sizing its next chunk

PrefillAdder.add_chunked_req sizes a continuing chunk from the SWA pool's free
rows, but ScheduleBatch.maybe_evict_swa retires the previous chunks'
out-of-window rows only later, in alloc_for_extend. At the SWA pool size
compute_swa_request_cap gives, every chunk after the second then alternates
between the slack and chunk - window tokens (a 127,852-token prompt on
MiMo-V2.6-Flash took 60 forwards instead of 32; gpt-oss-20b at page size 64,
14 instead of 9 on a 33,644-token prompt). Under overlap with a pool below two
chunks plus a window, no third chunk fits, so none is placed, maybe_evict_swa
never runs, and the request never runs again.

Retire them before the chunk is sized: retire_window_before_chunk in
mem_cache/allocation.py goes through retire_window, to the frontier
maybe_evict_swa uses for the same request in the same step. That frontier
(maybe_evict_swa's extend branch: none for a prefix-sharing cache, none before
the third chunk under overlap, else one chunk back) is now one function,
prefill_window_frontier, that both call, so the alloc-time pass finds nothing
left for this request. A per-request SWA ring is skipped (its chunks are not
sized from the free rows), and so is dLLM (its blocks are sized from its own
budget). PD prefill is not skipped: its sender reads window rows only on the
last chunk, from the retained tail.

The held-row admission test's tree_cache stub gains supports_swa, which the
new call reads, and test_request_demand patches get_schedule where
maybe_evict_swa now reads the chunk size (mem_cache.allocation).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bill-h-lin
bill-h-lin force-pushed the swa-evict-before-chunk-plan branch from 7463f1e to 213f3d8 Compare November 11, 2026 01:28
@bill-h-lin bill-h-lin changed the title [mem_cache] Free a chunked prefill's out-of-window SWA slots before sizing its next chunk [Fix] scheduler: Free a chunked prefill's out-of-window SWA slots before sizing its next chunk Nov 11, 2026
@bill-h-lin

Copy link
Copy Markdown
Author

Thanks for the review, and for the B200 run at page size 64. Rewritten on main 9fdd39206d and pushed (213f3d843e):

  • Rebase. The helper is retire_window_before_chunk in mem_cache/allocation.py and goes through retire_window. The test is ported with the changes you listed. It drives a real ScheduleBatch rather than binding the private trigger, and it is now test_retire_window_before_chunk.py. The held-row stub gets supports_swa=lambda: False. With PD prefill no longer skipped, both its NULL and PREFILL cases needed it. test_request_demand.py patched schedule_batch.get_schedule for maybe_evict_swa's chunk size, so it now patches mem_cache.allocation.get_schedule, where the shared frontier reads it.
  • Shared bound. maybe_evict_swa's extend branch is now prefill_window_frontier(req, prefix_len, tree_cache, *, enable_overlap) in allocation.py. It returns no frontier for a prefix-sharing cache, nothing before the third chunk under overlap, and one chunk back otherwise. maybe_evict_swa and the new helper both call it, and a test checks that the alloc-time pass then retires nothing for the chunked request.
  • SGLANG_SWA_EVICT_BEFORE_CHUNK_PLAN is gone.
  • PD prefill: no reason to skip it, so the exclusion is removed. The sender reads window rows only on the last chunk (_swa_payload under if last_chunk:, from the retained tail), and those rows are never retired. dLLM is still skipped, with the reason in the comment: _add_chunked_req sizes its blocks from _get_dllm_remain_tokens, not from the window's free rows, so retiring earlier buys it nothing.
  • New cases. Page size 64: the old-order control gives your sequence (4096, 4096, then 384 and 3968 alternating, 3308 last), and the change gives 8 × 4096 + 876 with and without overlap. The 6,200-token pool under overlap: the old order plans nothing after [4096, 2103]; with the change it completes, at 1,975 tokens per chunk after the second.
  • Results: 11 / 11; each of nine mutations fails it (no early retirement, overlap ignored, an early frontier behind maybe_evict_swa's, the shared frontier one chunk further back, and each of the five guards removed). The 56 upstream unit files that touch these paths (1,990 tests) give the same results as main. Lint clean.
  • Title updated.

On the demand.py TODO you cc'd ch-wan on: this keeps the scheduler-side rule (retirement keyed on prefix_len) and only moves the call ahead of chunk sizing. So window_retirement's prefill_retire_lag (one chunk under overlap, without prefix sharing) still describes it.

Ready for run-ci whenever you have a moment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants