Repository navigation
[Fix] scheduler: Free a chunked prefill's out-of-window SWA slots before sizing its next chunk - #41263
[Fix] scheduler: Free a chunked prefill's out-of-window SWA slots before sizing its next chunk#41263bill-h-lin wants to merge 1 commit into
Conversation
39528b1 to
1c6b653
Compare
|
Rebased onto The bug is still on @hnyls2002, you've merged most of the recent |
1c6b653 to
ffe840c
Compare
|
Ported onto
The bug is still on @hnyls2002, since this now sits on your |
ffe840c to
7463f1e
Compare
xiezhq-hermann
left a comment
There was a problem hiding this comment.
Thanks @bill-h-lin, and sorry for the slow reply to your 09-28 ping on #41580 (I reviewed #41580 today). This PR is independent of #41580/#42745. Those change the SWA lock walk in the prefix-sharing tree, and this one returns early for any cache that shares prefixes, so they can land in either order.
The bug is real and still on current main (1be97a0a631):
- End to end: gpt-oss-20b (window 128) on one B200,
--disable-radix-cache --max-running-requests 1 --chunked-prefill-size 4096, overlap on, page size 64 (trtllm_mha), SWA pool 8,640 tokens. One 33,644-token prompt runs as 4096, 4096, then 384 and 3968 alternating, and ends with 3308: 14 prefill forwards. With this PR ported onto main it runs as 8 × 4096 + 876 (9 forwards), with the same 10 greedy output tokens. So the fix also holds at page size > 1, which your tests don't cover. - Your unit test, ported to main: the old-order control still gives 4096, 4096, then 258 and 3968 alternating. With the change, all 9 tests pass.
- A pool below two chunks plus a window: at 6,200 tokens in the same harness with overlap on, the old order plans nothing after
[4096, 2103].add_chunked_reqreturns the request unscheduled, andmaybe_evict_swaonly runs once a chunk is allocated. That looks like the hang reported in #36458. With the early free it keeps going, so this would be a good test case to add.
Before we can merge:
-
Rebase. It conflicts in
scheduler.pyandmem_cache/common.py, and the APIs underneath have moved:evict_sliding_windowsis gone since #46991.maybe_evict_swanow goes throughretire_window(req, frontier, tree_cache)inmem_cache/allocation.py. Sinceallocation.pyimportscommon.py, the helper has to live inallocation.py.- In the test:
_swa_eviction_triggeris now_window_eviction_trigger.- Window cursors are keyed by sub-pool name since #48007 (
req.kv.get_evicted_seqlen(BaseSWAKVPool.WINDOW_SUB_POOL)). - The cache is built with
tree_slots=tree_slots_of((ComponentType.FULL, ComponentType.SWA), window=WINDOW). - The storage needs
kv_storage_of(req_to_token_pool, allocator).bind_window_retention(WINDOW, None).
test/registered/unit/managers/test_scheduler_held_row_admission.py(added after your base) stubstree_cachewithoutsupports_swa. Once the new call is in_get_new_batch_prefill_raw, its NULL-mode case fails with anAttributeError. Addingsupports_swa=lambda: Falseto that stub fixes it.
With exactly these changes on main + this PR, your test passes, and the 17 unit files that cover these paths (that one included) give the same results as main: 747 passed, 59 skipped.
-
Share the bound with
maybe_evict_swa. The new helper copies its extend branch (skip before the third chunk, one chunk back under overlap). Please move that into one function that both call, so the two can't drift and the alloc-time pass stays a no-op for this request. -
Drop
SGLANG_SWA_EVICT_BEFORE_CHUNK_PLAN. The change frees the same rows in the same scheduling step, just before the chunk is sized. I don't think we need a kill switch for that. -
PD prefill: is there a reason to skip it?
SWARequestCapPoolConfiguratoralso applies to a chunked prefill server with--disable-radix-cache, andmaybe_evict_swaalready frees these rows there in the same step. If you keep the exclusion, please say why in the code comment. -
Title: the "Check PR title" job needs the
[Type] scope: titleform, e.g.[Fix] scheduler: Free a chunked prefill's out-of-window SWA slots before sizing its next chunk.
CI hasn't run on this PR yet. Happy to start it once it's rebased.
cc @ch-wan: this moves window retirement for a chunked prefill ahead of admission, which relates to your TODO in kv/alloc/demand.py about moving retirement into the memory layer.
…ore sizing its next chunk PrefillAdder.add_chunked_req sizes a continuing chunk from the SWA pool's free rows, but ScheduleBatch.maybe_evict_swa retires the previous chunks' out-of-window rows only later, in alloc_for_extend. At the SWA pool size compute_swa_request_cap gives, every chunk after the second then alternates between the slack and chunk - window tokens (a 127,852-token prompt on MiMo-V2.6-Flash took 60 forwards instead of 32; gpt-oss-20b at page size 64, 14 instead of 9 on a 33,644-token prompt). Under overlap with a pool below two chunks plus a window, no third chunk fits, so none is placed, maybe_evict_swa never runs, and the request never runs again. Retire them before the chunk is sized: retire_window_before_chunk in mem_cache/allocation.py goes through retire_window, to the frontier maybe_evict_swa uses for the same request in the same step. That frontier (maybe_evict_swa's extend branch: none for a prefix-sharing cache, none before the third chunk under overlap, else one chunk back) is now one function, prefill_window_frontier, that both call, so the alloc-time pass finds nothing left for this request. A per-request SWA ring is skipped (its chunks are not sized from the free rows), and so is dLLM (its blocks are sized from its own budget). PD prefill is not skipped: its sender reads window rows only on the last chunk, from the retained tail. The held-row admission test's tree_cache stub gains supports_swa, which the new call reads, and test_request_demand patches get_schedule where maybe_evict_swa now reads the chunk size (mem_cache.allocation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7463f1e to
213f3d8
Compare
|
Thanks for the review, and for the B200 run at page size 64. Rewritten on
On the Ready for |
Motivation
On a hybrid-SWA model served with
--disable-radix-cache, when the SWA pool is sized bySWARequestCapPoolConfigurator, a long prompt's chunks shrink after the second one. On MiMo-V2.6-Flash (window 128,--chunked-prefill-size 4096,--max-running-requests 1, overlap on) the SWA pool is 8,451 tokens, and a 127,852-token prompt ran as 2 × 4,096 and then 258 and 3,968 alternating: 60 forwards instead of 32. On currentmain, gpt-oss-20b (window 128, page size 64, SWA pool 8,640) runs a 33,644-token prompt as 4096, 4096, then 384 and 3968 alternating, ending with 3308: 14 forwards instead of 9 (the reviewer's run on one B200).The cause is ordering.
PrefillAdder.add_chunked_reqsizes a continuing chunk fromSWAPrefillBudget.available_chunk_tokens, which is the SWA pool's free count. At that point the previous chunks' out-of-window rows are still held:ScheduleBatch.maybe_evict_swaretires them only later, inalloc_for_extend, once the chunk is placed.compute_swa_request_capalready assumes those rows are gone by then. Under overlap it budgets two chunks plus one window (chunks_in_flight = 2), which is what is live when chunk N+1 is placed while chunk N runs, once everything before chunk N's window has been retired. Without overlap it budgets one chunk, and the old order shrinks chunks there too (4096, 258, 3968, …; see the unit test).With a pool below two chunks plus a window, the same order stops the request for good: under overlap the first two chunks are both held when the third is sized, nothing fits,
add_chunked_reqreturns the request unscheduled, andmaybe_evict_swanever runs because no chunk is placed. At 6,200 tokens the unit test's old order plans nothing after [4096, 2103] (the hang in #36458 has the same mechanism; I have not run its DeepSeek-V4 repro).Modifications
mem_cache/allocation.py:prefill_window_frontier():maybe_evict_swa's extend branch as one function. No frontier for a cache that shares prefixes; without overlap, the chunk'sprefix_len; under overlap, nothing before the third chunk and one chunk back after it, because the chunk in flight still reads the window before its start.retire_window_before_chunk(): for a chunked request on an SWA cache, retires throughretire_windowto that frontier. It skips a per-request SWA ring, whose chunks are not sized from the free count (SWAPrefillBudget.available_chunk_tokensskips the window for a ring), and a request that holds no KV.managers/schedule_batch.py:maybe_evict_swa's extend branch callsprefill_window_frontier, with no change in behaviour.managers/scheduler.py:_get_new_batch_prefill_rawcallsretire_window_before_chunkbefore building the adder whenever a chunked request is in progress, PD prefill included (its sender reads window rows only on the last chunk, from the retained tail). dLLM is skipped:_add_chunked_reqsizes its blocks from_get_dllm_remain_tokens, not from the window's free rows.test_scheduler_held_row_admission.py: the scheduler stub'stree_cachegetssupports_swa, which the new call reads.test_request_demand.py: patchesget_scheduleinmem_cache.allocation, wheremaybe_evict_swanow reads the chunk size.Both callers take the same frontier, so this retires exactly the rows
maybe_evict_swawould retire later in the same step, and that pass then finds nothing left for this request. After a chunk cut short (e.g. byshortest_prefill_chunk_limit), the overlap frontier trails the chunk in flight, and what it retires is still enough: the rows held are then one chunk plus one window, which is whatcompute_swa_request_capbudgets. The unit test checks both.Accuracy Tests
The change moves chunk boundaries, so prompts longer than two chunks no longer get bit-identical output: different tokens share a forward. To check that the schedule is the only thing that changes, I ran a control: the old order with the SWA pool enlarged by one chunk, so the old planner produces the same full chunks. On the 127,852-token MiMo prompt the fix and the control gave identical output token ids and identical logprobs for all 64 output tokens, on each of two runs. Prompts that fit in two chunks were identical to the unpatched build. On gpt-oss-20b (current
main, page size 64) the reviewer's run gave the same 10 greedy output tokens before and after.The MiMo runs were on a local build at
542a043that also carries unrelated out-of-tree changes (a MoE expert offload path). Since thenmainhas moved under the branch several times; the current version is written on9fdd39206dafter review (the helper moved toallocation.pyand shares its frontier withmaybe_evict_swa, the kill switch is gone, PD prefill is no longer skipped). The MiMo run took the frontier from the start of the chunk in flight; with full chunks that equals the frontier above, which is what that run had at every step.Speed Tests and Profiling
MiMo-V2.6-Flash, one RTX PRO 6000 Blackwell, one request, 127,852-token prompt, two runs each:
This box streams MoE experts from host memory and SSD (out of tree), so the absolute throughput is not representative. The forward count is the part that carries over; gpt-oss-20b on a B200 (above) goes from 14 to 9.
Unit test
test/registered/unit/mem_cache/test_retire_window_before_chunk.pyruns on CPU. It drives one 33,644-token request through the realSWATokenToKVPoolAllocator,UnifiedRadixCachewithdisable=True(what--disable-radix-cachebuilds for a hybrid SWA model),PrefillAdder.add_chunked_reqandScheduleBatch.maybe_evict_swa, at the pool sizecompute_swa_request_capgives (chunk 4096, window 128, one running request), at page sizes 1 and 64:maybe_evict_swainalloc_for_extendthen finds nothing left to retire;The test fails for each of nine mutations: no early retirement; overlap ignored by the early retirement; an early frontier one chunk behind
maybe_evict_swa's; the shared frontier one more chunk back; and each of the five checks above removed. Upstream's unit tests that touch these paths (56 files, 1,990 tests, run with a GPU visible) give the same results as onmain.Not covered
Several concurrent chunked or decoding requests, dLLM, and the non-overlap path end to end (it is covered by the unit test only). The radix-cache path is unchanged, because the frontier is
Nonefor prefix-sharing caches; the scripted SWA chunked-prefill tests (test_scripted_swa_1gpu.py) run with the radix cache on, so they don't reach it.Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #42939266721
Latest PR Test (Extra): ❌ Run #42939266562
Latest PR Test (AMD ROCm 10): ❌ Run #42939267602
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: