Repository navigation
[Perf] hicache: Fence write_back eviction copies on the forward stream instead of blocking the scheduler - #50469
Open
weireweire wants to merge 6 commits into
Open
weireweire wants to merge 6 commits into
weireweire wants to merge 6 commits into
Conversation
…m instead of blocking the scheduler Root cause: the write_back eviction walk waits on the host (Event.synchronize) for its D2H backups before demoting the victims. The copies are fenced behind the in-flight forward, so on a busy prefill server the scheduler stalls until that forward ends; on DSv4.1-Flash prefill this wait was 50-56% of scheduler time. Fix: submit the queued copies, make the current (schedule) stream and the forward stream wait_event on each copy's finish event and finish the acks at once. Every later forward (on either stream, with or without overlap scheduling), every current-stream state copy, and through load_fence_stream every later load-back runs after the copy, so a freed device slot is never rewritten before its backup is read. Ack metrics are logged once the event completes. The blocking wait stays wherever freed slots or host pages can be touched outside forward/load stream order: L3 storage, the buffer pipeline, a cache linker, the unified memory pool (pages are moved and zeroed on the schedule stream), and PD decode (KV transfers write freed slots directly). Validation: DSv4.1-Flash GB300 disagg; GSM8K under full load 0.9735 / 0.9750; +3.0% e2e throughput, prefill 89.0k -> 94.3k tok/s/GPU.
weireweire
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
November 14, 2026 06:49
1 of 5 tasks
…m only without overlap scheduling Root cause: the current-stream wait added for the non-overlap path also ran under overlap scheduling. There the current stream is the schedule stream, so scheduler GPU work issued after an eviction waited for the in-flight forward and the copy, which cancelled the gain (same-day A/B: -0.5% e2e vs the bounded-reserve baseline). Fix: the scheduler records whether overlap scheduling is on. Only without it, where the forward and the Mamba state copies run on the current stream, does the eviction also make the current stream wait on the copies. With overlap the forward runs on forward_stream, which already waits. Validation: DSv4.1-Flash GB300 disagg, same-day A/B, 2 runs each: +2.9% e2e and +5.0% prefill tok/s/GPU over the bounded-reserve baseline; the unconditional current-stream wait gave -0.5%.
…ack-demote # Conflicts: # python/sglang/srt/mem_cache/unified_radix_cache.py
nvpohanh
reviewed
Nov 16, 2026
nvpohanh
left a comment
Collaborator
There was a problem hiding this comment.
[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.
…m-fenced demotion
4 of 5 tasks
nvpohanh
reviewed
Nov 16, 2026
nvpohanh
left a comment
Collaborator
There was a problem hiding this comment.
[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.
…allocator context in the fenced demotion test
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
@alphabetc1 hi, the ci looks good, could you have a look? The only fail is the env issue about numpy version. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With HiCache
write_back, the device eviction walk backs its victims up to host and then blocks the scheduler onEvent.synchronize()for those D2H copies before it demotes the victims. The copies are fenced behind the in-flight forward (write_fence_stream), so on a busy prefill server the scheduler cannot finish building the next batch until the current forward ends. On DSv4.1-Flash disaggregated prefill this wait was 50-56% of scheduler time, and the GPU sat idle while the next batch was prepared and launched.Modifications
_demote_backed_upno longer waits on the host. It submits the queued copies and makes the forward streamwait_eventon each copy's finish event, then finishes the acks and demotes the victims at once. Without overlap scheduling the forward runs on the current stream, so the current stream waits as well; the scheduler tells the cache controller which case applies.Ordering:
write_fence_streamfence), so it reads complete KV.no_bufferturns overlap off).load_fence_stream, which is the forward stream.The blocking wait is kept wherever freed slots or host pages can be touched outside forward/load stream order:
These are code conditions; there is no new flag. The other blocking
writing_check(write_back=True)callers are unchanged.Independent of #47650 (bounded write_back eviction reserve). The two stack: #47650 makes the waits less frequent, this change takes them off the host. Also different from #49498, which relaxes the copy's fence on the forward but keeps the host wait.
Accuracy Tests
DSv4.1-Flash, GB300, disaggregated prefill/decode with HiCache
write_back, simulated acceptance off. GSM8K (5-shot, temperature 0, concurrency 256) run while the full benchmark load was running, so later passes reuse prompt KV that had been demoted to host and loaded back under load:Speed Tests and Profiling
DSv4.1-Flash, GB300, disaggregated prefill/decode (14 prefill servers + DEP16 decode, overlap scheduling on), agentic multi-turn replay at a mid-curve concurrency, 1200 s per run. Same-day A/B on this PR's head, 2 runs per arm, same image, only this PR's diff differs; both arms include #47650:
Run-to-run spread within an arm is 0.3-0.6%. The gain comes from the prefill step rate: the eviction wait no longer sits between forwards, so the prefill queue drains faster (TTFT down) and decode runs fuller (TPOT up by ~1%). An earlier profile of this setup showed prefill GPU idle falling from 20.6% to 9.5% with this change.
An earlier A/B on an older stack also showed why the current (schedule) stream must not wait under overlap: with an unconditional current-stream wait the gain was gone (-0.5% vs. baseline, against +2.9% for this PR's rule there).
Checklist
test_write_back_eviction_batching.pychecks that the stream-fenced demotion never waits on the host and that the forward stream (and, without overlap, the current stream) waits on each copy before its victims' pages are freed.CI States
Latest PR Test (Base): ✅ Run #43999955293
Latest PR Test (Extra): ❌ Run #43999954985
Latest PR Test (AMD ROCm 10): ✅ Run #43999955222
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: