Repository navigation
[Perf] hicache: Space out write_back eviction waits with a bounded reserve - #47650
weireweire wants to merge 1 commit into
Conversation
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
| SGLANG_MAX_KV_CHUNK_CAPACITY = EnvInt(128 * 1024) | ||
| # Extra tokens a write_back HiCache eviction frees beyond the shortfall; each | ||
| # such eviction waits for the in-flight forward, so a reserve spaces the waits out. | ||
| SGLANG_OPT_HICACHE_WRITE_BACK_EVICTION_HEADROOM_TOKENS = EnvInt(0) |
|
Thanks @weireweire. Your DSV4.1-Flash PD A/B and the GPU-idle trace (prefill GPU idle 15.1% -> ~6%) are the clearest real-world motivation we have for this. The mechanism reproduced on 11-05 main (944549c, with #46930). Setup: Qwen2.5-1.5B, write_back, a 131K-token device pool, Rust core. This PR at 4096, at 80 req/s:
In a smaller-pool run at concurrency 8, eviction was only ~4% of wall time, and 4096 showed no gain (-3.7%, t ~2). So the benefit depends on how much of the step the wait takes. #48047 does the same thing at the same spot, and we only want one env var for this idea. So I'm consolidating on #48047 and will close this PR when it lands. Why that direction:
Both PRs now conflict with #48012. It moved allocation-driven eviction to a It would help a lot if you could:
|
…serve Root cause: with --hicache-write-policy write_back, a device eviction backs its victims up to host and blocks on that copy (writing_check(write_back=True) -> ack.finish_event.synchronize()). The D2H stream is fenced on the forward stream, so every eviction pass waits for the whole in-flight forward. On a saturated prefill instance the device pool is full and nearly every scheduling step evicts exactly its shortfall, so the scheduler loses its one-step lookahead each step. Under DP attention the peer rank then waits for it in the DP-attention all-gather, and the GPU idles between forwards. Fix: add opt-in SGLANG_OPT_HICACHE_WRITE_BACK_EVICTION_HEADROOM_TOKENS (default 0, behavior unchanged). Under write_back, UnifiedRadixCache.evict() and the per-pool shortfall eviction of evict_for_alloc() (evict_pool_shortfalls) grow a nonzero token/window shortfall by the reserve, capped by the evictable size; the shortfall eviction's stop demand grows by the same amount. This covers the paged allocator's eviction, HiCache load-back and the linker LOAD path. Joint reclaim is unchanged because the allocator owns that stop condition. Eviction order is unchanged; freed slots return to the free pool. The reserve is read at eviction time, so caches built without __init__ never touch it while it is 0. Validation (DSV4.1-Flash PD disaggregation on GB300; prefill 6x DP-attention DEP2 with write_back HiCache, decode DEP16; agentic multi-turn trace at a P-heavy mid-curve point, 20 min; reserve 262144 tokens, ~0.3% of the prefill pool; A/B run concurrently on the same image, both arms with sgl-project#46884): - Client output tput/GPU: +0.5% (min 8-20 bins) / +1.0% (whole run) at the same p90 interactivity (190.6 vs 191.8 tok/s/user); inside run-to-run noise. - TTFT p90 7.89 vs 8.45 s (-6.6%), p50 2.16 vs 2.28 s (-5%). - Prefill steps/s per rank 3.21 vs 3.01 (+6.6%); P queue p50 6 vs 11. - Prefix hit 0.9647 vs 0.9646; decode accept length 3.51 vs 3.51; 0 invalid records. - GSM8K (sgl-eval, 1319, 1P1D): 96.66%, 0% truncated/errors; baselines on the same build 96.51-96.97%. - Mechanism: in an nsys trace of the equivalent change with the reserve applied at the allocator call sites, prefill GPU idle fell from 15.1% to ~6%, and eviction stalls came every 4-5 forwards instead of every forward. A trace of this exact diff was not taken. - Not run: unit tests. pre-commit on the changed files passes.
97ec4ec to
5adcd7c
Compare
|
hi thabks for review! with the floor format, when we evict a big chunk, there will be no room for next round. should we do this special logic to the big chunk? about the perf difference, it may come from the less real eviction under the floor format, param tuning may fix,i can test. I'm not sure what's the theory to use floor instead of a fixed room. no unit test is intended as we already have many test, it is easy to add if you want. |
…falls Merge upstream main, which moved allocation-driven eviction to evict_pool_shortfalls(demand) (sgl-project#48012). There the walk stops at the first victim after which its pool serves the demand, so the floor now raises the token entry of the demand together with the token shortfall; raising only the shortfall would free nothing extra. - Rename the variable to SGLANG_HICACHE_WRITE_BACK_MIN_EVICT_TOKENS: a write_back-only minimum per allocation-driven eviction, Full KV only. - Check the shortfall before reading is_write_back, so a disabled floor never reaches the property. - Document the variable in environment_variables.mdx, adapted from the draft in sgl-project#47650 by @weireweire. - Replace the SimpleNamespace test with one on the write-back fixtures, run on both tree cores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I rebased and did some test, didn't see L1 cost under same parameter(see below). And the perf is also similar.(just for reference, as it's not apples to apples compare with same reserve parameter). What I care is how do we set the parameter, if we set the "floor"(minimal eviction) the perf is depends on the workload, for example, if we set to 16k, when real eviction is only 1k, it will evict 15k more, when real eviction is 30k, it will eviction 0 more. I don't know why we should take 1k/30k as different instead of just evict fixed space more. Also it may be hard to select the param as it need to consult the real chunk size. Single GPU, Qwen2.5-1.5B, write_back HiCache, Rust core, current main. Same value (16384) for both forms
DSV4.1-Flash PD on GB300 (6x DEP2 prefill with write_back HiCache, DEP16 decode, agentic trace, 20 min)
The PD workload above runs at c4370. Workload (b) on the single GPU only covers the long-prompt case. I didn't put the job IDs in the table. |
|
could we have a look at this pr again? @alphabetc1 @xiezhq-hermann |
Motivation
With
--hicache-write-policy write_back, a device eviction backs its victims up to host and blocks on that copy (writing_check(write_back=True)->ack.finish_event.synchronize()). The D2H stream is fenced on the forward stream (#44800), so every eviction pass waits for the whole in-flight forward.On a saturated prefill instance the device pool is full, so nearly every scheduling step evicts exactly its shortfall and pays that wait. The scheduler loses its one-step lookahead every step. Under DP attention the peer rank waits for it in the DP-attention all-gather, and the GPU idles between forwards. In a PD prefill trace this was 14-15% GPU idle, almost all of it between forwards.
#46884 removed an earlier form of this wait (the coarse WAR barrier behind the write-back index copy). The eviction ack wait remains. #41690 adds a reserve at some allocator call sites. This PR instead puts the reserve inside
UnifiedRadixCache.evict()and the per-pool shortfall eviction ofevict_for_alloc()(evict_pool_shortfalls), which stay exact-shortfall today. Those are reached by the paged allocator's eviction, HiCache load-back and the linker LOAD path.Modifications
SGLANG_OPT_HICACHE_WRITE_BACK_EVICTION_HEADROOM_TOKENS(default 0: unchanged).UnifiedRadixCache.evict()andevict_pool_shortfalls()grow a nonzero token / window shortfall by the reserve, capped by the evictable size. The shortfall eviction's stop demand grows by the same amount.reclaim_jointly) is unchanged: the allocator owns its stop condition.evict_to_free_tokens, decode prealloc reclaim and capacity checks, HiCache load-back, linker LOAD). It does not apply to write_through, to caches without HiCache, or to joint reclaim.Accuracy Tests
GSM8K (sgl-eval 0.1.3, 1319 examples, model preset, 1P1D PD with write_back HiCache): 96.66%, 0% truncated / errors. Same-build baselines without the reserve: 96.51-96.97%.
Speed Tests and Profiling
DeepSeek-V4.1-Flash PD disaggregation on GB300: 6 prefill instances (DP-attention, 2 ranks each, write_back HiCache), 1 DEP16 decode. Agentic multi-turn trace at a prefill-heavy mid-curve operating point, 20 minutes, metrics from minutes 8-20. Reserve 262144 tokens (~0.3% of the prefill device pool). Both arms ran concurrently on the same image with #46884.
The throughput change is within run-to-run noise at this point: prefill has spare capacity there, so the faster prefill shows up as lower TTFT and a shorter queue. Trace of the same mechanism with the reserve applied at the allocator call sites: prefill GPU idle 15.1% -> ~6%, and eviction stalls every 4-5 forwards instead of every forward. The remaining per-stall cost is the once-per-victim wait inside a larger pass, which #46930 addresses.
Checklist
CI States
Latest PR Test (Base): ❌ Run #42725270559
Latest PR Test (Extra): ❌ Run #42725270168
Latest PR Test (AMD ROCm 10): ❌ Run #42725270381
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: