Skip to content

[Perf] hicache: Space out write_back eviction waits with a bounded reserve - #47650

Open
weireweire wants to merge 1 commit into
sgl-project:mainfrom
weireweire:hicache-write-back-eviction-headroom
Open

weireweire wants to merge 1 commit into
sgl-project:mainfrom
weireweire:hicache-write-back-eviction-headroom

Conversation

@weireweire

@weireweire weireweire commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With --hicache-write-policy write_back, a device eviction backs its victims up to host and blocks on that copy (writing_check(write_back=True) -> ack.finish_event.synchronize()). The D2H stream is fenced on the forward stream (#44800), so every eviction pass waits for the whole in-flight forward.

On a saturated prefill instance the device pool is full, so nearly every scheduling step evicts exactly its shortfall and pays that wait. The scheduler loses its one-step lookahead every step. Under DP attention the peer rank waits for it in the DP-attention all-gather, and the GPU idles between forwards. In a PD prefill trace this was 14-15% GPU idle, almost all of it between forwards.

#46884 removed an earlier form of this wait (the coarse WAR barrier behind the write-back index copy). The eviction ack wait remains. #41690 adds a reserve at some allocator call sites. This PR instead puts the reserve inside UnifiedRadixCache.evict() and the per-pool shortfall eviction of evict_for_alloc() (evict_pool_shortfalls), which stay exact-shortfall today. Those are reached by the paged allocator's eviction, HiCache load-back and the linker LOAD path.

Modifications

  • New opt-in SGLANG_OPT_HICACHE_WRITE_BACK_EVICTION_HEADROOM_TOKENS (default 0: unchanged).
  • Under write_back, UnifiedRadixCache.evict() and evict_pool_shortfalls() grow a nonzero token / window shortfall by the reserve, capped by the evictable size. The shortfall eviction's stop demand grows by the same amount.
  • Joint reclaim (reclaim_jointly) is unchanged: the allocator owns its stop condition.
  • Eviction order is unchanged; freed slots return to the ordinary free pool.
  • Where it applies: prefill and decode eviction through these two entry points when write_back is active (allocator evict_to_free_tokens, decode prealloc reclaim and capacity checks, HiCache load-back, linker LOAD). It does not apply to write_through, to caches without HiCache, or to joint reclaim.
  • Trade-off: the coldest reserve tokens are demoted to host earlier, so a later hit on them loads back from host. Prefix hit was unchanged at the point below (0.9647 vs 0.9646); a much larger reserve would trade device hits for fewer stalls.
  • With [Perf] kv-cache: Amortize L1 eviction under sustained pressure #41690: separate env names. If both are set, their reserves stack (each capped by the evictable size); setting one is enough.

Accuracy Tests

GSM8K (sgl-eval 0.1.3, 1319 examples, model preset, 1P1D PD with write_back HiCache): 96.66%, 0% truncated / errors. Same-build baselines without the reserve: 96.51-96.97%.

Speed Tests and Profiling

DeepSeek-V4.1-Flash PD disaggregation on GB300: 6 prefill instances (DP-attention, 2 ranks each, write_back HiCache), 1 DEP16 decode. Agentic multi-turn trace at a prefill-heavy mid-curve operating point, 20 minutes, metrics from minutes 8-20. Reserve 262144 tokens (~0.3% of the prefill device pool). Both arms ran concurrently on the same image with #46884.

without reserve with reserve
Output tput/GPU (min 8-20) baseline +0.5%
Output tput/GPU (whole run) baseline +1.0%
p90 interactivity (tok/s/user) 191.8 190.6
TTFT p90 / p50 8.45 s / 2.28 s 7.89 s / 2.16 s (-6.6% / -5%)
Prefill steps/s per rank 3.01 3.21 (+6.6%)
Prefill queue p50 11 6
Prefix cache hit 0.9646 0.9647
Decode accept length 3.51 3.51

The throughput change is within run-to-run noise at this point: prefill has spare capacity there, so the faster prefill shows up as lower TTFT and a shorter queue. Trace of the same mechanism with the reserve applied at the allocator call sites: prefill GPU idle 15.1% -> ~6%, and eviction stalls every 4-5 forwards instead of every forward. The remaining per-stall cost is the once-per-victim wait inside a larger pass, which #46930 addresses.

Checklist

  • pre-commit on changed files
  • Accuracy (GSM8K above)
  • Unit test: none added; validated end to end on the deployment above

CI States

Latest PR Test (Base): ❌ Run #42725270559
Latest PR Test (Extra): ❌ Run #42725270168
Latest PR Test (AMD ROCm 10): ❌ Run #42725270381

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mintlify

mintlify Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 5adcd7c.

Approve preview

SGLANG_MAX_KV_CHUNK_CAPACITY = EnvInt(128 * 1024)
# Extra tokens a write_back HiCache eviction frees beyond the shortfall; each
# such eviction waits for the in-flight forward, so a reserve spaces the waits out.
SGLANG_OPT_HICACHE_WRITE_BACK_EVICTION_HEADROOM_TOKENS = EnvInt(0)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xiezhq-hermann

Copy link
Copy Markdown
Collaborator

Thanks @weireweire. Your DSV4.1-Flash PD A/B and the GPU-idle trace (prefill GPU idle 15.1% -> ~6%) are the clearest real-world motivation we have for this.

The mechanism reproduced on 11-05 main (944549c, with #46930). Setup: Qwen2.5-1.5B, write_back, a 131K-token device pool, Rust core. This PR at 4096, at 80 req/s:

  • eviction calls -58%, write_back wait -33%;
  • TTFT p50 -17%, ITL p99 -32%;
  • saturated throughput +6% (t ~4, against a GPU-matched baseline);
  • L1 hit -2.3% relative.

In a smaller-pool run at concurrency 8, eviction was only ~4% of wall time, and 4096 showed no gain (-3.7%, t ~2). So the benefit depends on how much of the step the wait takes.

#48047 does the same thing at the same spot, and we only want one env var for this idea. So I'm consolidating on #48047 and will close this PR when it lands. Why that direction:

Both PRs now conflict with #48012. It moved allocation-driven eviction to a KVDemand (what the allocation takes), and the walk stops once that demand fits. I've asked #48047 to raise the walk's target along with the request when it rebases.

It would help a lot if you could:

  1. Bring your environment_variables.mdx row over to [Perf] hicache: optional minimum tokens per write_back allocation-driven eviction #48047 (with co-author credit).
  2. Once [Perf] hicache: optional minimum tokens per write_back allocation-driven eviction #48047 is rebased, re-run the DSV4.1-Flash PD A/B on it against current main, with device and host hits reported separately. Long forwards are where the per-wait cost, and so the gain, should be largest; our 1.5B model can't show that.
  3. If explicit evict() callers or window (SWA) growth turn out to matter, propose them as a measured follow-up under the same env. Window growth should wait until write_back window backups are batched: they still wait once per node, and in our test a reserve raised the waits per pass from 2 to 8.

cc @alphabetc1 @nvpohanh

…serve

Root cause: with --hicache-write-policy write_back, a device eviction backs
its victims up to host and blocks on that copy (writing_check(write_back=True)
-> ack.finish_event.synchronize()). The D2H stream is fenced on the forward
stream, so every eviction pass waits for the whole in-flight forward. On a
saturated prefill instance the device pool is full and nearly every scheduling
step evicts exactly its shortfall, so the scheduler loses its one-step
lookahead each step. Under DP attention the peer rank then waits for it in the
DP-attention all-gather, and the GPU idles between forwards.

Fix: add opt-in SGLANG_OPT_HICACHE_WRITE_BACK_EVICTION_HEADROOM_TOKENS
(default 0, behavior unchanged). Under write_back, UnifiedRadixCache.evict()
and the per-pool shortfall eviction of evict_for_alloc()
(evict_pool_shortfalls) grow a nonzero token/window shortfall by the reserve,
capped by the evictable size; the shortfall eviction's stop demand grows by
the same amount. This covers the paged allocator's eviction, HiCache
load-back and the linker LOAD path. Joint reclaim is unchanged because the
allocator owns that stop condition. Eviction order is unchanged; freed slots
return to the free pool. The reserve is read at eviction time, so caches
built without __init__ never touch it while it is 0.

Validation (DSV4.1-Flash PD disaggregation on GB300; prefill 6x DP-attention
DEP2 with write_back HiCache, decode DEP16; agentic multi-turn trace at a
P-heavy mid-curve point, 20 min; reserve 262144 tokens, ~0.3% of the prefill
pool; A/B run concurrently on the same image, both arms with sgl-project#46884):
- Client output tput/GPU: +0.5% (min 8-20 bins) / +1.0% (whole run) at the
  same p90 interactivity (190.6 vs 191.8 tok/s/user); inside run-to-run noise.
- TTFT p90 7.89 vs 8.45 s (-6.6%), p50 2.16 vs 2.28 s (-5%).
- Prefill steps/s per rank 3.21 vs 3.01 (+6.6%); P queue p50 6 vs 11.
- Prefix hit 0.9647 vs 0.9646; decode accept length 3.51 vs 3.51; 0 invalid
  records.
- GSM8K (sgl-eval, 1319, 1P1D): 96.66%, 0% truncated/errors; baselines on the
  same build 96.51-96.97%.
- Mechanism: in an nsys trace of the equivalent change with the reserve
  applied at the allocator call sites, prefill GPU idle fell from 15.1% to
  ~6%, and eviction stalls came every 4-5 forwards instead of every forward.
  A trace of this exact diff was not taken.
- Not run: unit tests. pre-commit on the changed files passes.
@weireweire
weireweire force-pushed the hicache-write-back-eviction-headroom branch from 97ec4ec to 5adcd7c Compare November 10, 2026 00:02
@weireweire

Copy link
Copy Markdown
Contributor Author

hi thabks for review!

with the floor format, when we evict a big chunk, there will be no room for next round. should we do this special logic to the big chunk?

about the perf difference, it may come from the less real eviction under the floor format, param tuning may fix,i can test. I'm not sure what's the theory to use floor instead of a fixed room.

no unit test is intended as we already have many test, it is easy to add if you want.

llying-001 added a commit to llying-001/sglang that referenced this pull request Nov 10, 2026
…falls

Merge upstream main, which moved allocation-driven eviction to
evict_pool_shortfalls(demand) (sgl-project#48012). There the walk stops at the first
victim after which its pool serves the demand, so the floor now raises the
token entry of the demand together with the token shortfall; raising only
the shortfall would free nothing extra.

- Rename the variable to SGLANG_HICACHE_WRITE_BACK_MIN_EVICT_TOKENS: a
  write_back-only minimum per allocation-driven eviction, Full KV only.
- Check the shortfall before reading is_write_back, so a disabled floor
  never reaches the property.
- Document the variable in environment_variables.mdx, adapted from the
  draft in sgl-project#47650 by @weireweire.
- Replace the SimpleNamespace test with one on the write-back fixtures,
  run on both tree cores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@weireweire

Copy link
Copy Markdown
Contributor Author

I rebased and did some test, didn't see L1 cost under same parameter(see below). And the perf is also similar.(just for reference, as it's not apples to apples compare with same reserve parameter).

What I care is how do we set the parameter, if we set the "floor"(minimal eviction) the perf is depends on the workload, for example, if we set to 16k, when real eviction is only 1k, it will evict 15k more, when real eviction is 30k, it will eviction 0 more. I don't know why we should take 1k/30k as different instead of just evict fixed space more. Also it may be hard to select the param as it need to consult the real chunk size.

Single GPU, Qwen2.5-1.5B, write_back HiCache, Rust core, current main. Same value (16384) for both forms

workload per-step demand form write_back waits/s device hit
(a) short prompts, 80 req/s, 131K pool ~420 tokens additive R=16384 4.94 53.01%
floor F=16384 5.16 53.02%
(b) 16K prefix + 4K suffix, c128, 524K pool ~15K tokens additive R=16384 10.2 54.55%
floor F=16384 15.4 54.71%
  • (a): the two forms are equivalent. Both cut eviction calls by 90% and wait time by 45%, at -2.1% relative device hit vs main. The -4.3% vs -7.2% L1 gap did not reproduce.
  • (b): at the same value the floor leaves less spare room (6.5K vs 11.5K tokens), so it waits 50% more. At matched device hit, additive has 10-23% fewer waits.

DSV4.1-Flash PD on GB300 (6x DEP2 prefill with write_back HiCache, DEP16 decode, agentic trace, 20 min)

arm tput/GPU @ p90 intvty TTFT p90 / p50 prefill steps/s/rank waits / prefill step wait share of prefill time device / host hit
main (2 runs) 960.0k-965.7k @ 197.7-197.9 8.58-8.63 / 2.21-2.34 s 3.00-3.08 1.21-1.24 44.7-45.9% 0.862 / 0.101
additive R=262144 977.2k @ 200.2 7.81 / 2.03 s 3.27 0.23 14.6% 0.864 / 0.100
floor F=294912 977.3k @ 200.0 7.94 / 2.03 s 3.25 0.29 15.4% 0.863 / 0.100
  • Both forms cut the wait share of prefill time from ~45% to ~15%. Prefill steps/s rose ~7%, TTFT p90 fell 8-9%, and throughput rose 1.5%, with no device-hit cost.
  • The floor needed F = R + 32K (the max per-step demand) to tie. Even then it left less mean spare room (210K vs 275K tokens) and waited 24% more per step.

The PD workload above runs at c4370. Workload (b) on the single GPU only covers the long-prompt case. I didn't put the job IDs in the table.

@weireweire

Copy link
Copy Markdown
Contributor Author

could we have a look at this pr again? @alphabetc1 @xiezhq-hermann

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants