Skip to content

[Perf] hicache: Fence write_back eviction copies on the forward stream instead of blocking the scheduler - #50469

Open
weireweire wants to merge 6 commits into
sgl-project:mainfrom
weireweire:hicache-async-write-back-demote
Open

weireweire wants to merge 6 commits into
sgl-project:mainfrom
weireweire:hicache-async-write-back-demote

Conversation

@weireweire

@weireweire weireweire commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With HiCache write_back, the device eviction walk backs its victims up to host and then blocks the scheduler on Event.synchronize() for those D2H copies before it demotes the victims. The copies are fenced behind the in-flight forward (write_fence_stream), so on a busy prefill server the scheduler cannot finish building the next batch until the current forward ends. On DSv4.1-Flash disaggregated prefill this wait was 50-56% of scheduler time, and the GPU sat idle while the next batch was prepared and launched.

Modifications

_demote_backed_up no longer waits on the host. It submits the queued copies and makes the forward stream wait_event on each copy's finish event, then finishes the acks and demotes the victims at once. Without overlap scheduling the forward runs on the current stream, so the current stream waits as well; the scheduler tells the cache controller which case applies.

Ordering:

  • The copy still waits for the in-flight forward (unchanged write_fence_stream fence), so it reads complete KV.
  • The next forward waits for the copy. With overlap scheduling it runs on the forward stream. Without overlap it runs on the current stream, which then also waits, and so do current-stream writers such as a Mamba state copy into a freshly allocated slot (Mamba no_buffer turns overlap off).
  • With overlap the current (schedule) stream does not wait: making it wait would serialize the scheduler's GPU work behind the in-flight forward and the copy, which cancelled the gain in measurement (below).
  • Every later load-back waits for the copy, because the H2D stream waits on load_fence_stream, which is the forward stream.
  • So a freed device slot is never rewritten before its backup has been read. The host never blocks, so the scheduler can build the next batch while the copy and the in-flight forward run.
  • Backup metrics for these acks are recorded once their event has completed.

The blocking wait is kept wherever freed slots or host pages can be touched outside forward/load stream order:

  • an L3 storage backend (the storage backup reads the host pages),
  • the buffer pipeline,
  • a cache linker,
  • the unified memory pool (it relocates and zeroes pages outside this ordering),
  • PD decode (KV transfers write freed slots directly).

These are code conditions; there is no new flag. The other blocking writing_check(write_back=True) callers are unchanged.

Independent of #47650 (bounded write_back eviction reserve). The two stack: #47650 makes the waits less frequent, this change takes them off the host. Also different from #49498, which relaxes the copy's fence on the forward but keeps the host wait.

Accuracy Tests

DSv4.1-Flash, GB300, disaggregated prefill/decode with HiCache write_back, simulated acceptance off. GSM8K (5-shot, temperature 0, concurrency 256) run while the full benchmark load was running, so later passes reuse prompt KV that had been demoted to host and loaded back under load:

run strict-match flexible
pass 1 0.9735 0.9727
pass 2 0.9750 0.9742
separate run, unconditional current-stream wait 0.9712 0.9704

Speed Tests and Profiling

DSv4.1-Flash, GB300, disaggregated prefill/decode (14 prefill servers + DEP16 decode, overlap scheduling on), agentic multi-turn replay at a mid-curve concurrency, 1200 s per run. Same-day A/B on this PR's head, 2 runs per arm, same image, only this PR's diff differs; both arms include #47650:

baseline + this PR
end-to-end tok/s/GPU 958.8k (957.5k / 960.0k) 998.9k (+4.2%) (1001.9k / 996.0k)
prefill computed tok/s/GPU 92.8k 96.4k (+3.9%)
TTFT p50 / p90 4.26 s / 12.0 s 2.73 s / 9.6 s
TPOT p50 / p90 6.75 ms / 7.25 ms 6.92 ms / 7.32 ms

Run-to-run spread within an arm is 0.3-0.6%. The gain comes from the prefill step rate: the eviction wait no longer sits between forwards, so the prefill queue drains faster (TTFT down) and decode runs fuller (TPOT up by ~1%). An earlier profile of this setup showed prefill GPU idle falling from 20.6% to 9.5% with this change.

An earlier A/B on an older stack also showed why the current (schedule) stream must not wait under overlap: with an unconditional current-stream wait the gain was gone (-0.5% vs. baseline, against +2.9% for this PR's rule there).

Checklist


CI States

Latest PR Test (Base): ✅ Run #43999955293
Latest PR Test (Extra): ❌ Run #43999954985
Latest PR Test (AMD ROCm 10): ✅ Run #43999955222

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…m instead of blocking the scheduler

Root cause: the write_back eviction walk waits on the host (Event.synchronize)
for its D2H backups before demoting the victims. The copies are fenced behind
the in-flight forward, so on a busy prefill server the scheduler stalls until
that forward ends; on DSv4.1-Flash prefill this wait was 50-56% of scheduler
time.

Fix: submit the queued copies, make the current (schedule) stream and the
forward stream wait_event on each copy's finish event and finish the acks at
once. Every later forward (on either stream, with or without overlap
scheduling), every current-stream state copy, and through load_fence_stream
every later load-back runs after the copy, so a freed device slot is never
rewritten before its backup is read. Ack metrics
are logged once the event completes. The blocking wait stays wherever freed
slots or host pages can be touched outside forward/load stream order: L3
storage, the buffer pipeline, a cache linker, the unified memory pool (pages
are moved and zeroed on the schedule stream), and PD decode (KV transfers
write freed slots directly).

Validation: DSv4.1-Flash GB300 disagg; GSM8K under full load 0.9735 /
0.9750; +3.0% e2e throughput, prefill 89.0k -> 94.3k tok/s/GPU.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

…m only without overlap scheduling

Root cause: the current-stream wait added for the non-overlap path also ran
under overlap scheduling. There the current stream is the schedule stream,
so scheduler GPU work issued after an eviction waited for the in-flight
forward and the copy, which cancelled the gain (same-day A/B: -0.5% e2e vs
the bounded-reserve baseline).

Fix: the scheduler records whether overlap scheduling is on. Only without
it, where the forward and the Mamba state copies run on the current stream,
does the eviction also make the current stream wait on the copies. With
overlap the forward runs on forward_stream, which already waits.

Validation: DSv4.1-Flash GB300 disagg, same-day A/B, 2 runs each: +2.9%
e2e and +5.0% prefill tok/s/GPU over the bounded-reserve baseline; the
unconditional current-stream wait gave -0.5%.
@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Nov 14, 2026
@alphabetc1 alphabetc1 added run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) labels Nov 16, 2026
@alphabetc1 alphabetc1 self-assigned this Nov 16, 2026

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

…allocator context in the fenced demotion test
@weireweire

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@weireweire

weireweire commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

@alphabetc1 hi, the ci looks good, could you have a look? The only fail is the env issue about numpy version.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR run-ci-nv-extra CI: NVIDIA/shared Extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants