Repository navigation
[Fix] HiCache: stale host KV from write-through copies racing the in-flight forward - #42701
Merged
xiezhq-hermann merged 6 commits intoOct 22, 2026
Merged
Conversation
Under the overlap scheduler, process_batch_result for step N runs after step N+1 was launched. A request that finished at step N is still in batch N+1, whose forward writes the KV of its last output token on the forward stream, while release_kv_cache already inserts that token and starts the write-through. start_writing ordered the device-to-host copy only after an event on the scheduler stream, so the copy could read that token's KV half-written. A later load-back then restores the stale KV, and the next turn of the conversation diverges from what the same server produces without a load-back. Load-back already waits on the forward stream (load_fence_stream, see start_loading); make start_writing wait on it too. wait_stream orders the copy after the work queued so far, which at this point is the forward writing the node, so later forwards still overlap with the copy. Without overlap scheduling the forward stream is idle here and the wait is free. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
jumanzii
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
October 6, 2026 02:34
3 of 5 tasks
3 tasks done
Contributor
Author
|
@ispobock @xiezhq-hermann gentle ping on this one. small fix, repro numbers are in the description. would be great if someone can trigger ci 🙏 |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
xiezhq-hermann
approved these changes
Oct 16, 2026
This comment has been minimized.
This comment has been minimized.
Resolve against sgl-project#44800: start_writing fences every device-to-host copy on write_fence_stream, so the wait_for_forward flag and the separate load_fence_stream wait are no longer needed.
Collaborator
|
@jumanzii this branch conflicts with
Could you fast-forward the branch from it? |
jumanzii
requested review from
hanming-lu,
hzh0425,
ispobock and
yizhang2077
as code owners
October 19, 2026 23:54
jumanzii
requested review from
alphabetc1 and
huangtingwei9988
as code owners
October 19, 2026 23:54
Contributor
Author
|
@xiezhq-hermann thanks! fast-forwarded to |
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With HiCache on, a conversation's later turns can produce different output depending on whether its prefix was reused from device memory or loaded back from host memory. The host copy of the previous turn's last output token can be stale, so a load-back restores wrong KV for that token.
Under the overlap scheduler,
process_batch_resultfor step N runs after step N+1 was launched. A request that finished at step N is still in batch N+1, and that forward writes the KV of the request's last output token on the forward stream. Meanwhilerelease_kv_cachealready inserts that token into the tree and the write-through starts.HiCacheController.start_writing->L2TransferEngine.submit_device_to_hostorders the device-to-host copy only after an event recorded on the scheduler stream, never after the forward stream, so the copy can read that token's KV before or while it is written. Load-back already has the equivalent fence (load_fence_stream, set by the scheduler to the forward stream and used instart_loading); write-through had none.The stale token sits at the end of the previous reply, which is exactly where the next turn's prompt resumes, so the next turn diverges within its first few tokens.
Modifications
HiCacheController.start_writing: before submitting the copy, make the device-to-host stream wait onload_fence_stream(the forward stream).wait_streamorders the copy after the work queued so far, which at this point is the forward that writes the node's last token; forwards launched later still overlap with the copy. Whenload_fence_streamis unset (no scheduler wiring) nothing changes.load_fence_streamnow names both uses.test_write_copy_waits_for_queued_forwardsintest/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py: a write submitted through a realL2TransferEnginemust wait on the forward stream before its copy. It fails without the fix. Five existing tests in that file build controllers with__new__and now setload_fence_stream = None, as the load-path test there already does.This is the default behavior, not an option. The alternative is a data race that silently corrupts host KV, and the fence only adds a GPU-side stream dependency on work already queued (no CPU synchronization). Its costs: a write ack can arrive up to one forward later, so a just-written node stays unevictable a little longer; and on the paths that wait synchronously for a write (write-back on eviction), the scheduler also waits for the in-flight forward in that step. Without the overlap scheduler the forward stream is idle when the write is submitted, and the wait is free.
Other device-to-host copies checked:
flush_pending_backupsall go throughstart_writingand get the fence.UnifiedRadixCache.backup_kv_cache(PD decode retraction backup) copies onlyseqlen - 1tokens; the forward in flight at retraction writes positionseqlen - 1, so that copy is not exposed.DecodeKVCacheOffloadManagerbuilds its own controller withoutload_fence_stream, but offloadsorigin_input_ids + output_ids[:-1], so it is not exposed either.HiSparse orders its decode backups the same way (
decode_backup_stream.wait_stream(self.decode_producer_stream)inhisparse_coordinator.py).Related: #42702 orders PD decode retraction restores after the in-flight forward, the same kind of race on the restore side. The two PRs are independent.
Accuracy Tests
Greedy multi-turn outputs at concurrency 1 on a public model,
Qwen/Qwen3-8B(no sliding window), one RTX 5090. Six sessions x three turns. Each turn sends the previous turn'sinput_ids+ itsoutput_ids+ 200 new tokens, so a turn reuses the previous sequence including its last output token. Turns run turn-major (all sessions' turn 0, then turn 1, ...).--enable-hierarchical-cache --hicache-size 4 --hicache-write-policy write_through --max-total-tokens 8192. The small device pool evicts each session's prefix to host between its turns, so every later turn is a load-back. Defaults otherwise (flashinfer attention,kernelio backend,page_firstlayout, overlap scheduler on).Both arms report the same
cached_tokensfor every turn (3128 for turn 1, 3456 for turn 2).Reproduce:
multiturn_exact.py / compare_exact.py
Speed Tests and Profiling
Qwen/Qwen3-8Bon one RTX 5090 with--enable-hierarchical-cache --hicache-size 8 --hicache-write-policy write_throughand the default device pool, so every finished request starts a write-through. Load:python -m sglang.benchmark.serving --backend sglang --dataset-name random-ids --random-input-len 2048 --random-output-len 256 --random-range-ratio 1.0 --num-prompts 320 --max-concurrency 32 --request-rate inf --seed 1. Runs interleaved ABBA, a fresh server per run.Output throughput: main 772.6 mean, this PR 772.0 (-0.08%). That is inside the spread between the two
mainruns (0.13%), so no throughput cost is measurable at this load. Median TPOT is not worse.An earlier measurement on a heavier HiCache load: Gemma-4-26B-A4B-it (FP8) at 20-32 concurrent multi-turn sessions with frequent load-backs, one RTX 5090, single runs per point against the unfenced build. It showed -1.0% to -1.8% output tok/s, consistent across points. Without repeats that is close to noise, so a small real cost under heavy write-through cannot be excluded.
Checklist
Unit tests (pytest on a GPU host, rebased on f2f0be7):
test_hicache_staged_write_back_dispatch.pyandtest_hicache_pp_sync_drain.pypass (25). Before the rebase,test/registered/unit/managersandtest/registered/unit/mem_cachematchedmainplus the new case; the one failure there (test_mm_process_config.py::TestStreamOrderedMmFeaturePool::test_consumer_slot_uses_global_tp_rank) also fails onmainin the same environment.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): 🚫 Run #39344390123
Latest PR Test (Extra): ❌ Run #39344390070
Latest PR Test (AMD ROCm 10): ❌ Run #39344389912