Skip to content

[Fix] HiCache: stale host KV from write-through copies racing the in-flight forward - #42701

Merged
xiezhq-hermann merged 6 commits into
sgl-project:mainfrom
fractalyze:fix/hicache-write-through-fence
Oct 22, 2026
Merged

xiezhq-hermann merged 6 commits into
sgl-project:mainfrom
fractalyze:fix/hicache-write-through-fence

Conversation

@jumanzii

@jumanzii jumanzii commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With HiCache on, a conversation's later turns can produce different output depending on whether its prefix was reused from device memory or loaded back from host memory. The host copy of the previous turn's last output token can be stale, so a load-back restores wrong KV for that token.

Under the overlap scheduler, process_batch_result for step N runs after step N+1 was launched. A request that finished at step N is still in batch N+1, and that forward writes the KV of the request's last output token on the forward stream. Meanwhile release_kv_cache already inserts that token into the tree and the write-through starts. HiCacheController.start_writing -> L2TransferEngine.submit_device_to_host orders the device-to-host copy only after an event recorded on the scheduler stream, never after the forward stream, so the copy can read that token's KV before or while it is written. Load-back already has the equivalent fence (load_fence_stream, set by the scheduler to the forward stream and used in start_loading); write-through had none.

The stale token sits at the end of the previous reply, which is exactly where the next turn's prompt resumes, so the next turn diverges within its first few tokens.

Modifications

  • HiCacheController.start_writing: before submitting the copy, make the device-to-host stream wait on load_fence_stream (the forward stream). wait_stream orders the copy after the work queued so far, which at this point is the forward that writes the node's last token; forwards launched later still overlap with the copy. When load_fence_stream is unset (no scheduler wiring) nothing changes.
  • The comment on load_fence_stream now names both uses.
  • Test test_write_copy_waits_for_queued_forwards in test/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py: a write submitted through a real L2TransferEngine must wait on the forward stream before its copy. It fails without the fix. Five existing tests in that file build controllers with __new__ and now set load_fence_stream = None, as the load-path test there already does.

This is the default behavior, not an option. The alternative is a data race that silently corrupts host KV, and the fence only adds a GPU-side stream dependency on work already queued (no CPU synchronization). Its costs: a write ack can arrive up to one forward later, so a just-written node stays unevictable a little longer; and on the paths that wait synchronously for a write (write-back on eviction), the scheduler also waits for the in-flight forward in that step. Without the overlap scheduler the forward stream is idle when the write is submitted, and the wait is free.

Other device-to-host copies checked:

  • Write-back on eviction, buffer mode and flush_pending_backups all go through start_writing and get the fence.
  • UnifiedRadixCache.backup_kv_cache (PD decode retraction backup) copies only seqlen - 1 tokens; the forward in flight at retraction writes position seqlen - 1, so that copy is not exposed.
  • DecodeKVCacheOffloadManager builds its own controller without load_fence_stream, but offloads origin_input_ids + output_ids[:-1], so it is not exposed either.

HiSparse orders its decode backups the same way (decode_backup_stream.wait_stream(self.decode_producer_stream) in hisparse_coordinator.py).

Related: #42702 orders PD decode retraction restores after the in-flight forward, the same kind of race on the restore side. The two PRs are independent.

Accuracy Tests

Greedy multi-turn outputs at concurrency 1 on a public model, Qwen/Qwen3-8B (no sliding window), one RTX 5090. Six sessions x three turns. Each turn sends the previous turn's input_ids + its output_ids + 200 new tokens, so a turn reuses the previous sequence including its last output token. Turns run turn-major (all sessions' turn 0, then turn 1, ...).

  • Reference: no HiCache, default device pool, so every later turn reuses its prefix from device.
  • HiCache: --enable-hierarchical-cache --hicache-size 4 --hicache-write-policy write_through --max-total-tokens 8192. The small device pool evicts each session's prefix to host between its turns, so every later turn is a load-back. Defaults otherwise (flashinfer attention, kernel io backend, page_first layout, overlap scheduler on).

Both arms report the same cached_tokens for every turn (3128 for turn 1, 3456 for turn 2).

run (order) turn 0 identical to reference turns 1-2 identical to reference
main, 1st 6/6 0/12 (first differing token at 3-51)
this PR, 1st 6/6 12/12
this PR, 2nd 6/6 12/12
main, 2nd 6/6 0/12

Reproduce:

# reference
python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 30000
python multiturn_exact.py --model Qwen/Qwen3-8B --out ref.json
# HiCache with load-backs
python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 30000 \
  --enable-hierarchical-cache --hicache-size 4 --hicache-write-policy write_through \
  --max-total-tokens 8192
python multiturn_exact.py --model Qwen/Qwen3-8B --out hicache.json
python compare_exact.py ref.json hicache.json
multiturn_exact.py / compare_exact.py
# multiturn_exact.py
import argparse, json, random
import requests
from transformers import AutoTokenizer

p = argparse.ArgumentParser()
p.add_argument("--url", default="http://127.0.0.1:30000")
p.add_argument("--model", required=True)
p.add_argument("--sessions", type=int, default=6)
p.add_argument("--turns", type=int, default=3)
p.add_argument("--out", required=True)
args = p.parse_args()

tok = AutoTokenizer.from_pretrained(args.model)
vocab = [i for i in range(min(tok.vocab_size, 60000))
         if (s := tok.decode([i])).startswith(" ") and s[1:].isalpha() and len(s) > 3]
rng = random.Random(0)
seqs = [rng.choices(vocab, k=3000) for _ in range(args.sessions)]
records = []
for turn in range(args.turns):
    for s in range(args.sessions):
        if turn > 0:
            seqs[s] += rng.choices(vocab, k=200)
        r = requests.post(args.url + "/generate", json={
            "input_ids": seqs[s],
            "sampling_params": {"temperature": 0, "max_new_tokens": 128, "ignore_eos": True},
        }).json()
        records.append({"session": s, "turn": turn,
                        "cached_tokens": r["meta_info"]["cached_tokens"],
                        "output_ids": r["output_ids"]})
        seqs[s] += r["output_ids"]
json.dump(records, open(args.out, "w"))
# compare_exact.py
import json, sys
ref, run = (json.load(open(p)) for p in sys.argv[1:3])
for turn in sorted({r["turn"] for r in ref}):
    pairs = [(a, b) for a, b in zip(ref, run) if a["turn"] == turn]
    same = sum(a["output_ids"] == b["output_ids"] for a, b in pairs)
    print(f"turn {turn}: {same}/{len(pairs)} identical")

Speed Tests and Profiling

Qwen/Qwen3-8B on one RTX 5090 with --enable-hierarchical-cache --hicache-size 8 --hicache-write-policy write_through and the default device pool, so every finished request starts a write-through. Load: python -m sglang.benchmark.serving --backend sglang --dataset-name random-ids --random-input-len 2048 --random-output-len 256 --random-range-ratio 1.0 --num-prompts 320 --max-concurrency 32 --request-rate inf --seed 1. Runs interleaved ABBA, a fresh server per run.

run (order) output tok/s median TPOT (ms) median TTFT (ms)
main, 1st 772.1 24.07 3353
this PR, 1st 771.6 23.11 3364
this PR, 2nd 772.4 23.09 3356
main, 2nd 773.1 24.23 3350

Output throughput: main 772.6 mean, this PR 772.0 (-0.08%). That is inside the spread between the two main runs (0.13%), so no throughput cost is measurable at this load. Median TPOT is not worse.

An earlier measurement on a heavier HiCache load: Gemma-4-26B-A4B-it (FP8) at 20-32 concurrent multi-turn sessions with frequent load-backs, one RTX 5090, single runs per point against the unfenced build. It showed -1.0% to -1.8% output tok/s, consistent across points. Without repeats that is close to noise, so a small real cost under heavy write-through cannot be excluded.

Checklist

Unit tests (pytest on a GPU host, rebased on f2f0be7): test_hicache_staged_write_back_dispatch.py and test_hicache_pp_sync_drain.py pass (25). Before the rebase, test/registered/unit/managers and test/registered/unit/mem_cache matched main plus the new case; the one failure there (test_mm_process_config.py::TestStreamOrderedMmFeaturePool::test_consumer_slot_uses_global_tp_rank) also fails on main in the same environment.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): 🚫 Run #39344390123
Latest PR Test (Extra): ❌ Run #39344390070
Latest PR Test (AMD ROCm 10): ❌ Run #39344389912

Under the overlap scheduler, process_batch_result for step N runs after step
N+1 was launched. A request that finished at step N is still in batch N+1,
whose forward writes the KV of its last output token on the forward stream,
while release_kv_cache already inserts that token and starts the write-through.
start_writing ordered the device-to-host copy only after an event on the
scheduler stream, so the copy could read that token's KV half-written. A later
load-back then restores the stale KV, and the next turn of the conversation
diverges from what the same server produces without a load-back.

Load-back already waits on the forward stream (load_fence_stream, see
start_loading); make start_writing wait on it too. wait_stream orders the copy
after the work queued so far, which at this point is the forward writing the
node, so later forwards still overlap with the copy. Without overlap
scheduling the forward stream is idle here and the wait is free.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Oct 6, 2026
@jumanzii jumanzii changed the title [HiCache] Fence write-through copies behind the in-flight forward [HiCache] Fix stale host KV from write-through copies racing the in-flight forward Oct 6, 2026
@jumanzii

Copy link
Copy Markdown
Contributor Author

@ispobock @xiezhq-hermann gentle ping on this one. small fix, repro numbers are in the description. would be great if someone can trigger ci 🙏

@xiezhq-hermann xiezhq-hermann self-assigned this Oct 13, 2026
@xiezhq-hermann xiezhq-hermann added the highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed label Oct 15, 2026
@xiezhq-hermann xiezhq-hermann added the run-ci CI: run the baseline test suite on this PR label Oct 16, 2026
@github-actions

This comment has been minimized.

@xiezhq-hermann xiezhq-hermann changed the title [HiCache] Fix stale host KV from write-through copies racing the in-flight forward [Fix] HiCache stale host KV from write-through copies racing the in-flight forward Oct 16, 2026
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@xiezhq-hermann xiezhq-hermann changed the title [Fix] HiCache stale host KV from write-through copies racing the in-flight forward [Fix] HiCache: stale host KV from write-through copies racing the in-flight forward Oct 16, 2026
Resolve against sgl-project#44800: start_writing fences every device-to-host copy on write_fence_stream, so the wait_for_forward flag and the separate load_fence_stream wait are no longer needed.
@xiezhq-hermann

Copy link
Copy Markdown
Collaborator

@jumanzii this branch conflicts with main since #44800 landed (start_writing grew a wait_for_forward kwarg that only fences the blocking write-back drain). fractalyze/sglang is an organization-owned fork, so GitHub does not let upstream maintainers push the resolution here. I published the resolved merge as branch xiezhq/42701-merge-main on sgl-project/sglang (0a0ee5ff3b = this branch + main, a fast-forward of your branch):

  • start_writing() now always waits on write_fence_stream (covers this PR's write-through overlap race and [Fix] hicache: Order write-back host copies after the forward that produced the KV #44800's evicted-SWA-page case); the kwarg and its writing_check(write_back=True) call site are dropped, load_fence_stream is untouched.
  • this PR's test setups set controller.write_fence_stream = None; the new test fences via write_fence_stream.
  • test_hicache_staged_write_back_dispatch.py + test_unified_hicache_regressions.py: 29 passed, 1 skipped locally.

Could you fast-forward the branch from it?

git fetch https://github.com/sgl-project/sglang.git xiezhq/42701-merge-main
git push <fractalyze remote> FETCH_HEAD:fix/hicache-write-through-fence

@jumanzii

Copy link
Copy Markdown
Contributor Author

@xiezhq-hermann thanks! fast-forwarded to 0a0ee5ff3b.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants