Skip to content

[Perf] hicache: Add an opt-in admission throttle for the buffer-only write backlog - #44344

Draft
xiezhq-hermann wants to merge 1 commit into
hicache/backup-drop-visibilityfrom
hicache/backup-backlog-shedding
Draft

xiezhq-hermann wants to merge 1 commit into
hicache/backup-drop-visibilityfrom
hicache/backup-backlog-shedding

Conversation

@xiezhq-hermann

@xiezhq-hermann xiezhq-hermann commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #44339 (hicache/backup-drop-visibility): this branch is based on it and the PR base points at it, so the diff shown here is the throttle alone. Rebase onto main once #44339 lands.

Motivation

With --hicache-host-memory-mode buffer_only and write_through, a store that drains slower than prefill produces KV loses whole prompts from L3: a finished node's backup intent is queued unpinned until its D2H launches, host staging is held until the storage ack, and flush_pending_writes stops launching at live_cap, so the device LRU evicts exactly the queued leaves and the stale sweep retires them (source_evicted after #44339). The loss is bandwidth-bound: lost ~= produced - (staging window + drain x burst). Re-ordering the eviction walk or pinning queued nodes does not change those terms (pinning turns the device pool into a write buffer for the store's drain time), and back-pressure never reaches admission, so a slow store shows up as a silently collapsing L3 hit rate while TTFT stays flat. Deployments that would rather pay TTFT than lose coverage have no knob.

Modifications

  • Knob. "write_backlog_throttle": true in --hicache-storage-backend-extra-config (default false; parsed next to prefetch_threshold / prefetch_timeout_* in HybridCacheController.parse_storage_backend_extra_config, type-checked, applied at startup and on runtime attach). It lives with the other storage-controller policy keys rather than a new server arg because it only means something with a storage backend in buffer-only mode.
  • Hold. BufferModePipeline.write_backlog_hold_tokens() is the queued-but-unstaged backlog (write_backlog_tokens_, unique spans across pools) while the throttle is on and the write path is alive; BasePrefixCache.write_backlog_hold_tokens() (default 0) exposes it and PrefillAdder.__init__ charges it to total_offset / current_offset, so prefill admits only what free or already-written device memory can hold and recovers as writes stage. It is a budget charge, not a lock: the eviction walk is unchanged, in-flight chunked prefills and decode growth still evict queued spans (labeled drops), priority preemption still works against the hold, and a stale-swept span releases its share. There is no admission floor: with a live store, sustained admission equals the staging launch rate, which the window caps at the store's drain.
  • Why the whole backlog and not a cap. A cap at pool - max_request (the obvious wedge guard) does not pace: once the backlog exceeds it, each admission evicts the spans it then re-queues, so admission runs at full speed with drops (measured: 45% dropped with a 49k cap on a 65k pool at 0.3 GB/s, same as control; 16% with the real pool - max_request at the stock backend). Holding only while the running batch is non-empty degenerates to the cap in prefill-heavy traffic (the running batch is empty between prefill passes), i.e. exactly the PD-prefill shape this is for.
  • Safety from liveness, rank-agreed. write_path_alive() is each rank's verdict: a write launch or storage ack within write_backlog_stall_timeout_s (30 s, a class constant), or no backlog at all; stalled latches until the next launch or ack, so a dead store with a held window stays lapsed across sweeps and new backlogs instead of re-arming every timeout (a new backlog only restarts the clock when nothing is staged or in flight). The verdict rides the existing per-tick HiCache MIN all-reduce (_sync_hicache_ready_counts); in buffer-only PP the slot is additionally _pp_sync'd from PP0 because every stage schedules on its own clock. set_write_path_alive(False) lapses the hold with one WARNING and admission proceeds (drops labeled); progress re-engages it (INFO). The storage slots of the sync vector are now derived from one slot list (_STORAGE_QUEUE_SLOTS, extra pools, write_path_alive last), the PP placeholder from the same count, with a CPU test that placeholder and live lengths match. Slot order to coordinate with [Fix] hicache: Recover storage worker collectives and expose storage liveness #44342: [..., *extra_pools, write_path_alive, storage_healthy] keeps its [-1] parse valid. The bit is not added to drain_storage_control_queues (no callers under python/sglang/srt).
  • Scheduler latch. A NO_TOKEN with a non-empty running batch sets batch_is_full, which only a finished or retracted request cleared, so a hold-induced NO_TOKEN would have shut admission until the longest in-flight decode ended. Scheduler now records write_backlog_hold_tokens() at that NO_TOKEN and reopens admission at the top of _get_new_batch_prefill_raw once the hold has shrunk below it (one compare per pass; memory-bound NO_TOKENs keep the stock behavior).
  • Docs. hicache_design.mdx: the controller-consumed extra-config keys get their own paragraph; write_backlog_throttle says who should turn it on (coverage-first deployments behind slow or remote stores, e.g. PD prefill engines), who should not (latency tiers; anywhere the store drains faster than prefill), that every request queued behind the throttle pays, and the 30 s stall constant.
  • Tests. test/registered/unit/mem_cache/test_buffer_mode_backlog_throttle.py (CPU: default off holds nothing; the hold is the whole backlog and is released as intents stage or sweep; a stalled write path lapses the hold, stays latched through a sweep and a new backlog while the window is held, re-engages on a storage ack, restarts the clock only when idle; the extra-config key is consumed and type-checked), test/registered/unit/managers/test_scheduler_write_backlog_latch.py (CPU: the latch reopens when the hold shrinks and holds otherwise, through the real _get_new_batch_prefill_raw gate), test_prefill_adder.py::test_write_backlog_hold_is_charged_to_both_budgets, test_hicache_pp_sync_drain.py (PP0's liveness slot is _pp_sync'd in buffer-only PP; cache-mode placeholder length equals the live layout), test_unified_radix_cache_unittest.py::test_buffer_only_write_backlog_throttle_holds_queued_writes_until_staged (real cache + file backend across the suite's configs, on and off, including the stall release through check_hicache_events).

Accuracy Tests

Not applicable (admission pacing only; nothing changes in what is written, read or restored). The GPU runs below verify byte-identical greedy output cold vs L3 on the verify prompts.

Speed Tests and Profiling

Lane: Qwen2.5-1.5B-Instruct TP2 (B200), page 64, --max-total-tokens 65536 --hicache-ratio 2 (host 131,136 tokens, write window 118,023), python tree core, HybridCacheController, --hicache-storage-backend file, buffer_only, write_through, prefetch policy timeout; 40 cold prompts x 6-12k tokens, flush, verify, replay with a 64-token extension; per-rank counters (ranks identical). The stock HiCacheFile drains at ~1.6 GB/s/rank against ~4.3 GB/s/rank of prefill KV, so every seed overruns the window. All drops are source_evicted; the other four reasons stayed 0.

throttle conc seed tokens dropped / rank drop % replay full / zero / partial seed wall (s) seed TTFT p50 / p90 / max (s)
off 8 375,192 70,592 18.8% 31 / 5 / 4 1.25 0.222 / 0.259 / 0.345
on 8 369,304 22,656 6.1% 36 / 2 / 2 1.88 0.230 / 0.544 / 0.714
on (seed 2) 8 357,976 15,360 4.3% 37 / 1 / 2 1.74 0.221 / 0.433 / 0.738
on 4 352,664 39,744 11.3% 35 / 4 / 1 1.35 0.086 / 0.151 / 0.225

Earlier samples of the same code path at c4/c8 (before the squash) sit in the same bands: off 22-33% at c8 and 22-23% at c4; on 2.6-7.1% at c8 and 7-19% at c4.

Sustained back-pressure (store throttled to 0.3 GB/s with a fault-injecting HiCacheFile subclass; c8 seed plus a 256-token never-cached canary every 0.5 s for 70 s):

throttle dropped / rank drop % replay full / zero / partial seed wall (s) seed TTFT p50 / p90 / max (s) canary TTFT p50 / p90 / p99 (s), >1 s
off 160,768 44.9% 22 / 16 / 2 1.21 0.213 / 0.257 / 0.339 0.014 / 0.021 / 0.189, 0 of 140
on 35,520 9.6% 35 / 3 / 2 9.55 0.333 / 4.286 / 5.183 0.014 / 1.676 / 4.301, 16 of 140

Dead store (the first storage set hangs forever, so the single backup thread never acks again; c8 seed plus canaries for 120 s, throttle on): exactly one write backlog throttle released line per rank, 32 s after the seed started; no re-engage line (the latch holds); canary TTFT p50 15.4 s / max 29.9 s during the 30 s hold, p50 14 ms / p90 18 ms afterwards; all drops labeled source_evicted; no other WARNING/ERROR.

Reading: with the throttle on, the seed is paced to the store (9.6 s instead of 1.2 s at 0.3 GB/s), coverage goes from 22 to 35 full replays, and the cost lands on TTFT of everything queued behind the throttled admission, including unrelated short requests. That is the intended trade and why it is opt-in. Under a dead store the price is one stall of the timeout length, then admission with labeled drops.

Validation run (from the repo root, GPU index set for CUDA_VISIBLE_DEVICES):

  • pytest test_buffer_mode_backlog_throttle.py managers/test_scheduler_write_backlog_latch.py managers/test_prefill_adder.py test_prefill_memory_budget.py test_buffer_mode_sidecar.py test_buffer_mode_backup_drops.py test_storage_prefetch_lifecycle.py observability/test_stat_loggers_di.py test_hicache_pp_sync_drain.py -> 122 passed.
  • pytest test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py -k "buffer or backup or staged" (1 GPU) -> 278 passed, 329 skipped (the new case on 13 configs, on + off; Mamba configs skipped by the buffer fixture).
  • pre-commit run --files <changed files> -> passed.
  • Not run: full CI, AMD/NPU lanes, PP > 1 live (the sync-vector change is covered on CPU), the Rust tree core under the new suite case.

Checklist

@mintlify

mintlify Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
sglang-doc 🟢 Ready View Preview Oct 14, 2026, 10:03 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache labels Oct 14, 2026
…write backlog

With --hicache-host-memory-mode buffer_only and write_through, a queued
backup intent is unpinned until its D2H launches and the staging window
is held until the storage ack, so a store that drains slower than prefill
produces KV turns back-pressure into silent L3 coverage loss: the device
LRU evicts exactly the queued leaves and their spans are never written.
Deployments that would rather pay TTFT than lose coverage had no knob.

Add "write_backlog_throttle": true to --hicache-storage-backend-extra-
config (default false, nothing changes). When on, PrefillAdder charges the
queued-but-unstaged backlog to the admission budget, so prefill admits only
what written or free memory can hold and recovers as writes stage. It is a
budget charge, not a lock: the eviction walk is unchanged and a stale-swept
span releases its share. The whole backlog is held: a fixed cap cannot pace,
because once the backlog exceeds it each admission evicts the spans it then
re-queues. Safety comes from liveness instead: every rank reports whether
the write path launched or acked within 30 s, the per-tick HiCache sync
MIN-reduces it (and PP-syncs PP0's verdict in buffer-only PP), and the hold
lapses with a warning while the store is stalled, latched until the next
launch or ack so a dead store cannot re-arm the hold every timeout. A
hold-induced NO_TOKEN no longer latches batch_is_full until a request
finishes: the scheduler reopens admission once the hold shrinks.
@xiezhq-hermann
xiezhq-hermann force-pushed the hicache/backup-backlog-shedding branch from 1f598f6 to f89c1b4 Compare October 14, 2026 23:02
@xiezhq-hermann
xiezhq-hermann changed the base branch from main to hicache/backup-drop-visibility October 14, 2026 23:05
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed

This branch was successfully deployed

1 active deployment
staging - docs — f89c1b47 Deployed Oct 14, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant