Repository navigation
[Perf] hicache: Add an opt-in admission throttle for the buffer-only write backlog - #44344
Draft
xiezhq-hermann wants to merge 1 commit into
Draft
xiezhq-hermann wants to merge 1 commit into
xiezhq-hermann wants to merge 1 commit into
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
…write backlog With --hicache-host-memory-mode buffer_only and write_through, a queued backup intent is unpinned until its D2H launches and the staging window is held until the storage ack, so a store that drains slower than prefill produces KV turns back-pressure into silent L3 coverage loss: the device LRU evicts exactly the queued leaves and their spans are never written. Deployments that would rather pay TTFT than lose coverage had no knob. Add "write_backlog_throttle": true to --hicache-storage-backend-extra- config (default false, nothing changes). When on, PrefillAdder charges the queued-but-unstaged backlog to the admission budget, so prefill admits only what written or free memory can hold and recovers as writes stage. It is a budget charge, not a lock: the eviction walk is unchanged and a stale-swept span releases its share. The whole backlog is held: a fixed cap cannot pace, because once the backlog exceeds it each admission evicts the spans it then re-queues. Safety comes from liveness instead: every rank reports whether the write path launched or acked within 30 s, the per-tick HiCache sync MIN-reduces it (and PP-syncs PP0's verdict in buffer-only PP), and the hold lapses with a warning while the store is stalled, latched until the next launch or ack so a dead store cannot re-arm the hold every timeout. A hold-induced NO_TOKEN no longer latches batch_is_full until a request finishes: the scheduler reopens admission once the hold shrinks.
xiezhq-hermann
force-pushed
the
hicache/backup-backlog-shedding
branch
from
October 14, 2026 23:02
1f598f6 to
f89c1b4
Compare
xiezhq-hermann
changed the base branch from
main
to
hicache/backup-drop-visibility
October 14, 2026 23:05
2 tasks done
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
…gl-project#44462(317)/sgl-project#44342-trunk(318) in both the FAM-A3b block and the B6 re-triage verdicts; 10 dated corrections (mergeable flips: sgl-project#46191 REAL -> WATCH-HIGH with the re-derive-vs-QSA_RING_GROUPS=2 instruction, sgl-project#39845 DISPUTED as UNKNOWN-not-true plus the general 'UNKNOWN is no signal' rule; sgl-project#44344 -> SECTION R DOA with the entry-268 unconditional-clear subsumption citation and the mergeable-is-a-trap note; sgl-project#42525 target path moved fla->linear; sgl-project#45520 STILL OPEN -> re-apply rule ARMED FOR MERGE sgl-project#6; b12x-package archived claim DISPUTED with the frozen cutlass-dsl==4.8.0 pin risk standing; sgl-project#46153+sgl-project#41624 gated as ONE QAD bundle; sgl-project#37106 PARKED on eroded motivation with named revive conditions; sgl-project#32479 gated behind the PCIe-IPC measurement and DE-COUPLED from sgl-project#45391; sgl-project#44462's NEXT note spent); NEW S-item S-B12XDENSE filed
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #44339 (
hicache/backup-drop-visibility): this branch is based on it and the PR base points at it, so the diff shown here is the throttle alone. Rebase ontomainonce #44339 lands.Motivation
With
--hicache-host-memory-mode buffer_onlyandwrite_through, a store that drains slower than prefill produces KV loses whole prompts from L3: a finished node's backup intent is queued unpinned until its D2H launches, host staging is held until the storage ack, andflush_pending_writesstops launching atlive_cap, so the device LRU evicts exactly the queued leaves and the stale sweep retires them (source_evictedafter #44339). The loss is bandwidth-bound: lost ~= produced - (staging window + drain x burst). Re-ordering the eviction walk or pinning queued nodes does not change those terms (pinning turns the device pool into a write buffer for the store's drain time), and back-pressure never reaches admission, so a slow store shows up as a silently collapsing L3 hit rate while TTFT stays flat. Deployments that would rather pay TTFT than lose coverage have no knob.Modifications
"write_backlog_throttle": truein--hicache-storage-backend-extra-config(defaultfalse; parsed next toprefetch_threshold/prefetch_timeout_*inHybridCacheController.parse_storage_backend_extra_config, type-checked, applied at startup and on runtime attach). It lives with the other storage-controller policy keys rather than a new server arg because it only means something with a storage backend in buffer-only mode.BufferModePipeline.write_backlog_hold_tokens()is the queued-but-unstaged backlog (write_backlog_tokens_, unique spans across pools) while the throttle is on and the write path is alive;BasePrefixCache.write_backlog_hold_tokens()(default 0) exposes it andPrefillAdder.__init__charges it tototal_offset/current_offset, so prefill admits only what free or already-written device memory can hold and recovers as writes stage. It is a budget charge, not a lock: the eviction walk is unchanged, in-flight chunked prefills and decode growth still evict queued spans (labeled drops), priority preemption still works against the hold, and a stale-swept span releases its share. There is no admission floor: with a live store, sustained admission equals the staging launch rate, which the window caps at the store's drain.pool - max_request(the obvious wedge guard) does not pace: once the backlog exceeds it, each admission evicts the spans it then re-queues, so admission runs at full speed with drops (measured: 45% dropped with a 49k cap on a 65k pool at 0.3 GB/s, same as control; 16% with the realpool - max_requestat the stock backend). Holding only while the running batch is non-empty degenerates to the cap in prefill-heavy traffic (the running batch is empty between prefill passes), i.e. exactly the PD-prefill shape this is for.write_path_alive()is each rank's verdict: a write launch or storage ack withinwrite_backlog_stall_timeout_s(30 s, a class constant), or no backlog at all; stalled latches until the next launch or ack, so a dead store with a held window stays lapsed across sweeps and new backlogs instead of re-arming every timeout (a new backlog only restarts the clock when nothing is staged or in flight). The verdict rides the existing per-tick HiCache MIN all-reduce (_sync_hicache_ready_counts); in buffer-only PP the slot is additionally_pp_sync'd from PP0 because every stage schedules on its own clock.set_write_path_alive(False)lapses the hold with one WARNING and admission proceeds (drops labeled); progress re-engages it (INFO). The storage slots of the sync vector are now derived from one slot list (_STORAGE_QUEUE_SLOTS, extra pools,write_path_alivelast), the PP placeholder from the same count, with a CPU test that placeholder and live lengths match. Slot order to coordinate with [Fix] hicache: Recover storage worker collectives and expose storage liveness #44342:[..., *extra_pools, write_path_alive, storage_healthy]keeps its[-1]parse valid. The bit is not added todrain_storage_control_queues(no callers underpython/sglang/srt).batch_is_full, which only a finished or retracted request cleared, so a hold-induced NO_TOKEN would have shut admission until the longest in-flight decode ended.Schedulernow recordswrite_backlog_hold_tokens()at that NO_TOKEN and reopens admission at the top of_get_new_batch_prefill_rawonce the hold has shrunk below it (one compare per pass; memory-bound NO_TOKENs keep the stock behavior).hicache_design.mdx: the controller-consumed extra-config keys get their own paragraph;write_backlog_throttlesays who should turn it on (coverage-first deployments behind slow or remote stores, e.g. PD prefill engines), who should not (latency tiers; anywhere the store drains faster than prefill), that every request queued behind the throttle pays, and the 30 s stall constant.test/registered/unit/mem_cache/test_buffer_mode_backlog_throttle.py(CPU: default off holds nothing; the hold is the whole backlog and is released as intents stage or sweep; a stalled write path lapses the hold, stays latched through a sweep and a new backlog while the window is held, re-engages on a storage ack, restarts the clock only when idle; the extra-config key is consumed and type-checked),test/registered/unit/managers/test_scheduler_write_backlog_latch.py(CPU: the latch reopens when the hold shrinks and holds otherwise, through the real_get_new_batch_prefill_rawgate),test_prefill_adder.py::test_write_backlog_hold_is_charged_to_both_budgets,test_hicache_pp_sync_drain.py(PP0's liveness slot is_pp_sync'd in buffer-only PP; cache-mode placeholder length equals the live layout),test_unified_radix_cache_unittest.py::test_buffer_only_write_backlog_throttle_holds_queued_writes_until_staged(real cache + file backend across the suite's configs, on and off, including the stall release throughcheck_hicache_events).Accuracy Tests
Not applicable (admission pacing only; nothing changes in what is written, read or restored). The GPU runs below verify byte-identical greedy output cold vs L3 on the verify prompts.
Speed Tests and Profiling
Lane: Qwen2.5-1.5B-Instruct TP2 (B200), page 64,
--max-total-tokens 65536 --hicache-ratio 2(host 131,136 tokens, write window 118,023), python tree core,HybridCacheController,--hicache-storage-backend file,buffer_only,write_through, prefetch policytimeout; 40 cold prompts x 6-12k tokens, flush, verify, replay with a 64-token extension; per-rank counters (ranks identical). The stockHiCacheFiledrains at ~1.6 GB/s/rank against ~4.3 GB/s/rank of prefill KV, so every seed overruns the window. All drops aresource_evicted; the other four reasons stayed 0.Earlier samples of the same code path at c4/c8 (before the squash) sit in the same bands: off 22-33% at c8 and 22-23% at c4; on 2.6-7.1% at c8 and 7-19% at c4.
Sustained back-pressure (store throttled to 0.3 GB/s with a fault-injecting
HiCacheFilesubclass; c8 seed plus a 256-token never-cached canary every 0.5 s for 70 s):Dead store (the first storage
sethangs forever, so the single backup thread never acks again; c8 seed plus canaries for 120 s, throttle on): exactly onewrite backlog throttle releasedline per rank, 32 s after the seed started; no re-engage line (the latch holds); canary TTFT p50 15.4 s / max 29.9 s during the 30 s hold, p50 14 ms / p90 18 ms afterwards; all drops labeledsource_evicted; no other WARNING/ERROR.Reading: with the throttle on, the seed is paced to the store (9.6 s instead of 1.2 s at 0.3 GB/s), coverage goes from 22 to 35 full replays, and the cost lands on TTFT of everything queued behind the throttled admission, including unrelated short requests. That is the intended trade and why it is opt-in. Under a dead store the price is one stall of the timeout length, then admission with labeled drops.
Validation run (from the repo root, GPU index set for
CUDA_VISIBLE_DEVICES):pytest test_buffer_mode_backlog_throttle.py managers/test_scheduler_write_backlog_latch.py managers/test_prefill_adder.py test_prefill_memory_budget.py test_buffer_mode_sidecar.py test_buffer_mode_backup_drops.py test_storage_prefetch_lifecycle.py observability/test_stat_loggers_di.py test_hicache_pp_sync_drain.py-> 122 passed.pytest test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py -k "buffer or backup or staged"(1 GPU) -> 278 passed, 329 skipped (the new case on 13 configs, on + off; Mamba configs skipped by the buffer fixture).pre-commit run --files <changed files>-> passed.Checklist