Skip to content

[Fix] hicache: Bound storage backend calls and keep the scheduler alive when a worker wedges - #44343

Draft
xiezhq-hermann wants to merge 2 commits into
mainfrom
hicache/bounded-storage-calls
Draft

xiezhq-hermann wants to merge 2 commits into
mainfrom
hicache/bounded-storage-calls

Conversation

@xiezhq-hermann

@xiezhq-hermann xiezhq-hermann commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Revision 2 (force-pushed once, minutes after the draft opened, no external reviews yet). Restructured after an internal review: this PR is now only the bounded backend calls, the host-slot quarantine, their metrics and docs. The flush/reset hardening for a worker that cannot be joined (reset never raising, abandonment, Scheduler.flush_cache reporting) moved to a follow-up PR stacked on #44342 so the storage disable rides that PR's rank-agreed health bit. Changes from revision 1: per-slot pin counts instead of a pinned set (overlapping pins from a KV read and its sidecar read, or a whole-operation write over its per-batch writes, were released by the first unpin); the KV-derived sidecar read is skipped after a KV miss; a backend-raised TimeoutError is classified as an error, not as our deadline (same class on Python 3.11+); calls issued after a stop request return the miss without being dispatched; the one-caller-per-kind invariant is enforced; pools derived from a side pool are pinned explicitly; the refusal state is reported once as an ERROR; defaults lowered to 5 / 20 / 60 s; the unified page-envelope pool's clear() bounds its lease wait; the guard is an instance member.

Revision 3 (additional commit, after a second internal review). The one-caller rule had a second QUERY caller: the PD-decode scheduler thread probes L3 through query_storage_hit_length while the prefetch worker's query for the previous request may be in flight, so under --enable-decode-hicache the enforcement would have raised out of the scheduler loop. Scheduler-thread probes (that probe and PP0's ticket submission) now run on their own PROBE lane with the query deadline, and query_storage_hit_length turns any guard error into a miss. Also: the lane is freed before the orphan's spans are pinned (a pin failure can no longer leave a kind permanently busy), the deadline keys are validated before attach_storage_backend has any side effect, and the unified page-envelope wording below is corrected.

Motivation

The HiCache controller talks to the L3 storage backend from single worker threads (prefetch_thread_func, prefetch_io_aux_func, backup_thread_func) and the backend contract has no deadline. One backend call that never returns on one rank therefore wedges the whole prefetch/backup pipeline on every rank: the peer ranks block in the MIN collectives behind it (_sync_prefetch_hit_query, _reduce_prefetch_ack) and every later operation queues behind the stuck one. Under --hicache-storage-prefetch-policy timeout every later L3-hit request is deadline-cut and recomputed forever (plus a re-upload storm), under wait_complete every >= prefetch_threshold prompt hangs, and /health stays green because short prompts still run. #43550 made a raising backend call survivable; a hanging one (stuck RPC, slow scandir, dead metadata server) reproduces number for number.

Measured on a TP2 file-backend lane with one injected fault (Qwen2.5-1.5B, buffer_only, write_through; 16 seeded 6-8k prefixes, replay #1, then 8 healthy L3-hit requests at +10 s and 6 more at +2 min, then /flush_cache), this PR with its default bounds (5 / 20 / 60 s):

fault policy before (main) after
batch_get hangs on rank 1 timeout #1 2.92 s cached 0; all 14 later L3-hit requests cached 0 (2.5-3.0 s each) forever; /flush_cache kills both schedulers #1 2.92 s cached 0 (deadline cut); the 8 requests at +10 s are still cut (the GET lane is blocked until the 20 s bound expires); the 6 at +2 min get full hits (0.15-0.30 s); flush 200, alive
batch_get hangs on rank 1 wait_complete #1 stalls > 150 s, all 14 later requests stall #1 20.09 s cached 0 (= GET bound + recompute); the 8 at +10 s wait behind the hung read and get full hits at 10.2-10.4 s; the 6 at +2 min full hits in 0.1-0.3 s; flush 200
batch_exists raises on rank 0 timeout the faulted and every later >= 256-token request pays +1.0-1.2 s with cached 0 (query thread dead, peer in gloo); flush 200 then dead only the faulted request misses (0.05 s TTFT, recompute); 13/13 others full hits; flush 200
batch_exists raises on rank 0 wait_complete every later >= 256-token prompt stalls same as timeout: one miss, 13/13 full hits; flush 200

With tighter bounds (2 / 5 / 10 s, same guard, revision-1 build) the 8 requests at +10 s got full hits in all four scenarios as well; the default GET bound trades that head-of-line window for not cutting legitimately slow transfers.

Modifications

1. Bounded backend calls (python/sglang/srt/mem_cache/storage_call_guard.py, new). StorageCallGuard.run() executes one backend call on a per-kind daemon worker (hicache-storage-{query,probe,get,set}) while the controller thread waits with a deadline. On timeout or exception it logs a rate-limited WARNING (kind, op, key count), counts it, and returns the controller's own miss shape, so every per-batch PrefetchAck / hit count / backup ack is still produced and the peers' MIN reduces stay balanced:

call site miss shape
_storage_hit_query (batch_exists, hybrid batch_exists_v2) 0 / PoolTransferResult.empty()
_page_transfer_kv_batch (page_get_func, KV-derived batch_get_v2, skipped after a KV miss), hybrid _page_transfer_sidecar (batch_get_v2) 0 pages / all-False per transfer (pool_transfer_misses)
_page_backup (page_set_func), hybrid _page_backup_with_stable_layout (batch_set_v2) False / all-False

Each kind has exactly one calling thread, which the guard enforces: the controller's query, transfer and backup workers for QUERY / GET / SET, and the scheduler thread for PROBE, the existence probes it issues itself (the PD-decode hit probe in query_storage_hit_length, PP0's ticket submission in submit_prefetch), which share storage_query_timeout_s and therefore wait up to it on the scheduler thread. A probe may overlap the prefetch worker's query, hence the separate lane. Calls of one kind run one at a time unless the earlier one was orphaned. Deadlines come from --hicache-storage-backend-extra-config keys storage_query_timeout_s / storage_get_timeout_s / storage_set_timeout_s (popped before the backend sees the config; 0/null disables; defaults 5 / 20 / 60 s per batch call of up to 128 pages, i.e. one to two orders of magnitude above the healthy tails measured on remote backends). A backend-raised TimeoutError counts as an error, not as our deadline. Once a stop request (reset / detach) is posted, the call in flight gets a 5 s grace and later calls return the miss without being dispatched, so the controller threads stay joinable. Backend classes are untouched.

2. Orphan ownership. A timed-out call is not cancelled; it keeps running on its now-orphaned worker and a fresh worker serves the next call. Because zero-copy batch_get_v1/v2 write straight into host pool rows and batch_set* read them, those rows must not be reused while the orphan is alive. Before publishing the miss the guard quarantines the spans the call touches (HostSpan: the KV batch span in the anchor pool, each side pool's host_indices in the pool that allocated them, the KV span for KV-derived pools) via a new allocator-level HostSlotQuarantine (pool_host/base.py) wired into HostKVCache.free/clear, LogicalHostPool, DeepSeekV4PagedHostPool, MambaPoolHost and the unified page-envelope backing. Pins are counted per slot. The owner keeps freeing as usual (ack drain, hybrid discard, abort, staging release, L2 eviction, /flush_cache clear); the pinned part is held instead of returned to the free list and comes back when the orphan's done-callback calls release_quarantine. The hybrid worker also holds mem_pool_host.layout_lease() around the call so an orphan keeps the unified layout pinned; a clear() of that pool now bounds its wait for such a lease (10 s) and raises instead of blocking the scheduler forever. After max_orphans (4) hung calls of one kind, new calls of that kind return the miss immediately; that state is reported once as an ERROR, and the pinned-memory ceiling is max_orphans x 128 pages per kind and rank.

3. Observability. sglang:hicache_storage_call_timeouts_total{kind}, sglang:hicache_storage_call_errors_total{kind} (raised or refused) and the gauge sglang:hicache_storage_orphaned_calls{kind} in StorageMetricsCollector, fed from the guard on the existing per-tick log_storage_metrics path. Docs: a "Bounding Storage Backend Calls" subsection in hicache_best_practices.mdx.

Three existing unit fixtures that build bare controllers (test_buffer_mode_sidecar.py, test_pp_prefetch_ticket.py, test_hybrid_pool_assembler.py) gain one line each for the new call hook.

Points for reviewers:

  • Default bounds. Under wait_complete the GET bound is the TTFT of every request queued behind one hung read (measured 10.2-10.4 s with the 20 s default); under timeout those requests are deadline-cut until the bound expires. 5 / 20 / 60 s keeps >= 15x headroom over the measured healthy tails; wait_complete users with a fast backend can set storage_get_timeout_s to ~10x their p99.
  • Orphan ownership is implemented in the allocators rather than by making backends cancellable, so it covers every backend without per-backend work; the cost is host memory pinned for as long as a hung call lives (bounded by max_orphans) and a few lines in each slot-list allocator.
  • Unified page-envelope (mori) pool. An orphan holds a layout lease; compaction and clear() wait for it. The alloc-path wait is pre-existing behaviour. clear() now gives up after 10 s instead of hanging, but the RuntimeError it raises still leaves _reset_full -> Scheduler.flush_cache and exits the scheduler (as on main), so on this pool a /flush_cache issued while an orphaned call, or a merely slow backup once the 5 s stop grace has orphaned it, holds the lease still kills the server. Keeping quarantined pages immovable and not waiting for abandoned leases is part of the follow-up PR ([Fix] hicache: Flush the cache without dying when a storage worker cannot be stopped #44348), which removes that outcome.
  • A worker that cannot be joined (bound disabled, or a thread stuck in a gloo collective because its peer died) still makes reset() raise exactly as on main; the follow-up PR turns that into a reported, rank-agreed degradation.

Accuracy Tests

Not applicable: no change to model forward or to what a successful transfer writes. A timed-out or failed call is published as a miss (recompute), never as data.

Speed Tests and Profiling

No-fault control on the same lane (16 seeded prefixes replayed, timeout policy): per-request TTFT within +-0.03 s of main and identical hit token counts; the only added per-call work is one queue hand-off to the worker thread (tens of microseconds against a 128-page batch). Prod-scale perf validation was not run.

Unit tests (CPU, test/registered/unit/mem_cache/test_storage_call_guard.py, 14 cases, ~17 s): extra-config parsing (0/null disable, bad values rejected, probes share the query deadline); refusal when every worker is orphaned, reported once; a backend-raised TimeoutError counted as an error on the same worker; a second concurrent caller on one kind rejected; hung batch_exists on both controllers publishes [op0: 0 hits, op1: full] while the hang is still live; a scheduler-thread probe issued while the prefetch worker's query hangs runs on its own lane on both controllers (a hung probe is a miss within the bound, the next probe returns its hits, no one-caller error, the three calls land on the query / probe / probe workers); hung batch_get_v1 acks 0 tokens, the freed span is not allocatable and survives clear(), a raising read is the same miss, the orphan's return makes the span allocatable again; a KV miss skips the derived sidecar read; hung hybrid sidecar batch_get_v2 acks pool_hits={SWA: 0} and pins the SWA pool span; hung batch_set_v1 acks 0 tokens and pins its source span; after a stop request the in-flight call gets one grace period and queued calls are published as misses at once; overlapping pins are counted; the HostKVCache quarantine contract; the unified page-envelope backing pins pages and bounds clear().

Existing suites run green on this tree: test_storage_prefetch_lifecycle, test_buffer_mode_sidecar, test_hicache_storage_format_tag, test_hicache_file_lru_unit, test_hybrid_pool_assembler, test_pp_prefetch_ticket, test_hicache_staged_write_back_dispatch, test_mem_pool_host, test_hicache_host_register, test_hicache_pp_sync_drain, test_unified_hicache_transfer_lifecycle, test_unified_hicache_regressions, test_dsv4_hicache_l2 (CPU); test_unified_radix_cache_unittest -k "prefetch or storage or backup" (201 passed / 220 skipped) and test_mamba_unittest (12 passed) on one GPU.

GPU fault injection: the four rows of the table above (TP2, GPUs 0,1 of a B200 box, file backend wrapped by a fault-injecting dynamic backend, py-spy of every process at 60 s and after the flush: the orphaned hicache-storage-get worker is pinned in the fault next to a fresh worker, all controller threads idle, no Exception in thread, hicache_storage_call_timeouts_total{kind="get"} = 1 on the faulted rank). Revision 3 re-ran the batch_get hang / wait_complete row on the final head (4908f3a): #1 20.08 s cached 0 (page_get did not return within 20.0s ... orphaned on TP1), the 8 requests at +10 s full hits at 10.17-10.42 s, the 6 at +2 min full hits in 0.08-0.28 s, /flush_cache 200, /health 200, three post-flush requests 200 in 0.03 s, no Exception in thread.

Not run: real remote backends (mooncake / hf3fs / nixl / eic) under a hang, PP > 1, the unified page-envelope (mori) host pool end to end, and a prod-scale benchmark.

Checklist

@mintlify

mintlify Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
sglang-doc 🟢 Ready View Preview Oct 15, 2026, 12:17 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache memory-pool labels Oct 14, 2026
…ve when a worker wedges

One storage backend call that never returns on one rank wedges HiCache
prefetch/backup on every rank: the peers block in the MIN collectives
behind it and every later operation queues behind the stuck one, so the
engine keeps running with L3 permanently gone (timeout policy) or with
every L3-hit request stalled (wait_complete).

Run every backend call (query / get / set) on a per-kind worker with a
deadline (storage_{query,get,set}_timeout_s in
--hicache-storage-backend-extra-config, defaults 5 / 20 / 60 s, 0
disables). A timed-out or raising call is published as the controller's
own miss shape, so acks and hit counts stay count-stable across ranks;
the call keeps running on an orphaned worker and the host slots it may
still touch stay quarantined (per-slot pin counts) in their pool until
it returns, so a late zero-copy read or write can never land in memory
that was handed to another operation. Timeouts, errors and the orphan
count are exported as metrics. The flush/reset hardening for a worker
that cannot be joined is a separate PR.
… lane

The one-caller-per-kind rule missed a second QUERY caller: the PD-decode
scheduler thread probes L3 in query_storage_hit_length while the prefetch
worker's query for the previous request may be in flight, so under
--enable-decode-hicache the enforcement raised out of the scheduler loop.
Scheduler-thread probes (that probe and PP0's ticket submission) now run on
a PROBE lane with the query deadline, and the probe turns a guard error into
a miss. The lane is freed before an orphan's spans are pinned so a pin
failure cannot leave a kind busy forever, the deadline keys are validated
before attach has any side effect, and the docs state the unified
page-envelope flush outcome as it is.

This branch was successfully deployed

1 active deployment
staging - docs — 4908f3a3 Deployed Oct 15, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang memory-pool unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant