Repository navigation
[Fix] hicache: Bound storage backend calls and keep the scheduler alive when a worker wedges - #44343
Draft
xiezhq-hermann wants to merge 2 commits into
Draft
xiezhq-hermann wants to merge 2 commits into
xiezhq-hermann wants to merge 2 commits into
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
…ve when a worker wedges
One storage backend call that never returns on one rank wedges HiCache
prefetch/backup on every rank: the peers block in the MIN collectives
behind it and every later operation queues behind the stuck one, so the
engine keeps running with L3 permanently gone (timeout policy) or with
every L3-hit request stalled (wait_complete).
Run every backend call (query / get / set) on a per-kind worker with a
deadline (storage_{query,get,set}_timeout_s in
--hicache-storage-backend-extra-config, defaults 5 / 20 / 60 s, 0
disables). A timed-out or raising call is published as the controller's
own miss shape, so acks and hit counts stay count-stable across ranks;
the call keeps running on an orphaned worker and the host slots it may
still touch stay quarantined (per-slot pin counts) in their pool until
it returns, so a late zero-copy read or write can never land in memory
that was handed to another operation. Timeouts, errors and the orphan
count are exported as metrics. The flush/reset hardening for a worker
that cannot be joined is a separate PR.
xiezhq-hermann
force-pushed
the
hicache/bounded-storage-calls
branch
from
October 14, 2026 22:39
5a3e7fe to
b379b59
Compare
4 of 5 tasks
… lane The one-caller-per-kind rule missed a second QUERY caller: the PD-decode scheduler thread probes L3 in query_storage_hit_length while the prefetch worker's query for the previous request may be in flight, so under --enable-decode-hicache the enforcement raised out of the scheduler loop. Scheduler-thread probes (that probe and PP0's ticket submission) now run on a PROBE lane with the query deadline, and the probe turns a guard error into a miss. The lane is freed before an orphan's spans are pinned so a pin failure cannot leave a kind busy forever, the deadline keys are validated before attach has any side effect, and the docs state the unified page-envelope flush outcome as it is.
5 tasks done
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The HiCache controller talks to the L3 storage backend from single worker threads (
prefetch_thread_func,prefetch_io_aux_func,backup_thread_func) and the backend contract has no deadline. One backend call that never returns on one rank therefore wedges the whole prefetch/backup pipeline on every rank: the peer ranks block in the MIN collectives behind it (_sync_prefetch_hit_query,_reduce_prefetch_ack) and every later operation queues behind the stuck one. Under--hicache-storage-prefetch-policy timeoutevery later L3-hit request is deadline-cut and recomputed forever (plus a re-upload storm), underwait_completeevery >=prefetch_thresholdprompt hangs, and/healthstays green because short prompts still run. #43550 made a raising backend call survivable; a hanging one (stuck RPC, slowscandir, dead metadata server) reproduces number for number.Measured on a TP2 file-backend lane with one injected fault (Qwen2.5-1.5B, buffer_only, write_through; 16 seeded 6-8k prefixes, replay #1, then 8 healthy L3-hit requests at +10 s and 6 more at +2 min, then
/flush_cache), this PR with its default bounds (5 / 20 / 60 s):batch_gethangs on rank 1/flush_cachekills both schedulersbatch_gethangs on rank 1batch_existsraises on rank 0batch_existsraises on rank 0With tighter bounds (2 / 5 / 10 s, same guard, revision-1 build) the 8 requests at +10 s got full hits in all four scenarios as well; the default GET bound trades that head-of-line window for not cutting legitimately slow transfers.
Modifications
1. Bounded backend calls (
python/sglang/srt/mem_cache/storage_call_guard.py, new).StorageCallGuard.run()executes one backend call on a per-kind daemon worker (hicache-storage-{query,probe,get,set}) while the controller thread waits with a deadline. On timeout or exception it logs a rate-limited WARNING (kind, op, key count), counts it, and returns the controller's own miss shape, so every per-batchPrefetchAck/ hit count / backup ack is still produced and the peers' MIN reduces stay balanced:_storage_hit_query(batch_exists, hybridbatch_exists_v2)0/PoolTransferResult.empty()_page_transfer_kv_batch(page_get_func, KV-derivedbatch_get_v2, skipped after a KV miss), hybrid_page_transfer_sidecar(batch_get_v2)0pages / all-Falseper transfer (pool_transfer_misses)_page_backup(page_set_func), hybrid_page_backup_with_stable_layout(batch_set_v2)False/ all-FalseEach kind has exactly one calling thread, which the guard enforces: the controller's query, transfer and backup workers for
QUERY/GET/SET, and the scheduler thread forPROBE, the existence probes it issues itself (the PD-decode hit probe inquery_storage_hit_length, PP0's ticket submission insubmit_prefetch), which sharestorage_query_timeout_sand therefore wait up to it on the scheduler thread. A probe may overlap the prefetch worker's query, hence the separate lane. Calls of one kind run one at a time unless the earlier one was orphaned. Deadlines come from--hicache-storage-backend-extra-configkeysstorage_query_timeout_s/storage_get_timeout_s/storage_set_timeout_s(popped before the backend sees the config;0/nulldisables; defaults 5 / 20 / 60 s per batch call of up to 128 pages, i.e. one to two orders of magnitude above the healthy tails measured on remote backends). A backend-raisedTimeoutErrorcounts as an error, not as our deadline. Once a stop request (reset / detach) is posted, the call in flight gets a 5 s grace and later calls return the miss without being dispatched, so the controller threads stay joinable. Backend classes are untouched.2. Orphan ownership. A timed-out call is not cancelled; it keeps running on its now-orphaned worker and a fresh worker serves the next call. Because zero-copy
batch_get_v1/v2write straight into host pool rows andbatch_set*read them, those rows must not be reused while the orphan is alive. Before publishing the miss the guard quarantines the spans the call touches (HostSpan: the KV batch span in the anchor pool, each side pool'shost_indicesin the pool that allocated them, the KV span for KV-derived pools) via a new allocator-levelHostSlotQuarantine(pool_host/base.py) wired intoHostKVCache.free/clear,LogicalHostPool,DeepSeekV4PagedHostPool,MambaPoolHostand the unified page-envelope backing. Pins are counted per slot. The owner keeps freeing as usual (ack drain, hybrid discard, abort, staging release, L2 eviction,/flush_cacheclear); the pinned part is held instead of returned to the free list and comes back when the orphan's done-callback callsrelease_quarantine. The hybrid worker also holdsmem_pool_host.layout_lease()around the call so an orphan keeps the unified layout pinned; aclear()of that pool now bounds its wait for such a lease (10 s) and raises instead of blocking the scheduler forever. Aftermax_orphans(4) hung calls of one kind, new calls of that kind return the miss immediately; that state is reported once as an ERROR, and the pinned-memory ceiling ismax_orphansx 128 pages per kind and rank.3. Observability.
sglang:hicache_storage_call_timeouts_total{kind},sglang:hicache_storage_call_errors_total{kind}(raised or refused) and the gaugesglang:hicache_storage_orphaned_calls{kind}inStorageMetricsCollector, fed from the guard on the existing per-ticklog_storage_metricspath. Docs: a "Bounding Storage Backend Calls" subsection inhicache_best_practices.mdx.Three existing unit fixtures that build bare controllers (
test_buffer_mode_sidecar.py,test_pp_prefetch_ticket.py,test_hybrid_pool_assembler.py) gain one line each for the new call hook.Points for reviewers:
wait_completethe GET bound is the TTFT of every request queued behind one hung read (measured 10.2-10.4 s with the 20 s default); undertimeoutthose requests are deadline-cut until the bound expires. 5 / 20 / 60 s keeps >= 15x headroom over the measured healthy tails;wait_completeusers with a fast backend can setstorage_get_timeout_sto ~10x their p99.max_orphans) and a few lines in each slot-list allocator.clear()wait for it. The alloc-path wait is pre-existing behaviour.clear()now gives up after 10 s instead of hanging, but theRuntimeErrorit raises still leaves_reset_full->Scheduler.flush_cacheand exits the scheduler (as on main), so on this pool a/flush_cacheissued while an orphaned call, or a merely slow backup once the 5 s stop grace has orphaned it, holds the lease still kills the server. Keeping quarantined pages immovable and not waiting for abandoned leases is part of the follow-up PR ([Fix] hicache: Flush the cache without dying when a storage worker cannot be stopped #44348), which removes that outcome.reset()raise exactly as on main; the follow-up PR turns that into a reported, rank-agreed degradation.Accuracy Tests
Not applicable: no change to model forward or to what a successful transfer writes. A timed-out or failed call is published as a miss (recompute), never as data.
Speed Tests and Profiling
No-fault control on the same lane (16 seeded prefixes replayed,
timeoutpolicy): per-request TTFT within +-0.03 s of main and identical hit token counts; the only added per-call work is one queue hand-off to the worker thread (tens of microseconds against a 128-page batch). Prod-scale perf validation was not run.Unit tests (CPU,
test/registered/unit/mem_cache/test_storage_call_guard.py, 14 cases, ~17 s): extra-config parsing (0/null disable, bad values rejected, probes share the query deadline); refusal when every worker is orphaned, reported once; a backend-raisedTimeoutErrorcounted as an error on the same worker; a second concurrent caller on one kind rejected; hungbatch_existson both controllers publishes[op0: 0 hits, op1: full]while the hang is still live; a scheduler-thread probe issued while the prefetch worker's query hangs runs on its own lane on both controllers (a hung probe is a miss within the bound, the next probe returns its hits, no one-caller error, the three calls land on thequery/probe/probeworkers); hungbatch_get_v1acks 0 tokens, the freed span is not allocatable and survivesclear(), a raising read is the same miss, the orphan's return makes the span allocatable again; a KV miss skips the derived sidecar read; hung hybrid sidecarbatch_get_v2ackspool_hits={SWA: 0}and pins the SWA pool span; hungbatch_set_v1acks 0 tokens and pins its source span; after a stop request the in-flight call gets one grace period and queued calls are published as misses at once; overlapping pins are counted; theHostKVCachequarantine contract; the unified page-envelope backing pins pages and boundsclear().Existing suites run green on this tree: test_storage_prefetch_lifecycle, test_buffer_mode_sidecar, test_hicache_storage_format_tag, test_hicache_file_lru_unit, test_hybrid_pool_assembler, test_pp_prefetch_ticket, test_hicache_staged_write_back_dispatch, test_mem_pool_host, test_hicache_host_register, test_hicache_pp_sync_drain, test_unified_hicache_transfer_lifecycle, test_unified_hicache_regressions, test_dsv4_hicache_l2 (CPU); test_unified_radix_cache_unittest
-k "prefetch or storage or backup"(201 passed / 220 skipped) and test_mamba_unittest (12 passed) on one GPU.GPU fault injection: the four rows of the table above (TP2, GPUs 0,1 of a B200 box, file backend wrapped by a fault-injecting
dynamicbackend, py-spy of every process at 60 s and after the flush: the orphanedhicache-storage-getworker is pinned in the fault next to a fresh worker, all controller threads idle, noException in thread,hicache_storage_call_timeouts_total{kind="get"}= 1 on the faulted rank). Revision 3 re-ran thebatch_gethang /wait_completerow on the final head (4908f3a): #1 20.08 s cached 0 (page_get did not return within 20.0s ... orphanedon TP1), the 8 requests at +10 s full hits at 10.17-10.42 s, the 6 at +2 min full hits in 0.08-0.28 s,/flush_cache200,/health200, three post-flush requests 200 in 0.03 s, noException in thread.Not run: real remote backends (mooncake / hf3fs / nixl / eic) under a hang, PP > 1, the unified page-envelope (mori) host pool end to end, and a prod-scale benchmark.
Checklist