Skip to content

[Fix] hicache: Flush the cache without dying when a storage worker cannot be stopped - #44348

Draft
xiezhq-hermann wants to merge 6 commits into
mainfrom
hicache/flush-wedged-storage
Draft

xiezhq-hermann wants to merge 6 commits into
mainfrom
hicache/flush-wedged-storage

Conversation

@xiezhq-hermann

@xiezhq-hermann xiezhq-hermann commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #44342 (hicache/recoverable-storage-sync, head 74a0e6b) and #44343 (hicache/bounded-storage-calls, commits b379b59 + 4908f3a). Review only this PR's own commits (the first Flush the cache without dying ... commit and the revision-2 commit on top): they build on #44342's rank-agreed storage health bit and on #44343's host-slot quarantine. Draft until both land.

Revision 2 (additional commit, after a second internal review). Scheduler.flush_cache keeps returning bool (the registered flush-cache tests pass unchanged) and reports the wedge by raising HiCacheStorageWedgedError once the pools are flushed; the wrapper turns it into the Cache flushed, but ... response. Abandoned layout leases on the unified page-envelope pool are keyed by per-acquire tokens instead of thread idents, which Python recycles. A returning abandoned worker can no longer clear the live worker's in-flight slot, issues no further backend call once it learns it is dead, and the PP ticket thread checks its generation right after the broadcast. The two-rank test drives the production check_hicache_events round. Message and docs state the raise-policy, second-flush and never-returning-worker outcomes.

Motivation

/flush_cache is the operator's remediation for a wedged HiCache storage pipeline, and today it is the one thing that is guaranteed to kill the server: UnifiedRadixCache.reset() -> cache_controller.reset() joins the storage worker threads for 10 s and raises Failed to stop HiCache storage threads cleanly, which escapes Scheduler.flush_cache and exits the scheduler on every rank (reproduced 9/9 on a TP2 lane with one hung backend call, both on main and with #43550). #44343 bounds the backend calls so a hung call no longer wedges a worker, but a worker can still be unjoinable when the bound is disabled, when it is stuck in a rank-sync collective whose peer is gone, or when a backend ignores its deadline. Restarting new workers over a live one is not safe either: the stuck worker still owns the host rows of its operation, and when it finally returns it would publish a stale ack into the live queues (one extra ack on one rank misaligns the peers' MIN sequence) and release pages the flush already handed out.

Modifications

  1. reset() never raises; a stuck worker is abandoned, not restarted over. HiCacheController.reset() joins the workers (10 s each) and leaves the ones still alive on a dead generation: every worker loop and every publication site checks the generation it was started in, so an abandoned worker neither consumes from nor publishes into the live queues, drops its host releases, and issues no collective once it returns (_enter_storage_worker / _storage_worker_is_live / _publish). The operations the transfer and backup workers were inside (_inflight_prefetch_op / _inflight_backup_op) get their host spans quarantined through [Fix] hicache: Bound storage backend calls and keep the scheduler alive when a worker wedges #44343's HostSlotQuarantine; the stuck worker releases them in its finally when it returns. Fresh workers start on every rank, so no peer is left waiting on a rank without workers, and reset() returns the degradation message. UnifiedRadixCache._reset_full finishes the tree reset (the host pool is cleared through clear_host_pool(), after the quarantine), then raises HiCacheStorageWedgedError (defined in hicache_storage.py); Scheduler.flush_cache catches it, still flushes the device pools, and re-raises it at the end instead of dying, keeping its bool return for the ordinary outcomes. SchedulerFlushWrapper turns it into FlushCacheReqOutput(success=False, message="Cache flushed, but HiCache storage worker(s) [...] did not stop within 10s ..."), the weight updater logs it and lets the update stand (the cache is flushed), and the PD role switch catches it and continues the flip with L3 off. An abandoned worker that returns cannot clear the live worker's in-flight slot (compare-and-clear), issues no further backend call for the rest of its operation (so it never takes a guard lane the live worker needs), and the PP ticket thread re-checks its generation right after broadcast_pyobj and skips task_done() on the queue a reset replaced.

  2. The degradation is rank-agreed. The wedged rank sets storage_wedged_reason, which storage_workers_healthy() ([Fix] hicache: Recover storage worker collectives and expose storage liveness #44342) folds into the MIN-reduced health bit; at the next drain every rank flips storage_unavailable together under storage_failure_policy=disable (or fails loudly under raise). enable_storage is never touched rank-locally, so _sync_hicache_ready_counts keeps one tensor length on every rank. detach_storage_backend refuses while an abandoned worker is still alive (its sync groups and backend are in use) with a message naming the thread; detach and re-attach is the recovery once it has returned.

  3. Unified page-envelope (mori) host pool. An orphaned or abandoned call holds a layout lease. Leases are per-acquire tokens (_layout_holders[token] = thread ident); before the clear, clear_host_pool() converts the controller's orphaned and wedged thread idents into the tokens they hold at that moment (abandon_leases_of), so a thread ident recycled after the abandoned thread exits can never exempt a later lease. Compaction and clear() do not wait for abandoned tokens; the pages those calls may still write are immovable for compaction and survive clear() in place, unallocatable until their quarantine is released. After a flush this also removes the scheduler-thread wait on that call in the allocation-path compaction; an orphan created after the last flush is still waited for there, as on main.

  4. The two controller extra-config pops ([Fix] hicache: Recover storage worker collectives and expose storage liveness #44342's sync knobs, [Fix] hicache: Bound storage backend calls and keep the scheduler alive when a worker wedges #44343's call deadlines) are merged into _apply_storage_extra_config, validated before any side effect.

Operator story (message and hicache_best_practices.mdx): under storage_failure_policy=disable L3 is degraded on every rank until the backend is detached and re-attached, which needs the abandoned worker to return (detach is refused while it is alive; if it never returns, the recovery is a restart). A later /flush_cache returns 200 but L3 stays disabled until the re-attach. Under storage_failure_policy=raise the flush is followed by the scheduler failure at the next HiCache drain on every rank, as that policy asks.

Accuracy Tests

Not applicable: no change to model forward or to what a successful transfer writes.

Speed Tests and Profiling

No hot-path change: one thread-local read per worker publication; the generation check runs only on the worker threads. Not benchmarked at scale.

Unit tests (CPU, test/registered/unit/mem_cache/test_hicache_flush_wedged_storage.py, 6 cases, ~20 s): a flush with the bound disabled and a worker stuck in the backend returns the reason instead of raising, restarts a new generation, keeps the stuck operation's host memory quarantined across the pool clear, refuses detach while the worker lives, serves new backups meanwhile, the returning worker publishes nothing into the live queues and releases its quarantine, a reset with nothing in flight stays healthy and a later clean reset does not lift the degrade; an abandoned transfer worker's late acks and host releases are dropped; a two-rank gloo run where only rank 1 wedges at reset ends with storage_unavailable on both ranks after one production check_hicache_events round and enable_storage untouched; the unified page-envelope backing's clear() does not wait for an abandoned lease holder and keeps the pinned page mapped in place; compaction leaves a pinned page where it is; abandonment marks the leases held at marking time, so a lease taken afterwards under the same (recycled) thread ident is waited for and _begin_compaction still refuses. test/registered/unit/managers/test_scheduler_flush_cache.py (8, one new: the wedge message reaches FlushCacheReqOutput) and test_scheduler_flush_cache_after_retract.py (7, one new: the pools are cleared before the error surfaces) pass with flush_cache returning bool. Existing suites on this tree: test_storage_call_guard (14), test_hicache_storage_sync_recovery (the knob test also checks the deadline keys), test_pp_prefetch_ticket, test_hybrid_pool_assembler, test_buffer_mode_sidecar, test_hicache_pp_sync_drain, test_storage_prefetch_lifecycle, test_hicache_staged_write_back_dispatch, test_mem_pool_host, test_hicache_host_register, test_unified_hicache_transfer_lifecycle, test_unified_hicache_regressions, test_hicache_file_lru_unit, test_hicache_storage_format_tag, test_dsv4_hicache_l2, test_decode_hicache_tree_core, test_decode_queue_cleanup (205 passed / 3 skipped); test_unified_radix_cache_unittest -k "prefetch or storage or backup" (201 passed / 220 skipped) and test_mamba_unittest (12) on one GPU.

GPU fault injection (TP2, GPUs 0,1 of a B200 box, Qwen2.5-1.5B, buffer_only, write_through, file backend wrapped by a fault-injecting dynamic backend; pack-1 protocol, /flush_cache at the end, then three requests and /metrics):

run policy req #1 (t=0) #2-#9 (t+10 s) #10-#15 (t+2 min) /flush_cache after the flush
batch_get hang on rank 1, GET bound disabled (storage_get_timeout_s: 0) -- the wedge reset() must survive timeout cached 0, 2.92 s (policy cut) cached 0 x8 at 2.5-3.0 s (rank 1's transfer worker stays inside the hung read for the whole run, exactly like main) cached 0 x6 HTTP 400 Cache flushed, but HiCache storage worker(s) ['Thread-8 (prefetch_sync_thread_func)'] did not stop within 10s and were abandoned with their host memory quarantined; L3 is degraded on every rank until the storage backend is re-attached, which needs the abandoned worker(s) to return. (TP1 names Thread-7 (prefetch_io_aux_func)) /health 200, both schedulers alive; three /generate requests (600 / 600 / 40 words) answered with HTTP 200 in 0.03 s each; both ranks log the unhealthy pipeline in the same drain and sglang:hicache_storage_healthy = 0 on both; py-spy: the abandoned Thread-7 (prefetch_io_aux_func) is still parked in the hung read next to the fresh generation on TP1. On main (and with #43550) this same flush exits both schedulers.
batch_get hang on rank 1, default bounds timeout cached 0, 2.92 s (policy cut) cached 0 x8 at 2.5-3.0 s (GET lane blocked until the 20 s bound orphans the call) full x6, 0.18-0.34 s 200 requests served, hicache_storage_healthy = 1 on both ranks (an orphaned call is not a wedge)
batch_get hang on rank 1, default bounds wait_complete cached 0, 20.08 s (= GET bound; main hangs here forever) full x8 at 10.2-10.4 s full x6, 0.20-0.34 s 200 same
batch_exists raises on rank 0, default bounds timeout full, 0.14 s (the after=60 arm landed between req #1 and wave 1 in this run) the first wave-1 query raised on rank 0 -> that request cached 0 on both ranks, 0.04 s; the other seven full full x6 200 same
batch_exists raises on rank 0, default bounds wait_complete cached 0, 0.05 s (raised on rank 0, published as a miss on both ranks) full x8 full x6 200 same

Every row: no Exception in thread, no Failed to stop HiCache storage threads in the server log; the fault log shows exactly one injected fault per run. The rows were measured on the first commit of this PR; the wedge row was re-run on the final head (revision 2) with the same outcome: /flush_cache HTTP 400 Cache flushed, but HiCache storage worker(s) ['Thread-8 (prefetch_sync_thread_func)'] did not stop within 10s and were abandoned with their host memory quarantined; L3 is degraded on every rank until the storage backend is re-attached, which needs the abandoned worker(s) to return (restart the server if they never do)., /health 200, three post-flush requests HTTP 200 in 0.03 s, sglang:hicache_storage_healthy 0 on both ranks, py-spy showing the abandoned Thread-7 (prefetch_io_aux_func) still inside the hung read next to the new generation's Thread-11.

Not run: real remote backends, PP > 1, the mori pool end to end, the raise failure policy end to end.

Checklist

…liveness

A gloo timeout inside the HiCache storage worker collectives (the query
thread's hit MIN, the sync thread's ack MIN) killed the worker on every
rank, so one rank stuck in its backend left every rank unable to publish
L3 queries or acks: under wait_complete every later L3-hit request waited
forever, under timeout every one paid the full deadline. #43550 guards the
backend call only, not the collective, and nothing checked worker liveness.

Wrap every worker-thread collective: on failure log once, flag the
controller (storage_sync_failed), publish the operation as FAILED (0 hits,
0 completed tokens) and keep the thread alive; after the first failure the
broken groups are skipped so the per-rank publication sequence stays
intact for the queue-count drain. Fold a rank-local health bit (no sync
failure and all storage workers alive) into the MIN-reduce the scheduler
drain already issues; when the agreed bit drops, apply
storage_failure_policy: "disable" (default) degrades to no-L3 until the
backend is re-attached (no new prefetch/backup, in-flight prefetches cut
like a deadline, drained hits read as misses, progress frozen, gauge
sglang:hicache_storage_healthy -> 0), "raise" fails the scheduler. A new
storage_sync_timeout extra-config key bounds the sync groups' gloo
timeout so a hung peer is detected before torch's 30-minute default.
Review follow-up. Skipping every later collective after the first sync
failure was unsound: gloo closes only the pair of the group that timed
out, and the hit and completion sync groups are distinct even on TP2, so
a peer that had not failed blocked for the untouched group's full timeout
(30 min by default), during which reset/detach could not join and flush
raised out of the scheduler. A retry on the broken group raises at once
on every rank, so the collective is now always entered.

Also: take_ready_pp_prefetch waits in 1 s slices and fails loud when the
local ticket thread is dead; the buffer-only PP>1 health relay is limited
to the ticket-less shape (with tickets PP0 decides per request); the
sync knobs are validated before any attach side effect; the gauge drops
on detach; a WARNING once a minute while degraded; thread attributes are
initialised in __init__; reset logs that L3 stays disabled. The two-rank
test now runs its three cases in one spawn through check_hicache_events,
adds the cross-group case (an ack on the healthy group must reduce
promptly on both ranks after the hit group failed) and takes ~26 s.
…ve when a worker wedges

One storage backend call that never returns on one rank wedges HiCache
prefetch/backup on every rank: the peers block in the MIN collectives
behind it and every later operation queues behind the stuck one, so the
engine keeps running with L3 permanently gone (timeout policy) or with
every L3-hit request stalled (wait_complete).

Run every backend call (query / get / set) on a per-kind worker with a
deadline (storage_{query,get,set}_timeout_s in
--hicache-storage-backend-extra-config, defaults 5 / 20 / 60 s, 0
disables). A timed-out or raising call is published as the controller's
own miss shape, so acks and hit counts stay count-stable across ranks;
the call keeps running on an orphaned worker and the host slots it may
still touch stay quarantined (per-slot pin counts) in their pool until
it returns, so a late zero-copy read or write can never land in memory
that was handed to another operation. Timeouts, errors and the orphan
count are exported as metrics. The flush/reset hardening for a worker
that cannot be joined is a separate PR.
…nnot be stopped

/flush_cache is the operator's remediation for a wedged HiCache storage
pipeline and today it kills the scheduler: cache_controller.reset()
joins the storage workers for 10 s and raises, and the exception escapes
Scheduler.flush_cache on every rank. A worker can still be unjoinable
with the call deadlines of the previous change (bound disabled, stuck in
a rank-sync collective whose peer is gone, backend ignoring deadlines),
and restarting new workers over a live one is unsafe: it still owns its
operation's host rows and would publish stale acks into the live queues
when it returns.

reset() now joins the workers and leaves a stuck one on a dead worker
generation: generation checks keep it from consuming the live queues,
publishing into them, releasing host pages or entering a collective
once it returns, and the host spans of the operation it is inside stay
quarantined until then. Fresh workers start on every rank, the wedged
rank flags storage_workers_healthy() so the MIN-reduced health bit
degrades L3 on every rank together (enable_storage is never flipped
rank-locally), the flush completes and reports the stuck worker through
FlushCacheReqOutput.message, the PD role switch continues, and detach
refuses while the abandoned worker still lives. The unified
page-envelope host pool no longer waits for abandoned leases in
compaction and clear(); the pages such a call may still write stay in
place and quarantined. The two controller extra-config pops are merged.

Stacked on #44342 and #44343.
@mintlify

mintlify Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
sglang-doc 🟢 Ready View Preview Oct 15, 2026, 12:17 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache memory-pool labels Oct 14, 2026
… lane

The one-caller-per-kind rule missed a second QUERY caller: the PD-decode
scheduler thread probes L3 in query_storage_hit_length while the prefetch
worker's query for the previous request may be in flight, so under
--enable-decode-hicache the enforcement raised out of the scheduler loop.
Scheduler-thread probes (that probe and PP0's ticket submission) now run on
a PROBE lane with the query deadline, and the probe turns a guard error into
a miss. The lane is freed before an orphan's spans are pinned so a pin
failure cannot leave a kind busy forever, the deadline keys are validated
before attach has any side effect, and the docs state the unified
page-envelope flush outcome as it is.
…ses by token

flush_cache returns bool again and reports a storage worker it could not
stop by raising HiCacheStorageWedgedError once the pools are flushed; the
flush wrapper turns it into the "Cache flushed, but ..." response and the
weight updater logs it, so the registered flush-cache tests pass unchanged.
Abandoned layout leases on the unified page-envelope pool are per-acquire
tokens rather than thread idents, which are recycled once the abandoned
thread exits. A returning abandoned worker cannot clear the live worker's
in-flight slot, issues no further backend call, and the PP ticket thread
re-checks its generation after the broadcast. The two-rank test drives the
production drain; message and docs cover the raise policy, a second flush
and a worker that never returns.

This branch was successfully deployed

1 active deployment
staging - docs — 98469693 Deployed Oct 15, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang memory-pool unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant