Repository navigation
[Fix] hicache: Flush the cache without dying when a storage worker cannot be stopped - #44348
Draft
xiezhq-hermann wants to merge 6 commits into
Draft
xiezhq-hermann wants to merge 6 commits into
xiezhq-hermann wants to merge 6 commits into
Conversation
…liveness A gloo timeout inside the HiCache storage worker collectives (the query thread's hit MIN, the sync thread's ack MIN) killed the worker on every rank, so one rank stuck in its backend left every rank unable to publish L3 queries or acks: under wait_complete every later L3-hit request waited forever, under timeout every one paid the full deadline. #43550 guards the backend call only, not the collective, and nothing checked worker liveness. Wrap every worker-thread collective: on failure log once, flag the controller (storage_sync_failed), publish the operation as FAILED (0 hits, 0 completed tokens) and keep the thread alive; after the first failure the broken groups are skipped so the per-rank publication sequence stays intact for the queue-count drain. Fold a rank-local health bit (no sync failure and all storage workers alive) into the MIN-reduce the scheduler drain already issues; when the agreed bit drops, apply storage_failure_policy: "disable" (default) degrades to no-L3 until the backend is re-attached (no new prefetch/backup, in-flight prefetches cut like a deadline, drained hits read as misses, progress frozen, gauge sglang:hicache_storage_healthy -> 0), "raise" fails the scheduler. A new storage_sync_timeout extra-config key bounds the sync groups' gloo timeout so a hung peer is detected before torch's 30-minute default.
Review follow-up. Skipping every later collective after the first sync failure was unsound: gloo closes only the pair of the group that timed out, and the hit and completion sync groups are distinct even on TP2, so a peer that had not failed blocked for the untouched group's full timeout (30 min by default), during which reset/detach could not join and flush raised out of the scheduler. A retry on the broken group raises at once on every rank, so the collective is now always entered. Also: take_ready_pp_prefetch waits in 1 s slices and fails loud when the local ticket thread is dead; the buffer-only PP>1 health relay is limited to the ticket-less shape (with tickets PP0 decides per request); the sync knobs are validated before any attach side effect; the gauge drops on detach; a WARNING once a minute while degraded; thread attributes are initialised in __init__; reset logs that L3 stays disabled. The two-rank test now runs its three cases in one spawn through check_hicache_events, adds the cross-group case (an ack on the healthy group must reduce promptly on both ranks after the hit group failed) and takes ~26 s.
…ve when a worker wedges
One storage backend call that never returns on one rank wedges HiCache
prefetch/backup on every rank: the peers block in the MIN collectives
behind it and every later operation queues behind the stuck one, so the
engine keeps running with L3 permanently gone (timeout policy) or with
every L3-hit request stalled (wait_complete).
Run every backend call (query / get / set) on a per-kind worker with a
deadline (storage_{query,get,set}_timeout_s in
--hicache-storage-backend-extra-config, defaults 5 / 20 / 60 s, 0
disables). A timed-out or raising call is published as the controller's
own miss shape, so acks and hit counts stay count-stable across ranks;
the call keeps running on an orphaned worker and the host slots it may
still touch stay quarantined (per-slot pin counts) in their pool until
it returns, so a late zero-copy read or write can never land in memory
that was handed to another operation. Timeouts, errors and the orphan
count are exported as metrics. The flush/reset hardening for a worker
that cannot be joined is a separate PR.
…nnot be stopped /flush_cache is the operator's remediation for a wedged HiCache storage pipeline and today it kills the scheduler: cache_controller.reset() joins the storage workers for 10 s and raises, and the exception escapes Scheduler.flush_cache on every rank. A worker can still be unjoinable with the call deadlines of the previous change (bound disabled, stuck in a rank-sync collective whose peer is gone, backend ignoring deadlines), and restarting new workers over a live one is unsafe: it still owns its operation's host rows and would publish stale acks into the live queues when it returns. reset() now joins the workers and leaves a stuck one on a dead worker generation: generation checks keep it from consuming the live queues, publishing into them, releasing host pages or entering a collective once it returns, and the host spans of the operation it is inside stay quarantined until then. Fresh workers start on every rank, the wedged rank flags storage_workers_healthy() so the MIN-reduced health bit degrades L3 on every rank together (enable_storage is never flipped rank-locally), the flush completes and reports the stuck worker through FlushCacheReqOutput.message, the PD role switch continues, and detach refuses while the abandoned worker still lives. The unified page-envelope host pool no longer waits for abandoned leases in compaction and clear(); the pages such a call may still write stay in place and quarantined. The two controller extra-config pops are merged. Stacked on #44342 and #44343.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
… lane The one-caller-per-kind rule missed a second QUERY caller: the PD-decode scheduler thread probes L3 in query_storage_hit_length while the prefetch worker's query for the previous request may be in flight, so under --enable-decode-hicache the enforcement raised out of the scheduler loop. Scheduler-thread probes (that probe and PP0's ticket submission) now run on a PROBE lane with the query deadline, and the probe turns a guard error into a miss. The lane is freed before an orphan's spans are pinned so a pin failure cannot leave a kind busy forever, the deadline keys are validated before attach has any side effect, and the docs state the unified page-envelope flush outcome as it is.
…ses by token flush_cache returns bool again and reports a storage worker it could not stop by raising HiCacheStorageWedgedError once the pools are flushed; the flush wrapper turns it into the "Cache flushed, but ..." response and the weight updater logs it, so the registered flush-cache tests pass unchanged. Abandoned layout leases on the unified page-envelope pool are per-acquire tokens rather than thread idents, which are recycled once the abandoned thread exits. A returning abandoned worker cannot clear the live worker's in-flight slot, issues no further backend call, and the PP ticket thread re-checks its generation after the broadcast. The two-rank test drives the production drain; message and docs cover the raise policy, a second flush and a worker that never returns.
Draft
4 of 5 tasks
5 tasks done
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
/flush_cacheis the operator's remediation for a wedged HiCache storage pipeline, and today it is the one thing that is guaranteed to kill the server:UnifiedRadixCache.reset()->cache_controller.reset()joins the storage worker threads for 10 s and raisesFailed to stop HiCache storage threads cleanly, which escapesScheduler.flush_cacheand exits the scheduler on every rank (reproduced 9/9 on a TP2 lane with one hung backend call, both on main and with #43550). #44343 bounds the backend calls so a hung call no longer wedges a worker, but a worker can still be unjoinable when the bound is disabled, when it is stuck in a rank-sync collective whose peer is gone, or when a backend ignores its deadline. Restarting new workers over a live one is not safe either: the stuck worker still owns the host rows of its operation, and when it finally returns it would publish a stale ack into the live queues (one extra ack on one rank misaligns the peers' MIN sequence) and release pages the flush already handed out.Modifications
reset()never raises; a stuck worker is abandoned, not restarted over.HiCacheController.reset()joins the workers (10 s each) and leaves the ones still alive on a dead generation: every worker loop and every publication site checks the generation it was started in, so an abandoned worker neither consumes from nor publishes into the live queues, drops its host releases, and issues no collective once it returns (_enter_storage_worker/_storage_worker_is_live/_publish). The operations the transfer and backup workers were inside (_inflight_prefetch_op/_inflight_backup_op) get their host spans quarantined through [Fix] hicache: Bound storage backend calls and keep the scheduler alive when a worker wedges #44343'sHostSlotQuarantine; the stuck worker releases them in itsfinallywhen it returns. Fresh workers start on every rank, so no peer is left waiting on a rank without workers, andreset()returns the degradation message.UnifiedRadixCache._reset_fullfinishes the tree reset (the host pool is cleared throughclear_host_pool(), after the quarantine), then raisesHiCacheStorageWedgedError(defined inhicache_storage.py);Scheduler.flush_cachecatches it, still flushes the device pools, and re-raises it at the end instead of dying, keeping itsboolreturn for the ordinary outcomes.SchedulerFlushWrapperturns it intoFlushCacheReqOutput(success=False, message="Cache flushed, but HiCache storage worker(s) [...] did not stop within 10s ..."), the weight updater logs it and lets the update stand (the cache is flushed), and the PD role switch catches it and continues the flip with L3 off. An abandoned worker that returns cannot clear the live worker's in-flight slot (compare-and-clear), issues no further backend call for the rest of its operation (so it never takes a guard lane the live worker needs), and the PP ticket thread re-checks its generation right afterbroadcast_pyobjand skipstask_done()on the queue a reset replaced.The degradation is rank-agreed. The wedged rank sets
storage_wedged_reason, whichstorage_workers_healthy()([Fix] hicache: Recover storage worker collectives and expose storage liveness #44342) folds into the MIN-reduced health bit; at the next drain every rank flipsstorage_unavailabletogether understorage_failure_policy=disable(or fails loudly underraise).enable_storageis never touched rank-locally, so_sync_hicache_ready_countskeeps one tensor length on every rank.detach_storage_backendrefuses while an abandoned worker is still alive (its sync groups and backend are in use) with a message naming the thread; detach and re-attach is the recovery once it has returned.Unified page-envelope (mori) host pool. An orphaned or abandoned call holds a layout lease. Leases are per-acquire tokens (
_layout_holders[token] = thread ident); before the clear,clear_host_pool()converts the controller's orphaned and wedged thread idents into the tokens they hold at that moment (abandon_leases_of), so a thread ident recycled after the abandoned thread exits can never exempt a later lease. Compaction andclear()do not wait for abandoned tokens; the pages those calls may still write are immovable for compaction and surviveclear()in place, unallocatable until their quarantine is released. After a flush this also removes the scheduler-thread wait on that call in the allocation-path compaction; an orphan created after the last flush is still waited for there, as on main.The two controller extra-config pops ([Fix] hicache: Recover storage worker collectives and expose storage liveness #44342's sync knobs, [Fix] hicache: Bound storage backend calls and keep the scheduler alive when a worker wedges #44343's call deadlines) are merged into
_apply_storage_extra_config, validated before any side effect.Operator story (message and
hicache_best_practices.mdx): understorage_failure_policy=disableL3 is degraded on every rank until the backend is detached and re-attached, which needs the abandoned worker to return (detach is refused while it is alive; if it never returns, the recovery is a restart). A later/flush_cachereturns 200 but L3 stays disabled until the re-attach. Understorage_failure_policy=raisethe flush is followed by the scheduler failure at the next HiCache drain on every rank, as that policy asks.Accuracy Tests
Not applicable: no change to model forward or to what a successful transfer writes.
Speed Tests and Profiling
No hot-path change: one thread-local read per worker publication; the generation check runs only on the worker threads. Not benchmarked at scale.
Unit tests (CPU,
test/registered/unit/mem_cache/test_hicache_flush_wedged_storage.py, 6 cases, ~20 s): a flush with the bound disabled and a worker stuck in the backend returns the reason instead of raising, restarts a new generation, keeps the stuck operation's host memory quarantined across the pool clear, refuses detach while the worker lives, serves new backups meanwhile, the returning worker publishes nothing into the live queues and releases its quarantine, a reset with nothing in flight stays healthy and a later clean reset does not lift the degrade; an abandoned transfer worker's late acks and host releases are dropped; a two-rank gloo run where only rank 1 wedges at reset ends withstorage_unavailableon both ranks after one productioncheck_hicache_eventsround andenable_storageuntouched; the unified page-envelope backing'sclear()does not wait for an abandoned lease holder and keeps the pinned page mapped in place; compaction leaves a pinned page where it is; abandonment marks the leases held at marking time, so a lease taken afterwards under the same (recycled) thread ident is waited for and_begin_compactionstill refuses.test/registered/unit/managers/test_scheduler_flush_cache.py(8, one new: the wedge message reachesFlushCacheReqOutput) andtest_scheduler_flush_cache_after_retract.py(7, one new: the pools are cleared before the error surfaces) pass withflush_cachereturningbool. Existing suites on this tree: test_storage_call_guard (14), test_hicache_storage_sync_recovery (the knob test also checks the deadline keys), test_pp_prefetch_ticket, test_hybrid_pool_assembler, test_buffer_mode_sidecar, test_hicache_pp_sync_drain, test_storage_prefetch_lifecycle, test_hicache_staged_write_back_dispatch, test_mem_pool_host, test_hicache_host_register, test_unified_hicache_transfer_lifecycle, test_unified_hicache_regressions, test_hicache_file_lru_unit, test_hicache_storage_format_tag, test_dsv4_hicache_l2, test_decode_hicache_tree_core, test_decode_queue_cleanup (205 passed / 3 skipped); test_unified_radix_cache_unittest-k "prefetch or storage or backup"(201 passed / 220 skipped) and test_mamba_unittest (12) on one GPU.GPU fault injection (TP2, GPUs 0,1 of a B200 box, Qwen2.5-1.5B, buffer_only, write_through, file backend wrapped by a fault-injecting
dynamicbackend; pack-1 protocol,/flush_cacheat the end, then three requests and/metrics):/flush_cachebatch_gethang on rank 1, GET bound disabled (storage_get_timeout_s: 0) -- the wedgereset()must surviveCache flushed, but HiCache storage worker(s) ['Thread-8 (prefetch_sync_thread_func)'] did not stop within 10s and were abandoned with their host memory quarantined; L3 is degraded on every rank until the storage backend is re-attached, which needs the abandoned worker(s) to return.(TP1 namesThread-7 (prefetch_io_aux_func))/health200, both schedulers alive; three/generaterequests (600 / 600 / 40 words) answered with HTTP 200 in 0.03 s each; both ranks log the unhealthy pipeline in the same drain andsglang:hicache_storage_healthy= 0 on both; py-spy: the abandonedThread-7 (prefetch_io_aux_func)is still parked in the hung read next to the fresh generation on TP1. On main (and with #43550) this same flush exits both schedulers.batch_gethang on rank 1, default boundshicache_storage_healthy= 1 on both ranks (an orphaned call is not a wedge)batch_gethang on rank 1, default boundsbatch_existsraises on rank 0, default boundsafter=60arm landed between req #1 and wave 1 in this run)batch_existsraises on rank 0, default boundsEvery row: no
Exception in thread, noFailed to stop HiCache storage threadsin the server log; the fault log shows exactly one injected fault per run. The rows were measured on the first commit of this PR; the wedge row was re-run on the final head (revision 2) with the same outcome:/flush_cacheHTTP 400Cache flushed, but HiCache storage worker(s) ['Thread-8 (prefetch_sync_thread_func)'] did not stop within 10s and were abandoned with their host memory quarantined; L3 is degraded on every rank until the storage backend is re-attached, which needs the abandoned worker(s) to return (restart the server if they never do).,/health200, three post-flush requests HTTP 200 in 0.03 s,sglang:hicache_storage_healthy0 on both ranks, py-spy showing the abandonedThread-7 (prefetch_io_aux_func)still inside the hung read next to the new generation'sThread-11.Not run: real remote backends, PP > 1, the mori pool end to end, the
raisefailure policy end to end.Checklist