Repository navigation
[Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll - #44614
Merged
Merged
Conversation
…ked admission poll Under the `timeout` stop policy a storage prefetch was timed from the operation's own start, so every re-issued fetch for the same request began a fresh budget sized by the current, possibly shrunken, hit chain; a request could wait indefinitely across retries, and the per-request wait metric and the policy budget were timed by two separate mechanisms. A read cut by the policy was also reported as `storage_transfer`, which healed the storage-existence beliefs for pages the store still holds, and a rank that had not yet reached its admission poll could re-issue a prefetch its peers had already agreed to drop. - `PollTimer` (new `utils/poll_timer.py`) times an interval from its first blocked poll to its first ready poll. `SchedulerReqTimeStats` uses it for the storage prefetch and staging waits; the serialized fields and the `meta_info` keys are unchanged. - `UnifiedRadixCache` keeps a per-request `_StoragePrefetchWait` under the `timeout` policy: it starts at the first blocked admission poll, is sized from the span first requested (`prefetch_timeout_base + pages * prefetch_timeout_per_page`), survives re-issued fetches, and is closed by admission or abort. A complete KV read that only waits on a sidecar is never cut. - A rank-agreed expiry marks the operation `deadline_expired`, cancels its pending retries, and suppresses further fetches for the request on every rank, including one that had not yet started waiting. - A deadline-cut operation is resolved under the new `deadline` unfulfilled reason instead of `storage_transfer` and its storage beliefs are left untouched; `_invalidate_absent_from_hit_query` skips it because its agreed hit may be a MIN over skipped lookups. The operation accounting skips a deadline cut, which is not a storage verdict. - `PrefillAdder` closes storage-prefetch admission for a retracted request after discarding its accounting, so the wait budget is dropped with the rest of its per-request state. Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
metamergebot
requested review from
Ying1123,
alphabetc1,
fzyzcjy,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sufeng-buaa,
xiezhq-hermann and
yizhang2077
as code owners
October 16, 2026 00:35
Collaborator
Author
|
This port clears our bar on gate 20✓✗1 lint✓ build✓ A✓ B✓ other✓ · accel ✗amd,npu @fei-xx @merrymercy @xiezhq-hermann — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side. |
detain
pushed a commit
to detain/sglang
that referenced
this pull request
Oct 19, 2026
…ked admission poll (sgl-project#44614) Co-authored-by: metamergebot <metamergebot@users.noreply.github.com> Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com> (cherry picked from commit 453b251) (cherry picked from commit cf755bd21e8b11112ab407909867cc935f692169)
5 tasks done
xiezhq-hermann
added a commit
that referenced
this pull request
Oct 28, 2026
Resolves the rebase against #43344 (the PP ticket thread now runs the hit query itself, so the scheduler-thread reduce this branch guarded is gone and the ticket-thread failure hook moves to its unconditional exit), #44614 (the degrade cut reuses the deadline_expired machinery) and the shared-memory count exchange in _sync_hicache_ready_counts (the health bit is split out on that path too). Test fixtures gain the attributes main added.
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
Port of the upstream generalization of our entry-209 trailing-sidecar fix onto the landed stack. Union analysis per hunk (entry-209 = 8aab510 + 3fa5197, entry-217 = sgl-project#43108/sgl-project#44614 already occupy these regions): - allocate_storage_hit "pick the largest fitting verified endpoint BEFORE allocating, decline when none fits" -- REDUNDANT on our tree: our _trailing_endpoint_tokens (entry 209) already performs exactly that choice (plus the buffer-only sidecar_hash_values pin guard and the throttled _report_no_restorable_endpoint that upstream lacks). Kept ours; upstream's gate difference noted: their selection trigger is any TRAILING_PAGES transfer, ours additionally requires indices_from_pool is None (derived sidecars are covered through their source transfer). - _sync_prefetch_hit_query "retain the common endpoint set after the MIN reduction instead of collapsing to the deepest" -- REDUNDANT: our always-masked payload (1 + len(PoolName) + restorable_mask) already reduces and retains the intersected set. local_restorable_mask gained upstream's "no verified hit votes zero" invariant: a rank syncing with a folded count of 0 (terminated/skipped/failed query) can no longer vouch for a candidate set left by a partial earlier state. - assume_stored branch -- LANDED (semantics win over our blank): a trailing re-prefetch now reports [kv_hit_pages] (the original endpoint only, not shallower checkpoints) instead of None, so a capacity-shortened retry stops on the endpoint or declines, never on an unverified interior page. Non-trailing keeps None (contiguous contract). - unified_radix_cache aux staging -- LANDED (new here): _alloc_prefetch_aux_staging sizes to alloc_len (the endpoint the KV span got) instead of the full query hit; under mamba-host pressure the old full-span sizing failed the aux alloc and declined a prefetch a smaller endpoint would have fit. Tests: upstream's test_hybrid_storage_sparse_allocation.py (13 cases + 14 subtests) retargeted onto our field placement (operation.restorable_prefix_ pages), our packed collective layout, and page-0-always-zero mask (no sentinel); every allocation/query/drain expectation kept. The upstream PP companion test hunk (+42 in test_pp_prefetch_ticket.py) is NOT portable: this tree's PP broadcast path runs no mask collective (the ticket carries only the folded count, followers rebuild the operation without it), so its vote-shape assertions describe a payload our design does not have; flagged for the integrator. Our trailing_endpoint (17), storage_prefetch_lifecycle, buffer_mode_sidecar, pp_prefetch_ticket and unified_hicache suites all still pass. (cherry picked from commit a23ba83) PR: sgl-project#46954 (open, ported) (cherry picked from commit b6e7ed7cb232d6069ad2ac4d5be41c7bba606511)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
New --hicache-storage-prefetch-host-capacity-wait-ms (default 0.0, so the landed partial-staging behavior is byte-identical when unset). In single-engine CACHE mode a positive value makes a capacity-shortened full hit WAIT instead of declining: _try_alloc_storage_hit parks the operation in the new pending_storage_hits, retries each drain pass up to the bound (deadline + pacing from the same retry-max-attempts knob), and only then falls through to the existing host_capacity decline. allow_partial is inverted for those waits (full span or park), so the entry-209 endpoint-clamp machinery is untouched in the default mode and the buffer-only / TP / PP / CP paths keep their current behavior (enable_cache_mode_storage_hit_pending). Cancellation/expiry hardening riding along: - the hit-queue drain and _try_alloc_storage_hit now check info.operation is not operation, so a stale query callback from a revoked attempt can neither retire nor re-account a resubmitted op (this is the guard our entry-209 decline-and-retry loop needed: the retry re-issue and its late predecessor no longer collide); - an already-staged cache-mode op returns early instead of allocating a second bounce buffer. Interaction with sgl-project#44614's deadline anchor (entry 217, landed): the new wait deadline is per-operation (a fresh host_capacity_deadline on the first blocked alloc); on expiry the op takes the SAME _finish_storage_prefetch(host_capacity) + revoke + poll_miss path, whose retry budget stays anchored on the first blocked admission poll via PollTimer. The two counters (operation.host_capacity_retries vs StoragePrefetchRetries attempts) share the --hicache-storage-prefetch-retry-max-attempts cap. The timeout prefetch stop policy remains the earlier exit, as documented. This replaces the ad-hoc half of entry-209's host_capacity decline+requeue when opted in. Union notes: unified_radix_cache.py hunks applied over the landed entry-209/217/218/199 region (the poll_miss anchor lines inside the decline branch preserved); hybrid_cache_controller.py gains three PrefetchOperation fields only (no collision with the sparse-checkpoint regions); the lifecycle test hunk was anchored manually (upstream base carries a TestWaitingStorageSpan class this tree does not have, so the 202-line block was inserted at the same logical spot and both new imports retained; est_time stays 20 per our tree). Batteries: test_storage_prefetch_lifecycle 23P/28sub (was 15P/24sub; +8 all green), trailing_endpoint/buffer_mode_sidecar/sparse_allocation/urc/rust-urc 80P unchanged, server_args 430P/225sub, flag default+parse verified. (cherry picked from commit 9e61b77) PR: sgl-project#46451 (open, ported) (cherry picked from commit ff8bf1fe764856ad80ec95b1162251357ccba55d)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…217/293 cases the upstream rewrite dropped The class-A merge took UPSTREAM's replacement of this file (762 lines vs the fork's 990): green, but silently WITHOUT the fork's 12 cases — the entire capacity-wait/pending family (entry 293) and the timeout-budget anchors (entry 217/sgl-project#44614 + the metrics-episode persistence). DoD(c) names this file for at-or-above-295, and the cases exercise exactly the production families B3-3 planted, so they are re-homed here (B3 test semantics ONLY — the general URC test re-home stays B2's). Grafted from the 309 tip blob (git show 3cdba95:...), verbatim except: - _CapacityHostPool/_capacity_wait_fixture/_capacity_hit/_drain_capacity_hits helpers + class TestCacheModeCapacityWait (8 cases: expired-head double- submit, cancel head/middle, terminated-pending-no-transfer, cancelled-rid reuse, large-deferred-not-blocking-smaller, late-cancelled-attempt- identity-guard, deadline-no-spin-no-truncate, default-vs-distributed partial staging) -> ALL PASS against the B3-3 wait_full/pending_storage_ hits machinery UNMODIFIED (the re-home doubles as the machinery's proof). - TestStoragePrefetchDeadlineAnchor (4 cases: budget starts at first wait and survives refetch, timeout preserves complete-KV-with-pending-sidecar, stop policies distinguish deadline from storage failure, metric episodes do not restart the timeout) — the ONLY adaptation: the merged req_time_stats restructure renamed the scheduler export (convert_to_output_meta_info -> to_output_time_stats, Optional on metrics-off); the assertion's meaning (1.6 s episode sum / absent series) is unchanged. - CustomTestCase import added (the upstream file's classes use bare unittest.TestCase; the fork's used the retry wrapper). ACTUALS: test_storage_prefetch_lifecycle 16P/0F/13sub -> ** 28P/0F/31sub ** (295-era pin 23P/28sub EXCEEDED; every grafted case green first run against the planted B3-3 code). No GPU; no push.
detain
pushed a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
Review follow-up. Skipping every later collective after the first sync failure was unsound: gloo closes only the pair of the group that timed out, and the hit and completion sync groups are distinct even on TP2, so a peer that had not failed blocked for the untouched group's full timeout (30 min by default), during which reset/detach could not join and flush raised out of the scheduler. A retry on the broken group raises at once on every rank, so the collective is now always entered. Also: take_ready_pp_prefetch waits in 1 s slices and fails loud when the local ticket thread is dead; the buffer-only PP>1 health relay is limited to the ticket-less shape (with tickets PP0 decides per request); the sync knobs are validated before any attach side effect; the gauge drops on detach; a WARNING once a minute while degraded; thread attributes are initialised in __init__; reset logs that L3 stays disabled. The two-rank test now runs its three cases in one spawn through check_hicache_events, adds the cross-group case (an ack on the healthy group must reduce promptly on both ranks after the hit group failed) and takes ~26 s. Port adaptations (cherry-pick -x of 74a0e6b + the parts of the 60c5f3e main-sync resolution that are PR content, not main noise -- the merge commits themselves carry only upstream-main history already in this tree's merge sgl-project#5 base and are SKIPPED): - Rank-consistency preserved: the degraded cut now RIDES THE DEADLINE MACHINERY (the 60c5f3e resolution against sgl-project#44614, which this tree runs since entry 217): storage_unavailable -> _mark_prefetch_deadline_- expired -> the existing deadline_expired branch resolves the cut with NO poll_miss re-arm, honoring the entry-293 shared retry-cap rule with the timeout stop policy as the earlier, already-proven exit. - Fork stub convention on the new test's _scheduler_cache: pre-memoize _hcs_state=False (entry 294) and queued_backups={} (entry 199), plus the _storage_prefetch_wait fixture attribute the resolution itself records ("test fixtures gain the attributes main added"). - test_command_failure keeps the fork's ticket-thread shape: later submissions stay ticketed, the rank-agreed drop retires them. Gate at this commit: 88P/62sub green over trailing_endpoint, pp_prefetch_ticket, buffer_mode_sidecar, storage_prefetch_lifecycle, pp_sync_drain, async_ready, sync_recovery (2P/4sub), shm_tag_exchange, managers trio; import sglang proven; ruff+format clean. (cherry picked from commit 74a0e6b) (cherry picked from commit be1316edf74bd2139dc02c2294bc799af04205fd)
detain
pushed a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
… bounded ticket group - storage_workers_healthy() watches the worker threads the controller started; a controller whose storage I/O runs outside Python workers has none and reads healthy on their account instead of dead. - storage_sync_timeout defaults to 300 s instead of torch's 30-minute gloo default (null / absent -> default, non-finite rejected); a hung peer is noticed in minutes while staying far above any healthy batch call. - The PP ticket broadcast group is created when the storage workers start, after attach applied the timeout, so it is bounded like the other storage collectives (it used to be built before the config was parsed). - A prefetch cut by the rank-agreed degrade reports storage_unavailable in storage_prefetch_unfulfilled_tokens_total instead of borrowing deadline. Port of 6c37509 (the PR's third content commit; its two "Merge branch main" commits 60c5f3e/96850aef61 are pure upstream-main syncs -- all of their PR-relevant resolutions are already folded into this branch's commits dee1e2a872 (shm mins-path split, ticket-thread hook, fork-shape submission) and 8f479b20a9 (deadline-riding degrade cut, stub fixtures) and the merge commits themselves are NOT picked: they carry nothing but main history this tree already holds through MERGE sgl-project#5). Fork note: the new stop-reason rides the entry-217/sgl-project#44614 machinery -- storage_unavailable marks operation.deadline_expired, so the cut resolves through the existing deadline_expired branch with NO poll_miss re-arm (the entry-293 shared retry-max-attempts cap stays untouched; the timeout stop policy remains the earlier exit), and _prefetch_cut_reason routes that cut plus the hybrid discard and the token resolve to reason=storage_unavailable (gauge label registered additively). The fork-only parse_storage_backend_extra_config registration added in dee1e2a872 now mirrors the 300 s default semantics (absent -> default, non-finite rejected) without duplicating the constant. Gate at this commit: 90P/63sub green (trailing_endpoint 17P, pp_prefetch_ticket 17P/19sub, buffer_mode_sidecar 6P/6sub, storage_prefetch_lifecycle 28P/31sub, pp_sync_drain 4P, async_ready 7P, sync_recovery 4P/5sub, shm_tag_exchange, managers trio); import sglang proven; ruff+format clean. (cherry picked from commit 6c37509) (cherry picked from commit 5d1e36812a5fc252941d907ac815ddcee9e006cc)
3 of 5 tasks
5 tasks
detain
added a commit
to detain/sglang
that referenced
this pull request
Nov 11, 2026
…r/URC markers -- 3/18 census consumed, 3 gates in one CASCADE (start_writing(wait_for_forward) TypeError class, 68 distinct nodes): upstream sgl-project#48012's NEW fixture hicache_write_back_fixtures.InstantWriteBackController mirrored the pre-sgl-project#44800 no-kwarg surface while the fork's writing_check drain (entries 199/218: the batched-backup flush=False + ONE fenced start_writing) calls the real API. Def-site fixes, no fence weakened: - fixture start_writing gains the keyword (instant copies -> accepted-ignored, documented); - test_write_back_eviction_batching: record_submission wrapper FORWARDS the flag (it measures the very wait, so swallowing it would understate the fence); _FencedController gains load_fence_stream = forward (the fork's unconditional entry-168 fence reads it first; prod: both streams are the same forward stream) -- also silences the 16-test Rust TreeCore panic class whose "victims not demoted" assert was the aborted submission downstream. MARKERS (3): managers/cache_controller.py:1202 ADOPTED -- the dropped set_hicache_transfer_done_event write line is superseded by upstream c24285b(sgl-project#48008 per-band page ledger)+2cbb66f563(sgl-project#48012): record_host_transfer publishes the same signal per-page BOTH directions (load twin writes=True). unified_radix_cache.py:627 ADOPTED -- the buffer-mode threshold floor (resolve_prefetch_threshold) was DELIBERATELY deleted upstream by dfaca56(sgl-project#46934, restores sub-window SWA L3 hits); window sizing now lives per-allocation in BufferModePipeline; the per-call-site file_storage_path fold is consistent with the entry-253 funnel (cache_controller.py:948, intact). unified_radix_cache.py:3917 RE-PLANTED -- entry-293's cadence gate (park, do not re-enter allocate_storage_hit/evict_host inside the retry window) + the expired no-retry arm were dropped; wait_full folded into upstream's comp_xfers allow_partial rule (min_tokens form already equivalent). URC context guard: _deferred_ready_sync_group read get_memory() unguarded -- upstream's NEW _arm_tick CPU/spawn fixtures (their restructured timeout tests, sgl-project#44614) reach it without a published server config. Now published-check (schedule_policy.py:615 idiom): default-off CPU path, production unchanged. Deadline-anchor retargets (fork B3-9 family vs upstream's sgl-project#44614 vote-before- poll restructure; every assertion preserved): _vote_prefetch_deadlines drives the verdict before the expiry polls; failed-arm polled inside-budget so the storage failure stays the SOLE cut reason; fixture cc gains the sgl-project#44342 liveness attrs + pp_prefetch_command_thread; _arm_tick stub returns the fork's 5-tuple (health bit True); release_aborted_request -> finish(ABORT) per 8898429(sgl-project#48491). Battery actuals (CPU, CVD=9, prod venv, PYTHONPATH=wt/python): - the 5 cascade files 68F/39P -> 3F/70P (survivors = the sgl-project#48491/pp-thread/ budget-anchor fixes committed here; after them): test_storage_prefetch_ lifecycle ** 31P/31sub ** (entry-311 pin 28P/31sub, 295's >=23P/28sub EXCEEDED); decode_retraction_backup 6F + chunked_publish 2F = No-CUDA device class (env, reported). - combined gate run: trailing_endpoint + batching + parity + async_ready_sync + staged_write_back = 89P/1S/28sub 0F. Gates: ruff repo ruleset+format clean on touched files; registration trio rc=0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Under the
timeoutstop policy a storage prefetch is timed from the operation's own start. Every re-issued fetch for the same request therefore begins a fresh budget, sized by the current (possibly shrunken) hit chain, so a request can wait indefinitely across retries, and the per-request wait metric (storage_prefetch_wait) and the policy budget are timed by two unrelated mechanisms. Two further problems follow from how a cut read is handled: it is reported asstorage_transfer, which heals the storage-existence beliefs for pages the store still holds, and a rank that has not yet reached its admission poll can re-issue a prefetch its peers already agreed to drop.This PR also carries the deadline-cut attribution guards (the
deadline_expiredflag onPrefetchOperation, thedeadlineunfulfilled reason, the belief-healing skips, and the retracted-request admission close), which were previously proposed on their own in #44338 (closed); the timeout budget depends on them.Modifications
PollTimer(newpython/sglang/srt/utils/poll_timer.py) times an interval from its first blocked poll to its first ready poll.SchedulerReqTimeStatsuses it for the storage prefetch and staging waits in place of two hand-rolled start timestamps; the serialized fields and themeta_infokeys are unchanged.UnifiedRadixCachekeeps a per-request_StoragePrefetchWaitunder thetimeoutpolicy. It starts at the first blocked admission poll, is sized from the span first requested (prefetch_timeout_base + pages * prefetch_timeout_per_page), survives re-issued fetches, and is closed byfinish_storage_prefetch_admissionorrelease_aborted_request(and byStorageAttachmenton reset). A complete KV read that only waits on a sidecar is never cut._can_terminate_prefetchconsensus, now evaluating the shared budget on PP rank 0) marks the operationdeadline_expired, cancels its pending retries, and suppresses further fetches for the request on every rank, including a rank that had not yet started its own budget.deadlineunfulfilled reason instead ofstorage_transfer, its KV and auxiliary-pool storage beliefs are left untouched,_invalidate_absent_from_hit_queryskips it (its agreed hit may be a MIN over skipped lookups), and the prefetch-outcome counters skip it since a deadline is not a storage verdict.PrefillAddercloses storage-prefetch admission for a retracted request after discarding its accounting, so the wait budget is dropped with the rest of its per-request state.test_prefill_adder.pyasserts the call.TestStoragePrefetchTimeoutintest_storage_prefetch_lifecycle.pycovers budget start and survival across refetch, metric episodes not restarting the budget, the complete-KV-with-pending-sidecar guard, the stop-policy matrix, and a two-rank gloo case where only one rank has started waiting. ExistingSimpleNamespaceoperation fixtures gaindeadline_expired=False.Accuracy Tests
Not applicable: no change to model outputs.
Speed Tests and Profiling
Not applicable: the change only affects when a storage prefetch under the
timeoutpolicy is cut and how the cut is attributed.Unit tests run locally on this head (CPU,
pytest):test_storage_prefetch_lifecycle.py,test_hicache_pp_sync_drain.py,test_pp_prefetch_ticket.py,test_prefill_adder.py,test_req_time_stats.py,test_buffer_mode_sidecar.py,test_hybrid_pool_assembler.py,test_prefill_result_polling.py,test_admission_abort_not_enqueued.py,test_decode_hicache_tree_core.py,test_prefill_complete_integration.py,test_priority_scheduling_disaggregation.py,test_scheduler_chunked_abort_race.py,test_tokenizer_manager_rid_cleanup.py,test_scheduler_held_row_admission.py,test_scheduler_hicache_events.py: 309 passed, 277 subtests passed.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): 🚫 Run #38830377935
Latest PR Test (Extra): ❌ Run #38830378070
Latest PR Test (AMD ROCm 10): ❌ Run #38830377877