Skip to content

[Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll - #44614

Merged
cctry merged 1 commit into
mainfrom
oss-sync-634f306f15
Oct 16, 2026
Merged

cctry merged 1 commit into
mainfrom
oss-sync-634f306f15

Conversation

@metamergebot

@metamergebot metamergebot commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Under the timeout stop policy a storage prefetch is timed from the operation's own start. Every re-issued fetch for the same request therefore begins a fresh budget, sized by the current (possibly shrunken) hit chain, so a request can wait indefinitely across retries, and the per-request wait metric (storage_prefetch_wait) and the policy budget are timed by two unrelated mechanisms. Two further problems follow from how a cut read is handled: it is reported as storage_transfer, which heals the storage-existence beliefs for pages the store still holds, and a rank that has not yet reached its admission poll can re-issue a prefetch its peers already agreed to drop.

This PR also carries the deadline-cut attribution guards (the deadline_expired flag on PrefetchOperation, the deadline unfulfilled reason, the belief-healing skips, and the retracted-request admission close), which were previously proposed on their own in #44338 (closed); the timeout budget depends on them.

Modifications

  • PollTimer (new python/sglang/srt/utils/poll_timer.py) times an interval from its first blocked poll to its first ready poll. SchedulerReqTimeStats uses it for the storage prefetch and staging waits in place of two hand-rolled start timestamps; the serialized fields and the meta_info keys are unchanged.
  • UnifiedRadixCache keeps a per-request _StoragePrefetchWait under the timeout policy. It starts at the first blocked admission poll, is sized from the span first requested (prefetch_timeout_base + pages * prefetch_timeout_per_page), survives re-issued fetches, and is closed by finish_storage_prefetch_admission or release_aborted_request (and by StorageAttachment on reset). A complete KV read that only waits on a sidecar is never cut.
  • A rank-agreed expiry (the existing _can_terminate_prefetch consensus, now evaluating the shared budget on PP rank 0) marks the operation deadline_expired, cancels its pending retries, and suppresses further fetches for the request on every rank, including a rank that had not yet started its own budget.
  • A deadline-cut operation is resolved under the new deadline unfulfilled reason instead of storage_transfer, its KV and auxiliary-pool storage beliefs are left untouched, _invalidate_absent_from_hit_query skips it (its agreed hit may be a MIN over skipped lookups), and the prefetch-outcome counters skip it since a deadline is not a storage verdict.
  • PrefillAdder closes storage-prefetch admission for a retracted request after discarding its accounting, so the wait budget is dropped with the rest of its per-request state. test_prefill_adder.py asserts the call.
  • Tests: TestStoragePrefetchTimeout in test_storage_prefetch_lifecycle.py covers budget start and survival across refetch, metric episodes not restarting the budget, the complete-KV-with-pending-sidecar guard, the stop-policy matrix, and a two-rank gloo case where only one rank has started waiting. Existing SimpleNamespace operation fixtures gain deadline_expired=False.

Accuracy Tests

Not applicable: no change to model outputs.

Speed Tests and Profiling

Not applicable: the change only affects when a storage prefetch under the timeout policy is cut and how the cut is attributed.

Unit tests run locally on this head (CPU, pytest): test_storage_prefetch_lifecycle.py, test_hicache_pp_sync_drain.py, test_pp_prefetch_ticket.py, test_prefill_adder.py, test_req_time_stats.py, test_buffer_mode_sidecar.py, test_hybrid_pool_assembler.py, test_prefill_result_polling.py, test_admission_abort_not_enqueued.py, test_decode_hicache_tree_core.py, test_prefill_complete_integration.py, test_priority_scheduling_disaggregation.py, test_scheduler_chunked_abort_race.py, test_tokenizer_manager_rid_cleanup.py, test_scheduler_held_row_admission.py, test_scheduler_hicache_events.py: 309 passed, 277 subtests passed.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #38830377935
Latest PR Test (Extra): ❌ Run #38830378070
Latest PR Test (AMD ROCm 10): ❌ Run #38830377877

…ked admission poll

Under the `timeout` stop policy a storage prefetch was timed from the
operation's own start, so every re-issued fetch for the same request began
a fresh budget sized by the current, possibly shrunken, hit chain; a request
could wait indefinitely across retries, and the per-request wait metric and
the policy budget were timed by two separate mechanisms. A read cut by the
policy was also reported as `storage_transfer`, which healed the
storage-existence beliefs for pages the store still holds, and a rank that
had not yet reached its admission poll could re-issue a prefetch its peers
had already agreed to drop.

- `PollTimer` (new `utils/poll_timer.py`) times an interval from its first
  blocked poll to its first ready poll. `SchedulerReqTimeStats` uses it for
  the storage prefetch and staging waits; the serialized fields and the
  `meta_info` keys are unchanged.
- `UnifiedRadixCache` keeps a per-request `_StoragePrefetchWait` under the
  `timeout` policy: it starts at the first blocked admission poll, is sized
  from the span first requested (`prefetch_timeout_base + pages *
  prefetch_timeout_per_page`), survives re-issued fetches, and is closed by
  admission or abort. A complete KV read that only waits on a sidecar is
  never cut.
- A rank-agreed expiry marks the operation `deadline_expired`, cancels its
  pending retries, and suppresses further fetches for the request on every
  rank, including one that had not yet started waiting.
- A deadline-cut operation is resolved under the new `deadline` unfulfilled
  reason instead of `storage_transfer` and its storage beliefs are left
  untouched; `_invalidate_absent_from_hit_query` skips it because its agreed
  hit may be a MIN over skipped lookups. The operation accounting skips a
  deadline cut, which is not a storage verdict.
- `PrefillAdder` closes storage-prefetch admission for a retracted request
  after discarding its accounting, so the wait budget is dropped with the
  rest of its per-request state.

Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang unified-radix-cache labels Oct 16, 2026
@metamergebot metamergebot added run-ci CI: run the baseline test suite on this PR bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) labels Oct 16, 2026
@metamergebot

Copy link
Copy Markdown
Collaborator Author

This port clears our bar on 99b7852230be98a9595a12881a2be3fd931a4519: every base-a and base-b job is green.
base-c and the accelerator suites are reported but not gated, because they run
on scarce hardware and upstream merges through them routinely.

gate 20✓✗1 lint✓ build✓ A✓ B✓ other✓ · accel ✗amd,npu

@fei-xx @merrymercy @xiezhq-hermann — could one of you take a look and merge it if you are happy? Nothing else is waiting on our side.

@cctry
cctry merged commit 453b251 into main Oct 16, 2026
776 of 872 checks passed
@cctry
cctry deleted the oss-sync-634f306f15 branch October 16, 2026 17:37
detain pushed a commit to detain/sglang that referenced this pull request Oct 19, 2026
…ked admission poll (sgl-project#44614)

Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
(cherry picked from commit 453b251)
(cherry picked from commit cf755bd21e8b11112ab407909867cc935f692169)
xiezhq-hermann added a commit that referenced this pull request Oct 28, 2026
Resolves the rebase against #43344 (the PP ticket thread now runs the hit
query itself, so the scheduler-thread reduce this branch guarded is gone and
the ticket-thread failure hook moves to its unconditional exit), #44614 (the
degrade cut reuses the deadline_expired machinery) and the shared-memory
count exchange in _sync_hicache_ready_counts (the health bit is split out on
that path too). Test fixtures gain the attributes main added.
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
Port of the upstream generalization of our entry-209 trailing-sidecar fix
onto the landed stack. Union analysis per hunk (entry-209 = 8aab510 +
3fa5197, entry-217 = sgl-project#43108/sgl-project#44614 already occupy these regions):

- allocate_storage_hit "pick the largest fitting verified endpoint BEFORE
  allocating, decline when none fits" -- REDUNDANT on our tree: our
  _trailing_endpoint_tokens (entry 209) already performs exactly that
  choice (plus the buffer-only sidecar_hash_values pin guard and the
  throttled _report_no_restorable_endpoint that upstream lacks). Kept
  ours; upstream's gate difference noted: their selection trigger is any
  TRAILING_PAGES transfer, ours additionally requires indices_from_pool is
  None (derived sidecars are covered through their source transfer).
- _sync_prefetch_hit_query "retain the common endpoint set after the MIN
  reduction instead of collapsing to the deepest" -- REDUNDANT: our
  always-masked payload (1 + len(PoolName) + restorable_mask) already
  reduces and retains the intersected set. local_restorable_mask gained
  upstream's "no verified hit votes zero" invariant: a rank syncing with a
  folded count of 0 (terminated/skipped/failed query) can no longer vouch
  for a candidate set left by a partial earlier state.
- assume_stored branch -- LANDED (semantics win over our blank): a trailing
  re-prefetch now reports [kv_hit_pages] (the original endpoint only, not
  shallower checkpoints) instead of None, so a capacity-shortened retry
  stops on the endpoint or declines, never on an unverified interior page.
  Non-trailing keeps None (contiguous contract).
- unified_radix_cache aux staging -- LANDED (new here):
  _alloc_prefetch_aux_staging sizes to alloc_len (the endpoint the KV span
  got) instead of the full query hit; under mamba-host pressure the old
  full-span sizing failed the aux alloc and declined a prefetch a smaller
  endpoint would have fit.

Tests: upstream's test_hybrid_storage_sparse_allocation.py (13 cases + 14
subtests) retargeted onto our field placement (operation.restorable_prefix_
pages), our packed collective layout, and page-0-always-zero mask (no
sentinel); every allocation/query/drain expectation kept. The upstream PP
companion test hunk (+42 in test_pp_prefetch_ticket.py) is NOT portable:
this tree's PP broadcast path runs no mask collective (the ticket carries
only the folded count, followers rebuild the operation without it), so its
vote-shape assertions describe a payload our design does not have; flagged
for the integrator. Our trailing_endpoint (17), storage_prefetch_lifecycle,
buffer_mode_sidecar, pp_prefetch_ticket and unified_hicache suites all
still pass.

(cherry picked from commit a23ba83)
PR: sgl-project#46954 (open, ported)
(cherry picked from commit b6e7ed7cb232d6069ad2ac4d5be41c7bba606511)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
New --hicache-storage-prefetch-host-capacity-wait-ms (default 0.0, so the
landed partial-staging behavior is byte-identical when unset). In
single-engine CACHE mode a positive value makes a capacity-shortened full
hit WAIT instead of declining: _try_alloc_storage_hit parks the operation
in the new pending_storage_hits, retries each drain pass up to the bound
(deadline + pacing from the same retry-max-attempts knob), and only then
falls through to the existing host_capacity decline. allow_partial is
inverted for those waits (full span or park), so the entry-209
endpoint-clamp machinery is untouched in the default mode and the
buffer-only / TP / PP / CP paths keep their current behavior
(enable_cache_mode_storage_hit_pending).

Cancellation/expiry hardening riding along:
- the hit-queue drain and _try_alloc_storage_hit now check
  info.operation is not operation, so a stale query callback from a
  revoked attempt can neither retire nor re-account a resubmitted op
  (this is the guard our entry-209 decline-and-retry loop needed: the
  retry re-issue and its late predecessor no longer collide);
- an already-staged cache-mode op returns early instead of allocating a
  second bounce buffer.

Interaction with sgl-project#44614's deadline anchor (entry 217, landed): the new
wait deadline is per-operation (a fresh host_capacity_deadline on the
first blocked alloc); on expiry the op takes the SAME
_finish_storage_prefetch(host_capacity) + revoke + poll_miss path, whose
retry budget stays anchored on the first blocked admission poll via
PollTimer. The two counters (operation.host_capacity_retries vs
StoragePrefetchRetries attempts) share the
--hicache-storage-prefetch-retry-max-attempts cap. The timeout prefetch
stop policy remains the earlier exit, as documented. This replaces the
ad-hoc half of entry-209's host_capacity decline+requeue when opted in.

Union notes: unified_radix_cache.py hunks applied over the landed
entry-209/217/218/199 region (the poll_miss anchor lines inside the
decline branch preserved); hybrid_cache_controller.py gains three
PrefetchOperation fields only (no collision with the sparse-checkpoint
regions); the lifecycle test hunk was anchored manually (upstream base
carries a TestWaitingStorageSpan class this tree does not have, so the
202-line block was inserted at the same logical spot and both new imports
retained; est_time stays 20 per our tree). Batteries:
test_storage_prefetch_lifecycle 23P/28sub (was 15P/24sub; +8 all green),
trailing_endpoint/buffer_mode_sidecar/sparse_allocation/urc/rust-urc 80P
unchanged, server_args 430P/225sub, flag default+parse verified.

(cherry picked from commit 9e61b77)
PR: sgl-project#46451 (open, ported)
(cherry picked from commit ff8bf1fe764856ad80ec95b1162251357ccba55d)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…217/293 cases the upstream rewrite dropped

The class-A merge took UPSTREAM's replacement of this file (762 lines vs the
fork's 990): green, but silently WITHOUT the fork's 12 cases — the entire
capacity-wait/pending family (entry 293) and the timeout-budget anchors
(entry 217/sgl-project#44614 + the metrics-episode persistence). DoD(c) names this file
for at-or-above-295, and the cases exercise exactly the production families
B3-3 planted, so they are re-homed here (B3 test semantics ONLY — the
general URC test re-home stays B2's).

Grafted from the 309 tip blob (git show 3cdba95:...), verbatim except:
- _CapacityHostPool/_capacity_wait_fixture/_capacity_hit/_drain_capacity_hits
  helpers + class TestCacheModeCapacityWait (8 cases: expired-head double-
  submit, cancel head/middle, terminated-pending-no-transfer, cancelled-rid
  reuse, large-deferred-not-blocking-smaller, late-cancelled-attempt-
  identity-guard, deadline-no-spin-no-truncate, default-vs-distributed
  partial staging) -> ALL PASS against the B3-3 wait_full/pending_storage_
  hits machinery UNMODIFIED (the re-home doubles as the machinery's proof).
- TestStoragePrefetchDeadlineAnchor (4 cases: budget starts at first wait
  and survives refetch, timeout preserves complete-KV-with-pending-sidecar,
  stop policies distinguish deadline from storage failure, metric episodes
  do not restart the timeout) — the ONLY adaptation: the merged
  req_time_stats restructure renamed the scheduler export
  (convert_to_output_meta_info -> to_output_time_stats, Optional on
  metrics-off); the assertion's meaning (1.6 s episode sum / absent series)
  is unchanged.
- CustomTestCase import added (the upstream file's classes use bare
  unittest.TestCase; the fork's used the retry wrapper).

ACTUALS: test_storage_prefetch_lifecycle 16P/0F/13sub -> ** 28P/0F/31sub **
  (295-era pin 23P/28sub EXCEEDED; every grafted case green first run
  against the planted B3-3 code).
No GPU; no push.
detain pushed a commit to detain/sglang that referenced this pull request Oct 30, 2026
Review follow-up. Skipping every later collective after the first sync
failure was unsound: gloo closes only the pair of the group that timed
out, and the hit and completion sync groups are distinct even on TP2, so
a peer that had not failed blocked for the untouched group's full timeout
(30 min by default), during which reset/detach could not join and flush
raised out of the scheduler. A retry on the broken group raises at once
on every rank, so the collective is now always entered.

Also: take_ready_pp_prefetch waits in 1 s slices and fails loud when the
local ticket thread is dead; the buffer-only PP>1 health relay is limited
to the ticket-less shape (with tickets PP0 decides per request); the
sync knobs are validated before any attach side effect; the gauge drops
on detach; a WARNING once a minute while degraded; thread attributes are
initialised in __init__; reset logs that L3 stays disabled. The two-rank
test now runs its three cases in one spawn through check_hicache_events,
adds the cross-group case (an ack on the healthy group must reduce
promptly on both ranks after the hit group failed) and takes ~26 s.

Port adaptations (cherry-pick -x of 74a0e6b + the parts of the
60c5f3e main-sync resolution that are PR content, not main noise --
the merge commits themselves carry only upstream-main history already in
this tree's merge sgl-project#5 base and are SKIPPED):

- Rank-consistency preserved: the degraded cut now RIDES THE DEADLINE
  MACHINERY (the 60c5f3e resolution against sgl-project#44614, which this tree
  runs since entry 217): storage_unavailable -> _mark_prefetch_deadline_-
  expired -> the existing deadline_expired branch resolves the cut with
  NO poll_miss re-arm, honoring the entry-293 shared retry-cap rule with
  the timeout stop policy as the earlier, already-proven exit.
- Fork stub convention on the new test's _scheduler_cache: pre-memoize
  _hcs_state=False (entry 294) and queued_backups={} (entry 199), plus
  the _storage_prefetch_wait fixture attribute the resolution itself
  records ("test fixtures gain the attributes main added").
- test_command_failure keeps the fork's ticket-thread shape: later
  submissions stay ticketed, the rank-agreed drop retires them.

Gate at this commit: 88P/62sub green over trailing_endpoint,
pp_prefetch_ticket, buffer_mode_sidecar, storage_prefetch_lifecycle,
pp_sync_drain, async_ready, sync_recovery (2P/4sub), shm_tag_exchange,
managers trio; import sglang proven; ruff+format clean.

(cherry picked from commit 74a0e6b)
(cherry picked from commit be1316edf74bd2139dc02c2294bc799af04205fd)
detain pushed a commit to detain/sglang that referenced this pull request Oct 30, 2026
… bounded ticket group

- storage_workers_healthy() watches the worker threads the controller started;
  a controller whose storage I/O runs outside Python workers has none and
  reads healthy on their account instead of dead.
- storage_sync_timeout defaults to 300 s instead of torch's 30-minute gloo
  default (null / absent -> default, non-finite rejected); a hung peer is
  noticed in minutes while staying far above any healthy batch call.
- The PP ticket broadcast group is created when the storage workers start,
  after attach applied the timeout, so it is bounded like the other storage
  collectives (it used to be built before the config was parsed).
- A prefetch cut by the rank-agreed degrade reports storage_unavailable in
  storage_prefetch_unfulfilled_tokens_total instead of borrowing deadline.

Port of 6c37509 (the PR's third content commit; its two "Merge branch
main" commits 60c5f3e/96850aef61 are pure upstream-main syncs -- all of
their PR-relevant resolutions are already folded into this branch's
commits dee1e2a872 (shm mins-path split, ticket-thread hook, fork-shape
submission) and 8f479b20a9 (deadline-riding degrade cut, stub fixtures)
and the merge commits themselves are NOT picked: they carry nothing but
main history this tree already holds through MERGE sgl-project#5).

Fork note: the new stop-reason rides the entry-217/sgl-project#44614 machinery --
storage_unavailable marks operation.deadline_expired, so the cut resolves
through the existing deadline_expired branch with NO poll_miss re-arm
(the entry-293 shared retry-max-attempts cap stays untouched; the timeout
stop policy remains the earlier exit), and _prefetch_cut_reason routes
that cut plus the hybrid discard and the token resolve to
reason=storage_unavailable (gauge label registered additively). The
fork-only parse_storage_backend_extra_config registration added in
dee1e2a872 now mirrors the 300 s default semantics (absent -> default,
non-finite rejected) without duplicating the constant.

Gate at this commit: 90P/63sub green (trailing_endpoint 17P,
pp_prefetch_ticket 17P/19sub, buffer_mode_sidecar 6P/6sub,
storage_prefetch_lifecycle 28P/31sub, pp_sync_drain 4P, async_ready 7P,
sync_recovery 4P/5sub, shm_tag_exchange, managers trio); import sglang
proven; ruff+format clean.

(cherry picked from commit 6c37509)
(cherry picked from commit 5d1e36812a5fc252941d907ac815ddcee9e006cc)
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
…r/URC markers -- 3/18 census consumed, 3 gates in one

CASCADE (start_writing(wait_for_forward) TypeError class, 68 distinct nodes):
upstream sgl-project#48012's NEW fixture hicache_write_back_fixtures.InstantWriteBackController
mirrored the pre-sgl-project#44800 no-kwarg surface while the fork's writing_check drain
(entries 199/218: the batched-backup flush=False + ONE fenced start_writing)
calls the real API. Def-site fixes, no fence weakened:
- fixture start_writing gains the keyword (instant copies -> accepted-ignored,
  documented);
- test_write_back_eviction_batching: record_submission wrapper FORWARDS the
  flag (it measures the very wait, so swallowing it would understate the
  fence); _FencedController gains load_fence_stream = forward (the fork's
  unconditional entry-168 fence reads it first; prod: both streams are the
  same forward stream) -- also silences the 16-test Rust TreeCore panic class
  whose "victims not demoted" assert was the aborted submission downstream.

MARKERS (3): managers/cache_controller.py:1202 ADOPTED -- the dropped
set_hicache_transfer_done_event write line is superseded by upstream
c24285b(sgl-project#48008 per-band page ledger)+2cbb66f563(sgl-project#48012): record_host_transfer
publishes the same signal per-page BOTH directions (load twin writes=True).
unified_radix_cache.py:627 ADOPTED -- the buffer-mode threshold floor
(resolve_prefetch_threshold) was DELIBERATELY deleted upstream by
dfaca56(sgl-project#46934, restores sub-window SWA L3 hits); window sizing now lives
per-allocation in BufferModePipeline; the per-call-site file_storage_path fold
is consistent with the entry-253 funnel (cache_controller.py:948, intact).
unified_radix_cache.py:3917 RE-PLANTED -- entry-293's cadence gate (park, do
not re-enter allocate_storage_hit/evict_host inside the retry window) + the
expired no-retry arm were dropped; wait_full folded into upstream's
comp_xfers allow_partial rule (min_tokens form already equivalent).

URC context guard: _deferred_ready_sync_group read get_memory() unguarded --
upstream's NEW _arm_tick CPU/spawn fixtures (their restructured timeout tests,
sgl-project#44614) reach it without a published server config. Now published-check
(schedule_policy.py:615 idiom): default-off CPU path, production unchanged.

Deadline-anchor retargets (fork B3-9 family vs upstream's sgl-project#44614 vote-before-
poll restructure; every assertion preserved): _vote_prefetch_deadlines drives
the verdict before the expiry polls; failed-arm polled inside-budget so the
storage failure stays the SOLE cut reason; fixture cc gains the sgl-project#44342
liveness attrs + pp_prefetch_command_thread; _arm_tick stub returns the fork's
5-tuple (health bit True); release_aborted_request -> finish(ABORT) per
8898429(sgl-project#48491).

Battery actuals (CPU, CVD=9, prod venv, PYTHONPATH=wt/python):
- the 5 cascade files 68F/39P -> 3F/70P (survivors = the sgl-project#48491/pp-thread/
  budget-anchor fixes committed here; after them): test_storage_prefetch_
  lifecycle ** 31P/31sub ** (entry-311 pin 28P/31sub, 295's >=23P/28sub
  EXCEEDED); decode_retraction_backup 6F + chunked_publish 2F = No-CUDA
  device class (env, reported).
- combined gate run: trailing_endpoint + batching + parity + async_ready_sync
  + staged_write_back = 89P/1S/28sub 0F.
Gates: ruff repo ruleset+format clean on touched files; registration trio rc=0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants