Skip to content

[Fix] hicache: Defer query beliefs until the rank-synchronized hit drain - #43108

Merged
xiezhq-hermann merged 14 commits into
sgl-project:mainfrom
alphabetc1:fix/hicache-query-cancel-consensus
Oct 15, 2026
Merged

xiezhq-hermann merged 14 commits into
sgl-project:mainfrom
alphabetc1:fix/hicache-query-cancel-consensus

Conversation

@alphabetc1

@alphabetc1 alphabetc1 commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

In buffer_only mode, revoking a storage prefetch can update storage-existence beliefs from rank-local query results before their reduction. TP ranks can then disagree about whether a prefix is already stored and make different backup decisions. Cancellation can also cause a rank to skip the query; that lack of query evidence must not be treated as a storage miss.

This change updates beliefs only when consuming the rank-synchronized prefetch_hit_queue:

  • Remove query-driven belief updates from revoke_pending_prefetch.
  • Apply query evidence at the start of each hit-queue drain iteration, before handling removed requests, terminated operations, or below-threshold hits.
  • Skip belief updates when the reduced result has no KV verdict, indicating that a rank did not query storage. An explicit zero-page KV verdict remains a real miss and can invalidate existing beliefs.

The added sidecar unit test uses the actual query and synchronization methods with a stub storage backend and simulated MIN reduction. It checks that cancellation preserves existing KV beliefs when the peer skipped the query (-1), but invalidates them when the peer reports a miss (0). The terminated-query lifecycle fixture also gains all_hash_values=None for the belief-update path now reached during draining.

Belief updates reuse the existing synchronized drain and add no collective.


CI States

Latest PR Test (Base): 🚫 Run #38558881342
Latest PR Test (Extra): ✅ Run #38558881371
Latest PR Test (AMD ROCm 10): ❌ Run #38558881356

@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Oct 8, 2026
@alphabetc1 alphabetc1 added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) labels Oct 9, 2026
@xiezhq-hermann xiezhq-hermann self-assigned this Oct 12, 2026
if operation.all_hash_values is None:
# Cancellation can skip the local query. Every rank still needs
# the full chain to invalidate beliefs at the agreed boundary.
operation.all_hash_values = get_storage_hash_str(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the case of cancellation, we actually should not invalidate as we don't obtain any new information about existence believe

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed by using verdict to distinguish "query cancel" and "cache miss"

@alphabetc1 alphabetc1 changed the title [HiCache] Defer query beliefs until the rank-synchronized hit drain [Fix] hicache: Defer query beliefs until the rank-synchronized hit drain Oct 14, 2026
@alphabetc1
alphabetc1 enabled auto-merge (squash) October 14, 2026 15:54
@xiezhq-hermann
xiezhq-hermann merged commit 3c28614 into sgl-project:main Oct 15, 2026
199 of 213 checks passed
@alphabetc1
alphabetc1 deleted the fix/hicache-query-cancel-consensus branch October 15, 2026 01:44
detain pushed a commit to detain/sglang that referenced this pull request Oct 19, 2026
…ain (sgl-project#43108)

(cherry picked from commit 3c28614)
(cherry picked from commit 345659eb8fb0a1d00646b55029fdb2cab28b547b)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
Port of the upstream generalization of our entry-209 trailing-sidecar fix
onto the landed stack. Union analysis per hunk (entry-209 = 8aab510 +
3fa5197, entry-217 = sgl-project#43108/sgl-project#44614 already occupy these regions):

- allocate_storage_hit "pick the largest fitting verified endpoint BEFORE
  allocating, decline when none fits" -- REDUNDANT on our tree: our
  _trailing_endpoint_tokens (entry 209) already performs exactly that
  choice (plus the buffer-only sidecar_hash_values pin guard and the
  throttled _report_no_restorable_endpoint that upstream lacks). Kept
  ours; upstream's gate difference noted: their selection trigger is any
  TRAILING_PAGES transfer, ours additionally requires indices_from_pool is
  None (derived sidecars are covered through their source transfer).
- _sync_prefetch_hit_query "retain the common endpoint set after the MIN
  reduction instead of collapsing to the deepest" -- REDUNDANT: our
  always-masked payload (1 + len(PoolName) + restorable_mask) already
  reduces and retains the intersected set. local_restorable_mask gained
  upstream's "no verified hit votes zero" invariant: a rank syncing with a
  folded count of 0 (terminated/skipped/failed query) can no longer vouch
  for a candidate set left by a partial earlier state.
- assume_stored branch -- LANDED (semantics win over our blank): a trailing
  re-prefetch now reports [kv_hit_pages] (the original endpoint only, not
  shallower checkpoints) instead of None, so a capacity-shortened retry
  stops on the endpoint or declines, never on an unverified interior page.
  Non-trailing keeps None (contiguous contract).
- unified_radix_cache aux staging -- LANDED (new here):
  _alloc_prefetch_aux_staging sizes to alloc_len (the endpoint the KV span
  got) instead of the full query hit; under mamba-host pressure the old
  full-span sizing failed the aux alloc and declined a prefetch a smaller
  endpoint would have fit.

Tests: upstream's test_hybrid_storage_sparse_allocation.py (13 cases + 14
subtests) retargeted onto our field placement (operation.restorable_prefix_
pages), our packed collective layout, and page-0-always-zero mask (no
sentinel); every allocation/query/drain expectation kept. The upstream PP
companion test hunk (+42 in test_pp_prefetch_ticket.py) is NOT portable:
this tree's PP broadcast path runs no mask collective (the ticket carries
only the folded count, followers rebuild the operation without it), so its
vote-shape assertions describe a payload our design does not have; flagged
for the integrator. Our trailing_endpoint (17), storage_prefetch_lifecycle,
buffer_mode_sidecar, pp_prefetch_ticket and unified_hicache suites all
still pass.

(cherry picked from commit a23ba83)
PR: sgl-project#46954 (open, ported)
(cherry picked from commit b6e7ed7cb232d6069ad2ac4d5be41c7bba606511)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants