Repository navigation
[Fix] hicache: Defer query beliefs until the rank-synchronized hit drain - #43108
Merged
xiezhq-hermann merged 14 commits intoOct 15, 2026
Merged
xiezhq-hermann merged 14 commits into
xiezhq-hermann merged 14 commits into
Conversation
alphabetc1
requested review from
Ying1123,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 8, 2026 15:36
| if operation.all_hash_values is None: | ||
| # Cancellation can skip the local query. Every rank still needs | ||
| # the full chain to invalidate beliefs at the agreed boundary. | ||
| operation.all_hash_values = get_storage_hash_str( |
Collaborator
There was a problem hiding this comment.
in the case of cancellation, we actually should not invalidate as we don't obtain any new information about existence believe
Collaborator
Author
There was a problem hiding this comment.
fixed by using verdict to distinguish "query cancel" and "cache miss"
xiezhq-hermann
approved these changes
Oct 14, 2026
alphabetc1
enabled auto-merge (squash)
October 14, 2026 15:54
5 tasks done
xiezhq-hermann
disabled auto-merge
October 15, 2026 00:24
This was referenced Oct 15, 2026
detain
pushed a commit
to detain/sglang
that referenced
this pull request
Oct 19, 2026
…ain (sgl-project#43108) (cherry picked from commit 3c28614) (cherry picked from commit 345659eb8fb0a1d00646b55029fdb2cab28b547b)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
Port of the upstream generalization of our entry-209 trailing-sidecar fix onto the landed stack. Union analysis per hunk (entry-209 = 8aab510 + 3fa5197, entry-217 = sgl-project#43108/sgl-project#44614 already occupy these regions): - allocate_storage_hit "pick the largest fitting verified endpoint BEFORE allocating, decline when none fits" -- REDUNDANT on our tree: our _trailing_endpoint_tokens (entry 209) already performs exactly that choice (plus the buffer-only sidecar_hash_values pin guard and the throttled _report_no_restorable_endpoint that upstream lacks). Kept ours; upstream's gate difference noted: their selection trigger is any TRAILING_PAGES transfer, ours additionally requires indices_from_pool is None (derived sidecars are covered through their source transfer). - _sync_prefetch_hit_query "retain the common endpoint set after the MIN reduction instead of collapsing to the deepest" -- REDUNDANT: our always-masked payload (1 + len(PoolName) + restorable_mask) already reduces and retains the intersected set. local_restorable_mask gained upstream's "no verified hit votes zero" invariant: a rank syncing with a folded count of 0 (terminated/skipped/failed query) can no longer vouch for a candidate set left by a partial earlier state. - assume_stored branch -- LANDED (semantics win over our blank): a trailing re-prefetch now reports [kv_hit_pages] (the original endpoint only, not shallower checkpoints) instead of None, so a capacity-shortened retry stops on the endpoint or declines, never on an unverified interior page. Non-trailing keeps None (contiguous contract). - unified_radix_cache aux staging -- LANDED (new here): _alloc_prefetch_aux_staging sizes to alloc_len (the endpoint the KV span got) instead of the full query hit; under mamba-host pressure the old full-span sizing failed the aux alloc and declined a prefetch a smaller endpoint would have fit. Tests: upstream's test_hybrid_storage_sparse_allocation.py (13 cases + 14 subtests) retargeted onto our field placement (operation.restorable_prefix_ pages), our packed collective layout, and page-0-always-zero mask (no sentinel); every allocation/query/drain expectation kept. The upstream PP companion test hunk (+42 in test_pp_prefetch_ticket.py) is NOT portable: this tree's PP broadcast path runs no mask collective (the ticket carries only the folded count, followers rebuild the operation without it), so its vote-shape assertions describe a payload our design does not have; flagged for the integrator. Our trailing_endpoint (17), storage_prefetch_lifecycle, buffer_mode_sidecar, pp_prefetch_ticket and unified_hicache suites all still pass. (cherry picked from commit a23ba83) PR: sgl-project#46954 (open, ported) (cherry picked from commit b6e7ed7cb232d6069ad2ac4d5be41c7bba606511)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In
buffer_onlymode, revoking a storage prefetch can update storage-existence beliefs from rank-local query results before their reduction. TP ranks can then disagree about whether a prefix is already stored and make different backup decisions. Cancellation can also cause a rank to skip the query; that lack of query evidence must not be treated as a storage miss.This change updates beliefs only when consuming the rank-synchronized
prefetch_hit_queue:revoke_pending_prefetch.The added sidecar unit test uses the actual query and synchronization methods with a stub storage backend and simulated MIN reduction. It checks that cancellation preserves existing KV beliefs when the peer skipped the query (
-1), but invalidates them when the peer reports a miss (0). The terminated-query lifecycle fixture also gainsall_hash_values=Nonefor the belief-update path now reached during draining.Belief updates reuse the existing synchronized drain and add no collective.
CI States
Latest PR Test (Base): 🚫 Run #38558881342
Latest PR Test (Extra): ✅ Run #38558881371
Latest PR Test (AMD ROCm 10): ❌ Run #38558881356