Skip to content

[Fix] hicache: Preserve sparse restore checkpoints under host pressure - #46954

Open
pzcyyj wants to merge 1 commit into
sgl-project:mainfrom
pzcyyj:fix/hicache-sparse-restore-capacity
Open

pzcyyj wants to merge 1 commit into
sgl-project:mainfrom
pzcyyj:fix/hicache-sparse-restore-capacity

Conversation

@pzcyyj

@pzcyyj pzcyyj commented Oct 28, 2026 •

Copy link
Copy Markdown

Motivation

A hybrid L3 hit can be restorable only at sparse endpoints because it needs both a contiguous KV prefix and a trailing state/window snapshot. A smaller prefix is not necessarily another valid hit.

For example, a query can find complete checkpoints at pages 16 and 32. If host staging has room for only 24 pages, cache-mode partial allocation currently clips the hit to page 24. The subsequent trailing-pool read then targets a checkpoint that the query did not establish. The existing completeness check correctly rejects an incomplete restore, but the usable 16-page checkpoint is lost and KV reads can be wasted.

Modifications

  • Keep the common restorable endpoint set after the existing cross-rank query reduction, rather than keeping only its maximum and leaving a rank-local set behind.
  • When partial host allocation is necessary for a trailing-pool restore, select the largest common endpoint that fits the capacity bound before allocating any partial buffers. If no endpoint fits, use the existing host-capacity cleanup path without submitting the read.
  • Treat an assume_stored trailing hit as proof of its original endpoint only; it cannot establish intermediate checkpoints without a query. Plain KV retains its contiguous-prefix fast path.
  • Size cache-mode auxiliary staging from the selected restore length.

This preserves the all-pools completeness check, existing allocation ownership, and the current collective sequence. Full-hit allocation and allow_partial=False retain their existing behavior. Buffer-only head trimming continues to preserve the original sidecar endpoint; PP keeps its existing capacity-capped checkpoint vote. No backend/storage format or serving configuration changes are included.

Accuracy Tests

Local CPU regression control on Python 3.12.16 / PyTorch 2.14.1:

  • Patched: 14/14 tests passed, comprising 13 new controller/cache tests and one additional PP regression.
  • Unmodified base faa56e2b5c5c5dee60eb5dd7b6251944a21a6104: 8 distinct regression methods fail with the same tests and adapter; the other 6 controls pass. These are assertion failures on endpoint selection/retention or staging behavior, rather than import failures.
  • Repository pre-commit checks passed for all four changed files.
  • The existing buffer trim/stage regression also passed its 8 combinations. A separate shared-host real-pool neighbor was blocked by the existing host-memory reserve guard on this Mac, before reaching the behavior under test; it is not counted as passing.

The local macOS CPU process uses the repository's complete _platform_stubs.py and maybe_stub_sgl_kernel() for GPU-only imports. The MPS availability override is limited to compatibility installation and is restored before importing/running the targets; tensors and assertions run on CPU. This adapter is not part of the patch. New query tests mock the unrelated Linux-only native hash boundary. Target controller/cache methods are imported directly, not copied or AST-extracted.

In a configured Linux development environment, the added tests can be run with:

python3 test/registered/unit/mem_cache/test_hybrid_storage_sparse_allocation.py -v
python3 test/registered/unit/mem_cache/test_pp_prefetch_ticket.py TestPPPrefetchTicket.test_sparse_checkpoint_vote_does_not_invent_capacity_endpoint -v

The tests directly import the production controller/cache methods and use CPU tensors. Host capacity, storage queries, and collective boundaries are controlled test inputs; they do not reproduce a real allocator-pressure history or a multi-process Store deployment.

GPU/model accuracy, the complete CPU suite, and a full Linux integration run have not been performed. No available GPU is required for the added unit tests.

Speed Tests and Profiling

No end-to-end speedup is claimed. This fixes checkpoint selection under partial host capacity; serving performance and GPU L3-to-L1 validation remain unmeasured for this patch.

Checklist

  • Add CPU regression tests registered for base-a-test-cpu.
  • Run the repository pre-commit checks on all four changed files.
  • Complete GPU/model accuracy and integration validation.

AI assistance was used to prepare the patch and tests.


CI States

Latest PR Test (Base): ❌ Run #40640081762
Latest PR Test (Extra): ❌ Run #40640081524
Latest PR Test (AMD ROCm 10): ❌ Run #40640081521

@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang unified-radix-cache labels Oct 28, 2026
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
Port of the upstream generalization of our entry-209 trailing-sidecar fix
onto the landed stack. Union analysis per hunk (entry-209 = 8aab510 +
3fa5197, entry-217 = sgl-project#43108/sgl-project#44614 already occupy these regions):

- allocate_storage_hit "pick the largest fitting verified endpoint BEFORE
  allocating, decline when none fits" -- REDUNDANT on our tree: our
  _trailing_endpoint_tokens (entry 209) already performs exactly that
  choice (plus the buffer-only sidecar_hash_values pin guard and the
  throttled _report_no_restorable_endpoint that upstream lacks). Kept
  ours; upstream's gate difference noted: their selection trigger is any
  TRAILING_PAGES transfer, ours additionally requires indices_from_pool is
  None (derived sidecars are covered through their source transfer).
- _sync_prefetch_hit_query "retain the common endpoint set after the MIN
  reduction instead of collapsing to the deepest" -- REDUNDANT: our
  always-masked payload (1 + len(PoolName) + restorable_mask) already
  reduces and retains the intersected set. local_restorable_mask gained
  upstream's "no verified hit votes zero" invariant: a rank syncing with a
  folded count of 0 (terminated/skipped/failed query) can no longer vouch
  for a candidate set left by a partial earlier state.
- assume_stored branch -- LANDED (semantics win over our blank): a trailing
  re-prefetch now reports [kv_hit_pages] (the original endpoint only, not
  shallower checkpoints) instead of None, so a capacity-shortened retry
  stops on the endpoint or declines, never on an unverified interior page.
  Non-trailing keeps None (contiguous contract).
- unified_radix_cache aux staging -- LANDED (new here):
  _alloc_prefetch_aux_staging sizes to alloc_len (the endpoint the KV span
  got) instead of the full query hit; under mamba-host pressure the old
  full-span sizing failed the aux alloc and declined a prefetch a smaller
  endpoint would have fit.

Tests: upstream's test_hybrid_storage_sparse_allocation.py (13 cases + 14
subtests) retargeted onto our field placement (operation.restorable_prefix_
pages), our packed collective layout, and page-0-always-zero mask (no
sentinel); every allocation/query/drain expectation kept. The upstream PP
companion test hunk (+42 in test_pp_prefetch_ticket.py) is NOT portable:
this tree's PP broadcast path runs no mask collective (the ticket carries
only the folded count, followers rebuild the operation without it), so its
vote-shape assertions describe a payload our design does not have; flagged
for the integrator. Our trailing_endpoint (17), storage_prefetch_lifecycle,
buffer_mode_sidecar, pp_prefetch_ticket and unified_hicache suites all
still pass.

(cherry picked from commit a23ba83)
PR: sgl-project#46954 (open, ported)
(cherry picked from commit b6e7ed7cb232d6069ad2ac4d5be41c7bba606511)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…producer-dict closure — re-plant entries 24,199,209,217,219,291,292 (+168-verify)

** THE CLAMP IS BACK: trailing_endpoint 8P/9F -> 17P/0F (entry-209 gate). **

Re-plants the fork-hunks patch (replant/python__sglang__srt__mem_cache__
hybrid_cache__hybrid_cache_controller.py.fork-hunks.patch) onto the
#46970x-rewritten controller. B3 markers consumed: hybrid_cache_controller.py,
metrics_collector.py (entry-219 producer detail).

sgl-project#46954-IN-RANGE VERDICT + dedupe decision table (ledger entry 290 is the spec):
- sgl-project#46954 itself NEVER MERGED upstream (only our port b6e7ed7cb2 exists, on the
  unlanded port-hicache-ctrl branch); what arrived in-range is sgl-project#41187
  (c677171), whose _sync_prefetch_hit_query re-implements the mask reduce
  but COLLAPSES TO THE DEEPEST COMMON ENDPOINT. Per-hunk:
  (a) allocate-time endpoint choice -> RE-PLANTED (_trailing_endpoint_tokens:
      deepest VERIFIED endpoint <= alloc, decline-with-report when none; the
      entry-290(a) "BOTH OURS' ADDITIONS ARE KEPT": sidecar_hash_values pin
      guard + throttled _report_no_restorable_endpoint; the indices_from_pool
      IS None gate delta stays a documented non-union).
  (b) retain-common-endpoint-set -> OURS KEPT per the ledger; upstream's
      collapse loses shallower endpoints the host-capacity clamp needs.
      UNION REFINEMENT recorded here: the mask is packed only when the span
      has a TRAILING_PAGES transfer (trailing-ness is rank-invariant config,
      so shape consistency holds), which keeps the non-trailing collective
      BIT-IDENTICAL to upstream's 1+len(PoolName) payload and collapses the
      upstream PP-test breakage from 18F to the one genuinely trailing case.
      Fork layout kept where it matters: [folded] + verdicts + mask,
      page-0-always-zero (no sentinel), candidates NOT capacity-capped
      (the PP leader caps the folded entry itself before sync).
  (c) assume_stored endpoint-only proof -> RE-PLANTED ([kv_hit_pages] set for
      trailing spans, None otherwise; replaces the pre-sgl-project#46954 blank).
  (d) aux-staging sizing to the ACTUAL endpoint -> OWED B3-3: the producer
      lives in unified_radix_cache._alloc_prefetch_aux_staging (the single
      remaining sparse_allocation 1F is exactly this test).
- 217/218 rider semantics re-planted with the patch: restorable_prefix_pages
  operation field (both placements written: op attr for the clamp/mask,
  pool_storage_result for the upstream drain consumers), host_capacity_*
  retry fields + deadline_expired (producer side; the policy consumer rides
  B3-3 URC), entry-199 write(node_ids=...) ack list, entry-90 shortfall-only
  evict_host in allocate_storage_hit, sgl-project#46707 source_len SUM (multi-destination
  Mamba load-backs), sgl-project#43110 merge_source_transfers at BOTH derivation sites,
  sgl-project#39830 backup-thread survival comment, and the 3fa5197 reporting trio
  (_report_sidecar / _report_sidecar_endpoint_misses /
  _report_sidecar_write_shortfalls + class-level tallies for the /metrics
  hook). local_restorable_mask stays the merged-in module function (already
  byte-faithful to the patch).
- cache_controller.py fence VERIFIED, not re-planted (DRIVER:MANUAL held it):
  load_fence_stream/write_fence_stream live at :381-382/:997-1018/:1157-1162
  and test_hicache_staged_write_back_dispatch is 27P/1S/8sub.

Entry-219 PRODUCER-DICT closure (the narrowed marker, D3): sgl-project#43727 -- the
entry-189 PR -- LANDED UPSTREAM in-range (26a1e51), re-arriving as
hicache_host_mamba_{used,total}_states SchedulerStats fields, a reporter
entry_map read-back, and a TestHiCacheHostMambaStats class (the DI test file
was a union concatenation of BOTH class generations with a mid-file __main__
guard). Entry 219's ledger is authoritative: the per-pool
hicache_host_pool_{used,total}_tokens{pool} set is a strict superset, the
mamba-states layer was REMOVED and NO duplicate registration survives.
Deleted the gauge-less fields + reporter block (the arrival exported nothing:
log_stats never touched the fields), dropped the now-unused
PoolName/UnifiedRadixCache imports (entry-219's own wording), removed the
resurrected guard class, moved the stray __main__ guard to EOF, and re-wrote
hicache_best_practices.mdx's watch sentence onto the per-pool schema (B1 owns
that file's merge resolution; this is a one-line schema follow-up). The
{pool: -> dict} producer wiring (collect_host_pool_stats + derived_pools
sidecar exclusion) was already complete at the merge -- confirmed, not
duplicated.

ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after):
  test_hicache_trailing_endpoint            8P/9F  -> 17P/0F   ** GATE **
  test_pp_prefetch_ticket                  17P/0F* -> 17P/0F/19sub (*base green; 18F transient mid-replant, retargeted per entry-290 precedent: vote-shape assertion moved to OUR payload, every allocation/query/drain expectation KEPT)
  test_hybrid_storage_sparse_allocation     8P/13F -> 12P/1F (the 1F = entry-290(d) aux-staging sizing, URC producer, B3-3)
  test_buffer_mode_sidecar                          -> 6P/0F/6sub
  test_hicache_mamba_slot_side_states               -> 5P/0F
  test_storage_prefetch_lifecycle                    -> 16P/0F/13sub
  test_unified_hicache_transfer_lifecycle   4P/4F  -> 4P/4F (all 4F = cargo-build ENV class, documented byte-identical pre-merge)
  unit/observability test_stat_loggers_di + test_hicache_pool_stats: 7F -> 30P/0F
Lint: ruff format+F821/E9/F401 clean on all touched files; import OK.
No GPU executed; no push; tree otherwise untouched.
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…t) SESSION 2A COMPLETE

Entry 311 records: the 9 cluster commits (2a5646b..1dc41b2) with per-
cluster batteries; the sgl-project#46954 VERDICT + hunk-by-hunk DEDUPE TABLE (sgl-project#46954
itself NEVER MERGED upstream -- sgl-project#41187's collapsed-mask re-implementation is
what arrived; (a) OURS re-planted, (b) OURS kept with the recorded UNION
REFINEMENT -- mask packs only for TRAILING spans, non-trailing collectives
BIT-IDENTICAL to upstream's shape, (c) assume_stored endpoint-proof
re-planted, (d) aux-staging alloc_len sizing re-planted = the prod-highest-
value hunk); the entry-219 PRODUCER closure (sgl-project#43727 arrived in-range as
26a1e51, the gauge-less layer DELETED per entry 219's ledger, per-pool
serving schema CONFIRMED as the only producer, docs watch-sentence
re-pointed); the entry-196 rust ADAPTER-CONTRACT repairs (mamba_value_seqlen
kwarg + parent_node_id rename -- both live TypeErrors at the merged tip);
the boot-integrity lossless-ring planting (CPU-PROVEN construction; every
caller landed with THEIRS while the pool signatures were the dropped half);
the 12 dropped storage_prefetch_lifecycle cases RE-HOMED (28P/31sub >= the
23P/28sub pin, all green unmodified against the B3-3 machinery).

GATES: ** trailing_endpoint 17P/0F PROVEN at the final tree incl. the 2-rank
gloo cases, and at every cluster checkpoint after B3-2 ** (the entry-209 DoD
gate); the B3-2/B3-caused async_ready_sync collection error CLEARED (7P);
ZERO non-environment failures remain in unit/mem_cache.

NEW PIN (per-file, CVD=9, prod venv, new post-sgl-project#45232/sgl-project#45236 layout):
unit/mem_cache 1753P/1351F/364S/4439sub/4err -> ** 1861P/1209F/364S/4514sub/
2err **; the 1209F residue is 100% env class by per-file error taxonomy:
1056F = the upstream URC-split KITS' fixtures.py build_fixture get_device()
"No accelerator" -- ENTRY 310's "the URC split files themselves are GREEN,
honest skips" CLAIM IS CORRECTED IN PLACE (its own battery log shows core
27P/465F; the kits honest-skip DEFECT is B2 scope, recorded not fixed) --
130F+2err = cargo/torch-sys (byte-identical pre-merge, the documented
environment class incl. the draft_gate 4F / joint-allocation 47F / tri-reclaim
54F storm files reclassified EXPECTED-REPLANT->ENV), 23F = the No-CUDA class
(kv_layer_router 3F + kv_storage_lifecycle 9F + kv_layer_access 5F +
pricing_property 4F + decode_retraction 2F). Named-file pins recorded
(trailing_endpoint 17P, fast_file 54P/13sub, sparse 13P/14sub EXACT, donated
17P/14sub >= 16P, fence 15P EXACT, lifecycle 28P/31sub >= 23P/28sub, ...).

CAVEATS: the merge-era top bullet ANNOTATED (B3 landed, gate met, boot ban
STAYS -- the rule's HiCache half is proven but sessions 2B/3 remain and this
batch NEVER booted, per the era rule); the entry-310 damage-map mem_cache and
trailing_endpoint rows annotated RESOLVED-AT-B3 with the F-collapse table;
the synced-block INTEGRATION OWED line annotated B3 LANDED; marker census
correction noted (tree carried 15 B3 markers, not 12; ALL consumed -- B2/B4/
B5/B6 diff-audited INTACT).

Header: STACK -> qwennext-stack-311+merge5-b3-2026-10-29; +merge5-b3 wave
tag; entries 1-311; tip -> 1dc41b2 (B3-9 = true code tip, docs above).
wanted_stack: UNTOUCHED (no S-item changed by B3 findings -- recorded here).
Debt list carried in the entry: rust cargo UNVERIFIED 932P pin (session-3
seat), entry-198 scheduler-side CATCHER cross-noted for B4, kits get_device
skip defect for B2, NVFP4 dequant-workspace deliberately NOT restored
(superseded). NOTHING PUSHED; tree clean; no GPU executed all session.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant