Repository navigation
Conversation
pzcyyj
requested review from
Ying1123,
hanming-lu,
hnyls2002,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 28, 2026 05:48
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
Port of the upstream generalization of our entry-209 trailing-sidecar fix onto the landed stack. Union analysis per hunk (entry-209 = 8aab510 + 3fa5197, entry-217 = sgl-project#43108/sgl-project#44614 already occupy these regions): - allocate_storage_hit "pick the largest fitting verified endpoint BEFORE allocating, decline when none fits" -- REDUNDANT on our tree: our _trailing_endpoint_tokens (entry 209) already performs exactly that choice (plus the buffer-only sidecar_hash_values pin guard and the throttled _report_no_restorable_endpoint that upstream lacks). Kept ours; upstream's gate difference noted: their selection trigger is any TRAILING_PAGES transfer, ours additionally requires indices_from_pool is None (derived sidecars are covered through their source transfer). - _sync_prefetch_hit_query "retain the common endpoint set after the MIN reduction instead of collapsing to the deepest" -- REDUNDANT: our always-masked payload (1 + len(PoolName) + restorable_mask) already reduces and retains the intersected set. local_restorable_mask gained upstream's "no verified hit votes zero" invariant: a rank syncing with a folded count of 0 (terminated/skipped/failed query) can no longer vouch for a candidate set left by a partial earlier state. - assume_stored branch -- LANDED (semantics win over our blank): a trailing re-prefetch now reports [kv_hit_pages] (the original endpoint only, not shallower checkpoints) instead of None, so a capacity-shortened retry stops on the endpoint or declines, never on an unverified interior page. Non-trailing keeps None (contiguous contract). - unified_radix_cache aux staging -- LANDED (new here): _alloc_prefetch_aux_staging sizes to alloc_len (the endpoint the KV span got) instead of the full query hit; under mamba-host pressure the old full-span sizing failed the aux alloc and declined a prefetch a smaller endpoint would have fit. Tests: upstream's test_hybrid_storage_sparse_allocation.py (13 cases + 14 subtests) retargeted onto our field placement (operation.restorable_prefix_ pages), our packed collective layout, and page-0-always-zero mask (no sentinel); every allocation/query/drain expectation kept. The upstream PP companion test hunk (+42 in test_pp_prefetch_ticket.py) is NOT portable: this tree's PP broadcast path runs no mask collective (the ticket carries only the folded count, followers rebuild the operation without it), so its vote-shape assertions describe a payload our design does not have; flagged for the integrator. Our trailing_endpoint (17), storage_prefetch_lifecycle, buffer_mode_sidecar, pp_prefetch_ticket and unified_hicache suites all still pass. (cherry picked from commit a23ba83) PR: sgl-project#46954 (open, ported) (cherry picked from commit b6e7ed7cb232d6069ad2ac4d5be41c7bba606511)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…producer-dict closure — re-plant entries 24,199,209,217,219,291,292 (+168-verify) ** THE CLAMP IS BACK: trailing_endpoint 8P/9F -> 17P/0F (entry-209 gate). ** Re-plants the fork-hunks patch (replant/python__sglang__srt__mem_cache__ hybrid_cache__hybrid_cache_controller.py.fork-hunks.patch) onto the #46970x-rewritten controller. B3 markers consumed: hybrid_cache_controller.py, metrics_collector.py (entry-219 producer detail). sgl-project#46954-IN-RANGE VERDICT + dedupe decision table (ledger entry 290 is the spec): - sgl-project#46954 itself NEVER MERGED upstream (only our port b6e7ed7cb2 exists, on the unlanded port-hicache-ctrl branch); what arrived in-range is sgl-project#41187 (c677171), whose _sync_prefetch_hit_query re-implements the mask reduce but COLLAPSES TO THE DEEPEST COMMON ENDPOINT. Per-hunk: (a) allocate-time endpoint choice -> RE-PLANTED (_trailing_endpoint_tokens: deepest VERIFIED endpoint <= alloc, decline-with-report when none; the entry-290(a) "BOTH OURS' ADDITIONS ARE KEPT": sidecar_hash_values pin guard + throttled _report_no_restorable_endpoint; the indices_from_pool IS None gate delta stays a documented non-union). (b) retain-common-endpoint-set -> OURS KEPT per the ledger; upstream's collapse loses shallower endpoints the host-capacity clamp needs. UNION REFINEMENT recorded here: the mask is packed only when the span has a TRAILING_PAGES transfer (trailing-ness is rank-invariant config, so shape consistency holds), which keeps the non-trailing collective BIT-IDENTICAL to upstream's 1+len(PoolName) payload and collapses the upstream PP-test breakage from 18F to the one genuinely trailing case. Fork layout kept where it matters: [folded] + verdicts + mask, page-0-always-zero (no sentinel), candidates NOT capacity-capped (the PP leader caps the folded entry itself before sync). (c) assume_stored endpoint-only proof -> RE-PLANTED ([kv_hit_pages] set for trailing spans, None otherwise; replaces the pre-sgl-project#46954 blank). (d) aux-staging sizing to the ACTUAL endpoint -> OWED B3-3: the producer lives in unified_radix_cache._alloc_prefetch_aux_staging (the single remaining sparse_allocation 1F is exactly this test). - 217/218 rider semantics re-planted with the patch: restorable_prefix_pages operation field (both placements written: op attr for the clamp/mask, pool_storage_result for the upstream drain consumers), host_capacity_* retry fields + deadline_expired (producer side; the policy consumer rides B3-3 URC), entry-199 write(node_ids=...) ack list, entry-90 shortfall-only evict_host in allocate_storage_hit, sgl-project#46707 source_len SUM (multi-destination Mamba load-backs), sgl-project#43110 merge_source_transfers at BOTH derivation sites, sgl-project#39830 backup-thread survival comment, and the 3fa5197 reporting trio (_report_sidecar / _report_sidecar_endpoint_misses / _report_sidecar_write_shortfalls + class-level tallies for the /metrics hook). local_restorable_mask stays the merged-in module function (already byte-faithful to the patch). - cache_controller.py fence VERIFIED, not re-planted (DRIVER:MANUAL held it): load_fence_stream/write_fence_stream live at :381-382/:997-1018/:1157-1162 and test_hicache_staged_write_back_dispatch is 27P/1S/8sub. Entry-219 PRODUCER-DICT closure (the narrowed marker, D3): sgl-project#43727 -- the entry-189 PR -- LANDED UPSTREAM in-range (26a1e51), re-arriving as hicache_host_mamba_{used,total}_states SchedulerStats fields, a reporter entry_map read-back, and a TestHiCacheHostMambaStats class (the DI test file was a union concatenation of BOTH class generations with a mid-file __main__ guard). Entry 219's ledger is authoritative: the per-pool hicache_host_pool_{used,total}_tokens{pool} set is a strict superset, the mamba-states layer was REMOVED and NO duplicate registration survives. Deleted the gauge-less fields + reporter block (the arrival exported nothing: log_stats never touched the fields), dropped the now-unused PoolName/UnifiedRadixCache imports (entry-219's own wording), removed the resurrected guard class, moved the stray __main__ guard to EOF, and re-wrote hicache_best_practices.mdx's watch sentence onto the per-pool schema (B1 owns that file's merge resolution; this is a one-line schema follow-up). The {pool: -> dict} producer wiring (collect_host_pool_stats + derived_pools sidecar exclusion) was already complete at the merge -- confirmed, not duplicated. ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after): test_hicache_trailing_endpoint 8P/9F -> 17P/0F ** GATE ** test_pp_prefetch_ticket 17P/0F* -> 17P/0F/19sub (*base green; 18F transient mid-replant, retargeted per entry-290 precedent: vote-shape assertion moved to OUR payload, every allocation/query/drain expectation KEPT) test_hybrid_storage_sparse_allocation 8P/13F -> 12P/1F (the 1F = entry-290(d) aux-staging sizing, URC producer, B3-3) test_buffer_mode_sidecar -> 6P/0F/6sub test_hicache_mamba_slot_side_states -> 5P/0F test_storage_prefetch_lifecycle -> 16P/0F/13sub test_unified_hicache_transfer_lifecycle 4P/4F -> 4P/4F (all 4F = cargo-build ENV class, documented byte-identical pre-merge) unit/observability test_stat_loggers_di + test_hicache_pool_stats: 7F -> 30P/0F Lint: ruff format+F821/E9/F401 clean on all touched files; import OK. No GPU executed; no push; tree otherwise untouched.
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…t) SESSION 2A COMPLETE Entry 311 records: the 9 cluster commits (2a5646b..1dc41b2) with per- cluster batteries; the sgl-project#46954 VERDICT + hunk-by-hunk DEDUPE TABLE (sgl-project#46954 itself NEVER MERGED upstream -- sgl-project#41187's collapsed-mask re-implementation is what arrived; (a) OURS re-planted, (b) OURS kept with the recorded UNION REFINEMENT -- mask packs only for TRAILING spans, non-trailing collectives BIT-IDENTICAL to upstream's shape, (c) assume_stored endpoint-proof re-planted, (d) aux-staging alloc_len sizing re-planted = the prod-highest- value hunk); the entry-219 PRODUCER closure (sgl-project#43727 arrived in-range as 26a1e51, the gauge-less layer DELETED per entry 219's ledger, per-pool serving schema CONFIRMED as the only producer, docs watch-sentence re-pointed); the entry-196 rust ADAPTER-CONTRACT repairs (mamba_value_seqlen kwarg + parent_node_id rename -- both live TypeErrors at the merged tip); the boot-integrity lossless-ring planting (CPU-PROVEN construction; every caller landed with THEIRS while the pool signatures were the dropped half); the 12 dropped storage_prefetch_lifecycle cases RE-HOMED (28P/31sub >= the 23P/28sub pin, all green unmodified against the B3-3 machinery). GATES: ** trailing_endpoint 17P/0F PROVEN at the final tree incl. the 2-rank gloo cases, and at every cluster checkpoint after B3-2 ** (the entry-209 DoD gate); the B3-2/B3-caused async_ready_sync collection error CLEARED (7P); ZERO non-environment failures remain in unit/mem_cache. NEW PIN (per-file, CVD=9, prod venv, new post-sgl-project#45232/sgl-project#45236 layout): unit/mem_cache 1753P/1351F/364S/4439sub/4err -> ** 1861P/1209F/364S/4514sub/ 2err **; the 1209F residue is 100% env class by per-file error taxonomy: 1056F = the upstream URC-split KITS' fixtures.py build_fixture get_device() "No accelerator" -- ENTRY 310's "the URC split files themselves are GREEN, honest skips" CLAIM IS CORRECTED IN PLACE (its own battery log shows core 27P/465F; the kits honest-skip DEFECT is B2 scope, recorded not fixed) -- 130F+2err = cargo/torch-sys (byte-identical pre-merge, the documented environment class incl. the draft_gate 4F / joint-allocation 47F / tri-reclaim 54F storm files reclassified EXPECTED-REPLANT->ENV), 23F = the No-CUDA class (kv_layer_router 3F + kv_storage_lifecycle 9F + kv_layer_access 5F + pricing_property 4F + decode_retraction 2F). Named-file pins recorded (trailing_endpoint 17P, fast_file 54P/13sub, sparse 13P/14sub EXACT, donated 17P/14sub >= 16P, fence 15P EXACT, lifecycle 28P/31sub >= 23P/28sub, ...). CAVEATS: the merge-era top bullet ANNOTATED (B3 landed, gate met, boot ban STAYS -- the rule's HiCache half is proven but sessions 2B/3 remain and this batch NEVER booted, per the era rule); the entry-310 damage-map mem_cache and trailing_endpoint rows annotated RESOLVED-AT-B3 with the F-collapse table; the synced-block INTEGRATION OWED line annotated B3 LANDED; marker census correction noted (tree carried 15 B3 markers, not 12; ALL consumed -- B2/B4/ B5/B6 diff-audited INTACT). Header: STACK -> qwennext-stack-311+merge5-b3-2026-10-29; +merge5-b3 wave tag; entries 1-311; tip -> 1dc41b2 (B3-9 = true code tip, docs above). wanted_stack: UNTOUCHED (no S-item changed by B3 findings -- recorded here). Debt list carried in the entry: rust cargo UNVERIFIED 932P pin (session-3 seat), entry-198 scheduler-side CATCHER cross-noted for B4, kits get_device skip defect for B2, NVFP4 dequant-workspace deliberately NOT restored (superseded). NOTHING PUSHED; tree clean; no GPU executed all session.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A hybrid L3 hit can be restorable only at sparse endpoints because it needs both a contiguous KV prefix and a trailing state/window snapshot. A smaller prefix is not necessarily another valid hit.
For example, a query can find complete checkpoints at pages 16 and 32. If host staging has room for only 24 pages, cache-mode partial allocation currently clips the hit to page 24. The subsequent trailing-pool read then targets a checkpoint that the query did not establish. The existing completeness check correctly rejects an incomplete restore, but the usable 16-page checkpoint is lost and KV reads can be wasted.
Modifications
assume_storedtrailing hit as proof of its original endpoint only; it cannot establish intermediate checkpoints without a query. Plain KV retains its contiguous-prefix fast path.This preserves the all-pools completeness check, existing allocation ownership, and the current collective sequence. Full-hit allocation and
allow_partial=Falseretain their existing behavior. Buffer-only head trimming continues to preserve the original sidecar endpoint; PP keeps its existing capacity-capped checkpoint vote. No backend/storage format or serving configuration changes are included.Accuracy Tests
Local CPU regression control on Python 3.12.16 / PyTorch 2.14.1:
faa56e2b5c5c5dee60eb5dd7b6251944a21a6104: 8 distinct regression methods fail with the same tests and adapter; the other 6 controls pass. These are assertion failures on endpoint selection/retention or staging behavior, rather than import failures.The local macOS CPU process uses the repository's complete
_platform_stubs.pyandmaybe_stub_sgl_kernel()for GPU-only imports. The MPS availability override is limited to compatibility installation and is restored before importing/running the targets; tensors and assertions run on CPU. This adapter is not part of the patch. New query tests mock the unrelated Linux-only native hash boundary. Target controller/cache methods are imported directly, not copied or AST-extracted.In a configured Linux development environment, the added tests can be run with:
The tests directly import the production controller/cache methods and use CPU tensors. Host capacity, storage queries, and collective boundaries are controlled test inputs; they do not reproduce a real allocator-pressure history or a multi-process Store deployment.
GPU/model accuracy, the complete CPU suite, and a full Linux integration run have not been performed. No available GPU is required for the added unit tests.
Speed Tests and Profiling
No end-to-end speedup is claimed. This fixes checkpoint selection under partial host capacity; serving performance and GPU L3-to-L1 validation remain unmeasured for this patch.
Checklist
base-a-test-cpu.AI assistance was used to prepare the patch and tests.
CI States
Latest PR Test (Base): ❌ Run #40640081762
Latest PR Test (Extra): ❌ Run #40640081524
Latest PR Test (AMD ROCm 10): ❌ Run #40640081521