Repository navigation
Conversation
Preserve default partial staging, bound opt-in full-span waits, and protect pending operations from cancellation and stale callbacks. Signed-off-by: donghanyu <donghanyu@meituan.com>
donghyq
requested review from
JustinTong0323,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
sogalin,
wisclmy0611,
xiezhq-hermann,
yizhang2077 and
zijiexia
as code owners
October 25, 2026 16:41
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
5 tasks done
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
New --hicache-storage-prefetch-host-capacity-wait-ms (default 0.0, so the landed partial-staging behavior is byte-identical when unset). In single-engine CACHE mode a positive value makes a capacity-shortened full hit WAIT instead of declining: _try_alloc_storage_hit parks the operation in the new pending_storage_hits, retries each drain pass up to the bound (deadline + pacing from the same retry-max-attempts knob), and only then falls through to the existing host_capacity decline. allow_partial is inverted for those waits (full span or park), so the entry-209 endpoint-clamp machinery is untouched in the default mode and the buffer-only / TP / PP / CP paths keep their current behavior (enable_cache_mode_storage_hit_pending). Cancellation/expiry hardening riding along: - the hit-queue drain and _try_alloc_storage_hit now check info.operation is not operation, so a stale query callback from a revoked attempt can neither retire nor re-account a resubmitted op (this is the guard our entry-209 decline-and-retry loop needed: the retry re-issue and its late predecessor no longer collide); - an already-staged cache-mode op returns early instead of allocating a second bounce buffer. Interaction with sgl-project#44614's deadline anchor (entry 217, landed): the new wait deadline is per-operation (a fresh host_capacity_deadline on the first blocked alloc); on expiry the op takes the SAME _finish_storage_prefetch(host_capacity) + revoke + poll_miss path, whose retry budget stays anchored on the first blocked admission poll via PollTimer. The two counters (operation.host_capacity_retries vs StoragePrefetchRetries attempts) share the --hicache-storage-prefetch-retry-max-attempts cap. The timeout prefetch stop policy remains the earlier exit, as documented. This replaces the ad-hoc half of entry-209's host_capacity decline+requeue when opted in. Union notes: unified_radix_cache.py hunks applied over the landed entry-209/217/218/199 region (the poll_miss anchor lines inside the decline branch preserved); hybrid_cache_controller.py gains three PrefetchOperation fields only (no collision with the sparse-checkpoint regions); the lifecycle test hunk was anchored manually (upstream base carries a TestWaitingStorageSpan class this tree does not have, so the 202-line block was inserted at the same logical spot and both new imports retained; est_time stays 20 per our tree). Batteries: test_storage_prefetch_lifecycle 23P/28sub (was 15P/24sub; +8 all green), trailing_endpoint/buffer_mode_sidecar/sparse_allocation/urc/rust-urc 80P unchanged, server_args 430P/225sub, flag default+parse verified. (cherry picked from commit 9e61b77) PR: sgl-project#46451 (open, ported) (cherry picked from commit ff8bf1fe764856ad80ec95b1162251357ccba55d)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
check_hicache_events runs a blocking gloo MIN-all-reduce of the write-back/load-back ready counts every scheduler step; on TP2 short-decode steps that CPU collective is a large share of the step. With the new flag (default False) the same reduction is issued asynchronously on its own gloo group in step N and consumed in step N+1: acks complete in queue order and queues only grow at the tail, so the agreed ready prefix stays valid one step later, and _deferred_consume_count re-finds the agreed last-ready ack by identity so a blocking drain in between is never double-consumed. A deferred digest pair keeps the write_back duplicate-reclaim victim-order assertion across ranks. Runtime storage attach settles the in-flight sync and falls back to synchronous for good (the deferred path skips the storage-queue drain). reset() drains the pending op. Union notes: lands after the sgl-project#46451 port in unified_radix_cache.py — the insertion sits between flush_pending_backups() (entry-199 batched-backup drain; its SGLANG_ENABLE_HICACHE_BATCHED_BACKUP default-OFF semantics are untouched: the new flag is an orthogonal memory-group bool, flag-off is the unchanged synchronous path) and the four-tuple _sync_hicache_ready_counts call our entry-217/218 region built on. The deferred gate refuses storage/buffer_only/PP/attn-CP/single-rank setups with a warning, matching the landed _sync_hicache_ready_counts payload (storage-queue sizes stay on the sync path). Ports the full head (29e8125 + 39a0c0d incl. the runtime-attach refactor + docs); the 244-line CPU test suite (7P) drives two simulated ranks through the real methods. Docs rows land next to the entry-219 mamba-sizing note. Fork-stub adaptation: our test_hicache_pp_sync_drain __new__-built stubs now pre-memoize _hcs_state=False (same as upstream's own async harness pre-setting the state) so the per-step gate never resolves the memory bag these stubs never publish; 4P. (cherry picked from commit 6ffe9cd) PR: sgl-project#45795 (open, ported) (cherry picked from commit b357ce2961bdfa723091e0b3bd44b989243756d8)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 29, 2026
…),199,217,218,219,253,290d,293,294 Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the #46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py. What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired, the reason="deadline" bookkeeping, belief-heal suppression on deadline, the _invalidate_absent_from_hit_query move to the drain head + KV-verdict guard + deadline gate, the release_aborted_request / finish_storage_prefetch_admission wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800 (unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P). Re-planted here: - entry 294 (sgl-project#45795): the FULL deferred ready-count sync family (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback, _check_hicache_events_deferred one-op-per-step on its own gloo group, _consume_deferred_ready_sync with the digest MIN/-MAX assert, _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync collection error (the B3-caused 1err in the merge gates). - entries 199+218: the batched write-through backup family VERBATIM from the patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups, _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node, _parent_backed_up, _release_queued_backup, _backup_queued_batch, _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the node_ids= ack-list through _execute_kv_backup into the controller write (B3-2), the flush_pending_backups guard, and the writing_check(write_back) drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule). _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record). - entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache reads with the nonnegative-finite ValueError, wait_full, the retry cadence (host_capacity_deadline/next_retry/retries — fields planted on the operation in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with identity snapshots, the info.operation-is-not-operation late-query guards, and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical. - entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining prefetches a smaller endpoint would fit). - entry 219 producer: _record_host_pool_evictions on evict_host (before _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}. - entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through merge_source_transfers (every destination, not matching_sources[0]; the two AssertionErrors it subsumes are structurally guaranteed by the resolver). - entry 30 rider: load_back unwinds (drop both locks, report miss) when any component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba component sets -- component side rides B3-5's mamba.py cluster). - entry 253: the per-call-site file_storage_path fold DELETED here (it moved into _generate_storage_config, every attach path, B3-1-verified). - test_chunked_publish_release_disjointness.py (entry 307's tripwire): re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...) per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line is the documented cargo/torch-sys ENV class (fails identically at base). ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after): test_hicache_async_ready_sync 1err -> 7P/0F ** B3-caused error CLEARED ** test_hybrid_storage_sparse_allocation 12P/1F -> 13P/0F/14sub ** entry-290(d) closed ** test_storage_prefetch_lifecycle -> 16P/0F/13sub test_hicache_trailing_endpoint (gate) -> 17P/0F (unmoved by this cluster; controller-side B3-2) test_buffer_mode_sidecar -> 6P/0F/6sub test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green) test_mamba_path_state_cap -> 5P/5S (skips = honest, env) test_decode_retraction_backup 2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log) test_hicache_mamba_sizing -> 30P/0F/17sub (unmoved) URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S UNMOVED — 100% of the errors are 'No accelerator ... available' from the upstream kits' fixtures.py build_fixture calling get_device() (entry 310's "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's own per-file numbers (core 27P/465F etc.) match the damage totals exactly; this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here). Lint: ruff format + F821/E9 clean on both files; py_compile + import OK. No GPU executed; no push.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Supersedes #40361.
An L3 hit can be available in storage while the host pool temporarily lacks room to stage it. Add an opt-in wait for full staging so short-lived pressure can clear before the request falls back to recomputation.
Modifications
--hicache-storage-prefetch-host-capacity-wait-ms(default0). When enabled, retry full-span allocations until the deadline or retry cap is reached. The policy applies to single-engine cache mode.Accuracy Tests
test/registered/unit/mem_cache/test_storage_prefetch_lifecycle.py: 24 passed, including eight new regressions for cancellation, request reuse, stale notifications, allocation order and timeout cleanup.test/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py: 24 passed, 1 skipped.Speed Tests and Profiling
One H20-3e, TP1, Qwen3.5-35B-A3B, Triton attention/MoE,
page_first_direct, private TCP Mooncake. Replay a stored 16,384-token prefix with one additional input token and 16 output tokens, alongside a concurrent short request.Host-pool reservations leave 4,096 tokens available for staging. For transient pressure, release the reservation on the first scheduler drain after 25 ms; for sustained pressure, hold it until cleanup. Bounded wait uses 300 ms and eight allocation retries. The immediate-miss control uses zero allocation retries, with the same storage-miss polling.
Long-request TTFT medians from five interleaved repetitions per case after warmup:
Waiting reduces TTFT by 78.9% under transient pressure and increases it by 17.9% under sustained pressure, so it remains opt-in. Concurrent short-request TTFT under transient pressure changes from 232.4 ms to 143.0 ms.
On this hybrid model, partial staging reads 80 MiB without yielding a usable cached prefix; full restore reads 381.4 MiB across KV and Mamba state.
Benchmark runtime: Torch 2.13.0+cu129 and sglang-kernel 0.4.6.post1+cu129, with kernel version checking disabled.
Checklist
Review and Merge Process
Repository test CI is waiting for the
run-cilabel.CI States
Latest PR Test (Base): ❌ Run #40212919560
Latest PR Test (Extra): ❌ Run #40212919332
Latest PR Test (AMD ROCm 10): ❌ Run #40212919344