Skip to content

[Perf] hicache: bound full-hit staging waits - #46451

Open
donghyq wants to merge 1 commit into
sgl-project:mainfrom
donghyq:codex/hicache-bounded-storage-hit-wait
Open

donghyq wants to merge 1 commit into
sgl-project:mainfrom
donghyq:codex/hicache-bounded-storage-hit-wait

Conversation

@donghyq

@donghyq donghyq commented Oct 25, 2026 •

Copy link
Copy Markdown

Motivation

Supersedes #40361.

An L3 hit can be available in storage while the host pool temporarily lacks room to stage it. Add an opt-in wait for full staging so short-lived pressure can clear before the request falls back to recomputation.

Modifications

  • Add --hicache-storage-prefetch-host-capacity-wait-ms (default 0). When enabled, retry full-span allocations until the deadline or retry cap is reached. The policy applies to single-engine cache mode.
  • Fix cancellation and expiry handling so each pending operation is submitted once, and stale notifications cannot revoke its replacement.
  • Let smaller hits proceed while a larger hit waits for capacity.
  • Keep partial staging as the default and preserve buffer-only and distributed behavior.

Accuracy Tests

  • test/registered/unit/mem_cache/test_storage_prefetch_lifecycle.py: 24 passed, including eight new regressions for cancellation, request reuse, stale notifications, allocation order and timeout cleanup.
  • test/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py: 24 passed, 1 skipped.
  • CLI help, changed-file pre-commit and repository Lint passed.
  • All 45 benchmark runs matched the reference's 16 generated token IDs and logprobs. No operation obtained staging twice.

Speed Tests and Profiling

One H20-3e, TP1, Qwen3.5-35B-A3B, Triton attention/MoE, page_first_direct, private TCP Mooncake. Replay a stored 16,384-token prefix with one additional input token and 16 output tokens, alongside a concurrent short request.

Host-pool reservations leave 4,096 tokens available for staging. For transient pressure, release the reservation on the first scheduler drain after 25 ms; for sustained pressure, hold it until cleanup. Bounded wait uses 300 ms and eight allocation retries. The immediate-miss control uses zero allocation retries, with the same storage-miss polling.

Long-request TTFT medians from five interleaved repetitions per case after warmup:

Host capacity Partial staging (default) Immediate full-span miss Bounded full-span wait
Unconstrained 247.8 ms 249.7 ms 249.6 ms
Transient pressure 1445.6 ms 1413.7 ms 304.9 ms
Sustained pressure 1425.4 ms 1343.1 ms 1680.6 ms

Waiting reduces TTFT by 78.9% under transient pressure and increases it by 17.9% under sustained pressure, so it remains opt-in. Concurrent short-request TTFT under transient pressure changes from 232.4 ms to 143.0 ms.

On this hybrid model, partial staging reads 80 MiB without yielding a usable cached prefix; full restore reads 381.4 MiB across KV and Mamba state.

Benchmark runtime: Torch 2.13.0+cu129 and sglang-kernel 0.4.6.post1+cu129, with kernel version checking disabled.

Checklist

  • Format changed code with pre-commit.
  • Add and run relevant unit tests.
  • Update configuration documentation.
  • Provide accuracy and speed results.
  • Follow the SGLang code style.

Review and Merge Process

Repository test CI is waiting for the run-ci label.


CI States

Latest PR Test (Base): ❌ Run #40212919560
Latest PR Test (Extra): ❌ Run #40212919332
Latest PR Test (AMD ROCm 10): ❌ Run #40212919344

Preserve default partial staging, bound opt-in full-span waits, and protect pending operations from cancellation and stale callbacks.

Signed-off-by: donghanyu <donghanyu@meituan.com>
@mintlify

mintlify Bot commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 9e61b77.

Approve preview

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache labels Oct 25, 2026
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
New --hicache-storage-prefetch-host-capacity-wait-ms (default 0.0, so the
landed partial-staging behavior is byte-identical when unset). In
single-engine CACHE mode a positive value makes a capacity-shortened full
hit WAIT instead of declining: _try_alloc_storage_hit parks the operation
in the new pending_storage_hits, retries each drain pass up to the bound
(deadline + pacing from the same retry-max-attempts knob), and only then
falls through to the existing host_capacity decline. allow_partial is
inverted for those waits (full span or park), so the entry-209
endpoint-clamp machinery is untouched in the default mode and the
buffer-only / TP / PP / CP paths keep their current behavior
(enable_cache_mode_storage_hit_pending).

Cancellation/expiry hardening riding along:
- the hit-queue drain and _try_alloc_storage_hit now check
  info.operation is not operation, so a stale query callback from a
  revoked attempt can neither retire nor re-account a resubmitted op
  (this is the guard our entry-209 decline-and-retry loop needed: the
  retry re-issue and its late predecessor no longer collide);
- an already-staged cache-mode op returns early instead of allocating a
  second bounce buffer.

Interaction with sgl-project#44614's deadline anchor (entry 217, landed): the new
wait deadline is per-operation (a fresh host_capacity_deadline on the
first blocked alloc); on expiry the op takes the SAME
_finish_storage_prefetch(host_capacity) + revoke + poll_miss path, whose
retry budget stays anchored on the first blocked admission poll via
PollTimer. The two counters (operation.host_capacity_retries vs
StoragePrefetchRetries attempts) share the
--hicache-storage-prefetch-retry-max-attempts cap. The timeout prefetch
stop policy remains the earlier exit, as documented. This replaces the
ad-hoc half of entry-209's host_capacity decline+requeue when opted in.

Union notes: unified_radix_cache.py hunks applied over the landed
entry-209/217/218/199 region (the poll_miss anchor lines inside the
decline branch preserved); hybrid_cache_controller.py gains three
PrefetchOperation fields only (no collision with the sparse-checkpoint
regions); the lifecycle test hunk was anchored manually (upstream base
carries a TestWaitingStorageSpan class this tree does not have, so the
202-line block was inserted at the same logical spot and both new imports
retained; est_time stays 20 per our tree). Batteries:
test_storage_prefetch_lifecycle 23P/28sub (was 15P/24sub; +8 all green),
trailing_endpoint/buffer_mode_sidecar/sparse_allocation/urc/rust-urc 80P
unchanged, server_args 430P/225sub, flag default+parse verified.

(cherry picked from commit 9e61b77)
PR: sgl-project#46451 (open, ported)
(cherry picked from commit ff8bf1fe764856ad80ec95b1162251357ccba55d)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
check_hicache_events runs a blocking gloo MIN-all-reduce of the
write-back/load-back ready counts every scheduler step; on TP2 short-decode
steps that CPU collective is a large share of the step. With the new flag
(default False) the same reduction is issued asynchronously on its own
gloo group in step N and consumed in step N+1: acks complete in queue
order and queues only grow at the tail, so the agreed ready prefix stays
valid one step later, and _deferred_consume_count re-finds the agreed
last-ready ack by identity so a blocking drain in between is never
double-consumed. A deferred digest pair keeps the write_back
duplicate-reclaim victim-order assertion across ranks. Runtime storage
attach settles the in-flight sync and falls back to synchronous for good
(the deferred path skips the storage-queue drain). reset() drains the
pending op.

Union notes: lands after the sgl-project#46451 port in unified_radix_cache.py — the
insertion sits between flush_pending_backups() (entry-199 batched-backup
drain; its SGLANG_ENABLE_HICACHE_BATCHED_BACKUP default-OFF semantics are
untouched: the new flag is an orthogonal memory-group bool, flag-off is
the unchanged synchronous path) and the four-tuple
_sync_hicache_ready_counts call our entry-217/218 region built on. The
deferred gate refuses storage/buffer_only/PP/attn-CP/single-rank setups
with a warning, matching the landed _sync_hicache_ready_counts payload
(storage-queue sizes stay on the sync path). Ports the full head
(29e8125 + 39a0c0d incl. the runtime-attach refactor + docs); the
244-line CPU test suite (7P) drives two simulated ranks through the real
methods. Docs rows land next to the entry-219 mamba-sizing note.
Fork-stub adaptation: our test_hicache_pp_sync_drain __new__-built stubs
now pre-memoize _hcs_state=False (same as upstream's own async harness
pre-setting the state) so the per-step gate never resolves the memory bag
these stubs never publish; 4P.

(cherry picked from commit 6ffe9cd)
PR: sgl-project#45795 (open, ported)
(cherry picked from commit b357ce2961bdfa723091e0b3bd44b989243756d8)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
…),199,217,218,219,253,290d,293,294

Re-plants the fork-hunks patch (1013 lines, the deepest file) onto the
#46970x-rewritten URC. B3 marker consumed in unified_radix_cache.py.

What the merged tree had ALREADY absorbed (deduped, not duplicated — per-hunk
verdicts recorded): the whole deadline/stop-policy family (_StoragePrefetchWait
msgspec/PollTimer, _start_storage_prefetch_wait, _prefetch_deadline_expired
REPLACING _prefetch_timeout_check_linear_func, _mark_prefetch_deadline_expired,
the reason="deadline" bookkeeping, belief-heal suppression on deadline, the
_invalidate_absent_from_hit_query move to the drain head + KV-verdict guard +
deadline gate, the release_aborted_request / finish_storage_prefetch_admission
wait pops, and the _should_backup_after_insert write-back half of sgl-project#44800
(unified_tree_core.py:1447). The sgl-project#45703 short-full-match seam did NOT
obsolete entry 209's clamp (it lives controller-side; B3-2 gate 17P).

Re-planted here:
- entry 294 (sgl-project#45795): the FULL deferred ready-count sync family
  (_deferred_ready_sync_group gate ladder incl. the runtime-attach fallback,
  _check_hicache_events_deferred one-op-per-step on its own gloo group,
  _consume_deferred_ready_sync with the digest MIN/-MAX assert,
  _deferred_consume_count, _drop_deferred_ready_sync) + the check_hicache_events
  branch + the reset() settle. This CLEARS the test_hicache_async_ready_sync
  collection error (the B3-caused 1err in the merge gates).
- entries 199+218: the batched write-through backup family VERBATIM from the
  patch (_NodeBackupSpec, _queue_write_through_backup, _drain_queued_backups,
  _ancestors_first incl. the sgl-project#40363 delta walked-set, _is_unbacked_node,
  _parent_backed_up, _release_queued_backup, _backup_queued_batch,
  _build_node_backup_spec, _backup_specs, _merge_backup_transfers, the
  node_ids= ack-list through _execute_kv_backup into the controller write
  (B3-2), the flush_pending_backups guard, and the writing_check(write_back)
  drain + start_writing(wait_for_forward=True) — the ONE-fence ordering rule).
  _batched_backup reads SGLANG_ENABLE_HICACHE_BATCHED_BACKUP, DEFAULT FALSE
  (upstream's 0ab8654 default-ON stays NOT taken, entry-218's record).
- entry 293 (sgl-project#46451): the opt-in host-capacity WAIT: fields + init_hicache
  reads with the nonnegative-finite ValueError, wait_full, the retry cadence
  (host_capacity_deadline/next_retry/retries — fields planted on the operation
  in B3-2), allow_partial=not wait_full, pending_storage_hits park/drain with
  identity snapshots, the info.operation-is-not-operation late-query guards,
  and the revoke/alloc-already-owned guards. DEFAULT 0.0 = byte-identical.
- entry 290(d): _alloc_prefetch_aux_staging now sizes to alloc_len, the
  endpoint the KV span ACTUALLY got (mamba-host-full prod boots stop declining
  prefetches a smaller endpoint would fit).
- entry 219 producer: _record_host_pool_evictions on evict_host (before
  _apply_eviction_result) feeding hicache_host_pool_evicted_tokens_total{pool}.
- entry 292 (sgl-project#43110): _build_sidecar_transfers resolves derived pools through
  merge_source_transfers (every destination, not matching_sources[0]; the two
  AssertionErrors it subsumes are structurally guaranteed by the resolver).
- entry 30 rider: load_back unwinds (drop both locks, report miss) when any
  component PREPARED-REJECTS (the sgl-project#39830 poison/degrade contract the mamba
  component sets -- component side rides B3-5's mamba.py cluster).
- entry 253: the per-call-site file_storage_path fold DELETED here (it moved
  into _generate_storage_config, every attach path, B3-1-verified).
- test_chunked_publish_release_disjointness.py (entry 307's tripwire):
  re-anchored CacheInitParams(tree_components=...) -> tree_slots=tree_slots_of(...)
  per the upstream slots refactor; the tree_core_backend='rust' SUBFAILED line
  is the documented cargo/torch-sys ENV class (fails identically at base).

ACTUALS (CVD=9, prod venv, PYTHONPATH=python, per-file, damage -> after):
  test_hicache_async_ready_sync          1err -> 7P/0F   ** B3-caused error CLEARED **
  test_hybrid_storage_sparse_allocation  12P/1F -> 13P/0F/14sub  ** entry-290(d) closed **
  test_storage_prefetch_lifecycle                -> 16P/0F/13sub
  test_hicache_trailing_endpoint (gate)          -> 17P/0F (unmoved by this cluster; controller-side B3-2)
  test_buffer_mode_sidecar                        -> 6P/0F/6sub
  test_chunked_publish_release_disjointness 4P/2F -> 4P+1sub/1F (the F now the rust-cargo ENV class; the python-backend half RE-ANCHORED green)
  test_mamba_path_state_cap                      -> 5P/5S (skips = honest, env)
  test_decode_retraction_backup             2P/2F -> 2P/2F (both F = No-CUDA ENV class, byte-identical at base; file is B2's per the resolution log)
  test_hicache_mamba_sizing                      -> 30P/0F/17sub (unmoved)
  URC topic dir (unified_radix_cache/*.py): 1056F/40P/199S -> 1056F/40P/199S
    UNMOVED — 100% of the errors are 'No accelerator ... available' from the
    upstream kits' fixtures.py build_fixture calling get_device() (entry 310's
    "GREEN, honest skips" claim does NOT reproduce at this seat; the battery's
    own per-file numbers (core 27P/465F etc.) match the damage totals exactly;
    this is a kit no-CUDA skip DEFECT = B2/kit scope, recorded not fixed here).
Lint: ruff format + F821/E9 clean on both files; py_compile + import OK.
No GPU executed; no push.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant