Skip to content

[Perf] hicache: bound full-hit staging waits (superseded by #46451) - #40361

Closed
donghyq wants to merge 2 commits into
sgl-project:mainfrom
donghyq:codex/hicache-cache-mode-pending-storage-hit
Closed

donghyq wants to merge 2 commits into
sgl-project:mainfrom
donghyq:codex/hicache-cache-mode-pending-storage-hit

Conversation

@donghyq

@donghyq donghyq commented Sep 19, 2026 •

Copy link
Copy Markdown

Superseded by #46451, which is open and ready for review. GitHub rejected reopening this closed PR; the updated implementation, regression tests and H20 results are being reviewed in #46451. The original diff is retained here for history.

Motivation

In single-engine cache mode, short-lived host-pool pressure can prevent an otherwise reusable storage-hit span from being staged completely. Add an opt-in, bounded wait for complete staging while keeping the existing partial-prefix policy as the default.

Pending operations must also survive cancellation safely: revoking one operation must not submit its successor twice, and a late query notification from an old attempt must not revoke or change the accounting of its replacement.

Modifications

  • Track parked cache-mode hits by request handle and validate operation identity before processing hit notifications or allocating staging.
  • Remove revoked hits in place and continue checking smaller hits when a larger span cannot fit.
  • Add --hicache-storage-prefetch-host-capacity-wait-ms (default 0). Positive values enable full-span staging with a per-hit wall-clock deadline and the existing allocation retry cap. Reaching either cap ends that prefetch operation and permits recomputation; the existing stop/admission policy still applies.
  • Keep partial staging as the default. The new wait policy is disabled for buffer-only mode and TP/PP/CP groups larger than one.
  • Add eight CPU regression cases using real revoke, allocation and accounting paths, covering expired heads, cancellation at different positions, terminated operations, request-ID reuse, late replaced attempts, smaller-request progress, and deadline cleanup.
  • Document the new flag and its sustained-pressure tradeoff.

Accuracy Tests

  • test/registered/unit/mem_cache/test_storage_prefetch_lifecycle.py: 24 passed, including eight new cases.
  • test/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py: 24 passed, 1 skipped.
  • CLI help recognizes the new option; changed-file pre-commit checks passed.
  • Full Mintlify validation could not start because installing the validator failed to resolve unist-util-visit-parents@^6.1.0; documentation CI remains pending.
  • The old pending-queue implementation reproduced two transfers for the same successor after its predecessor was revoked. The late-attempt regression also failed before the identity guard and passed afterward.
  • Controlled native H20 comparison: 45 measured runs, each with 16 greedy output tokens. All generated token IDs and token logprobs exactly matched the full-restore reference. No operation obtained staging twice.

The cache/controller and lifecycle-test source bytes used for these results are unchanged by the final main refresh. Standard repository CI is still required before merge.

Speed Tests and Profiling

Controlled experiment: one NVIDIA H20-3e, TP1, Qwen3.5-35B-A3B, Triton attention/MoE, direct IO with page_first_direct, and an isolated TCP Mooncake instance. A stored 16,384-token prefix is replayed with one additional input token and 16 output tokens, alongside one concurrent short request. Each case has five interleaved measured repetitions after a warm round.

Pressure is imposed by real host-pool reservations, leaving 4,096 staging tokens available. In the transient case the reservation becomes releasable after 25 ms and is freed on the next scheduler drain; this is not a fixed 25 ms actual-release guarantee. The bounded policy uses a 300 ms per-hit wait and eight allocation retries. The immediate full-miss control has zero allocation retries, while storage-miss polling is held constant.

Long-request TTFT medians:

Host capacity Partial staging (default) Immediate full-span miss Bounded full-span wait
Unconstrained 247.8 ms 249.7 ms 249.6 ms
Transient pressure 1445.6 ms 1413.7 ms 304.9 ms
Sustained pressure 1425.4 ms 1343.1 ms 1680.6 ms

In this controlled transient case the bounded policy reduces long-request TTFT by 78.9% versus partial staging. The concurrent short-request median changes from 232.4 ms to 143.0 ms. Under sustained pressure, long-request TTFT increases by 17.9% versus partial staging, supporting the opt-in default.

A full restore is credited as storage=16384, device=0, host=0 and reads 381.4 MiB across KV and Mamba state. On this hybrid model the 4,096-token partial transfer reads 80 MiB but is not credited as a usable cached prefix. These results do not establish the same gain for ordinary full-attention models, remote/RDMA storage, production traffic, or throughput/SLO improvements.

Runtime boundary: the H20 experiment reuses an existing Torch 2.13.0+cu129 / sglang-kernel 0.4.6.post1+cu129 environment with SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1. It exercises the candidate cache code against base 30dc2f94303e339431cbd51489b2d1719678b0c1; it does not validate the project's standard CUDA13 dependency stack. The final main refresh does not change the tested cache/controller source bytes.

Checklist

  • Format changed code with pre-commit.
  • Add and run relevant unit tests.
  • Update user-facing configuration documentation.
  • Provide configuration-specific correctness and speed results with tradeoffs and limits.
  • Follow the SGLang code style.

Review and Merge Process

Maintainer review and repository CI, including the standard dependency environment, remain required before merge. No merge or production deployment is requested by this PR.


CI States

Latest PR Test (Base): ❌ Run #35453198814
Latest PR Test (Extra): ❌ Run #35453198683
Latest PR Test (AMD ROCm 10): ❌ Run #35453198802

@github-actions github-actions Bot added hicache Hierarchical Caching for SGLang unified-radix-cache labels Sep 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks @donghyq. Closing this because it is still a draft and has not been updated in 29 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Oct 19, 2026
@donghyq donghyq changed the title [HiCache] Retry parked cache-mode storage hits on host capacity [Perf] hicache: bound full-hit staging waits (superseded by #46451) Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant