Repository navigation
Conversation
Contributor
|
Thanks @donghyq. Closing this because it is still a draft and has not been updated in 29 days. Reopen it if the work is still relevant. Some directories moved recently, so an older branch may need retargeting: |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #46451, which is open and ready for review. GitHub rejected reopening this closed PR; the updated implementation, regression tests and H20 results are being reviewed in #46451. The original diff is retained here for history.
Motivation
In single-engine cache mode, short-lived host-pool pressure can prevent an otherwise reusable storage-hit span from being staged completely. Add an opt-in, bounded wait for complete staging while keeping the existing partial-prefix policy as the default.
Pending operations must also survive cancellation safely: revoking one operation must not submit its successor twice, and a late query notification from an old attempt must not revoke or change the accounting of its replacement.
Modifications
--hicache-storage-prefetch-host-capacity-wait-ms(default0). Positive values enable full-span staging with a per-hit wall-clock deadline and the existing allocation retry cap. Reaching either cap ends that prefetch operation and permits recomputation; the existing stop/admission policy still applies.Accuracy Tests
test/registered/unit/mem_cache/test_storage_prefetch_lifecycle.py: 24 passed, including eight new cases.test/registered/unit/mem_cache/test_hicache_staged_write_back_dispatch.py: 24 passed, 1 skipped.unist-util-visit-parents@^6.1.0; documentation CI remains pending.The cache/controller and lifecycle-test source bytes used for these results are unchanged by the final main refresh. Standard repository CI is still required before merge.
Speed Tests and Profiling
Controlled experiment: one NVIDIA H20-3e, TP1, Qwen3.5-35B-A3B, Triton attention/MoE, direct IO with
page_first_direct, and an isolated TCP Mooncake instance. A stored 16,384-token prefix is replayed with one additional input token and 16 output tokens, alongside one concurrent short request. Each case has five interleaved measured repetitions after a warm round.Pressure is imposed by real host-pool reservations, leaving 4,096 staging tokens available. In the transient case the reservation becomes releasable after 25 ms and is freed on the next scheduler drain; this is not a fixed 25 ms actual-release guarantee. The bounded policy uses a 300 ms per-hit wait and eight allocation retries. The immediate full-miss control has zero allocation retries, while storage-miss polling is held constant.
Long-request TTFT medians:
In this controlled transient case the bounded policy reduces long-request TTFT by 78.9% versus partial staging. The concurrent short-request median changes from 232.4 ms to 143.0 ms. Under sustained pressure, long-request TTFT increases by 17.9% versus partial staging, supporting the opt-in default.
A full restore is credited as
storage=16384, device=0, host=0and reads 381.4 MiB across KV and Mamba state. On this hybrid model the 4,096-token partial transfer reads 80 MiB but is not credited as a usable cached prefix. These results do not establish the same gain for ordinary full-attention models, remote/RDMA storage, production traffic, or throughput/SLO improvements.Runtime boundary: the H20 experiment reuses an existing Torch 2.13.0+cu129 / sglang-kernel 0.4.6.post1+cu129 environment with
SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK=1. It exercises the candidate cache code against base30dc2f94303e339431cbd51489b2d1719678b0c1; it does not validate the project's standard CUDA13 dependency stack. The final main refresh does not change the tested cache/controller source bytes.Checklist
Review and Merge Process
Maintainer review and repository CI, including the standard dependency environment, remain required before merge. No merge or production deployment is requested by this PR.
CI States
Latest PR Test (Base): ❌ Run #35453198814
Latest PR Test (Extra): ❌ Run #35453198683
Latest PR Test (AMD ROCm 10): ❌ Run #35453198802