Repository navigation
[Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes - #43548
Merged
Merged
Conversation
alphabetc1
requested review from
ByronHsu,
Duyi-Wang,
HaiShaw,
ShangmingCai,
hnyls2002 and
sogalin
as code owners
October 10, 2026 11:01
fivechenxi
pushed a commit
to fivechenxi/sglang
that referenced
this pull request
Oct 16, 2026
Adapt the logical storage_hit_end approach from upstream sgl-project#43548 to the pinned prefetch API. Keep bigram boundary IDs, honor declined prefetches before preallocation, and reject short restored coverage before enqueuing DMA. Add real CUDA/file round-trip regressions for boundary, partial L1, and short-restore cases.
ispobock
approved these changes
Oct 23, 2026
This was referenced Oct 24, 2026
Venkat2811
added a commit
to datacrunch-research/sglang
that referenced
this pull request
Oct 27, 2026
…e slots (upstream sgl-project#41695) Layer 12 of the Kimi-K3 nightly line on upstream main 5884e4f. The prefill/decode part of upstream sgl-project#41695 that 5884e4f does not have. A decode server with --disaggregation-decode-enable-radix-cache and HiCache restores the cached prefix locally while prefill sends the rest: - The transfer-side load-back handed the request to init_load_back, so for a hybrid (Mamba/KDA) model it also copied the host checkpoint at the prefix end into the request's own state slot. That slot receives the live state prefill computes for the whole prompt; whichever copy lands last wins. InitLoadBackParams.restore_request_state=False restores the canonical cache slot only. - A hybrid restore needs one canonical state slot besides the request's live slot. Admission keeps a slot free for each queued restore (ensure_mamba_capacity(..., reserved_slots=)) and promises prefill only the device-resident prefix when no slot can be reclaimed; before, the promise went out and the restore failed after prefill had skipped those tokens. - Under DCP, a decode server with an L3 backend requires --hicache-storage-prefetch-policy wait_complete (decode promises the probed L3 span before the read completes) and refuses --disaggregation-decode-enable-offload-kvcache, as sgl-project#41695 did. Checked at startup (resolve_hicache_dcp_compatibility) and on runtime attach, including the same-backend policy update (StorageAttachment.attach; HiCacheController.attach_storage_backend never sees the effective policy). Source: 91ecb13 on the kimi-k3-nightly-3c560089 line, which equals upstream sgl-project#41695's own commits over sgl-project#41694 (head ed85005). Dropped because 5884e4f has it: starting the L3 prefetch before preallocation and promising only the span it took (sgl-project#43548). The sgl-project#41693/sgl-project#41694 ports that 91ecb13 sat on are superseded by sgl-project#41187 (no P/D refusal, no decode-offload case), so the guards are re-expressed on upstream's structure and the tests are rewritten on 5884e4f's fixtures. No change for a decode server without the decode radix cache, which is every K3 recipe today. Tests (CPU), this commit's test files on the parent: test_server_args.py TestHiCacheArgs::test_dcp_decode_storage_requires_wait_complete_without_offload 3 subtests fail (NotImplementedError not raised for timeout, best_effort and decode offload); test_hicache_dcp_host_pool.py TestDcpDecodeStorageAttach 6 subtests fail (attach accepted an early-stopping policy, attached or not); test_mamba_capacity.py test_pd_restore_promise_keeps_a_canonical_state_slot 2 subtests fail (full and pending_restore promised 6 tokens, not 2). With this commit all pass. Tests (GPU, one B300): test_unified_radix_cache_load_back.py Test_FULL_MAMBA_ps1::test_pd_decode_restore_keeps_the_live_request_state fails on the parent (the live slot ends with the checkpoint's marker 11, not the live 37) and passes with this commit. Fork PR (v0.5.22 line): #19 Upstream: sgl-project#41695 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Venkat2811
added a commit
to datacrunch-research/sglang
that referenced
this pull request
Oct 27, 2026
…e slots (upstream sgl-project#41695) Layer 12 of the Kimi-K3 nightly line on upstream main 5884e4f. The prefill/decode part of upstream sgl-project#41695 that 5884e4f does not have. A decode server with --disaggregation-decode-enable-radix-cache and HiCache restores the cached prefix locally while prefill sends the rest: - The transfer-side load-back handed the request to init_load_back, so for a hybrid (Mamba/KDA) model it also copied the host checkpoint at the prefix end into the request's own state slot. That slot receives the live state prefill computes for the whole prompt; whichever copy lands last wins. InitLoadBackParams.restore_request_state=False restores the canonical cache slot only. - A hybrid restore needs one canonical state slot besides the request's live slot. Admission keeps a slot free for each queued restore (ensure_mamba_capacity(..., reserved_slots=)) and promises prefill only the device-resident prefix when no slot can be reclaimed; before, the promise went out and the restore failed after prefill had skipped those tokens. - Under DCP, a decode server with an L3 backend requires --hicache-storage-prefetch-policy wait_complete (decode promises the probed L3 span before the read completes) and refuses --disaggregation-decode-enable-offload-kvcache, as sgl-project#41695 did. Checked at startup (resolve_hicache_dcp_compatibility) and on runtime attach, including the same-backend policy update (StorageAttachment.attach; HiCacheController.attach_storage_backend never sees the effective policy). Source: 91ecb13 on the kimi-k3-nightly-3c560089 line, which equals upstream sgl-project#41695's own commits over sgl-project#41694 (head ed85005). Dropped because 5884e4f has it: starting the L3 prefetch before preallocation and promising only the span it took (sgl-project#43548). The sgl-project#41693/sgl-project#41694 ports that 91ecb13 sat on are superseded by sgl-project#41187 (no P/D refusal, no decode-offload case), so the guards are re-expressed on upstream's structure and the tests are rewritten on 5884e4f's fixtures. No change for a decode server without the decode radix cache, which is every K3 recipe today. Tests (CPU), this commit's test files on the parent: test_server_args.py TestHiCacheArgs::test_dcp_decode_storage_requires_wait_complete_without_offload 3 subtests fail (NotImplementedError not raised for timeout, best_effort and decode offload); test_hicache_dcp_host_pool.py TestDcpDecodeStorageAttach 6 subtests fail (attach accepted an early-stopping policy, attached or not); test_mamba_capacity.py test_pd_restore_promise_keeps_a_canonical_state_slot 2 subtests fail (full and pending_restore promised 6 tokens, not 2). With this commit all pass. Tests (GPU, one B300): test_unified_radix_cache_load_back.py Test_FULL_MAMBA_ps1::test_pd_decode_restore_keeps_the_live_request_state fails on the parent (the live slot ends with the checkpoint's marker 11, not the live 37) and passes with this commit. Fork PR (v0.5.22 line): #19 Upstream: sgl-project#41695 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes the request failures reported in #30322 (
HiCache load_back failed ... expected decode_prefix_len=64 (l1=0, l2=0, l3=64)), which the stale #32278 tried to address by deferring the metadata send.With a decode-side radix cache and an L3 backend, decode promises prefill a prefix that includes the L3 hit (
decode_prefix_len) before the L3 prefetch is accepted:_build_decode_prefix_matchqueries L3 with the whole suffix.query_storage_hit_lengthonly requires the suffix to reachprefetch_threshold(default 256), so the returned hit can be any page multiple below it._pre_allocand_send_kv_metadatacommitL1 + L2 + L3to prefill, so prefill skips those tokens._start_hicache_prefetchthen prefetches only the hit span.prefetch_from_storagedeclines spans shorter thanprefetch_threshold(and rate-limited ones) without registering anything.check_prefetch_progressreturns True for the unregistered request, load-back covers only L1 + L2, the restore is markedFAILED, and the request is aborted with a 500.So every request whose L3 hit is shorter than
prefetch_threshold(e.g. one shared 128-token system prompt at page size 64) fails deterministically.Modifications
_start_hicache_prefetchnow uses the prefill scheduler's span shape: the whole suffix,matched_prefix_tokensandstorage_hit_end. The prefetch reuses the queried hit (no second L3 query) and carries the bigram boundary token, asScheduler._prefetch_kvcachedoes._start_hicache_prefetchruns before_pre_alloc, andtotal_prefix_lenis re-read afterwards, sodecode_prefix_len,cache_protected_lenand the restore reservation only cover what will actually load.An L3 span that is accepted but later completes short (e.g. the object is evicted between query and read) still fails the restore. That is a narrower race than the deterministic failure fixed here.
Tests
test_decode_promises_only_the_prefetched_l3_span(test_unified_radix_cache_unittest.py) drives the real_build_decode_prefix_matchand_start_hicache_prefetchagainst a realUnifiedRadixCacheand file L3 backend (prefetch_threshold=16):decode_prefix_len == 0);On
mainthe test fails with8 != 0. With this change it passes, along withtest/registered/unit/disaggregation(686 passed),test_storage_prefetch_lifecycle.py, and the L3 / prefetch / decode subset oftest_unified_radix_cache_unittest.py(135 passed).test_decode_hicache_tree_core.pyis updated for the new call shape.CI States
Latest PR Test (Base): ✅ Run #39717056163
Latest PR Test (Extra): ❌ Run #39717056113
Latest PR Test (AMD ROCm 10): ❌ Run #39717056095