Skip to content

[Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes - #43548

Merged
ispobock merged 6 commits into
sgl-project:mainfrom
alphabetc1:fix/decode-l3-promise
Oct 23, 2026
Merged

ispobock merged 6 commits into
sgl-project:mainfrom
alphabetc1:fix/decode-l3-promise

Conversation

@alphabetc1

@alphabetc1 alphabetc1 commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Fixes the request failures reported in #30322 (HiCache load_back failed ... expected decode_prefix_len=64 (l1=0, l2=0, l3=64)), which the stale #32278 tried to address by deferring the metadata send.

With a decode-side radix cache and an L3 backend, decode promises prefill a prefix that includes the L3 hit (decode_prefix_len) before the L3 prefetch is accepted:

  1. _build_decode_prefix_match queries L3 with the whole suffix. query_storage_hit_length only requires the suffix to reach prefetch_threshold (default 256), so the returned hit can be any page multiple below it.
  2. _pre_alloc and _send_kv_metadata commit L1 + L2 + L3 to prefill, so prefill skips those tokens.
  3. _start_hicache_prefetch then prefetches only the hit span. prefetch_from_storage declines spans shorter than prefetch_threshold (and rate-limited ones) without registering anything.
  4. check_prefetch_progress returns True for the unregistered request, load-back covers only L1 + L2, the restore is marked FAILED, and the request is aborted with a 500.

So every request whose L3 hit is shorter than prefetch_threshold (e.g. one shared 128-token system prompt at page size 64) fails deterministically.

Modifications

  • _start_hicache_prefetch now uses the prefill scheduler's span shape: the whole suffix, matched_prefix_tokens and storage_hit_end. The prefetch reuses the queried hit (no second L3 query) and carries the bigram boundary token, as Scheduler._prefetch_kvcache does.
  • If the prefetch is neither registered nor ticketed, the L3 span is cleared.
  • _start_hicache_prefetch runs before _pre_alloc, and total_prefix_len is re-read afterwards, so decode_prefix_len, cache_protected_len and the restore reservation only cover what will actually load.

An L3 span that is accepted but later completes short (e.g. the object is evicted between query and read) still fails the restore. That is a narrower race than the deterministic failure fixed here.

Tests

test_decode_promises_only_the_prefetched_l3_span (test_unified_radix_cache_unittest.py) drives the real _build_decode_prefix_match and _start_hicache_prefetch against a real UnifiedRadixCache and file L3 backend (prefetch_threshold=16):

  • an 8-token L3 hit is queried (8) but no longer promised (decode_prefix_len == 0);
  • a 24-token hit is promised, prefetched, and lands as a 24-token host hit.

On main the test fails with 8 != 0. With this change it passes, along with test/registered/unit/disaggregation (686 passed), test_storage_prefetch_lifecycle.py, and the L3 / prefetch / decode subset of test_unified_radix_cache_unittest.py (135 passed). test_decode_hicache_tree_core.py is updated for the new call shape.


CI States

Latest PR Test (Base): ✅ Run #39717056163
Latest PR Test (Extra): ❌ Run #39717056113
Latest PR Test (AMD ROCm 10): ❌ Run #39717056095

@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Oct 10, 2026
@alphabetc1 alphabetc1 changed the title [PD][HiCache] Promise prefill only the L3 span the decode prefetch takes [Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes Oct 14, 2026
fivechenxi pushed a commit to fivechenxi/sglang that referenced this pull request Oct 16, 2026
Adapt the logical storage_hit_end approach from upstream sgl-project#43548 to the pinned prefetch API. Keep bigram boundary IDs, honor declined prefetches before preallocation, and reject short restored coverage before enqueuing DMA. Add real CUDA/file round-trip regressions for boundary, partial L1, and short-restore cases.
@alphabetc1 alphabetc1 added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) labels Oct 17, 2026
@alphabetc1 alphabetc1 added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 17, 2026
@hzh0425 hzh0425 self-assigned this Oct 19, 2026
@ispobock
ispobock merged commit 77bc8ca into sgl-project:main Oct 23, 2026
265 of 281 checks passed
@alphabetc1
alphabetc1 deleted the fix/decode-l3-promise branch October 23, 2026 10:07
Venkat2811 added a commit to datacrunch-research/sglang that referenced this pull request Oct 27, 2026
…e slots (upstream sgl-project#41695)

Layer 12 of the Kimi-K3 nightly line on upstream main 5884e4f.

The prefill/decode part of upstream sgl-project#41695 that 5884e4f does not have. A
decode server with --disaggregation-decode-enable-radix-cache and HiCache
restores the cached prefix locally while prefill sends the rest:

- The transfer-side load-back handed the request to init_load_back, so for a
  hybrid (Mamba/KDA) model it also copied the host checkpoint at the prefix
  end into the request's own state slot. That slot receives the live state
  prefill computes for the whole prompt; whichever copy lands last wins.
  InitLoadBackParams.restore_request_state=False restores the canonical
  cache slot only.
- A hybrid restore needs one canonical state slot besides the request's live
  slot. Admission keeps a slot free for each queued restore
  (ensure_mamba_capacity(..., reserved_slots=)) and promises prefill only
  the device-resident prefix when no slot can be reclaimed; before, the
  promise went out and the restore failed after prefill had skipped those
  tokens.
- Under DCP, a decode server with an L3 backend requires
  --hicache-storage-prefetch-policy wait_complete (decode promises the probed
  L3 span before the read completes) and refuses
  --disaggregation-decode-enable-offload-kvcache, as sgl-project#41695 did. Checked at
  startup (resolve_hicache_dcp_compatibility) and on runtime attach,
  including the same-backend policy update (StorageAttachment.attach;
  HiCacheController.attach_storage_backend never sees the effective policy).

Source: 91ecb13 on the kimi-k3-nightly-3c560089 line, which equals
upstream sgl-project#41695's own commits over sgl-project#41694 (head ed85005). Dropped because
5884e4f has it: starting the L3 prefetch before preallocation and
promising only the span it took (sgl-project#43548). The sgl-project#41693/sgl-project#41694 ports that
91ecb13 sat on are superseded by sgl-project#41187 (no P/D refusal, no
decode-offload case), so the guards are re-expressed on upstream's structure
and the tests are rewritten on 5884e4f's fixtures. No change for a decode
server without the decode radix cache, which is every K3 recipe today.

Tests (CPU), this commit's test files on the parent: test_server_args.py
TestHiCacheArgs::test_dcp_decode_storage_requires_wait_complete_without_offload
3 subtests fail (NotImplementedError not raised for timeout, best_effort and
decode offload); test_hicache_dcp_host_pool.py TestDcpDecodeStorageAttach 6
subtests fail (attach accepted an early-stopping policy, attached or not);
test_mamba_capacity.py test_pd_restore_promise_keeps_a_canonical_state_slot
2 subtests fail (full and pending_restore promised 6 tokens, not 2). With
this commit all pass.
Tests (GPU, one B300): test_unified_radix_cache_load_back.py
Test_FULL_MAMBA_ps1::test_pd_decode_restore_keeps_the_live_request_state
fails on the parent (the live slot ends with the checkpoint's marker 11,
not the live 37) and passes with this commit.

Fork PR (v0.5.22 line): #19
Upstream: sgl-project#41695

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Venkat2811 added a commit to datacrunch-research/sglang that referenced this pull request Oct 27, 2026
…e slots (upstream sgl-project#41695)

Layer 12 of the Kimi-K3 nightly line on upstream main 5884e4f.

The prefill/decode part of upstream sgl-project#41695 that 5884e4f does not have. A
decode server with --disaggregation-decode-enable-radix-cache and HiCache
restores the cached prefix locally while prefill sends the rest:

- The transfer-side load-back handed the request to init_load_back, so for a
  hybrid (Mamba/KDA) model it also copied the host checkpoint at the prefix
  end into the request's own state slot. That slot receives the live state
  prefill computes for the whole prompt; whichever copy lands last wins.
  InitLoadBackParams.restore_request_state=False restores the canonical
  cache slot only.
- A hybrid restore needs one canonical state slot besides the request's live
  slot. Admission keeps a slot free for each queued restore
  (ensure_mamba_capacity(..., reserved_slots=)) and promises prefill only
  the device-resident prefix when no slot can be reclaimed; before, the
  promise went out and the restore failed after prefill had skipped those
  tokens.
- Under DCP, a decode server with an L3 backend requires
  --hicache-storage-prefetch-policy wait_complete (decode promises the probed
  L3 span before the read completes) and refuses
  --disaggregation-decode-enable-offload-kvcache, as sgl-project#41695 did. Checked at
  startup (resolve_hicache_dcp_compatibility) and on runtime attach,
  including the same-backend policy update (StorageAttachment.attach;
  HiCacheController.attach_storage_backend never sees the effective policy).

Source: 91ecb13 on the kimi-k3-nightly-3c560089 line, which equals
upstream sgl-project#41695's own commits over sgl-project#41694 (head ed85005). Dropped because
5884e4f has it: starting the L3 prefetch before preallocation and
promising only the span it took (sgl-project#43548). The sgl-project#41693/sgl-project#41694 ports that
91ecb13 sat on are superseded by sgl-project#41187 (no P/D refusal, no
decode-offload case), so the guards are re-expressed on upstream's structure
and the tests are rewritten on 5884e4f's fixtures. No change for a decode
server without the decode radix cache, which is every K3 recipe today.

Tests (CPU), this commit's test files on the parent: test_server_args.py
TestHiCacheArgs::test_dcp_decode_storage_requires_wait_complete_without_offload
3 subtests fail (NotImplementedError not raised for timeout, best_effort and
decode offload); test_hicache_dcp_host_pool.py TestDcpDecodeStorageAttach 6
subtests fail (attach accepted an early-stopping policy, attached or not);
test_mamba_capacity.py test_pd_restore_promise_keeps_a_canonical_state_slot
2 subtests fail (full and pending_restore promised 6 tokens, not 2). With
this commit all pass.
Tests (GPU, one B300): test_unified_radix_cache_load_back.py
Test_FULL_MAMBA_ps1::test_pd_decode_restore_keeps_the_live_request_state
fails on the parent (the live slot ends with the checkpoint's marker 11,
not the live 37) and passes with this commit.

Fork PR (v0.5.22 line): #19
Upstream: sgl-project#41695

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants