Repository navigation
[Fix] hicache: Restore sub-window SWA L3 hits under a mid-tree anchor - #46934
Merged
Merged
Conversation
An L3 hit shorter than one sliding window under a mid-tree anchor was dropped. It is the usual hit when one writer stores a continuation of a prefix the reader already holds (for example, a PD decode offloading its output KV). - Cache mode fetched it and then dropped it: prepare_prefetch staged no SWA for the span, and the SWA prefetch commit released a window shorter than full under a non-root anchor. The window head is the anchor path's own SWA, and the match validator only accepts a boundary with a full window of contiguous SWA, so stage and attach min(span, window) pages at any anchor (Python and Rust tree cores). - Buffer mode never looked it up: every lookup needed a full window. A staged hold pins FULL only, so the head of a sub-window hit can be evicted while it waits. Drop the full-window minimum and refetch at admission when this pass's joint match no longer reaches the start of the staged window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pengwu22
requested review from
Jialin,
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
hzh0425,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
October 28, 2026 04:45
Contributor
Author
|
/tag-run-ci-label |
This was referenced Oct 28, 2026
4 of 5 tasks
Contributor
Author
|
/rerun-failed-ci |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/run-cpu-ci |
Contributor
|
🚀 Running |
The full window is counted in cache pages (full_window_pages comes from the cache page size), so revert the previous commit and give the short SWA tail test fixture a real page size instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-sub-window-l3-restore Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/rerun-failed-ci |
1 similar comment
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-test registered/hicache/test_hicache_variants.py registered/e2e/hicache/test_hicache_storage_lora.py registered/kernels/ops/kvcache/test_hicache.py registered/kernels/ops/kvcache/test_hicache_page_first_write_back.py registered/unit/mem_cache/test_swa_unittest.py |
Contributor
Author
|
/rerun-failed-ci |
Contributor
|
🚀 |
Contributor
Author
|
/rerun-failed-ci |
1 similar comment
Contributor
Author
|
/rerun-failed-ci |
Contributor
Author
|
/rerun-test registered/e2e/models/test_deepseek_v41_flash_gb300_dp2ep2.py |
Contributor
Author
|
/rerun-failed-ci |
Contributor
|
🚀 |
alphabetc1
approved these changes
Nov 9, 2026
6 tasks done
detain
added a commit
to detain/sglang
that referenced
this pull request
Nov 11, 2026
…r/URC markers -- 3/18 census consumed, 3 gates in one CASCADE (start_writing(wait_for_forward) TypeError class, 68 distinct nodes): upstream sgl-project#48012's NEW fixture hicache_write_back_fixtures.InstantWriteBackController mirrored the pre-sgl-project#44800 no-kwarg surface while the fork's writing_check drain (entries 199/218: the batched-backup flush=False + ONE fenced start_writing) calls the real API. Def-site fixes, no fence weakened: - fixture start_writing gains the keyword (instant copies -> accepted-ignored, documented); - test_write_back_eviction_batching: record_submission wrapper FORWARDS the flag (it measures the very wait, so swallowing it would understate the fence); _FencedController gains load_fence_stream = forward (the fork's unconditional entry-168 fence reads it first; prod: both streams are the same forward stream) -- also silences the 16-test Rust TreeCore panic class whose "victims not demoted" assert was the aborted submission downstream. MARKERS (3): managers/cache_controller.py:1202 ADOPTED -- the dropped set_hicache_transfer_done_event write line is superseded by upstream c24285b(sgl-project#48008 per-band page ledger)+2cbb66f563(sgl-project#48012): record_host_transfer publishes the same signal per-page BOTH directions (load twin writes=True). unified_radix_cache.py:627 ADOPTED -- the buffer-mode threshold floor (resolve_prefetch_threshold) was DELIBERATELY deleted upstream by dfaca56(sgl-project#46934, restores sub-window SWA L3 hits); window sizing now lives per-allocation in BufferModePipeline; the per-call-site file_storage_path fold is consistent with the entry-253 funnel (cache_controller.py:948, intact). unified_radix_cache.py:3917 RE-PLANTED -- entry-293's cadence gate (park, do not re-enter allocate_storage_hit/evict_host inside the retry window) + the expired no-retry arm were dropped; wait_full folded into upstream's comp_xfers allow_partial rule (min_tokens form already equivalent). URC context guard: _deferred_ready_sync_group read get_memory() unguarded -- upstream's NEW _arm_tick CPU/spawn fixtures (their restructured timeout tests, sgl-project#44614) reach it without a published server config. Now published-check (schedule_policy.py:615 idiom): default-off CPU path, production unchanged. Deadline-anchor retargets (fork B3-9 family vs upstream's sgl-project#44614 vote-before- poll restructure; every assertion preserved): _vote_prefetch_deadlines drives the verdict before the expiry polls; failed-arm polled inside-budget so the storage failure stays the SOLE cut reason; fixture cc gains the sgl-project#44342 liveness attrs + pp_prefetch_command_thread; _arm_tick stub returns the fork's 5-tuple (health bit True); release_aborted_request -> finish(ABORT) per 8898429(sgl-project#48491). Battery actuals (CPU, CVD=9, prod venv, PYTHONPATH=wt/python): - the 5 cascade files 68F/39P -> 3F/70P (survivors = the sgl-project#48491/pp-thread/ budget-anchor fixes committed here; after them): test_storage_prefetch_ lifecycle ** 31P/31sub ** (entry-311 pin 28P/31sub, 295's >=23P/28sub EXCEEDED); decode_retraction_backup 6F + chunked_publish 2F = No-CUDA device class (env, reported). - combined gate run: trailing_endpoint + batching + parity + async_ready_sync + staged_write_back = 89P/1S/28sub 0F. Gates: ruff repo ruleset+format clean on touched files; registration trio rc=0.
luoroger37
added a commit
to bytedance-iaas/sglang
that referenced
this pull request
Nov 16, 2026
Adapt the relevant upstream change to poc_dsv4.1. Upstream-PR: sgl-project#46934 Upstream-commit: dfaca56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
An L3 hit shorter than one sliding window, under a mid-tree anchor, is dropped in both host-memory modes. It is the usual hit when a reader holds a prefix on device and another writer stored its continuation, e.g. a PD decode offloading its output KV.
End to end (PD with decode KV offload, hybrid SWA + Mamba model, window 512, page 128; a 512-token prompt whose decode wrote KV up to 768), the next turn's prefill reuse:
cache(write_back)buffer_onlyEvery turn's logprobs match a no-cache reference (max drift 4.4e-16). Run on a fork carrying this change.
Modifications
SWAComponent.prepare_prefetchstagesmin(span, window)pages at any anchor, and the SWA prefetch commit no longer drops a short window under a non-root anchor. The window head is the anchor path's own SWA; the match validator (create_match_validator) accepts a boundary only after a full contiguous SWA window, so a short window whose head is gone never matches.resolve_prefetch_threshold). A staged hold pins FULL, not SWA, so the head of a staged short window can be evicted while it waits (the concern behind [HiCache] Rework the buffer-mode storage prefetch pipeline and retry bookkeeping #39283); admission now refetches when this pass's joint match no longer reaches the window's start.Accuracy Tests
No kernel or forward change.
test_hicache_l3_prefetch_attaches_a_sub_window_suffix_under_a_device_anchor(new)test_buffer_only_hit_restores_a_window_shorter_than_full(new; root, and mid-tree withwindow - 1andwindowpages)test_buffer_only_sub_window_hit_refetches_when_its_window_head_is_evicted(new)test_mid_tree_sub_window_swa_prefetch_is_attached(replaces..._is_released; both tree cores) and Rustprefetch_commit_attaches_a_sub_window_hit_under_a_non_root_anchortest_buffer_only_swa_window_semanticsis removed: it pinned the full-window minimum.test/registered/unit/mem_cache/unified_radix_cache/: 1080 passed, 0 failed (main: 1065, 0).sglang-radix-treelib tests: 942 passed.Not run here: the GPU suites (
test_unified_radix_cache_kl_hybrid_bitexact.py, incl.TestUnifiedHybridBufferOnlyBitExact).Related: #46934, #46935, #46936, #46940.
Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #42611727167
Latest PR Test (Extra): ❌ Run #42611727183
Latest PR Test (AMD ROCm 10): ⏳ Run #42611726946
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: