Skip to content

[Fix] hicache: Restore sub-window SWA L3 hits under a mid-tree anchor - #46934

Merged
alphabetc1 merged 6 commits into
sgl-project:mainfrom
pengwu22:swa-sub-window-l3-restore
Nov 9, 2026
Merged

alphabetc1 merged 6 commits into
sgl-project:mainfrom
pengwu22:swa-sub-window-l3-restore

Conversation

@pengwu22

@pengwu22 pengwu22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

An L3 hit shorter than one sliding window, under a mid-tree anchor, is dropped in both host-memory modes. It is the usual hit when a reader holds a prefix on device and another writer stored its continuation, e.g. a PD decode offloading its output KV.

End to end (PD with decode KV offload, hybrid SWA + Mamba model, window 512, page 128; a 512-token prompt whose decode wrote KV up to 768), the next turn's prefill reuse:

prefill host memory before after
cache (write_back) 512 (its own prompt; the 256 stored tokens are dropped) 768
buffer_only 512 (the lookup is declined) 768

Every turn's logprobs match a no-cache reference (max drift 4.4e-16). Run on a fork carrying this change.

Modifications

  • Cache mode (Python and Rust tree cores): SWAComponent.prepare_prefetch stages min(span, window) pages at any anchor, and the SWA prefetch commit no longer drops a short window under a non-root anchor. The window head is the anchor path's own SWA; the match validator (create_match_validator) accepts a boundary only after a full contiguous SWA window, so a short window whose head is gone never matches.
  • Buffer mode: remove the full-window lookup minimum (resolve_prefetch_threshold). A staged hold pins FULL, not SWA, so the head of a staged short window can be evicted while it waits (the concern behind [HiCache] Rework the buffer-mode storage prefetch pipeline and retry bookkeeping #39283); admission now refetches when this pass's joint match no longer reaches the window's start.

Accuracy Tests

No kernel or forward change.

test before after
test_hicache_l3_prefetch_attaches_a_sub_window_suffix_under_a_device_anchor (new) fails passes
test_buffer_only_hit_restores_a_window_shorter_than_full (new; root, and mid-tree with window - 1 and window pages) fails (declined) passes
test_buffer_only_sub_window_hit_refetches_when_its_window_head_is_evicted (new) fails without the admission check passes
test_mid_tree_sub_window_swa_prefetch_is_attached (replaces ..._is_released; both tree cores) and Rust prefetch_commit_attaches_a_sub_window_hit_under_a_non_root_anchor encoded the drop pass

test_buffer_only_swa_window_semantics is removed: it pinned the full-window minimum. test/registered/unit/mem_cache/unified_radix_cache/: 1080 passed, 0 failed (main: 1065, 0). sglang-radix-tree lib tests: 942 passed.

Not run here: the GPU suites (test_unified_radix_cache_kl_hybrid_bitexact.py, incl. TestUnifiedHybridBufferOnlyBitExact).

Related: #46934, #46935, #46936, #46940.

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ✅ Run #42611727167
Latest PR Test (Extra): ❌ Run #42611727183
Latest PR Test (AMD ROCm 10): ⏳ Run #42611726946

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

An L3 hit shorter than one sliding window under a mid-tree anchor was dropped.
It is the usual hit when one writer stores a continuation of a prefix the
reader already holds (for example, a PD decode offloading its output KV).

- Cache mode fetched it and then dropped it: prepare_prefetch staged no SWA for
  the span, and the SWA prefetch commit released a window shorter than full
  under a non-root anchor. The window head is the anchor path's own SWA, and
  the match validator only accepts a boundary with a full window of contiguous
  SWA, so stage and attach min(span, window) pages at any anchor (Python and
  Rust tree cores).
- Buffer mode never looked it up: every lookup needed a full window. A staged
  hold pins FULL only, so the head of a sub-window hit can be evicted while it
  waits. Drop the full-window minimum and refetch at admission when this
  pass's joint match no longer reaches the start of the staged window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pengwu22

Copy link
Copy Markdown
Contributor Author

/tag-run-ci-label

@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pengwu22

Copy link
Copy Markdown
Contributor Author

/run-cpu-ci

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Running base-a-test-cpu on 5a610e474b27: View workflow run

pengwu22 and others added 2 commits October 29, 2026 14:44
The full window is counted in cache pages (full_window_pages comes from
the cache page size), so revert the previous commit and give the short
SWA tail test fixture a real page size instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-sub-window-l3-restore

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-test registered/hicache/test_hicache_variants.py registered/e2e/hicache/test_hicache_storage_lora.py registered/kernels/ops/kvcache/test_hicache.py registered/kernels/ops/kvcache/test_hicache_page_first_write_back.py registered/unit/mem_cache/test_swa_unittest.py

@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-h100 (5 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_variants.py
cd test/ && python3 registered/e2e/hicache/test_hicache_storage_lora.py
cd test/ && python3 registered/kernels/ops/kvcache/test_hicache.py
cd test/ && python3 registered/kernels/ops/kvcache/test_hicache_page_first_write_back.py
cd test/ && python3 registered/unit/mem_cache/test_swa_unittest.py

@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

1 similar comment
@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-test registered/e2e/models/test_deepseek_v41_flash_gb300_dp2ep2.py

@pengwu22

Copy link
Copy Markdown
Contributor Author

/rerun-failed-ci

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🚀 2-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v41_flash_gb300_dp2ep2.py

@alphabetc1
alphabetc1 merged commit dfaca56 into sgl-project:main Nov 9, 2026
155 of 167 checks passed
detain added a commit to detain/sglang that referenced this pull request Nov 11, 2026
…r/URC markers -- 3/18 census consumed, 3 gates in one

CASCADE (start_writing(wait_for_forward) TypeError class, 68 distinct nodes):
upstream sgl-project#48012's NEW fixture hicache_write_back_fixtures.InstantWriteBackController
mirrored the pre-sgl-project#44800 no-kwarg surface while the fork's writing_check drain
(entries 199/218: the batched-backup flush=False + ONE fenced start_writing)
calls the real API. Def-site fixes, no fence weakened:
- fixture start_writing gains the keyword (instant copies -> accepted-ignored,
  documented);
- test_write_back_eviction_batching: record_submission wrapper FORWARDS the
  flag (it measures the very wait, so swallowing it would understate the
  fence); _FencedController gains load_fence_stream = forward (the fork's
  unconditional entry-168 fence reads it first; prod: both streams are the
  same forward stream) -- also silences the 16-test Rust TreeCore panic class
  whose "victims not demoted" assert was the aborted submission downstream.

MARKERS (3): managers/cache_controller.py:1202 ADOPTED -- the dropped
set_hicache_transfer_done_event write line is superseded by upstream
c24285b(sgl-project#48008 per-band page ledger)+2cbb66f563(sgl-project#48012): record_host_transfer
publishes the same signal per-page BOTH directions (load twin writes=True).
unified_radix_cache.py:627 ADOPTED -- the buffer-mode threshold floor
(resolve_prefetch_threshold) was DELIBERATELY deleted upstream by
dfaca56(sgl-project#46934, restores sub-window SWA L3 hits); window sizing now lives
per-allocation in BufferModePipeline; the per-call-site file_storage_path fold
is consistent with the entry-253 funnel (cache_controller.py:948, intact).
unified_radix_cache.py:3917 RE-PLANTED -- entry-293's cadence gate (park, do
not re-enter allocate_storage_hit/evict_host inside the retry window) + the
expired no-retry arm were dropped; wait_full folded into upstream's
comp_xfers allow_partial rule (min_tokens form already equivalent).

URC context guard: _deferred_ready_sync_group read get_memory() unguarded --
upstream's NEW _arm_tick CPU/spawn fixtures (their restructured timeout tests,
sgl-project#44614) reach it without a published server config. Now published-check
(schedule_policy.py:615 idiom): default-off CPU path, production unchanged.

Deadline-anchor retargets (fork B3-9 family vs upstream's sgl-project#44614 vote-before-
poll restructure; every assertion preserved): _vote_prefetch_deadlines drives
the verdict before the expiry polls; failed-arm polled inside-budget so the
storage failure stays the SOLE cut reason; fixture cc gains the sgl-project#44342
liveness attrs + pp_prefetch_command_thread; _arm_tick stub returns the fork's
5-tuple (health bit True); release_aborted_request -> finish(ABORT) per
8898429(sgl-project#48491).

Battery actuals (CPU, CVD=9, prod venv, PYTHONPATH=wt/python):
- the 5 cascade files 68F/39P -> 3F/70P (survivors = the sgl-project#48491/pp-thread/
  budget-anchor fixes committed here; after them): test_storage_prefetch_
  lifecycle ** 31P/31sub ** (entry-311 pin 28P/31sub, 295's >=23P/28sub
  EXCEEDED); decode_retraction_backup 6F + chunked_publish 2F = No-CUDA
  device class (env, reported).
- combined gate run: trailing_endpoint + batching + parity + async_ready_sync
  + staged_write_back = 89P/1S/28sub 0F.
Gates: ruff repo ruleset+format clean on touched files; registration trio rc=0.
luoroger37 added a commit to bytedance-iaas/sglang that referenced this pull request Nov 16, 2026
Adapt the relevant upstream change to poc_dsv4.1.

Upstream-PR: sgl-project#46934
Upstream-commit: dfaca56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants