Skip to content

[HiCache] Rework the buffer-mode storage prefetch pipeline and retry bookkeeping - #39283

Merged
xiezhq-hermann merged 2 commits into
mainfrom
hicache-buffer-mode-prefetch-pipeline
Sep 15, 2026
Merged

xiezhq-hermann merged 2 commits into
mainfrom
hicache-buffer-mode-prefetch-pipeline

Conversation

@xiezhq-hermann

@xiezhq-hermann xiezhq-hermann commented Sep 13, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Buffer-mode HiCache (host memory used as transport staging, no L2 tier) decided a staged L3 hit's fate inside the prefill adder, after admission had already charged host-hit tokens that a shrunk device prefix could no longer splice. Under load this produced phantom host-hit accounting, holds that wedged admission, and unbounded storage-prefetch retries. PD-prefill and PP schedulers had also stopped draining HiCache events after a scheduler refactor.

Modifications

Prefill adder / scheduler

  • PrefillAdder.add_one_req selects a prefill shape without allocating (_select_prefill_admission), materializes host hits under the prefix lock via init_load_back, then commits (_commit_prefill_admission). init_load_back may return None to request another admission attempt; FULL over-delivery shrinks the planned extend range, under-delivery re-selects with no host hit.
  • The SWA budget gate is factored into _swa_admission_gate and keeps the ring-slot exact-fit comparison for the unified-KV SWA ring.
  • Storage prefetch retries move into StoragePrefetchRetries (mem_cache/storage_prefetch.py): paced miss polls, immediate re-issues for known hits whose anchor moved, and a per-request cap via --hicache-storage-prefetch-retry-max-attempts (default 8; --hicache-storage-prefetch-retry-poll-interval default becomes 8). Requests past the cap are admitted with whatever the device holds.
  • HiCache event draining is factored into _process_hicache_events and called from the PD-prefill and PP scheduler loops.

Buffer-mode pipeline

  • Staged holds are anchored by a FULL-only pin (match_full_device_prefix, inc_full_pin, dec_full_pin on the Python and Rust tree cores), so the device prefix under a parked fetch cannot be evicted while the SWA segment lock is left alone.
  • Aux components size prefetch staging in prepare_prefetch (PreparePrefetchResult.staging_tokens) and allocate it once the hit is known (alloc_prefetch_staging).
  • Admission-time device-capacity deferrals are bounded (max_staged_admission_defers = 32); past the cap the hold is dropped and the request recomputes.
  • Parked hits keep their turn ahead of newer hits, stale prefetch acks are ignored once a request re-issued its query, misses are accounted from the published hit count, and EAGLE bigram keys keep their boundary token when a prefetch span is trimmed or staged.
  • swa_transient_size exposes SWA slots owned by staged holds to the scheduler invariant checker.

Adaptations to upstream

  • _swa_admission_gate incorporates the _swa_req_ring exact-fit comparison added upstream in [AMD][DSV4] Reland unified-KV pool sizing and SWA ring accounting, fully gated #38192.
  • The server args land in arg_groups/fields/memory.py (upstream moved the field declarations there).
  • Unfulfilled-prefetch reason labels keep the existing upstream names (shrunk).
  • The Rust match_full_device_prefix walks the full radix namespace (extra key + cache salt) like match_prefix does upstream.
  • The Rust tree core gets the same staging_tokens contract as the Python components: SWA/Mamba PREFETCH builds size their placeholder keys from the planned staging and carry no host buffer, which the Python pipeline attaches once the hit is known (alloc_prefetch_staging). Without this the Rust adapter rejected the new keyword and the Rust-backed shared suite failed.
  • Known gap (Rust tree core): buffer-mode consumption now repairs SWA tombstones under the loaded window through two new Python tree-core helpers, swa_tombstone_ranges and attach_swa_window, which have no Rust port yet. The Rust adapter exposes them as explicit NotImplementedError stubs (same pattern as its root_node), and the three shared-suite tests that exercise that path (test_buffer_only_load_back_uses_full_behind_swa_tombstone, test_buffer_only_load_back_reuses_partial_masked_full, test_buffer_only_load_back_trims_head_published_by_sibling) are skipped under the Rust backend. The last one passed under Rust before this PR, so this is a deliberate coverage regression for buffer-only + SWA + L3 on the Rust core until the helpers are ported; the Python core is unaffected.

Original commits

  • 0e78bbe270
  • bdc1fb6985
  • 57b0a7f139

Accuracy Tests

N/A (scheduler / cache bookkeeping change; no numerics affected).

Benchmarking and Profiling

Validated on an internal buffer-mode + L3 setup: warm-cache prefix hit rate no longer collapses under high concurrency and no admission wedges were observed. No absolute numbers are included here.

Checklist

Local validation (details in the first comment): Rust cargo test --locked (865 passed incl. the new pin test), cargo check --features python-extension,inspection, cargo fmt --check, cargo clippy -D warnings; Python unit suites test_unified_radix_cache_unittest.py, test_rust_unified_radix_cache_unittest.py, test_storage_prefetch_lifecycle.py, test_prefill_adder.py, test_scheduler_hicache_events.py, test_buffer_mode_sidecar.py, test_hicache_staged_write_back_dispatch.py, test_rust_tree_core.py, test_tree_core_registry.py; ruff, ruff format, isort, codespell on changed files.


CI States

Latest PR Test (Base): ✅ Run #34939088034
Latest PR Test (Extra): ❌ Run #34939087207
Latest PR Test (AMD ROCm 10): ⏳ Run #34939087808

@xiezhq-hermann

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@xiezhq-hermann

Copy link
Copy Markdown
Collaborator Author

Local validation (GB300 devserver, prod venv):

  • Rust crate rust/sglang-radix-tree: cargo test --locked 865 passed (includes the new FULL-pin test and the reworked SWA/Mamba prefetch-build tests); cargo check --features python-extension,inspection; cargo fmt --check; cargo clippy -D warnings on the default, python-extension,inspection, and --no-default-features --features tch/doc-only shapes.
  • test_unified_radix_cache_unittest.py (Python core): 1251 passed, 1435 skipped, 70 subtests.
  • test_rust_unified_radix_cache_unittest.py (Rust core): 2686 ran, OK, 1456 skipped (pristine main baseline: 2610 ran, OK, 1388 skipped). The 3 buffer-mode SWA window-repair tests are skipped under Rust as described above.
  • test_storage_prefetch_lifecycle.py, test_prefill_adder.py, test_scheduler_hicache_events.py, test_buffer_mode_sidecar.py, test_hicache_staged_write_back_dispatch.py, test_scheduler_chunked_req_gate.py, test_invariant_checker.py, test_stat_loggers_di.py, test_rust_tree_core.py, test_tree_core_registry.py: all green (98 tests).
  • ruff check --select F401,F821,UP037, ruff format --check, isort --check-only, codespell on the changed files: clean.
  • Not run locally: GPU end-to-end HiCache L3 serving; covered by an internal setup, see the PR description.

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Sep 13, 2026
@Jiminator Jiminator closed this Sep 14, 2026
@Jiminator
Jiminator deleted the hicache-buffer-mode-prefetch-pipeline branch September 14, 2026 04:44
@alexnails
alexnails restored the hicache-buffer-mode-prefetch-pipeline branch September 14, 2026 05:44
@hnyls2002 hnyls2002 removed the run-ci CI: run the baseline test suite on this PR label Sep 14, 2026
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Sep 17, 2026
Origin: sgl-project#39701
"[HiCache] Retry failed storage backup batches and expose terminal failures"

Squashed upstream commits:
- c311e1c fix(hicache): retry failed storage backup batches
- 3edaf3b fix(hicache): harden storage backup retry path

Manual merge notes:
- Applied as one squashed diff over our fork state. cache_controller.py,
  hiradix_cache.py, unified_radix_cache.py, metrics_collector.py and the new
  test landed cleanly; only hybrid_cache_controller.py conflicted.
- Conflict 1 (method signature): kept our fork-local `trim_prefetch_full_head`
  (buffer-mode storage-prefetch pipeline, sgl-project#39283/sgl-project#39050 lineage) directly
  above the upstream `def _page_backup(self, operation) -> bool:` return type.
  Both are preserved.
- Conflict 2 (sidecar write body): upstream moved the batch_set_v2 call into a
  nested `write_sidecars()` closure so `_retry_storage_write` can retry it.
  Our fork-local sgl-project#39050 `extra_info=HiCacheStorageExtraInfo(prefix_keys=...)`
  argument is kept INSIDE that closure, so retries still forward prefix keys.
- Upstream deletes `HybridCacheController.backup_thread_func` (the base
  implementation now handles sidecar-only ranks and terminal-failure logging).
  Deletion accepted; the unused `Empty` import was dropped with it. This does
  not regress fork-local behavior: the base thread still calls the overridden
  `_page_backup`, which keeps the rank-sharded-sidecar `should_backup` path.
- Fork-local `_resolve_sidecar_kv_derived_pool_transfers` keeps
  `transfer.keys = operation.hash_value` (unconditional) instead of upstream's
  `if transfer.keys is None` guard; that is pre-existing fork behavior from
  sgl-project#39283, untouched by this PR.
- Also kept from the fork: `dp_rank` / `kv_cache_dtype` storage-config args,
  `prefix_keys is not None` copy semantics and `batch_get_v2(extra_info=...)`.
- Test registered under test/registered/unit/mem_cache/ (valid taxonomy);
  12 tests pass locally (CPU-only harness).
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Sep 17, 2026
Origin: sgl-project#39701
"[HiCache] Retry failed storage backup batches and expose terminal failures"

Squashed upstream commits:
- c311e1c fix(hicache): retry failed storage backup batches
- 3edaf3b fix(hicache): harden storage backup retry path

Manual merge notes:
- Applied as one squashed diff over our fork state. cache_controller.py,
  hiradix_cache.py, unified_radix_cache.py, metrics_collector.py and the new
  test landed cleanly; only hybrid_cache_controller.py conflicted.
- Conflict 1 (method signature): kept our fork-local `trim_prefetch_full_head`
  (buffer-mode storage-prefetch pipeline, sgl-project#39283/sgl-project#39050 lineage) directly
  above the upstream `def _page_backup(self, operation) -> bool:` return type.
  Both are preserved.
- Conflict 2 (sidecar write body): upstream moved the batch_set_v2 call into a
  nested `write_sidecars()` closure so `_retry_storage_write` can retry it.
  Our fork-local sgl-project#39050 `extra_info=HiCacheStorageExtraInfo(prefix_keys=...)`
  argument is kept INSIDE that closure, so retries still forward prefix keys.
- Upstream deletes `HybridCacheController.backup_thread_func` (the base
  implementation now handles sidecar-only ranks and terminal-failure logging).
  Deletion accepted; the unused `Empty` import was dropped with it. This does
  not regress fork-local behavior: the base thread still calls the overridden
  `_page_backup`, which keeps the rank-sharded-sidecar `should_backup` path.
- Fork-local `_resolve_sidecar_kv_derived_pool_transfers` keeps
  `transfer.keys = operation.hash_value` (unconditional) instead of upstream's
  `if transfer.keys is None` guard; that is pre-existing fork behavior from
  sgl-project#39283, untouched by this PR.
- Also kept from the fork: `dp_rank` / `kv_cache_dtype` storage-config args,
  `prefix_keys is not None` copy semantics and `batch_get_v2(extra_info=...)`.
- Test registered under test/registered/unit/mem_cache/ (valid taxonomy);
  12 tests pass locally (CPU-only harness).
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 18, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Sep 18, 2026
Origin: sgl-project#39701
"[HiCache] Retry failed storage backup batches and expose terminal failures"

Squashed upstream commits:
- c311e1c fix(hicache): retry failed storage backup batches
- 3edaf3b fix(hicache): harden storage backup retry path

Manual merge notes:
- Applied as one squashed diff over our fork state. cache_controller.py,
  hiradix_cache.py, unified_radix_cache.py, metrics_collector.py and the new
  test landed cleanly; only hybrid_cache_controller.py conflicted.
- Conflict 1 (method signature): kept our fork-local `trim_prefetch_full_head`
  (buffer-mode storage-prefetch pipeline, sgl-project#39283/sgl-project#39050 lineage) directly
  above the upstream `def _page_backup(self, operation) -> bool:` return type.
  Both are preserved.
- Conflict 2 (sidecar write body): upstream moved the batch_set_v2 call into a
  nested `write_sidecars()` closure so `_retry_storage_write` can retry it.
  Our fork-local sgl-project#39050 `extra_info=HiCacheStorageExtraInfo(prefix_keys=...)`
  argument is kept INSIDE that closure, so retries still forward prefix keys.
- Upstream deletes `HybridCacheController.backup_thread_func` (the base
  implementation now handles sidecar-only ranks and terminal-failure logging).
  Deletion accepted; the unused `Empty` import was dropped with it. This does
  not regress fork-local behavior: the base thread still calls the overridden
  `_page_backup`, which keeps the rank-sharded-sidecar `should_backup` path.
- Fork-local `_resolve_sidecar_kv_derived_pool_transfers` keeps
  `transfer.keys = operation.hash_value` (unconditional) instead of upstream's
  `if transfer.keys is None` guard; that is pre-existing fork behavior from
  sgl-project#39283, untouched by this PR.
- Also kept from the fork: `dp_rank` / `kv_cache_dtype` storage-config args,
  `prefix_keys is not None` copy semantics and `batch_get_v2(extra_info=...)`.
- Test registered under test/registered/unit/mem_cache/ (valid taxonomy);
  12 tests pass locally (CPU-only harness).
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 21, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 22, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 22, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 22, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 23, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 24, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 24, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 24, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 26, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Jialin added a commit to Jialin/sglang that referenced this pull request Sep 26, 2026
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
@Yoongger

Copy link
Copy Markdown

@xiezhq-hermann Heads-up: the load-back completeness check added in this PR (promised_host_hit = req.host_hit_length + the RuntimeError in add_one_req) fires on a valid partial load-back and takes down the whole scheduler.

With HiCache on a hybrid-SWA model, a host hit can include FULL tokens that are already device-resident (behind a host-only window/state entry). init_load_back then legitimately splices only the evicted rows — exactly the full_load_tokens value that prepare_load_back was budgeted with a few lines above — so host_loaded_length < host_hit_length and the assert kills every TP rank.

Observed on main @ 2026-11-16:

RuntimeError: HiCache load-back must commit all promised FULL tokens or none:
  req=... promised=136192 loaded=41728

(promised - loaded = 94464 tokens were already on device; only 41728 rows needed H2D.)

Fix we are running locally is the one-line compare against the transferred row count:

-                    promised_host_hit = req.host_hit_length
+                    promised_host_hit = full_load_tokens

This keeps the fail-fast for a genuinely short load-back and matches full_load_back_tokens() / build_hicache_transfers(LOAD_BACK), whose own docstring notes those rows "take no new device slots".

Full report with repro, traceback and verification: #50892

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants