Repository navigation
[HiCache] Rework the buffer-mode storage prefetch pipeline and retry bookkeeping - #39283
Conversation
|
/tag-and-rerun-ci |
|
Local validation (GB300 devserver, prod venv):
|
Origin: sgl-project#39701 "[HiCache] Retry failed storage backup batches and expose terminal failures" Squashed upstream commits: - c311e1c fix(hicache): retry failed storage backup batches - 3edaf3b fix(hicache): harden storage backup retry path Manual merge notes: - Applied as one squashed diff over our fork state. cache_controller.py, hiradix_cache.py, unified_radix_cache.py, metrics_collector.py and the new test landed cleanly; only hybrid_cache_controller.py conflicted. - Conflict 1 (method signature): kept our fork-local `trim_prefetch_full_head` (buffer-mode storage-prefetch pipeline, sgl-project#39283/sgl-project#39050 lineage) directly above the upstream `def _page_backup(self, operation) -> bool:` return type. Both are preserved. - Conflict 2 (sidecar write body): upstream moved the batch_set_v2 call into a nested `write_sidecars()` closure so `_retry_storage_write` can retry it. Our fork-local sgl-project#39050 `extra_info=HiCacheStorageExtraInfo(prefix_keys=...)` argument is kept INSIDE that closure, so retries still forward prefix keys. - Upstream deletes `HybridCacheController.backup_thread_func` (the base implementation now handles sidecar-only ranks and terminal-failure logging). Deletion accepted; the unused `Empty` import was dropped with it. This does not regress fork-local behavior: the base thread still calls the overridden `_page_backup`, which keeps the rank-sharded-sidecar `should_backup` path. - Fork-local `_resolve_sidecar_kv_derived_pool_transfers` keeps `transfer.keys = operation.hash_value` (unconditional) instead of upstream's `if transfer.keys is None` guard; that is pre-existing fork behavior from sgl-project#39283, untouched by this PR. - Also kept from the fork: `dp_rank` / `kv_cache_dtype` storage-config args, `prefix_keys is not None` copy semantics and `batch_get_v2(extra_info=...)`. - Test registered under test/registered/unit/mem_cache/ (valid taxonomy); 12 tests pass locally (CPU-only harness).
Origin: sgl-project#39701 "[HiCache] Retry failed storage backup batches and expose terminal failures" Squashed upstream commits: - c311e1c fix(hicache): retry failed storage backup batches - 3edaf3b fix(hicache): harden storage backup retry path Manual merge notes: - Applied as one squashed diff over our fork state. cache_controller.py, hiradix_cache.py, unified_radix_cache.py, metrics_collector.py and the new test landed cleanly; only hybrid_cache_controller.py conflicted. - Conflict 1 (method signature): kept our fork-local `trim_prefetch_full_head` (buffer-mode storage-prefetch pipeline, sgl-project#39283/sgl-project#39050 lineage) directly above the upstream `def _page_backup(self, operation) -> bool:` return type. Both are preserved. - Conflict 2 (sidecar write body): upstream moved the batch_set_v2 call into a nested `write_sidecars()` closure so `_retry_storage_write` can retry it. Our fork-local sgl-project#39050 `extra_info=HiCacheStorageExtraInfo(prefix_keys=...)` argument is kept INSIDE that closure, so retries still forward prefix keys. - Upstream deletes `HybridCacheController.backup_thread_func` (the base implementation now handles sidecar-only ranks and terminal-failure logging). Deletion accepted; the unused `Empty` import was dropped with it. This does not regress fork-local behavior: the base thread still calls the overridden `_page_backup`, which keeps the rank-sharded-sidecar `should_backup` path. - Fork-local `_resolve_sidecar_kv_derived_pool_transfers` keeps `transfer.keys = operation.hash_value` (unconditional) instead of upstream's `if transfer.keys is None` guard; that is pre-existing fork behavior from sgl-project#39283, untouched by this PR. - Also kept from the fork: `dp_rank` / `kv_cache_dtype` storage-config args, `prefix_keys is not None` copy semantics and `batch_get_v2(extra_info=...)`. - Test registered under test/registered/unit/mem_cache/ (valid taxonomy); 12 tests pass locally (CPU-only harness).
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Origin: sgl-project#39701 "[HiCache] Retry failed storage backup batches and expose terminal failures" Squashed upstream commits: - c311e1c fix(hicache): retry failed storage backup batches - 3edaf3b fix(hicache): harden storage backup retry path Manual merge notes: - Applied as one squashed diff over our fork state. cache_controller.py, hiradix_cache.py, unified_radix_cache.py, metrics_collector.py and the new test landed cleanly; only hybrid_cache_controller.py conflicted. - Conflict 1 (method signature): kept our fork-local `trim_prefetch_full_head` (buffer-mode storage-prefetch pipeline, sgl-project#39283/sgl-project#39050 lineage) directly above the upstream `def _page_backup(self, operation) -> bool:` return type. Both are preserved. - Conflict 2 (sidecar write body): upstream moved the batch_set_v2 call into a nested `write_sidecars()` closure so `_retry_storage_write` can retry it. Our fork-local sgl-project#39050 `extra_info=HiCacheStorageExtraInfo(prefix_keys=...)` argument is kept INSIDE that closure, so retries still forward prefix keys. - Upstream deletes `HybridCacheController.backup_thread_func` (the base implementation now handles sidecar-only ranks and terminal-failure logging). Deletion accepted; the unused `Empty` import was dropped with it. This does not regress fork-local behavior: the base thread still calls the overridden `_page_backup`, which keeps the rank-sharded-sidecar `should_backup` path. - Fork-local `_resolve_sidecar_kv_derived_pool_transfers` keeps `transfer.keys = operation.hash_value` (unconditional) instead of upstream's `if transfer.keys is None` guard; that is pre-existing fork behavior from sgl-project#39283, untouched by this PR. - Also kept from the fork: `dp_rank` / `kv_cache_dtype` storage-config args, `prefix_keys is not None` copy semantics and `batch_get_v2(extra_info=...)`. - Test registered under test/registered/unit/mem_cache/ (valid taxonomy); 12 tests pass locally (CPU-only harness).
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
Implement read-only tombstone span discovery and atomic loaded-window attachment across native node splits. Preserve Full-KV ownership, lock accounting, namespace and bigram semantics, and pending write-through actions. Remove the temporary Python fallback and Rust skips in shared buffer-window tests.
|
@xiezhq-hermann Heads-up: the load-back completeness check added in this PR ( With HiCache on a hybrid-SWA model, a host hit can include FULL tokens that are already device-resident (behind a host-only window/state entry). Observed on main @ 2026-11-16: ( Fix we are running locally is the one-line compare against the transferred row count: - promised_host_hit = req.host_hit_length
+ promised_host_hit = full_load_tokensThis keeps the fail-fast for a genuinely short load-back and matches Full report with repro, traceback and verification: #50892 |
Motivation
Buffer-mode HiCache (host memory used as transport staging, no L2 tier) decided a staged L3 hit's fate inside the prefill adder, after admission had already charged host-hit tokens that a shrunk device prefix could no longer splice. Under load this produced phantom host-hit accounting, holds that wedged admission, and unbounded storage-prefetch retries. PD-prefill and PP schedulers had also stopped draining HiCache events after a scheduler refactor.
Modifications
Prefill adder / scheduler
PrefillAdder.add_one_reqselects a prefill shape without allocating (_select_prefill_admission), materializes host hits under the prefix lock viainit_load_back, then commits (_commit_prefill_admission).init_load_backmay returnNoneto request another admission attempt; FULL over-delivery shrinks the planned extend range, under-delivery re-selects with no host hit._swa_admission_gateand keeps the ring-slot exact-fit comparison for the unified-KV SWA ring.StoragePrefetchRetries(mem_cache/storage_prefetch.py): paced miss polls, immediate re-issues for known hits whose anchor moved, and a per-request cap via--hicache-storage-prefetch-retry-max-attempts(default 8;--hicache-storage-prefetch-retry-poll-intervaldefault becomes 8). Requests past the cap are admitted with whatever the device holds._process_hicache_eventsand called from the PD-prefill and PP scheduler loops.Buffer-mode pipeline
match_full_device_prefix,inc_full_pin,dec_full_pinon the Python and Rust tree cores), so the device prefix under a parked fetch cannot be evicted while the SWA segment lock is left alone.prepare_prefetch(PreparePrefetchResult.staging_tokens) and allocate it once the hit is known (alloc_prefetch_staging).max_staged_admission_defers = 32); past the cap the hold is dropped and the request recomputes.swa_transient_sizeexposes SWA slots owned by staged holds to the scheduler invariant checker.Adaptations to upstream
_swa_admission_gateincorporates the_swa_req_ringexact-fit comparison added upstream in [AMD][DSV4] Reland unified-KV pool sizing and SWA ring accounting, fully gated #38192.arg_groups/fields/memory.py(upstream moved the field declarations there).shrunk).match_full_device_prefixwalks the full radix namespace (extra key + cache salt) likematch_prefixdoes upstream.staging_tokenscontract as the Python components: SWA/MambaPREFETCHbuilds size their placeholder keys from the planned staging and carry no host buffer, which the Python pipeline attaches once the hit is known (alloc_prefetch_staging). Without this the Rust adapter rejected the new keyword and the Rust-backed shared suite failed.swa_tombstone_rangesandattach_swa_window, which have no Rust port yet. The Rust adapter exposes them as explicitNotImplementedErrorstubs (same pattern as itsroot_node), and the three shared-suite tests that exercise that path (test_buffer_only_load_back_uses_full_behind_swa_tombstone,test_buffer_only_load_back_reuses_partial_masked_full,test_buffer_only_load_back_trims_head_published_by_sibling) are skipped under the Rust backend. The last one passed under Rust before this PR, so this is a deliberate coverage regression for buffer-only + SWA + L3 on the Rust core until the helpers are ported; the Python core is unaffected.Original commits
0e78bbe270bdc1fb698557b0a7f139Accuracy Tests
N/A (scheduler / cache bookkeeping change; no numerics affected).
Benchmarking and Profiling
Validated on an internal buffer-mode + L3 setup: warm-cache prefix hit rate no longer collapses under high concurrency and no admission wedges were observed. No absolute numbers are included here.
Checklist
Local validation (details in the first comment): Rust
cargo test --locked(865 passed incl. the new pin test),cargo check --features python-extension,inspection,cargo fmt --check,cargo clippy -D warnings; Python unit suitestest_unified_radix_cache_unittest.py,test_rust_unified_radix_cache_unittest.py,test_storage_prefetch_lifecycle.py,test_prefill_adder.py,test_scheduler_hicache_events.py,test_buffer_mode_sidecar.py,test_hicache_staged_write_back_dispatch.py,test_rust_tree_core.py,test_tree_core_registry.py;ruff,ruff format,isort,codespellon changed files.CI States
Latest PR Test (Base): ✅ Run #34939088034
Latest PR Test (Extra): ❌ Run #34939087207
Latest PR Test (AMD ROCm 10): ⏳ Run #34939087808