Repository navigation
[HiCache][KVCR] Use named composite pools for hybrid cache - #2
Draft
ziqifan617 wants to merge 18 commits into
Draft
ziqifan617 wants to merge 18 commits into
ziqifan617 wants to merge 18 commits into
Conversation
Adds nvidia-kvcr as a HiCacheStorage backend so a worker can serve a prefix another worker computed, without recomputing it and without a shared store in the middle. A router that already knows which instance holds a prefix (dynamo's KV router) passes that knowledge down as a per-request hint; the backend uses it to pull the pages peer-to-peer over NIXL. Three parts: * `mem_cache/storage/kvcr/` -- the backend itself: config validation, the HiCacheStorage implementation, and the host-pool pin adapter. Runs in either HiCache cache mode or buffer-only mode. * Router-hint plumbing -- a per-request `kv_hints` envelope threaded from the entrypoints through the scheduler into `HiCacheStorageExtraInfo`, so a storage backend can locate a remote source. The controllers stay backend-agnostic: the hint is an opaque dict to everything between the entrypoint and the backend. * KV-event alignment -- the block-stored/removed events grow the positional fields routers already parse, so an external router can track what each instance holds. The backend is optional: without `nvidia-kvcr` installed the import is skipped and the suite stays green.
nvidia-kvcr moved hint parsing into the core, which reshapes three things the backend touches: * `submit_hint` now takes the hint as a plain dict and parses it itself, so it raises on a hint it rejects instead of degrading to local-only. We validate the same fields first, so a raise means the two parsers disagree -- count it as a fault rather than fail the batch. * `KeyHintAdapter.matches(key, hint)` is gone. The core holds the block hashes as a `frozenset[int]` and tests `KeyAdapter.decode(key) in hashes`, so the adapter only translates keys now; `StrKeyHintAdapter` becomes `StrKeyAdapter` and the binding kwarg follows. * `LocalDramInfo` is renamed to `LocalDramOptions`; the fields are unchanged. Both sides of that new membership test are produced here, and both must be **unsigned**: the core validates every wire hash into `0 <= h < 1<<64`, while SGLang's `hash_str_to_int64` yields a signed int64 from the same 16 hex chars. Decoding through the hex prefix rather than through that helper keeps the two on the value the router actually indexed -- a signed value on either side makes every hinted block miss with nothing logged, which is why the schema test now pins what `to_kvcr_hint()` emits against what `decode()` answers.
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
linhu-nv
force-pushed
the
linhu/kvcr-hicache-backend
branch
from
September 18, 2026 05:45
931c056 to
930783d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency and review scope
This remains a stacked draft on sgl-project/sglang#36409, using
linhu-nv/sglang:linhu/kvcr-hicache-backendat931c05606as its base.ai-dynamo/kvcr#19 is now merged. It supplies the named-layout and descriptor-list public API used here, but the merged runtime still rejects more than one layout or descriptor. DSV4 therefore also needs the follow-up multipool runtime. The GPU validation used the ai-dynamo/kvcr#20 squash rebased onto current KVCR main plus its singleton G3 adapter; that runtime is not merged.
The adapter now probes this runtime capability before allocating composite HiCache CPU pools. Installing PR sgl-project#19 alone produces an actionable construction error instead of a late allocation or transfer failure.
Related original main-targeted draft: sgl-project/sglang#38034.
Summary
Represent each physical HiCache pool page as one atomic KVCR record whose descriptor list contains all of that page's non-contiguous components.
<physical-pool>:<component-size>. Repeated equal-size components within one physical pool share that allocator; equal sizes in different physical pools keep separate capacity.v5pool-page key namespace tied to the hybrid cache compatibility digest.deliver()directly into HiCache host pages; this adapter does not use KVCRfetch().kv.fetch@1.0envelope, retain legacy input shapes for rolling upgrades, preserve Dynamomessage_id/action_id, send unsigned u64 block hashes, and emit KVCR's exact two-field payload.Hint integration boundary
Dynamo PR sgl-project#13134 now emits the typed envelope in the top-level singular request field
kv_hint. SGLang RFC sgl-project#36224 exposes the envelope as pluralkv_hints. The Dynamo SGLang bridge must translate the field name without merging or rebuilding the envelope for both native and non-native request paths.The current head of Dynamo PR sgl-project#13591 still reads the older nested
extra_args.kv_transfer_params.router_hintlocation. Until that PR is rebased, the direct/generatecanary below proves the SGLang -> HiCache -> KVCR -> NIXL path, not a Dynamo-router-generated hint reaching SGLang.Correctness boundary
KVCR owns atomicity within a physical-pool page. SGLang still restores different physical pools independently because their usable prefix windows can differ. The compatibility digest remains responsible for distinguishing component/layer ordering when equal-sized descriptors alone cannot do so.
Validation
DSV4 GPU canary
Passed with two same-node B200-backed SGLang workers running
nvidia/DeepSeek-V4-Flash-NVFP4, TP=1 and DP=1:256-token pages.
kv.fetch@1.0envelope inkv_hintsand restored the five complete pages: 1,280 tokens from storage, computing only the final 256-token page.The causal transport evidence is six matching
source_write/remote_deliveroperation IDs across all six DSV4 pools: 18/18 blocks and 12,720,960/12,720,960 bytes. HiCache reportedcompleted=1280 matched=0 loaded=1280; all five control/reuse requests returned identical output token IDs; the harness exited zero with no recorded errors.This is a correctness canary, not a performance result. It used a localhost control endpoint on one node, so it does not prove Dynamo routing, disaggregated serving, cross-node RDMA, DSV4-Pro, concurrency, or load behavior.
Focused tests
Known KVCR validation-runtime boundary
The full KVCR sweep reached 309 passes, with three failures and two teardown errors confined to Guard integration. One real-UCX Guard failure reproduces on current KVCR main. The other two are shutdown behavior from unrelated terminal-resource-retention code bundled in the multipool validation squash. Multipool DSV4 rejects Guard/G3 and did not execute those paths in the passing DSV4 canary, but the runtime should be split from those unrelated changes before it is proposed for merge.
CI States
Latest PR Test (Base): ❌ Run #34289496873
Latest PR Test (Extra): ❌ Run #34289496449