Skip to content

[HiCache][KVCR] Use named composite pools for hybrid cache - #2

Draft
ziqifan617 wants to merge 18 commits into
linhu-nv:linhu/kvcr-hicache-backendfrom
ziqifan617:ziqifan617/kvcr-dsv4-hicache
Draft

ziqifan617 wants to merge 18 commits into
linhu-nv:linhu/kvcr-hicache-backendfrom
ziqifan617:ziqifan617/kvcr-dsv4-hicache

Conversation

@ziqifan617

@ziqifan617 ziqifan617 commented Sep 8, 2026 •

Copy link
Copy Markdown

Dependency and review scope

This remains a stacked draft on sgl-project/sglang#36409, using linhu-nv/sglang:linhu/kvcr-hicache-backend at 931c05606 as its base.

ai-dynamo/kvcr#19 is now merged. It supplies the named-layout and descriptor-list public API used here, but the merged runtime still rejects more than one layout or descriptor. DSV4 therefore also needs the follow-up multipool runtime. The GPU validation used the ai-dynamo/kvcr#20 squash rebased onto current KVCR main plus its singleton G3 adapter; that runtime is not merged.

The adapter now probes this runtime capability before allocating composite HiCache CPU pools. Installing PR sgl-project#19 alone produces an actionable construction error instead of a late allocation or transfer failure.

Related original main-targeted draft: sgl-project/sglang#38034.

Summary

Represent each physical HiCache pool page as one atomic KVCR record whose descriptor list contains all of that page's non-contiguous components.

  • Defer KVCR construction until HiCache has registered and validated the complete physical-pool topology.
  • Register every framework-owned host-memory region with KVCR/NIXL.
  • Create named local pools as <physical-pool>:<component-size>. Repeated equal-size components within one physical pool share that allocator; equal sizes in different physical pools keep separate capacity.
  • Size each named pool for the same whole-page capacity. Both hybrid and ordinary configurations round down to complete pages and reject a budget that cannot hold one page.
  • Use a v5 pool-page key namespace tied to the hybrid cache compatibility digest.
  • Deposit and deliver one descriptor list per physical-pool page. Remote misses use deliver() directly into HiCache host pages; this adapter does not use KVCR fetch().
  • Consume the canonical Dynamo kv.fetch@1.0 envelope, retain legacy input shapes for rolling upgrades, preserve Dynamo message_id/action_id, send unsigned u64 block hashes, and emit KVCR's exact two-field payload.
  • Retain the existing TP/CP ownership, terminal-transfer lifetime, and fail-closed behavior.

Hint integration boundary

Dynamo PR sgl-project#13134 now emits the typed envelope in the top-level singular request field kv_hint. SGLang RFC sgl-project#36224 exposes the envelope as plural kv_hints. The Dynamo SGLang bridge must translate the field name without merging or rebuilding the envelope for both native and non-native request paths.

The current head of Dynamo PR sgl-project#13591 still reads the older nested extra_args.kv_transfer_params.router_hint location. Until that PR is rebased, the direct /generate canary below proves the SGLang -> HiCache -> KVCR -> NIXL path, not a Dynamo-router-generated hint reaching SGLang.

Correctness boundary

KVCR owns atomicity within a physical-pool page. SGLang still restores different physical pools independently because their usable prefix windows can differ. The compatibility digest remains responsible for distinguishing component/layer ordering when equal-sized descriptors alone cannot do so.

Validation

DSV4 GPU canary

Passed with two same-node B200-backed SGLang workers running nvidia/DeepSeek-V4-Flash-NVFP4, TP=1 and DP=1:

  1. Worker A computed a 1,536-token prompt and deposited its five reusable
    256-token pages.
  2. Both workers' device and HiCache host tiers were flushed.
  3. Worker B received the canonical kv.fetch@1.0 envelope in kv_hints and restored the five complete pages: 1,280 tokens from storage, computing only the final 256-token page.
  4. Worker B's next request reused the same 1,280 tokens from its device cache.

The causal transport evidence is six matching source_write/remote_deliver operation IDs across all six DSV4 pools: 18/18 blocks and 12,720,960/12,720,960 bytes. HiCache reported completed=1280 matched=0 loaded=1280; all five control/reuse requests returned identical output token IDs; the harness exited zero with no recorded errors.

This is a correctness canary, not a performance result. It used a localhost control endpoint on one node, so it does not prove Dynamo routing, disaggregated serving, cross-node RDMA, DSV4-Pro, concurrency, or load behavior.

Focused tests

  • 77 focused KVCR composite, G3, Guard, and hint tests passed.
  • 108 focused SGLang tests and 39 subtests passed, covering hint schema, capability gates, composite descriptors, TP/concurrency, HiCache sidecar, and hybrid-storage lifecycle.
  • The final canonical-payload delta passed 16 tests and 13 subtests.
  • Python bytecode compilation and diff checks passed.

Known KVCR validation-runtime boundary

The full KVCR sweep reached 309 passes, with three failures and two teardown errors confined to Guard integration. One real-UCX Guard failure reproduces on current KVCR main. The other two are shutdown behavior from unrelated terminal-resource-retention code bundled in the multipool validation squash. Multipool DSV4 rejects Guard/G3 and did not execute those paths in the passing DSV4 canary, but the runtime should be split from those unrelated changes before it is proposed for merge.


CI States

Latest PR Test (Base): ❌ Run #34289496873
Latest PR Test (Extra): ❌ Run #34289496449

linhu-nv and others added 13 commits September 6, 2026 11:04
Adds nvidia-kvcr as a HiCacheStorage backend so a worker can serve a prefix
another worker computed, without recomputing it and without a shared store in
the middle. A router that already knows which instance holds a prefix (dynamo's
KV router) passes that knowledge down as a per-request hint; the backend uses it
to pull the pages peer-to-peer over NIXL.

Three parts:

* `mem_cache/storage/kvcr/` -- the backend itself: config validation, the
  HiCacheStorage implementation, and the host-pool pin adapter. Runs in either
  HiCache cache mode or buffer-only mode.

* Router-hint plumbing -- a per-request `kv_hints` envelope threaded from the
  entrypoints through the scheduler into `HiCacheStorageExtraInfo`, so a storage
  backend can locate a remote source. The controllers stay backend-agnostic: the
  hint is an opaque dict to everything between the entrypoint and the backend.

* KV-event alignment -- the block-stored/removed events grow the positional
  fields routers already parse, so an external router can track what each
  instance holds.

The backend is optional: without `nvidia-kvcr` installed the import is skipped
and the suite stays green.
nvidia-kvcr moved hint parsing into the core, which reshapes three things the
backend touches:

* `submit_hint` now takes the hint as a plain dict and parses it itself, so it
  raises on a hint it rejects instead of degrading to local-only. We validate
  the same fields first, so a raise means the two parsers disagree -- count it
  as a fault rather than fail the batch.

* `KeyHintAdapter.matches(key, hint)` is gone. The core holds the block hashes
  as a `frozenset[int]` and tests `KeyAdapter.decode(key) in hashes`, so the
  adapter only translates keys now; `StrKeyHintAdapter` becomes `StrKeyAdapter`
  and the binding kwarg follows.

* `LocalDramInfo` is renamed to `LocalDramOptions`; the fields are unchanged.

Both sides of that new membership test are produced here, and both must be
**unsigned**: the core validates every wire hash into `0 <= h < 1<<64`, while
SGLang's `hash_str_to_int64` yields a signed int64 from the same 16 hex chars.
Decoding through the hex prefix rather than through that helper keeps the two
on the value the router actually indexed -- a signed value on either side makes
every hinted block miss with nothing logged, which is why the schema test now
pins what `to_kvcr_hint()` emits against what `decode()` answers.
Signed-off-by: Ziqi Fan <ziqif@nvidia.com>
@ziqifan617 ziqifan617 changed the title [HiCache][DSV4] Add hybrid-pool support to KVCR [HiCache][KVCR] Use named composite pools for hybrid cache Sep 8, 2026
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 8, 2026
@linhu-nv
linhu-nv force-pushed the linhu/kvcr-hicache-backend branch from 931c056 to 930783d Compare September 18, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation hicache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants