Skip to content

feat(kvbm-engine): integrate Mooncake Store as G4 object storage backend - #9786

Open
Dayuxiaoshui wants to merge 19 commits into
ai-dynamo:mainfrom
yuechen-sys:main
Open

Dayuxiaoshui wants to merge 19 commits into
ai-dynamo:mainfrom
yuechen-sys:main

Conversation

@Dayuxiaoshui

@Dayuxiaoshui Dayuxiaoshui commented May 20, 2026 •

Copy link
Copy Markdown

This commit adds Mooncake Store (distributed KV cache storage from Moonshot AI) as a G4 tier backend for Dynamo KVBM, enabling cross-node KV cache sharing.

Phase 1: TCP copy path

  • MooncakeObjectBlockClient implementing ObjectBlockOps trait
  • MooncakeObjectConfig with protocol selection (tcp/rdma)
  • MooncakeLockManager with optimistic locking (idempotent writes)

Phase 2: RDMA zero-copy + Batch API

  • with_zero_copy() builder and register_layout() for buffer registration
  • batch_put_from / batch_get_into for contiguous layouts (1 RPC batch)
  • Automatic fallback to copy path for non-contiguous layouts

Phase 3: KV Router G4 event linkage

  • OffloadEngineBuilder::with_event_sender()
  • ObjectTransferExecutor emits KvCacheEvent::Create on successful offload

Testing:


Open in Devin Review

Summary by CodeRabbit

  • New Features

    • Added Mooncake object-storage backend with zero-copy RDMA support for distributed block operations
    • Added event notification system to track offload completion status
  • Configuration

    • Extended configuration support for Mooncake backend settings via environment variables and configuration files

Review Change Stack

Dayuxiaoshui and others added 4 commits May 5, 2026 14:56
This commit adds Mooncake Store (distributed KV cache storage from Moonshot AI)
as a G4 tier backend for Dynamo KVBM, enabling cross-node KV cache sharing.

Phase 1: TCP copy path
- MooncakeObjectBlockClient implementing ObjectBlockOps trait
- MooncakeObjectConfig with protocol selection (tcp/rdma)
- MooncakeLockManager with optimistic locking (idempotent writes)

Phase 2: RDMA zero-copy + Batch API
- with_zero_copy() builder and register_layout() for buffer registration
- batch_put_from / batch_get_into for contiguous layouts (1 RPC batch)
- Automatic fallback to copy path for non-contiguous layouts

Phase 3: KV Router G4 event linkage
- OffloadEngineBuilder::with_event_sender()
- ObjectTransferExecutor emits KvCacheEvent::Create on successful offload

Testing:
- 3 integration tests pass (copy + zero-copy roundtrip)
- cargo check with --features mooncake,testing clean
feat(kvbm-engine): integrate Mooncake Store as G4 object storage backend
- Bugfix: has_blocks() now preserves non-existing keys in result
- Safety: prevent null pointer from being passed to Mooncake C library
  in batch zero-copy path; fall through to single-block path on error
- Refactor: extract format_storage_key() helper to reduce duplication
- Refactor: change MooncakeObjectConfig.protocol from String to
  MooncakeProtocol enum for compile-time validation
- Remove redundant #[cfg(feature = "mooncake")] guards inside
  already-gated mooncake module
- Move integration test from examples/ to tests/
- Document: out-of-tree Mooncake path dependency in Cargo.toml
- Document: Linux-only /proc/sys/kernel/hostname default
- Document: intentional error-swallowing in MooncakeLockManager::has_meta
fix(kvbm-engine): resolve Mooncake Store code review issues
@copy-pr-bot

copy-pr-bot Bot commented May 20, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi Dayuxiaoshui! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added feat external-contribution Pull request is from an external contributor labels May 20, 2026
@coderabbitai

coderabbitai Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

This PR integrates Mooncake distributed KV store as an alternative object-storage backend for KVBM. It adds Mooncake configuration types with environment variable support, implements a complete object client with zero-copy RDMA and CPU fallback block operations, provides optimistic lock management, wires factory functions to dispatch between S3 and Mooncake, adds G4 offload completion event notifications, and includes integration tests and layout helpers.

Changes

Mooncake Backend Support

Layer / File(s) Summary
Configuration Schema and Environment Variables
lib/kvbm-config/src/lib.rs, lib/kvbm-config/src/object.rs
MooncakeProtocol enum and MooncakeObjectConfig struct added with serde defaults; hostname reads from /proc/sys/kernel/hostname with "localhost" fallback. Environment variables prefixed KVBM_OBJECT_CLIENT_ are merged into object.client.* config namespace.
Cargo Dependencies and Features
lib/kvbm-engine/Cargo.toml
Optional mooncake_store dependency added via local path; new mooncake feature enables mooncake_store, rayon, and tokio-rayon.
MooncakeObjectBlockClient Implementation
lib/kvbm-engine/src/object/mooncake/client.rs
Core client with centralized key formatting, RDMA buffer registration, zero-copy batch put/get with per-block CPU fallback, layout serialization for fully-contiguous and layer-separated layouts, and ObjectBlockOps trait providing concurrent has_blocks with size lookups.
MooncakeLockManager Implementation
lib/kvbm-engine/src/object/mooncake/mod.rs, lib/kvbm-engine/src/object/mooncake/lock.rs
Optimistic lock manager with <hash>.meta existence checks, unconditional overwrite semantics, and error-as-not-found behavior for transient failures.
Object Client and Lock Manager Factory Functions
lib/kvbm-engine/src/object/mod.rs
create_object_client and create_lock_manager expanded from s3-only to dispatch both S3 and Mooncake backends; explicit error paths for unimplemented Nixl and missing features.
G4 Offload Completion Event Notifications
lib/kvbm-engine/src/offload/engine.rs, lib/kvbm-engine/src/offload/pipeline.rs
Optional broadcast sender wired through OffloadEngineBuilder → ObjectPipeline → ObjectTransferExecutor to emit KvCacheEvent::Create per successful block upload; unit tests updated with event_sender: None.
Integration Tests and Layout Utilities
lib/kvbm-engine/tests/mooncake_integration.rs, lib/kvbm-physical/src/testing/physical.rs
Integration test performs raw store lifecycle and has_blocks checks. Test helpers create_fc_layout_system and create_lw_layout_system construct system-memory layouts without requiring Nixl agent.

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The PR title accurately summarizes the main change: adding Mooncake Store as a G4 object storage backend to kvbm-engine, which is the primary objective of this changeset.
Description check ✅ Passed The PR description is comprehensive and addresses all required template sections with clear details about phases, testing, and linked issues.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
lib/kvbm-engine/src/offload/engine.rs (1)

462-469: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Wire event_sender into remote G4 offloads too.

Right now the sender is only passed to the local ObjectPipeline. If callers use with_enable_remote_g4(true), remote_g4_offload_task() still completes uploads without ever emitting KvCacheEvent::Create, so distributed G4 offloads stay invisible to KV Router subscribers.

Suggested wiring
             Some(ObjectPipeline::new(
                 config,
                 object_ops,
                 LogicalLayoutHandle::G2,
                 self.leader.clone(),
                 runtime.clone(),
-                self.event_sender,
+                self.event_sender.clone(),
             ))
-            Some(runtime.spawn(remote_g4_offload_task(rx, self.leader.clone())))
+            Some(runtime.spawn(remote_g4_offload_task(
+                rx,
+                self.leader.clone(),
+                self.event_sender,
+            )))

Then emit KvCacheEvent::Create for request.keys after notification.await succeeds inside remote_g4_offload_task().

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/kvbm-engine/src/offload/engine.rs` around lines 462 - 469, The
event_sender is only wired into the local pipeline; update the remote G4 offload
path so the ObjectPipeline constructed for LogicalLayoutHandle::G4 (where
remote_g4_offload_task is used) also receives self.event_sender (i.e., pass
event_sender into the ObjectPipeline::new call used for G4 offloads or otherwise
make event_sender available to remote_g4_offload_task). Then, inside
remote_g4_offload_task, after awaiting the upload notification success, emit
KvCacheEvent::Create for request.keys via the same event_sender so distributed
G4 offloads generate the same Create events as local pipelines; ensure you
reference with_enable_remote_g4 and the existing event_sender variable when
wiring.
🧹 Nitpick comments (2)
lib/kvbm-engine/src/offload/pipeline.rs (1)

2049-2055: ⚡ Quick win

Add one test that actually exercises event emission.

All of the updated executor tests still construct SharedObjectExecutorState with event_sender: None, so the new KvCacheEvent::Create path can regress without any test failing. A small broadcast::channel assertion in the partial-failure case would cover the important contract: emit only successful hashes.

Also applies to: 2126-2132, 2195-2201

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/kvbm-engine/src/offload/pipeline.rs` around lines 2049 - 2055, Tests
construct SharedObjectExecutorState with event_sender: None, so the new
KvCacheEvent::Create path isn't exercised; add a test that supplies a real
broadcast::channel sender to SharedObjectExecutorState and asserts emitted
events in the partial-failure case. Specifically, in the executor test that
exercises object creation (the tests that build SharedObjectExecutorState and
call the executor path that can partially fail), create a
tokio::sync::broadcast::channel, pass Some(sender.clone()) into
SharedObjectExecutorState.event_sender, run the executor to trigger
KvCacheEvent::Create, and assert the receiver only receives the successful
hashes (no failed ones); mirror this pattern for the other two similar locations
referenced (around the SharedObjectExecutorState constructions at the other
ranges).
lib/kvbm-engine/src/object/mooncake/mod.rs (1)

4-5: ⚡ Quick win

Add module doc include to satisfy module-doc policy.

Please add a module-level #[doc = include_str!(...)] and corresponding docs file for this new module.

As per coding guidelines, “Module documentation should live in docs/ and be included via #[doc = include_str!("../docs/...")] in module files. When modifying a module, update the corresponding doc file.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/kvbm-engine/src/object/mooncake/mod.rs` around lines 4 - 5, Add a
module-level documentation include for the mooncake module: create a
docs/mooncake.md file with the module docs and add a top-level #[doc =
include_str!("../docs/mooncake.md")] to
lib/kvbm-engine/src/object/mooncake/mod.rs above the pub mod declarations (the
module exposes client and lock), ensuring the doc file text describes the module
and follows project style; update the docs file whenever the module changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/kvbm-engine/Cargo.toml`:
- Around line 47-51: The local path dependency mooncake_store in Cargo.toml
breaks CI; replace the path = "../../../Mooncake/mooncake-store/rust" entry for
the mooncake_store dependency with a git-based source (e.g. git =
"https://github.com/kvcache-ai/Mooncake" plus a specific tag or rev such as tag
= "vX.Y.Z" or rev = "<commit>") while keeping optional = true so CI can resolve
it; after making the change, update Cargo.lock (run cargo update or relevant
tooling) to ensure the workspace builds in CI.

In `@lib/kvbm-engine/src/object/mooncake/client.rs`:
- Around line 242-243: Before zipping keys and block_ids into work_items,
validate that keys.len() == block_ids.len() and return or propagate an explicit
error (or panic/log) if they differ to avoid silent truncation; update the two
places where work_items is created (the expression using
keys.into_iter().zip(block_ids) and the similar zip at the other occurrence) to
perform the length check first and only zip after validating lengths, including
a clear error message that references the operation (e.g., "upload/download
block count mismatch") so callers can detect and handle the input error.
- Around line 507-535: The current loop that builds size queries and then uses
buffer_unordered can change ordering and may drop entries if exist_results
length differs; fix has_blocks by preserving 1:1 positional correspondence:
iterate keys.enumerate() to pair each (index, (hash, key)) and look up the
matching exist flag via index (or explicitly validate exist_results.len() ==
keys.len() and handle mismatches by pushing None for missing entries), then when
creating tasks include the original index (e.g., tasks produce (index, hash,
Option<size>)), run the concurrent stream, collect all results, sort them by
index to restore deterministic order, and finally map to Vec<(SequenceHash,
Option<usize>)> in the original order so results align exactly with the input
keys (update references to keys, formatted, exist_results, size_queries, tasks,
size_results accordingly).

---

Outside diff comments:
In `@lib/kvbm-engine/src/offload/engine.rs`:
- Around line 462-469: The event_sender is only wired into the local pipeline;
update the remote G4 offload path so the ObjectPipeline constructed for
LogicalLayoutHandle::G4 (where remote_g4_offload_task is used) also receives
self.event_sender (i.e., pass event_sender into the ObjectPipeline::new call
used for G4 offloads or otherwise make event_sender available to
remote_g4_offload_task). Then, inside remote_g4_offload_task, after awaiting the
upload notification success, emit KvCacheEvent::Create for request.keys via the
same event_sender so distributed G4 offloads generate the same Create events as
local pipelines; ensure you reference with_enable_remote_g4 and the existing
event_sender variable when wiring.

---

Nitpick comments:
In `@lib/kvbm-engine/src/object/mooncake/mod.rs`:
- Around line 4-5: Add a module-level documentation include for the mooncake
module: create a docs/mooncake.md file with the module docs and add a top-level
#[doc = include_str!("../docs/mooncake.md")] to
lib/kvbm-engine/src/object/mooncake/mod.rs above the pub mod declarations (the
module exposes client and lock), ensuring the doc file text describes the module
and follows project style; update the docs file whenever the module changes.

In `@lib/kvbm-engine/src/offload/pipeline.rs`:
- Around line 2049-2055: Tests construct SharedObjectExecutorState with
event_sender: None, so the new KvCacheEvent::Create path isn't exercised; add a
test that supplies a real broadcast::channel sender to SharedObjectExecutorState
and asserts emitted events in the partial-failure case. Specifically, in the
executor test that exercises object creation (the tests that build
SharedObjectExecutorState and call the executor path that can partially fail),
create a tokio::sync::broadcast::channel, pass Some(sender.clone()) into
SharedObjectExecutorState.event_sender, run the executor to trigger
KvCacheEvent::Create, and assert the receiver only receives the successful
hashes (no failed ones); mirror this pattern for the other two similar locations
referenced (around the SharedObjectExecutorState constructions at the other
ranges).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3561ae62-dfe6-4191-a621-1d1f619b9c3b

📥 Commits

Reviewing files that changed from the base of the PR and between a9984dc and abed905.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (11)
  • lib/kvbm-config/src/lib.rs
  • lib/kvbm-config/src/object.rs
  • lib/kvbm-engine/Cargo.toml
  • lib/kvbm-engine/src/object/mod.rs
  • lib/kvbm-engine/src/object/mooncake/client.rs
  • lib/kvbm-engine/src/object/mooncake/lock.rs
  • lib/kvbm-engine/src/object/mooncake/mod.rs
  • lib/kvbm-engine/src/offload/engine.rs
  • lib/kvbm-engine/src/offload/pipeline.rs
  • lib/kvbm-engine/tests/mooncake_integration.rs
  • lib/kvbm-physical/src/testing/physical.rs

Comment thread lib/kvbm-engine/Cargo.toml Resolved
Dayuxiaoshui and others added 2 commits May 21, 2026 12:41
- has_blocks(): preserve 1:1 cardinality and deterministic ordering
- put/get_blocks_with_layout(): add keys/block_ids length mismatch guard
- Fix buffer_unordered ordering for SpmdParallelWorkers (index tracking)
- Wire event_sender to remote G4 offload task for distributed mode
- Add test verifying KvCacheEvent::Create emission in partial-failure path
- Add mooncake module docs following project convention
- Add CI path dependency guidance in Cargo.toml
- Add #![cfg(feature = "mooncake")] guard to integration test
fix(kvbm-engine): resolve Mooncake Store code review issues
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 21, 2026
Signed-off-by: Dayuxiaoshui <158081477+Dayuxiaoshui@users.noreply.github.com>
@Dayuxiaoshui
Dayuxiaoshui requested a review from a team as a code owner May 21, 2026 05:04
Dayuxiaoshui and others added 4 commits May 21, 2026 13:16
Signed-off-by: Dayuxiaoshui <158081477+Dayuxiaoshui@users.noreply.github.com>
- Add Mooncake clone step to rust-clippy job (before cargo metadata)
- Fix formatting in kvbm-physical/src/testing/physical.rs
fix(ci): resolve cargo fmt/metadata failures on Mooncake path dep   @Dayuxiaoshui
@Dayuxiaoshui

Copy link
Copy Markdown
Author

cc @dynamo-ops

@Dayuxiaoshui

Copy link
Copy Markdown
Author

cc @dynamo-ops

Comment thread lib/kvbm-engine/docs/mooncake.md Resolved
Comment thread lib/kvbm-engine/docs/mooncake.md Resolved
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions Bot added the Stale label Nov 6, 2026
…yout failure

register_layout registered buffers in a loop, so an error on a later buffer
returned early and leaked every registration made before it. Wrap the loop in
a guard that unregisters the already-registered buffers on drop unless the
whole layout succeeded, making the call all-or-nothing on error and on panic.

unregister_layout had the mirror-image problem: it bailed out on the first
error, leaving later buffers registered. Attempt every buffer and return the
first error instead.

Addresses the review comment on client.rs:94.
bench_mooncake measures G4 put/get/has throughput and p50/p99 latency using
Qwen3-30B-A3B KV geometry (48 layers, 4 KV heads, head_dim 128, bf16 = 96
KiB/token) across fully-contiguous and layer-separate layouts, for both the
TCP copy path and the RDMA zero-copy path.

Expose create_{fc,lw}_layout_system_with_config from kvbm-physical testing so
the bench can build model-shaped layouts without a NIXL agent.
…e tests

Drop the unused TransferStatus import from four offload pipeline tests and
bind the unused TransferState handle to _handle.
…oughput

Add docs/integrations/mooncake-store-integration.md covering prerequisites,
build, cluster setup, configuration, G4 wiring from Rust, and troubleshooting.
It also resolves the dangling reference from kvbm-engine/Cargo.toml.

Fold the headline numbers into the KVBM guide as a G4 Object Storage Tier
section so the deployment docs no longer stop at G3, and register the new
page in the docs nav.

Measured on one H20 with Qwen3-30B-A3B KV geometry (96 KiB/token), 32K-token
prefix: RDMA zero-copy is 119 ms put / 116 ms get on one 400 Gb/s NIC (45/42
ms across four), against 1658/1703 ms for the TCP copy path.

Document two API traps found while benchmarking: use_zero_copy is a builder
flag that create_object_client never sets, so a factory-built client takes
the copy path even with protocol: rdma, and register_layout is not called
implicitly before transfers.
@Dayuxiaoshui
Dayuxiaoshui requested review from a team as code owners November 9, 2026 13:20
@Dayuxiaoshui
Dayuxiaoshui temporarily deployed to external_collaborator November 9, 2026 13:20 — with GitHub Actions Inactive
@github-actions github-actions Bot removed the Stale label Nov 10, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

External PR triage (automated, 2026-11-11)

Blocker: merge conflict with main (dirty, last push 2026-11-09). Trusted CI never ran. DCO is action_required, and Require an issue link fails. @ryanolson and @cheese-head are requested as reviewers.

Next action. Owner: @Dayuxiaoshui

  1. Rebase onto current main, fix DCO sign-off, add an issue link, and pin the Mooncake clone (see audit).
  2. Then @ai-dynamo/dynamo-ops-codeowners must sign off on the workflow change before anyone posts /ok to test. After that, @ryanolson / @ai-dynamo/dynamo-kv-memory-codeowners review.

Supply-chain audit: HOLD. .github/workflows/pre-merge.yml adds an unpinned git clone --depth 1 https://github.com/kvcache-ai/Mooncake.git in two CI jobs. The new optional mooncake_store path dependency builds native FFI from that mutable checkout, and Cargo.lock adds bindgen 0.70.1. CI would run untrusted, unpinned external code. Pin it to a reviewed commit SHA, or have ops explicitly accept the risk. No explicit secrets access and no binary files.

Written by Devin

This branch was previously deployed

1 inactive deployment
external_collaborator — 05a20dd2 Deployed Nov 9, 2026 by Dayuxiaoshui via ok-to-test #48644
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants