Repository navigation
feat(kvbm-engine): integrate Mooncake Store as G4 object storage backend - #9786
Dayuxiaoshui wants to merge 19 commits into
Conversation
This commit adds Mooncake Store (distributed KV cache storage from Moonshot AI) as a G4 tier backend for Dynamo KVBM, enabling cross-node KV cache sharing. Phase 1: TCP copy path - MooncakeObjectBlockClient implementing ObjectBlockOps trait - MooncakeObjectConfig with protocol selection (tcp/rdma) - MooncakeLockManager with optimistic locking (idempotent writes) Phase 2: RDMA zero-copy + Batch API - with_zero_copy() builder and register_layout() for buffer registration - batch_put_from / batch_get_into for contiguous layouts (1 RPC batch) - Automatic fallback to copy path for non-contiguous layouts Phase 3: KV Router G4 event linkage - OffloadEngineBuilder::with_event_sender() - ObjectTransferExecutor emits KvCacheEvent::Create on successful offload Testing: - 3 integration tests pass (copy + zero-copy roundtrip) - cargo check with --features mooncake,testing clean
feat(kvbm-engine): integrate Mooncake Store as G4 object storage backend
- Bugfix: has_blocks() now preserves non-existing keys in result - Safety: prevent null pointer from being passed to Mooncake C library in batch zero-copy path; fall through to single-block path on error - Refactor: extract format_storage_key() helper to reduce duplication - Refactor: change MooncakeObjectConfig.protocol from String to MooncakeProtocol enum for compile-time validation - Remove redundant #[cfg(feature = "mooncake")] guards inside already-gated mooncake module - Move integration test from examples/ to tests/ - Document: out-of-tree Mooncake path dependency in Cargo.toml - Document: Linux-only /proc/sys/kernel/hostname default - Document: intentional error-swallowing in MooncakeLockManager::has_meta
fix(kvbm-engine): resolve Mooncake Store code review issues
|
👋 Hi Dayuxiaoshui! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
WalkthroughThis PR integrates Mooncake distributed KV store as an alternative object-storage backend for KVBM. It adds Mooncake configuration types with environment variable support, implements a complete object client with zero-copy RDMA and CPU fallback block operations, provides optimistic lock management, wires factory functions to dispatch between S3 and Mooncake, adds G4 offload completion event notifications, and includes integration tests and layout helpers. ChangesMooncake Backend Support
🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
lib/kvbm-engine/src/offload/engine.rs (1)
462-469:⚠️ Potential issue | 🟠 Major | ⚡ Quick winWire
event_senderinto remote G4 offloads too.Right now the sender is only passed to the local
ObjectPipeline. If callers usewith_enable_remote_g4(true),remote_g4_offload_task()still completes uploads without ever emittingKvCacheEvent::Create, so distributed G4 offloads stay invisible to KV Router subscribers.Suggested wiring
Some(ObjectPipeline::new( config, object_ops, LogicalLayoutHandle::G2, self.leader.clone(), runtime.clone(), - self.event_sender, + self.event_sender.clone(), ))- Some(runtime.spawn(remote_g4_offload_task(rx, self.leader.clone()))) + Some(runtime.spawn(remote_g4_offload_task( + rx, + self.leader.clone(), + self.event_sender, + )))Then emit
KvCacheEvent::Createforrequest.keysafternotification.awaitsucceeds insideremote_g4_offload_task().🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/kvbm-engine/src/offload/engine.rs` around lines 462 - 469, The event_sender is only wired into the local pipeline; update the remote G4 offload path so the ObjectPipeline constructed for LogicalLayoutHandle::G4 (where remote_g4_offload_task is used) also receives self.event_sender (i.e., pass event_sender into the ObjectPipeline::new call used for G4 offloads or otherwise make event_sender available to remote_g4_offload_task). Then, inside remote_g4_offload_task, after awaiting the upload notification success, emit KvCacheEvent::Create for request.keys via the same event_sender so distributed G4 offloads generate the same Create events as local pipelines; ensure you reference with_enable_remote_g4 and the existing event_sender variable when wiring.
🧹 Nitpick comments (2)
lib/kvbm-engine/src/offload/pipeline.rs (1)
2049-2055: ⚡ Quick winAdd one test that actually exercises event emission.
All of the updated executor tests still construct
SharedObjectExecutorStatewithevent_sender: None, so the newKvCacheEvent::Createpath can regress without any test failing. A smallbroadcast::channelassertion in the partial-failure case would cover the important contract: emit only successful hashes.Also applies to: 2126-2132, 2195-2201
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/kvbm-engine/src/offload/pipeline.rs` around lines 2049 - 2055, Tests construct SharedObjectExecutorState with event_sender: None, so the new KvCacheEvent::Create path isn't exercised; add a test that supplies a real broadcast::channel sender to SharedObjectExecutorState and asserts emitted events in the partial-failure case. Specifically, in the executor test that exercises object creation (the tests that build SharedObjectExecutorState and call the executor path that can partially fail), create a tokio::sync::broadcast::channel, pass Some(sender.clone()) into SharedObjectExecutorState.event_sender, run the executor to trigger KvCacheEvent::Create, and assert the receiver only receives the successful hashes (no failed ones); mirror this pattern for the other two similar locations referenced (around the SharedObjectExecutorState constructions at the other ranges).lib/kvbm-engine/src/object/mooncake/mod.rs (1)
4-5: ⚡ Quick winAdd module doc include to satisfy module-doc policy.
Please add a module-level
#[doc = include_str!(...)]and corresponding docs file for this new module.As per coding guidelines, “Module documentation should live in
docs/and be included via#[doc = include_str!("../docs/...")]in module files. When modifying a module, update the corresponding doc file.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/kvbm-engine/src/object/mooncake/mod.rs` around lines 4 - 5, Add a module-level documentation include for the mooncake module: create a docs/mooncake.md file with the module docs and add a top-level #[doc = include_str!("../docs/mooncake.md")] to lib/kvbm-engine/src/object/mooncake/mod.rs above the pub mod declarations (the module exposes client and lock), ensuring the doc file text describes the module and follows project style; update the docs file whenever the module changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/kvbm-engine/Cargo.toml`:
- Around line 47-51: The local path dependency mooncake_store in Cargo.toml
breaks CI; replace the path = "../../../Mooncake/mooncake-store/rust" entry for
the mooncake_store dependency with a git-based source (e.g. git =
"https://github.com/kvcache-ai/Mooncake" plus a specific tag or rev such as tag
= "vX.Y.Z" or rev = "<commit>") while keeping optional = true so CI can resolve
it; after making the change, update Cargo.lock (run cargo update or relevant
tooling) to ensure the workspace builds in CI.
In `@lib/kvbm-engine/src/object/mooncake/client.rs`:
- Around line 242-243: Before zipping keys and block_ids into work_items,
validate that keys.len() == block_ids.len() and return or propagate an explicit
error (or panic/log) if they differ to avoid silent truncation; update the two
places where work_items is created (the expression using
keys.into_iter().zip(block_ids) and the similar zip at the other occurrence) to
perform the length check first and only zip after validating lengths, including
a clear error message that references the operation (e.g., "upload/download
block count mismatch") so callers can detect and handle the input error.
- Around line 507-535: The current loop that builds size queries and then uses
buffer_unordered can change ordering and may drop entries if exist_results
length differs; fix has_blocks by preserving 1:1 positional correspondence:
iterate keys.enumerate() to pair each (index, (hash, key)) and look up the
matching exist flag via index (or explicitly validate exist_results.len() ==
keys.len() and handle mismatches by pushing None for missing entries), then when
creating tasks include the original index (e.g., tasks produce (index, hash,
Option<size>)), run the concurrent stream, collect all results, sort them by
index to restore deterministic order, and finally map to Vec<(SequenceHash,
Option<usize>)> in the original order so results align exactly with the input
keys (update references to keys, formatted, exist_results, size_queries, tasks,
size_results accordingly).
---
Outside diff comments:
In `@lib/kvbm-engine/src/offload/engine.rs`:
- Around line 462-469: The event_sender is only wired into the local pipeline;
update the remote G4 offload path so the ObjectPipeline constructed for
LogicalLayoutHandle::G4 (where remote_g4_offload_task is used) also receives
self.event_sender (i.e., pass event_sender into the ObjectPipeline::new call
used for G4 offloads or otherwise make event_sender available to
remote_g4_offload_task). Then, inside remote_g4_offload_task, after awaiting the
upload notification success, emit KvCacheEvent::Create for request.keys via the
same event_sender so distributed G4 offloads generate the same Create events as
local pipelines; ensure you reference with_enable_remote_g4 and the existing
event_sender variable when wiring.
---
Nitpick comments:
In `@lib/kvbm-engine/src/object/mooncake/mod.rs`:
- Around line 4-5: Add a module-level documentation include for the mooncake
module: create a docs/mooncake.md file with the module docs and add a top-level
#[doc = include_str!("../docs/mooncake.md")] to
lib/kvbm-engine/src/object/mooncake/mod.rs above the pub mod declarations (the
module exposes client and lock), ensuring the doc file text describes the module
and follows project style; update the docs file whenever the module changes.
In `@lib/kvbm-engine/src/offload/pipeline.rs`:
- Around line 2049-2055: Tests construct SharedObjectExecutorState with
event_sender: None, so the new KvCacheEvent::Create path isn't exercised; add a
test that supplies a real broadcast::channel sender to SharedObjectExecutorState
and asserts emitted events in the partial-failure case. Specifically, in the
executor test that exercises object creation (the tests that build
SharedObjectExecutorState and call the executor path that can partially fail),
create a tokio::sync::broadcast::channel, pass Some(sender.clone()) into
SharedObjectExecutorState.event_sender, run the executor to trigger
KvCacheEvent::Create, and assert the receiver only receives the successful
hashes (no failed ones); mirror this pattern for the other two similar locations
referenced (around the SharedObjectExecutorState constructions at the other
ranges).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3561ae62-dfe6-4191-a621-1d1f619b9c3b
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
lib/kvbm-config/src/lib.rslib/kvbm-config/src/object.rslib/kvbm-engine/Cargo.tomllib/kvbm-engine/src/object/mod.rslib/kvbm-engine/src/object/mooncake/client.rslib/kvbm-engine/src/object/mooncake/lock.rslib/kvbm-engine/src/object/mooncake/mod.rslib/kvbm-engine/src/offload/engine.rslib/kvbm-engine/src/offload/pipeline.rslib/kvbm-engine/tests/mooncake_integration.rslib/kvbm-physical/src/testing/physical.rs
- has_blocks(): preserve 1:1 cardinality and deterministic ordering - put/get_blocks_with_layout(): add keys/block_ids length mismatch guard - Fix buffer_unordered ordering for SpmdParallelWorkers (index tracking) - Wire event_sender to remote G4 offload task for distributed mode - Add test verifying KvCacheEvent::Create emission in partial-failure path - Add mooncake module docs following project convention - Add CI path dependency guidance in Cargo.toml - Add #![cfg(feature = "mooncake")] guard to integration test
fix(kvbm-engine): resolve Mooncake Store code review issues
Signed-off-by: Dayuxiaoshui <158081477+Dayuxiaoshui@users.noreply.github.com>
Update pre-merge.yml
Signed-off-by: Dayuxiaoshui <158081477+Dayuxiaoshui@users.noreply.github.com>
- Add Mooncake clone step to rust-clippy job (before cargo metadata) - Fix formatting in kvbm-physical/src/testing/physical.rs
fix(ci): resolve cargo fmt/metadata failures on Mooncake path dep @Dayuxiaoshui
|
cc @dynamo-ops |
fix(kvbm-engine): remove unused max_concurrent in has_blocks
|
cc @dynamo-ops |
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
…yout failure register_layout registered buffers in a loop, so an error on a later buffer returned early and leaked every registration made before it. Wrap the loop in a guard that unregisters the already-registered buffers on drop unless the whole layout succeeded, making the call all-or-nothing on error and on panic. unregister_layout had the mirror-image problem: it bailed out on the first error, leaving later buffers registered. Attempt every buffer and return the first error instead. Addresses the review comment on client.rs:94.
bench_mooncake measures G4 put/get/has throughput and p50/p99 latency using
Qwen3-30B-A3B KV geometry (48 layers, 4 KV heads, head_dim 128, bf16 = 96
KiB/token) across fully-contiguous and layer-separate layouts, for both the
TCP copy path and the RDMA zero-copy path.
Expose create_{fc,lw}_layout_system_with_config from kvbm-physical testing so
the bench can build model-shaped layouts without a NIXL agent.
…e tests Drop the unused TransferStatus import from four offload pipeline tests and bind the unused TransferState handle to _handle.
…oughput Add docs/integrations/mooncake-store-integration.md covering prerequisites, build, cluster setup, configuration, G4 wiring from Rust, and troubleshooting. It also resolves the dangling reference from kvbm-engine/Cargo.toml. Fold the headline numbers into the KVBM guide as a G4 Object Storage Tier section so the deployment docs no longer stop at G3, and register the new page in the docs nav. Measured on one H20 with Qwen3-30B-A3B KV geometry (96 KiB/token), 32K-token prefix: RDMA zero-copy is 119 ms put / 116 ms get on one 400 Gb/s NIC (45/42 ms across four), against 1658/1703 ms for the TCP copy path. Document two API traps found while benchmarking: use_zero_copy is a builder flag that create_object_client never sets, so a factory-built client takes the copy path even with protocol: rdma, and register_layout is not called implicitly before transfers.
External PR triage (automated, 2026-11-11)Blocker: merge conflict with Next action. Owner: @Dayuxiaoshui
Supply-chain audit: HOLD. |
This commit adds Mooncake Store (distributed KV cache storage from Moonshot AI) as a G4 tier backend for Dynamo KVBM, enabling cross-node KV cache sharing.
Phase 1: TCP copy path
Phase 2: RDMA zero-copy + Batch API
Phase 3: KV Router G4 event linkage
Testing:
3 integration tests pass (copy + zero-copy roundtrip)
cargo check with --features mooncake,testing clean
closes GitHub issue: [RFC]: Dynamo KVBM + Mooncake Store Integration Design kvcache-ai/Mooncake#1979
Summary by CodeRabbit
New Features
Configuration