Repository navigation
feat(kvbm-engine): integrate Mooncake Store as G4 object storage backend - #1
Conversation
This commit adds Mooncake Store (distributed KV cache storage from Moonshot AI) as a G4 tier backend for Dynamo KVBM, enabling cross-node KV cache sharing. Phase 1: TCP copy path - MooncakeObjectBlockClient implementing ObjectBlockOps trait - MooncakeObjectConfig with protocol selection (tcp/rdma) - MooncakeLockManager with optimistic locking (idempotent writes) Phase 2: RDMA zero-copy + Batch API - with_zero_copy() builder and register_layout() for buffer registration - batch_put_from / batch_get_into for contiguous layouts (1 RPC batch) - Automatic fallback to copy path for non-contiguous layouts Phase 3: KV Router G4 event linkage - OffloadEngineBuilder::with_event_sender() - ObjectTransferExecutor emits KvCacheEvent::Create on successful offload Testing: - 3 integration tests pass (copy + zero-copy roundtrip) - cargo check with --features mooncake,testing clean
stmatengss
left a comment
There was a problem hiding this comment.
Mooncake Store G4 Backend — Code Review
Solid integration overall. The phased approach (TCP copy → RDMA zero-copy → event linkage) is well-structured, and the trait-based pluggability via ObjectBlockOps is clean. Found several issues ranging from a data-loss bug to safety concerns in unsafe code.
Summary:
- 1 bug (
has_blocksdrops non-existing keys from result) - 2 safety issues (null pointer in batch path, hostname portability)
- 2 design concerns (out-of-tree path dep, error-swallowing lock manager)
- Several minor suggestions
| .into_iter() | ||
| .zip(formatted.into_iter()) | ||
| .zip(exist_results.into_iter()) | ||
| .filter_map(|((hash, key), exists)| if exists { Some((hash, key)) } else { None }) |
There was a problem hiding this comment.
🐛 Bug: has_blocks() drops non-existing keys from the result.
The filter_map here filters out non-existing keys, and keys was consumed by into_iter() above. The final collect() only returns entries for keys that exist in the store. Non-existing keys (which should appear as (hash, None)) are silently lost from the result.
This violates the trait contract — callers expect a result entry for every input key.
Fix: Preserve the non-existing keys in the result:
let mut results: Vec<(SequenceHash, Option<usize>)> = Vec::with_capacity(keys.len());
let mut size_queries = Vec::new();
for ((hash, key), exists) in keys.into_iter().zip(formatted).zip(exist_results) {
if exists {
size_queries.push((hash, key));
} else {
results.push((hash, None));
}
}
// ... query sizes for size_queries, then extend results| .map(|(_, block_id)| { | ||
| layout.memory_region(*block_id, 0, 0) | ||
| .map(|r| r.addr as *mut std::ffi::c_void) | ||
| .unwrap_or(std::ptr::null_mut()) |
There was a problem hiding this comment.
memory_region failure.
.unwrap_or(std::ptr::null_mut())If layout.memory_region() fails for any block, a null pointer is silently inserted into the buffers vec and passed to store.batch_put_from(). The Mooncake C library will likely segfault on this.
The same pattern appears in get_blocks_with_layout at the analogous batch path.
Suggestion: Collect into Result<Vec<_>> and fall through to the single-block path on error:
let buffers: Result<Vec<_>> = work_items.iter()
.map(|(_, block_id)| {
layout.memory_region(*block_id, 0, 0)
.map(|r| r.addr as *mut std::ffi::c_void)
})
.collect();
let Ok(buffers) = buffers else {
// fall through to single-block path
...
};| rayon = { version = "1", optional = true } | ||
| tokio-rayon = { version = "2", optional = true } | ||
| chrono = { version = "0.4", optional = true } | ||
| mooncake_store = { path = "../../../Mooncake/mooncake-store/rust", optional = true } |
There was a problem hiding this comment.
mooncake_store = { path = "../../../Mooncake/mooncake-store/rust", optional = true }This assumes the Mooncake repo is checked out at a specific sibling directory relative to the workspace root (../../Mooncake/). This will break for any contributor who does not have that exact layout, and will fail in CI unless the CI script clones Mooncake to that exact path.
Consider:
- Publishing
mooncake_storeto a private registry or crates.io - Using a git dependency:
mooncake_store = { git = "https://github.com/...", optional = true } - At minimum, documenting the required checkout layout in the PR description or a README
| } | ||
|
|
||
| fn default_hostname() -> String { | ||
| std::fs::read_to_string("/proc/sys/kernel/hostname") |
There was a problem hiding this comment.
/proc/sys/kernel/hostname does not exist on macOS/Windows.
The fallback to "localhost" will silently fire on non-Linux platforms, which may cause confusing RDMA addressing issues if someone develops or tests on macOS.
Suggestion: Use gethostname(2) via the libc crate (already a dependency) or the hostname crate for portable behavior. At minimum, add a comment noting this is Linux-only by design.
| pub namespace: Option<String>, | ||
|
|
||
| /// Maximum concurrent requests. | ||
| #[serde(default = "default_max_concurrent")] |
There was a problem hiding this comment.
Possible compilation error: missing default_max_concurrent function.
This field uses #[serde(default = "default_max_concurrent")] but I don't see a default_max_concurrent() function defined in this file. The other defaults (default_master_server_addr, default_hostname, default_mooncake_protocol, default_segment_size, default_buffer_size) are all defined below — but default_max_concurrent appears to be missing.
Is it defined in another module, or was it accidentally omitted from this PR?
|
|
||
| /// Transport protocol: "tcp" or "rdma". | ||
| #[serde(default = "default_mooncake_protocol")] | ||
| pub protocol: String, |
There was a problem hiding this comment.
Nit: protocol should be an enum, not a free-form String.
Currently accepts any string value, but only "tcp" and "rdma" are valid per the doc comment. A typo like "tcpp" would pass config validation and fail at runtime deep in the Mooncake C library.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MooncakeProtocol {
Tcp,
Rdma,
}This gives compile-time + deserialization-time validation instead of a runtime C-library error.
| Ok(exists) => Ok(exists), | ||
| Err(e) => { | ||
| tracing::warn!(key = %key, error = %e, "has_meta check failed"); | ||
| Ok(false) |
There was a problem hiding this comment.
Design: has_meta swallows errors and returns Ok(false).
Returning Ok(false) on error means a transient network failure is indistinguishable from "meta doesn't exist." This could cause the system to re-upload blocks that already exist (wasted bandwidth) or re-acquire locks unnecessarily.
The optimistic-locking rationale (idempotent writes) makes duplicate uploads safe, so this isn't a correctness bug — but propagating the error would let callers make better decisions. At minimum, document this as an intentional design choice in the doc comment.
| // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| #[cfg(feature = "mooncake")] |
There was a problem hiding this comment.
Nit: Redundant #[cfg(feature = "mooncake")] guards.
This module is already gated at the parent level in object/mod.rs:
#[cfg(feature = "mooncake")]
pub mod mooncake;The inner #[cfg] guards here are redundant — the compiler will never reach this file without the feature enabled. Not harmful, but unnecessary noise. Consider removing them.
| if is_contiguous && use_zero_copy && !work_items.is_empty() { | ||
| let keys_str: Vec<String> = work_items | ||
| .iter() | ||
| .map(|(hash, _)| match &namespace { |
There was a problem hiding this comment.
Nit: Duplicated key formatting logic.
The format_key helper exists as a method on MooncakeObjectBlockClient (line ~72), but the closures inside put_blocks_with_layout, get_blocks_with_layout, and has_blocks all re-implement the namespace-prefixing logic inline:
match &namespace {
Some(ns) => format!("{}/{}", ns, formatter.format_key(hash)),
None => formatter.format_key(hash),
}This pattern appears ~6 times. Consider extracting a standalone function that takes &namespace and &formatter to reduce repetition and risk of divergence.
| // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| //! Integration test: verify Mooncake Store via raw API and MooncakeObjectBlockClient. |
There was a problem hiding this comment.
Nit: This is an integration test disguised as an example.
This file lives in examples/ but is functionally a test (uses assert!, assert_eq!, prints pass/fail). It also requires a running Mooncake server to pass.
The #[ignore] tests in client.rs already cover the same scenarios (copy roundtrip, zero-copy roundtrip, has_blocks) more thoroughly with proper test framework support.
Consider either moving this to tests/mooncake_integration.rs or removing it in favor of the existing #[ignore] tests.
This commit adds Mooncake Store (distributed KV cache storage from Moonshot AI) as a G4 tier backend for Dynamo KVBM, enabling cross-node KV cache sharing.
Phase 1: TCP copy path
Phase 2: RDMA zero-copy + Batch API
Phase 3: KV Router G4 event linkage
Testing:
Overview:
Details:
Where should the reviewer start?
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)