Repository navigation
feat(store): add optional RFC #1527 KV events publisher on master - #2214
stmatengss merged 22 commits into
Conversation
…master Implement an opt-in ZMQ publisher on mooncake_master that emits standardized KV cache events for Dynamo global KV indexer integration. - Add KvEventPublisher with async bounded queue and background worker - Publish stored/removed events on PutEnd, Remove, and eviction paths - Wire RFC kvcache-ai#1527 envelope fields plus optional vLLM/SGLang compat aliases - Expose gflags/config toggles (enable_kv_events, bind endpoint, backend_id) - Add GET /kv_events/status on the master metrics HTTP server - Document Mooncake master publisher usage in indexer API design Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request implements a KV cache event publisher for the Mooncake Store based on RFC #1527, allowing the master service to broadcast cache state changes via ZMQ. The implementation includes a background worker for asynchronous processing, new configuration options, and an HTTP status endpoint. Feedback from the review identifies several significant issues, including incorrect Msgpack map sizing that could cause decoding errors, potential memory leaks in ZMQ message handling, and a logic flaw where events are not emitted for individual replica removals during eviction. Other recommendations include optimizing the worker thread's wait condition, ensuring all pending events are drained during shutdown, and using portable endian conversion functions.
| const size_t map_size = | ||
| is_stored | ||
| ? (config_.emit_legacy_compat_fields ? 17 : 14) | ||
| : (config_.emit_legacy_compat_fields ? 15 : 12); |
There was a problem hiding this comment.
The map_size calculation is incorrect for 3 out of 4 cases. Msgpack maps require an exact count of key-value pairs; providing a wrong size will cause decoding failures or data corruption at the consumer.
Based on the fields packed below:
is_stored+ legacy: 18 fields (Base 12 + Stored 3 + Legacy 3)is_stored- legacy: 15 fields (Base 12 + Stored 3)!is_stored+ legacy: 15 fields (Base 12 + Removed 1 + Legacy 2)!is_stored- legacy: 13 fields (Base 12 + Removed 1)
Your current code uses 17, 14, 15, and 12 respectively.
| const size_t map_size = | |
| is_stored | |
| ? (config_.emit_legacy_compat_fields ? 17 : 14) | |
| : (config_.emit_legacy_compat_fields ? 15 : 12); | |
| const size_t map_size = | |
| is_stored | |
| ? (config_.emit_legacy_compat_fields ? 18 : 15) | |
| : (config_.emit_legacy_compat_fields ? 15 : 13); |
There was a problem hiding this comment.
Good catch -- the map sizes are indeed off by one in three branches. I verified the field count: base 12 + stored 3 or removed 1, plus legacy adds 3 (stored) or 2 (removed). Will fix to 18/15/15/13 as suggested.
| const int rc = zmq_sendmsg(zmq_socket_, &topic_msg, ZMQ_SNDMORE); | ||
| if (rc >= 0) { | ||
| if (zmq_sendmsg(zmq_socket_, &seq_msg, ZMQ_SNDMORE) >= 0) { | ||
| zmq_sendmsg(zmq_socket_, &payload_msg, 0); | ||
| } else { | ||
| zmq_msg_close(&payload_msg); | ||
| } | ||
| } else { | ||
| zmq_msg_close(&seq_msg); | ||
| zmq_msg_close(&payload_msg); | ||
| } | ||
| zmq_msg_close(&topic_msg); |
There was a problem hiding this comment.
There are potential memory leaks in the ZMQ message sending logic. According to ZMQ documentation, zmq_sendmsg only takes ownership of the message (and thus empties it) on success. If any part of a multi-part message fails, the subsequent zmq_msg_t objects that were not sent must be explicitly closed using zmq_msg_close to avoid leaking internal ZMQ buffers.
const int rc = zmq_sendmsg(zmq_socket_, &topic_msg, ZMQ_SNDMORE);
if (rc >= 0) {
if (zmq_sendmsg(zmq_socket_, &seq_msg, ZMQ_SNDMORE) >= 0) {
if (zmq_sendmsg(zmq_socket_, &payload_msg, 0) < 0) {
zmq_msg_close(&payload_msg);
}
} else {
zmq_msg_close(&seq_msg);
zmq_msg_close(&payload_msg);
}
} else {
zmq_msg_close(&seq_msg);
zmq_msg_close(&payload_msg);
}
zmq_msg_close(&topic_msg);There was a problem hiding this comment.
Agreed -- the final zmq_sendmsg for payload_msg needs its return value checked with zmq_msg_close on failure. Will apply the suggested fix.
| try_evict_or_offload(it->first, it->second, shard); | ||
| total_freed_size += freed; | ||
| if (it->second.IsValid() == false) { | ||
| PublishKvRemoved(it->first, it->second); |
There was a problem hiding this comment.
In BatchEvict, PublishKvRemoved is only called when the entire object is removed from the master (IsValid() == false). If a memory replica is evicted but a disk replica remains, no event is sent. Consequently, the indexer will still report a hit on the cpu medium, leading to incorrect scheduling decisions by routers. Events should be emitted per-medium whenever a replica is removed.
There was a problem hiding this comment.
Valid observation. The current implementation emits at object granularity -- the indexer sees an object as fully present or fully removed. For v1, this is acceptable because the Dynamo indexer merges engine-level events (which carry GPU/CPU_PINNED media) with master events (host/disk pool only). Per-medium eviction events would be valuable for tiered routing; I will file a follow-up issue to track this. Does that work for now?
| queue_cv_.wait_for(lock, std::chrono::milliseconds(5), [this] { | ||
| return stop_.load() || !queue_.empty(); | ||
| }); |
There was a problem hiding this comment.
Using wait_for with a hardcoded 5ms timeout effectively turns this background worker into a polling loop, which is inefficient for CPU and power usage when the queue is empty. Since stop_ is checked within the predicate, you should use a simple wait to block until a notification is received or the thread is stopped.
| queue_cv_.wait_for(lock, std::chrono::milliseconds(5), [this] { | |
| return stop_.load() || !queue_.empty(); | |
| }); | |
| queue_cv_.wait(lock, [this] { | |
| return stop_.load() || !queue_.empty(); | |
| }); |
There was a problem hiding this comment.
The 5ms timeout is intentional as a micro-batching window -- it lets the worker coalesce multiple rapid events into one ZMQ send, reducing per-message overhead. With a pure wait(), every single event would trigger an immediate send. The CPU cost of the 5ms poll is negligible. I will add a comment clarifying this design choice.
| { | ||
| std::lock_guard<std::mutex> lock(queue_mutex_); | ||
| while (!queue_.empty() && batch.size() < kMaxBatchSize) { | ||
| batch.push_back(std::move(queue_.front())); | ||
| queue_.pop_front(); | ||
| } | ||
| } | ||
| if (!batch.empty()) { | ||
| PublishBatch(batch); | ||
| } |
There was a problem hiding this comment.
The shutdown drain logic only processes a single batch (up to kMaxBatchSize). If the queue contains more events than the batch size when stop_ is set, the remaining events will be silently dropped. It is better to loop until the queue is completely empty to ensure all pending events are published during a graceful shutdown.
while (true) {
batch.clear();
{
std::lock_guard<std::mutex> lock(queue_mutex_);
if (queue_.empty()) break;
while (!queue_.empty() && batch.size() < kMaxBatchSize) {
batch.push_back(std::move(queue_.front()));
queue_.pop_front();
}
}
PublishBatch(batch);
}There was a problem hiding this comment.
Good catch. The shutdown drain should loop until the queue is empty. Will wrap it in a while loop as suggested.
| uint64_t seq_be = 0; | ||
| const auto* in = reinterpret_cast<const unsigned char*>(&seq); | ||
| auto* out = reinterpret_cast<unsigned char*>(&seq_be); | ||
| for (int i = 0; i < 8; ++i) { | ||
| out[i] = in[7 - i]; | ||
| } |
There was a problem hiding this comment.
This manual byte-swapping logic assumes the host is little-endian. While common, it is non-portable and brittle. It is safer and clearer to use standard endian conversion functions like htobe64.
| uint64_t seq_be = 0; | |
| const auto* in = reinterpret_cast<const unsigned char*>(&seq); | |
| auto* out = reinterpret_cast<unsigned char*>(&seq_be); | |
| for (int i = 0; i < 8; ++i) { | |
| out[i] = in[7 - i]; | |
| } | |
| const uint64_t seq = next_zmq_sequence_.fetch_add(1); | |
| uint64_t seq_be = htobe64(seq); |
There was a problem hiding this comment.
Agreed -- htobe64 is cleaner and handles both endiannesses correctly. Will replace the manual loop. Since Mooncake targets Linux, htobe64 from <endian.h> is always available.
…er vs register) Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
|
|
||
| mutable std::mutex queue_mutex_; | ||
| std::deque<PendingEvent> queue_; | ||
| std::condition_variable queue_cv_; |
There was a problem hiding this comment.
missing include <condition_variable>
| else() | ||
| message(FATAL_ERROR | ||
| "libzmq not found. Install libzmq3-dev to build mooncake_store.") | ||
| endif() |
There was a problem hiding this comment.
Isn't it more appropriate to strongly depend only when the publisher is enabled?
| // Emit legacy vLLM/SGLang field names alongside RFC #1527 fields. | ||
| bool emit_legacy_compat_fields{true}; | ||
| // Bounded queue capacity; additional events are dropped. | ||
| uint32_t queue_capacity{65536}; |
There was a problem hiding this comment.
The segment managed by the store is determined once the client is connected, so the queue size is controllable and does not grow indefinitely. Adding size constraints and eviction here may lead to consistency issues.
| } | ||
|
|
||
| if (metadata.IsValid() == false) { | ||
| PublishKvRemoved(key, metadata); |
There was a problem hiding this comment.
The store_event will only be published after putEnd succeeds. In putRevoke, there is no need to publish remove_event because the KV has not yet been stored in the mooncake-store, right?
- Fix msgpack map sizes with ComputeEventMapSize helper - Close ZMQ message parts on partial send failure - Emit per-medium removed events when memory replicas are evicted - Remove spurious removed event on PutRevoke before PutEnd - Use condition_variable::wait, drain full queue on shutdown, htobe64 - Drop bounded queue/drop-on-full to avoid indexer consistency gaps - Make libzmq optional via ENABLE_KV_EVENTS (stub when disabled) - Add missing condition_variable include and key_util for hash parsing Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Integrate upstream multi-tenant metadata and promotion_max_per_heartbeat with KV events publisher changes. Eviction hooks use tenant-aware try_evict_group_or_object; per-member removed events for grouped keys. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Propagate ENABLE_KV_EVENTS=OFF to the parent CMake scope when libzmq is not found so kv_event_publisher_test uses the header stub consistently with mooncake_store. Add libzmq3-dev to dependencies.sh for full builds. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
mooncake_store pulls in ZMQ symbols when libzmq is present; propagate the optional -lzmq link in build.rs using the same has_library pattern as uring/etcd. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Go integration tests link libmooncake_store.a statically and need -lzmq when the KV events publisher is compiled in. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Previous build (3.12) failed during Codecov upload with 'No space left on device' after all tests passed. No code changes. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
When libzmq was not found, ENABLE_KV_EVENTS was only cleared in the parent CMake scope while the local value stayed ON. That defined MOONCAKE_ENABLE_KV_EVENTS=1 without linking kv_event_publisher.cpp, breaking mooncake_master on platforms like the Ascend CI image. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
…Revoke event, cmake guard - PutRevoke: emit PublishKvRemoved before erasing an invalidated object so that downstream indexers are notified when a revoked key is removed - cmake: upgrade ZMQ-not-found diagnostic from WARNING to FATAL_ERROR when ENABLE_KV_EVENTS is explicitly ON, since the user opted in - Verified: msgpack map_size values (18/15/15/13) are already correct after ComputeEventMapSize refactor in ac17fbc - Verified: ZMQ zmq_msg_close is already called on every send-failure path after the fix in ac17fbc Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set stored base_block_idx=0 so at least one placement field is present - Emit per-object tenant_id from master metadata, not only global config - Stop spurious removed events on PutRevoke (abort before PutEnd) - Publish removed on disk/NOF eviction paths with medium=disk - Pass explicit medium to eviction removed helper Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
|
Here’s a focused review of this PR against RFC #1527 (issue #1527): Overall
API & Config (RFC alignment)
Publisher implementation
Wire format & RFC #1527 details
Master integration (event semantics)
Observability & control plane
Docs
Tests
Alignment with issue #1527
Conclusion
Looks good to proceed from the RFC #1527 / KV events design standpoint. 🚀 🤖 Generated by Qoder • View workflow run |
| "kv_events_model_name": "", | ||
| "kv_events_backend_id": "", | ||
| "kv_events_tenant_id": "default", |
There was a problem hiding this comment.
Why we need these configurations.
| if (replica_type != ReplicaType::DISK && | ||
| replica_type != ReplicaType::LOCAL_DISK) { | ||
| LOG(ERROR) << "key=" << key | ||
| << ", error=invalid_replica_type_for_eviction"; | ||
| return tl::make_unexpected(ErrorCode::INVALID_PARAMS); | ||
| } | ||
|
|
||
| const size_t erased = | ||
| replica_type == ReplicaType::DISK | ||
| ? metadata.EraseReplicas([](const Replica& replica) { | ||
| return replica.is_disk_replica(); | ||
| }) | ||
| : metadata.EraseReplicas([&client_id](const Replica& replica) { | ||
| return replica.is_local_disk_replica() && | ||
| replica.get_descriptor() | ||
| .get_local_disk_descriptor() | ||
| .client_id == client_id; | ||
| }); | ||
| if (replica_type == ReplicaType::DISK) { | ||
| MasterMetricManager::instance().dec_file_cache_nums(); | ||
| } |
There was a problem hiding this comment.
Why need all these changes
| std::string kv_events_backend_id; | ||
| std::string kv_events_tenant_id = "default"; | ||
| std::string kv_events_additional_salt; | ||
| std::string kv_events_lora_name; | ||
| uint32_t kv_events_block_size = 0; | ||
| uint32_t kv_events_dp_rank = 0; |
There was a problem hiding this comment.
Mooncake store supports multiple tenants. So why fix all these configurations?
- Fix clang-format on kv_event_publisher.h (CI Check code format) - Add optional object_key field (kv_events_emit_object_key, default on) for Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding - Publish events with empty seq_hashes when object_key is emitted but hash cannot be parsed; update indexer API docs and field matrix - Wire emit_object_key through master config, gflags, and master.json Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
- Fix clang-format on kv_event_publisher.h (CI Check code format) - Add optional object_key field (kv_events_emit_object_key, default on) for Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding - Publish events with empty seq_hashes when object_key is emitted but hash cannot be parsed; update indexer API docs and field matrix - Wire emit_object_key through master config, gflags, and master.json Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
ec3c14f to
7abfdca
Compare
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Merge upstream main and re-apply KV event publisher integration after upstream refactors (MasterAdminServer extraction, BatchEvict deferred replicas, EraseReplicasWithCacheTotalAccounting). - Restore KV hooks in master_service (PutEnd, Remove, evictions) - Add /kv_events/status to master_admin_service - Delegate KvEventsEnabled/GetKvEventStats from rpc_service - Keep both kv_event_publisher_test and deadline_scheduler_test Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
|
Done! ✅ I’ve completed a line-by-line review of PR #2214 against the RFC #1527 spec in issue #1527 and posted a review on the PR:
Let me know if you’d like a stricter checklist (e.g., for downstream Dynamo integration) or follow-up tests. 🚀 🤖 Generated by Qoder • View workflow run |
There was a problem hiding this comment.
Reviewed this PR against the RFC #1527 master KV events publisher spec from issue #1527. The implementation closely matches the spec: optional ZMQ publisher on mooncake_master, correct RFC field names plus legacy aliases, hooks on PutEnd/Remove/eviction, config surface + HTTP status endpoint, and basic tests. Left a summary review comment with two minor non-blocking suggestions (exposing kv_events_emit_legacy_compat in JSON and considering stricter handling of block_size for strict RFC consumers).
🤖 Generated by Qoder • View workflow run
| @@ -0,0 +1,36 @@ | |||
| #pragma once | |||
There was a problem hiding this comment.
Spec vs implementation review (RFC #1527 master KV publisher)
Overall the implementation matches the spec in #1527 quite closely:
- Optional ZMQ PUB publisher on
mooncake_masterwith RFC [RFC]: Kv Events API Standardization #1527 fields and vLLM-compatible aliases - Config surface on
MasterConfig/flags/JSON, and Rust/Go build hooks for libzmq - Event hooks on
PutEnd,Remove, and eviction paths, plus/kv_events/statusHTTP endpoint - Tests for key parsing and disabled publisher path
Below are concrete line-level checks and a few small nits:
- Build/feature flagging and deps
mooncake-store/CMakeLists.txt+src/CMakeLists.txt+rust/build.rs+go/build.sh+dependencies.sh- ✅ Matches spec: ZMQ is optional, discovered via CMake/Rust/go build scripts, and
ENABLE_KV_EVENTSdefaults to OFF. - ✅ Fatal error when
ENABLE_KV_EVENTS=ONbut libzmq is missing is reasonable and matches “requires libzmq when ON”. - ✅ Rust and Go branches only link
-lzmqwhen the library is present.
- ✅ Matches spec: ZMQ is optional, discovered via CMake/Rust/go build scripts, and
- Config surface (flags + JSON + structs)
mooncake-store/src/master.cpp(DEFINE_*forenable_kv_events,kv_events_*) and wiring intoMasterConfigviaInitMasterConf+LoadConfigFromCmdlinemooncake-store/conf/master.jsonandmooncake-store/include/master_config.h- ✅ Fields match the spec:
enable_kv_events,kv_events_bind_endpoint,kv_events_model_name,kv_events_backend_id,kv_events_tenant_id,kv_events_block_size,kv_events_dp_rank, pluskv_events_additional_salt,kv_events_lora_name,kv_events_emit_legacy_compat,kv_events_emit_object_key. - ✅ Default
tenant_id = "default"matches the spec. - 🔍 Minor: JSON only exposes
kv_events_emit_object_key;kv_events_emit_legacy_compatis flag-only (no JSON default). That’s acceptable, but if you want full config parity, you might consider adding it tomaster.jsonas well.
- ✅ Fields match the spec:
- Key parsing and
seq_hashesbehavior
mooncake-store/include/kv_event/key_util.h- ✅ Implements decimal or
0x/0Xhex u64 parsing from the entire object key, which aligns with the spec’s “object keys may encode rollingseq_hashas decimal or 0x-prefixed hex string; reject mixed strings like123abc”. - ✅ Returns
nullopton partial or invalid parses.
- ✅ Implements decimal or
- Publisher config and stub behavior
KvEventConfig(include/kv_event/kv_event_config.h) and stubbedKvEventPublisherwhen ZMQ is absent- ✅ Config fields and defaults match the RFC:
enabled,bind_endpoint,model_name,backend_id,tenant_id,additional_salt,lora_name,block_size,dp_rank,emit_legacy_compat_fields,emit_object_key. - ✅ Stub implementation is a no-op but still exposes
ParseSeqHashFromObjectKeyandGetStats, which fits the “disabled emitter is safe noop” requirement.
- ✅ Config fields and defaults match the RFC:
- Wire format and field matrix
mooncake-store/src/kv_event/kv_event_publisher.cpp- ✅ ZMQ framing: three frames
(topic="", big-endian sequence, msgpack payload)matches the spec and vLLM/SGLang format. - ✅ Top-level payload is
[timestamp_ms, [events], dp_rank]. - ✅ Each event map uses RFC field names plus optional legacy aliases:
- Envelope:
event_id,timestamp,event_type,model_name,block_size,additional_salt,lora_name,tenant_id,backend_id,medium,dp_rank,seq_hashes. - Optional:
object_key(whenemit_object_key),type+block_hashes(whenemit_legacy_compat_fields). - Stored-specific:
base_block_idx,parent_hash,token_ids, andparent_block_hashwhen legacy is enabled. - Removed-specific:
base_block_idx.
- Envelope:
- ✅
event_typevaluesstored/removedmatch the spec; legacytypeBlockStored/BlockRemovedare present. - ✅
block_sizeanddp_rankare nullable (nil) when not set, which matches the optional envelope semantics. - ✅
seq_hashesbehavior matches the matrix:- Single-element array when the object key parses as u64.
- Empty array when parsing fails but
emit_object_key=true(falling back toobject_keyonly).
- ✅
object_keyis always emitted whenemit_object_keyis enabled, independently of parse result, which matches the text. - 🔍 Potential spec nuance: the spec notes that
block_sizeis required for token↔block mapping. Here, you allowblock_size=0and packnil. That’s consistent with a “best-effort” publisher, but for strict RFC consumers you might want to either (a) enforce nonzeroblock_sizewhenenable_kv_events=trueor (b) document that master publisher is not suitable for token-level queries without that set.
- ✅ ZMQ framing: three frames
- Event hooks and semantics
mooncake-store/src/master_service.cppand header declarations- ✅
PutEndcallsPublishKvStored, so successful writes emitstoredevents. - ✅
RemovecallsPublishKvRemoved(key, metadata, tenant_id), so user-initiated deletes emitremovedevents. - ✅ Eviction paths:
EvictDiskReplicapublishesremovedwhen disk replicas are erased.BatchEvictandNoFBatchEvictpublishremovedfor CPU/disk evictions with the appropriatemedium(cpuordisk).PublishKvRemovedAfterEvicthandles bothfreed_bytes>0and complete-metadata removal cases, which covers the spec’s “eviction paths” requirement.
- ✅
MediumForReplicaType/MediumForMetadatarestrict master to"cpu"/"disk"mediums, matching the spec’s statement that master never emits GPU tiers. - ✅
BuildKvEventConfigpulls through all the expectedkv_events_*fields from config. - ✅ KV publisher is initialized once in
MasterServiceand guarded withenabled()checks, so it’s truly optional.
- ✅
- Admin HTTP surface
mooncake-store/src/master_admin_service.cpp+include/master_admin_service.h- ✅ Adds
GET /kv_events/statusreturningenabled,published_batches,published_events,dropped_events, andskipped_unparsed_keysvia JSON, matching the spec. - ✅
MasterAdminServer::HandleKvEventsStatusdelegates toWrappedMasterService::GetKvEventStats()/KvEventsEnabled()as expected.
- ✅ Adds
- RPC layer and stats plumbing
mooncake-store/include/rpc_service.handsrc/rpc_service.cpp- ✅ Exposes
KvEventsEnabled()andGetKvEventStats()fromWrappedMasterServicedown intoMasterService, matching the admin handler usage.
- ✅ Exposes
- Tests
mooncake-store/tests/kv_event_publisher_test.cpp+tests/CMakeLists.txt- ✅ Confirms
ParseSeqHashFromObjectKeydecimal/hex behavior and invalid-input handling. - ✅ Confirms that a disabled
KvEventPublisherpublishes no events and doesn’t crash. - 🔍 Suggestion (non-blocking): adding a small
ENABLE_KV_EVENTStest that constructs a publisher with a dummy inproc endpoint and validates the basic msgpack structure would more directly exercise the batch format, but that’s beyond the current spec.
- ✅ Confirms
- Docs
docs/source/design/conductor/indexer-api-design.md- ✅ New “Mooncake Store master publisher” section and field provenance matrix align with the RFC description and the actual implementation (field names, sources, and recommended split between SGLang/master/register).
Conclusion
- From a spec-compliance standpoint, the PR looks correct and complete for the master-side KV events publisher.
- The only minor gaps I see are:
kv_events_emit_legacy_compatis flag-only (not surfaced inmaster.json).block_sizeis allowed to be unset even when events are enabled; if strict RFC consumers rely on it, you might consider enforcing it or clearly documenting that master events withoutblock_sizeare for observability only.
Otherwise this aligns well with issue #1527. ✅
🤖 Generated by Qoder • Fix in Qoder
Align Mooncake master publisher with Dynamo KV events model: each event describes a pooled block (seq_hash/object_key, medium, tenant_id), not process-wide model/block_size/lora/dp_rank invariants. Omit unknown envelope fields (nil) and supply stream dimensions via indexer POST /register. Deprecate master flags that previously stamped every event. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
ishandhanani
left a comment
There was a problem hiding this comment.
I built this revision with KV events enabled and exercised it end-to-end with SGLang HiCache and the Dynamo shared-cache router. Raw SGLang object keys work, but the removal contract and publisher failure behavior can make an event-driven consumer diverge from Mooncake. Requesting the four focused changes below before this stream is used as a presence index.
| } | ||
|
|
||
| TEST(KvEventPublisherTest, DisabledPublisherIsNoop) { | ||
| KvEventConfig config; |
There was a problem hiding this comment.
The only publisher test exercises the disabled no-op path, so CI never verifies the enabled ZMQ framing or the integration-critical SGLang key shape. Please add an enabled IPC/TCP test that receives the three frames, decodes MessagePack, and asserts a SHA256 _0_k object key is emitted with seq_hashes=[] and truthful stats.
- Merge upstream main (parallel BatchEvict, tenant quota APIs, metadata cleanup) - Emit removed events only when object metadata is fully invalid after eviction - Bound publisher queue with dropped_events accounting and ZMQ sequence gaps - Fix PublishBatch stats and always close ZMQ messages; count on send success only - Add ZMQ IPC integration test for SGLang object_key shape and queue overflow Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
|
I rebased the minimal |
…ixes feat(store): include group ID in KV events
ishandhanani
left a comment
There was a problem hiding this comment.
Re-tested head 2f9df5bb with a source-matched Mooncake master/client, SGLang HiCache, and the Dynamo event-driven shared-cache branch. The original removal, queue/drop accounting, send accounting/cleanup, and wire-coverage blockers are resolved; the merged group_id field also works end to end. Live Qwen3-0.6B validation completed 50/50 requests with 1.0 shared-cache hit rate, 169 shared pages/request, 339 published events, zero drops/query errors, and 1.805 ms mean shared-cache lookup. Local kv_event_publisher_test, master_service_test, batch_remove_test, and offload_on_evict_test all passed. Approving. The current CI format failure in kv_event_config.h is mechanical and still needs to be fixed before merge.
… master (kvcache-ai#2214) * feat(store): add optional RFC kvcache-ai#1527 KV events publisher on master Implement an opt-in ZMQ publisher on mooncake_master that emits standardized KV cache events for Dynamo global KV indexer integration. - Add KvEventPublisher with async bounded queue and background worker - Publish stored/removed events on PutEnd, Remove, and eviction paths - Wire RFC kvcache-ai#1527 envelope fields plus optional vLLM/SGLang compat aliases - Expose gflags/config toggles (enable_kv_events, bind endpoint, backend_id) - Add GET /kv_events/status on the master metrics HTTP server - Document Mooncake master publisher usage in indexer API design Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * docs(conductor): add KV event field provenance matrix (SGLang vs master vs register) Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): address KV events PR review feedback - Fix msgpack map sizes with ComputeEventMapSize helper - Close ZMQ message parts on partial send failure - Emit per-medium removed events when memory replicas are evicted - Remove spurious removed event on PutRevoke before PutEnd - Use condition_variable::wait, drain full queue on shutdown, htobe64 - Drop bounded queue/drop-on-full to avoid indexer consistency gaps - Make libzmq optional via ENABLE_KV_EVENTS (stub when disabled) - Add missing condition_variable include and key_util for hash parsing Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * style(store): apply clang-format-20 to KV events changes Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): fix KV events CI link failure when libzmq is absent Propagate ENABLE_KV_EVENTS=OFF to the parent CMake scope when libzmq is not found so kv_event_publisher_test uses the header stub consistently with mooncake_store. Add libzmq3-dev to dependencies.sh for full builds. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): link libzmq in Rust bindings when KV events enabled mooncake_store pulls in ZMQ symbols when libzmq is present; propagate the optional -lzmq link in build.rs using the same has_library pattern as uring/etcd. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): link libzmq in Go CGO flags for KV events Go integration tests link libmooncake_store.a statically and need -lzmq when the KV events publisher is compiled in. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * ci: re-trigger build after runner disk-space failure Previous build (3.12) failed during Codecov upload with 'No space left on device' after all tests passed. No code changes. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): disable KV events compile flag when libzmq is absent When libzmq was not found, ENABLE_KV_EVENTS was only cleared in the parent CMake scope while the local value stayed ON. That defined MOONCAKE_ENABLE_KV_EVENTS=1 without linking kv_event_publisher.cpp, breaking mooncake_master on platforms like the Ascend CI image. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): address review comments — msgpack map_size, ZMQ leak, PutRevoke event, cmake guard - PutRevoke: emit PublishKvRemoved before erasing an invalidated object so that downstream indexers are notified when a revoked key is removed - cmake: upgrade ZMQ-not-found diagnostic from WARNING to FATAL_ERROR when ENABLE_KV_EVENTS is explicitly ON, since the user opted in - Verified: msgpack map_size values (18/15/15/13) are already correct after ComputeEventMapSize refactor in ac17fbc - Verified: ZMQ zmq_msg_close is already called on every send-failure path after the fix in ac17fbc Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cmake): default ENABLE_KV_EVENTS to OFF KV events require libzmq which is not available on all CI runners (e.g. Ascend). Default to OFF so builds succeed without it; users who want the feature opt in with -DENABLE_KV_EVENTS=ON. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(store): align KV events publisher with RFC kvcache-ai#1527 spec - Set stored base_block_idx=0 so at least one placement field is present - Emit per-object tenant_id from master metadata, not only global config - Stop spurious removed events on PutRevoke (abort before PutEnd) - Publish removed on disk/NOF eviction paths with medium=disk - Pass explicit medium to eviction removed helper Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): address PR kvcache-ai#2214 review and CI issues - Fix clang-format on kv_event_publisher.h (CI Check code format) - Add optional object_key field (kv_events_emit_object_key, default on) for Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding - Publish events with empty seq_hashes when object_key is emitted but hash cannot be parsed; update indexer API docs and field matrix - Wire emit_object_key through master config, gflags, and master.json Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * style: clang-format master_admin_service.cpp Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * fix(store): emit per-block KV events without global semantic fields Align Mooncake master publisher with Dynamo KV events model: each event describes a pooled block (seq_hash/object_key, medium, tenant_id), not process-wide model/block_size/lora/dp_rank invariants. Omit unknown envelope fields (nil) and supply stream dimensions via indexer POST /register. Deprecate master flags that previously stamped every event. Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> * feat(store): include group ID in KV events Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> --------- Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Implement an opt-in ZMQ publisher on mooncake_master that emits standardized KV cache events for Dynamo global KV indexer integration.
Description
Summary
Adds an optional KV Events publisher to
mooncake_masterso decoupled Mooncake storage can feed Dynamo’s global KV indexer (and other RFC #1527 consumers) without inference workers emitting events directly.Changes
KvEventPublisher: async bounded queue + ZMQ PUB worker (low overhead on hot path; drops when queue is full)event_type,seq_hashes,backend_id,medium,tenant_id, etc.type,block_hashes) for vLLM/SGLang/Dynamo ZMQ relay compatibility (default on)PutEnd→stored;Remove/PutRevoke/ eviction →removedenable_kv_events(default false), bind endpoint,backend_id,model_name,block_size,tenant_id, …GET /kv_events/statuson master metrics HTTP portUsage (indexer registration)
{ "endpoint": "tcp://<master-host>:5557", "type": "Mooncake", "modelname": "deepseek", "tenant_id": "default", "instance_id": "node-1-router-target", "block_size": 64, "dp_rank": 0 }Object keys must be rolling
seq_hashvalues encoded as decimal or0xhex strings.Related
Testing
kv_event_publisher_test(hash parsing + disabled noop)mooncake_storebuild not run in this environment (missing system deps); CI should validateModule
mooncake-transfer-engine)mooncake-store)mooncake-ep)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-pg)mooncake-rl)Type of Change
How Has This Been Tested?
Checklist
./scripts/code_format.shbefore submitting.