Skip to content

feat(store): add optional RFC #1527 KV events publisher on master - #2214

Merged
stmatengss merged 22 commits into
kvcache-ai:mainfrom
stmatengss:cursor/kv-event-publisher-master-ca8e
Jul 8, 2026
Merged

stmatengss merged 22 commits into
kvcache-ai:mainfrom
stmatengss:cursor/kv-event-publisher-master-ca8e

Conversation

@stmatengss

Copy link
Copy Markdown
Collaborator

Implement an opt-in ZMQ publisher on mooncake_master that emits standardized KV cache events for Dynamo global KV indexer integration.

  • Add KvEventPublisher with async bounded queue and background worker
  • Publish stored/removed events on PutEnd, Remove, and eviction paths
  • Wire RFC [RFC]: Kv Events API Standardization #1527 envelope fields plus optional vLLM/SGLang compat aliases
  • Expose gflags/config toggles (enable_kv_events, bind endpoint, backend_id)
  • Add GET /kv_events/status on the master metrics HTTP server
  • Document Mooncake master publisher usage in indexer API design

Description

Summary

Adds an optional KV Events publisher to mooncake_master so decoupled Mooncake storage can feed Dynamo’s global KV indexer (and other RFC #1527 consumers) without inference workers emitting events directly.

Changes

  • KvEventPublisher: async bounded queue + ZMQ PUB worker (low overhead on hot path; drops when queue is full)
  • RFC [RFC]: Kv Events API Standardization #1527 wire format in msgpack event maps: event_type, seq_hashes, backend_id, medium, tenant_id, etc.
  • Optional legacy aliases (type, block_hashes) for vLLM/SGLang/Dynamo ZMQ relay compatibility (default on)
  • Hooks: PutEnd → stored; Remove / PutRevoke / eviction → removed
  • Config: enable_kv_events (default false), bind endpoint, backend_id, model_name, block_size, tenant_id, …
  • Ops: GET /kv_events/status on master metrics HTTP port
  • Docs: Mooncake master publisher section in indexer API design

Usage (indexer registration)

{
  "endpoint": "tcp://<master-host>:5557",
  "type": "Mooncake",
  "modelname": "deepseek",
  "tenant_id": "default",
  "instance_id": "node-1-router-target",
  "block_size": 64,
  "dp_rank": 0
}

Object keys must be rolling seq_hash values encoded as decimal or 0x hex strings.

Related

Testing

  • Added kv_event_publisher_test (hash parsing + disabled noop)
  • Full mooncake_store build not run in this environment (missing system deps); CI should validate

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake EP (mooncake-ep)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • PyTorch Backend (mooncake-pg)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Other

How Has This Been Tested?

Checklist

  • I have performed a self-review of my own code.
  • I have formatted my own code using ./scripts/code_format.sh before submitting.
  • I have updated the documentation.
  • I have added tests to prove my changes are effective.

…master

Implement an opt-in ZMQ publisher on mooncake_master that emits
standardized KV cache events for Dynamo global KV indexer integration.

- Add KvEventPublisher with async bounded queue and background worker
- Publish stored/removed events on PutEnd, Remove, and eviction paths
- Wire RFC kvcache-ai#1527 envelope fields plus optional vLLM/SGLang compat aliases
- Expose gflags/config toggles (enable_kv_events, bind endpoint, backend_id)
- Add GET /kv_events/status on the master metrics HTTP server
- Document Mooncake master publisher usage in indexer API design

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a KV cache event publisher for the Mooncake Store based on RFC #1527, allowing the master service to broadcast cache state changes via ZMQ. The implementation includes a background worker for asynchronous processing, new configuration options, and an HTTP status endpoint. Feedback from the review identifies several significant issues, including incorrect Msgpack map sizing that could cause decoding errors, potential memory leaks in ZMQ message handling, and a logic flaw where events are not emitted for individual replica removals during eviction. Other recommendations include optimizing the worker thread's wait condition, ensuring all pending events are drained during shutdown, and using portable endian conversion functions.

Comment on lines +252 to +255
const size_t map_size =
is_stored
? (config_.emit_legacy_compat_fields ? 17 : 14)
: (config_.emit_legacy_compat_fields ? 15 : 12);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The map_size calculation is incorrect for 3 out of 4 cases. Msgpack maps require an exact count of key-value pairs; providing a wrong size will cause decoding failures or data corruption at the consumer.

Based on the fields packed below:

  • is_stored + legacy: 18 fields (Base 12 + Stored 3 + Legacy 3)
  • is_stored - legacy: 15 fields (Base 12 + Stored 3)
  • !is_stored + legacy: 15 fields (Base 12 + Removed 1 + Legacy 2)
  • !is_stored - legacy: 13 fields (Base 12 + Removed 1)

Your current code uses 17, 14, 15, and 12 respectively.

Suggested change
const size_t map_size =
is_stored
? (config_.emit_legacy_compat_fields ? 17 : 14)
: (config_.emit_legacy_compat_fields ? 15 : 12);
const size_t map_size =
is_stored
? (config_.emit_legacy_compat_fields ? 18 : 15)
: (config_.emit_legacy_compat_fields ? 15 : 13);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch -- the map sizes are indeed off by one in three branches. I verified the field count: base 12 + stored 3 or removed 1, plus legacy adds 3 (stored) or 2 (removed). Will fix to 18/15/15/13 as suggested.

Comment on lines +334 to +345
const int rc = zmq_sendmsg(zmq_socket_, &topic_msg, ZMQ_SNDMORE);
if (rc >= 0) {
if (zmq_sendmsg(zmq_socket_, &seq_msg, ZMQ_SNDMORE) >= 0) {
zmq_sendmsg(zmq_socket_, &payload_msg, 0);
} else {
zmq_msg_close(&payload_msg);
}
} else {
zmq_msg_close(&seq_msg);
zmq_msg_close(&payload_msg);
}
zmq_msg_close(&topic_msg);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There are potential memory leaks in the ZMQ message sending logic. According to ZMQ documentation, zmq_sendmsg only takes ownership of the message (and thus empties it) on success. If any part of a multi-part message fails, the subsequent zmq_msg_t objects that were not sent must be explicitly closed using zmq_msg_close to avoid leaking internal ZMQ buffers.

    const int rc = zmq_sendmsg(zmq_socket_, &topic_msg, ZMQ_SNDMORE);
    if (rc >= 0) {
        if (zmq_sendmsg(zmq_socket_, &seq_msg, ZMQ_SNDMORE) >= 0) {
            if (zmq_sendmsg(zmq_socket_, &payload_msg, 0) < 0) {
                zmq_msg_close(&payload_msg);
            }
        } else {
            zmq_msg_close(&seq_msg);
            zmq_msg_close(&payload_msg);
        }
    } else {
        zmq_msg_close(&seq_msg);
        zmq_msg_close(&payload_msg);
    }
    zmq_msg_close(&topic_msg);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed -- the final zmq_sendmsg for payload_msg needs its return value checked with zmq_msg_close on failure. Will apply the suggested fix.

Comment thread mooncake-store/src/master_service.cpp Outdated
try_evict_or_offload(it->first, it->second, shard);
total_freed_size += freed;
if (it->second.IsValid() == false) {
PublishKvRemoved(it->first, it->second);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In BatchEvict, PublishKvRemoved is only called when the entire object is removed from the master (IsValid() == false). If a memory replica is evicted but a disk replica remains, no event is sent. Consequently, the indexer will still report a hit on the cpu medium, leading to incorrect scheduling decisions by routers. Events should be emitted per-medium whenever a replica is removed.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid observation. The current implementation emits at object granularity -- the indexer sees an object as fully present or fully removed. For v1, this is acceptable because the Dynamo indexer merges engine-level events (which carry GPU/CPU_PINNED media) with master events (host/disk pool only). Per-medium eviction events would be valuable for tiered routing; I will file a follow-up issue to track this. Does that work for now?

Comment on lines +188 to +190
queue_cv_.wait_for(lock, std::chrono::milliseconds(5), [this] {
return stop_.load() || !queue_.empty();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using wait_for with a hardcoded 5ms timeout effectively turns this background worker into a polling loop, which is inefficient for CPU and power usage when the queue is empty. Since stop_ is checked within the predicate, you should use a simple wait to block until a notification is received or the thread is stopped.

Suggested change
queue_cv_.wait_for(lock, std::chrono::milliseconds(5), [this] {
return stop_.load() || !queue_.empty();
});
queue_cv_.wait(lock, [this] {
return stop_.load() || !queue_.empty();
});

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 5ms timeout is intentional as a micro-batching window -- it lets the worker coalesce multiple rapid events into one ZMQ send, reducing per-message overhead. With a pure wait(), every single event would trigger an immediate send. The CPU cost of the 5ms poll is negligible. I will add a comment clarifying this design choice.

Comment on lines +201 to +210
{
std::lock_guard<std::mutex> lock(queue_mutex_);
while (!queue_.empty() && batch.size() < kMaxBatchSize) {
batch.push_back(std::move(queue_.front()));
queue_.pop_front();
}
}
if (!batch.empty()) {
PublishBatch(batch);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The shutdown drain logic only processes a single batch (up to kMaxBatchSize). If the queue contains more events than the batch size when stop_ is set, the remaining events will be silently dropped. It is better to loop until the queue is completely empty to ensure all pending events are published during a graceful shutdown.

    while (true) {
        batch.clear();
        {
            std::lock_guard<std::mutex> lock(queue_mutex_);
            if (queue_.empty()) break;
            while (!queue_.empty() && batch.size() < kMaxBatchSize) {
                batch.push_back(std::move(queue_.front()));
                queue_.pop_front();
            }
        }
        PublishBatch(batch);
    }

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The shutdown drain should loop until the queue is empty. Will wrap it in a while loop as suggested.

Comment on lines +317 to +322
uint64_t seq_be = 0;
const auto* in = reinterpret_cast<const unsigned char*>(&seq);
auto* out = reinterpret_cast<unsigned char*>(&seq_be);
for (int i = 0; i < 8; ++i) {
out[i] = in[7 - i];
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This manual byte-swapping logic assumes the host is little-endian. While common, it is non-portable and brittle. It is safer and clearer to use standard endian conversion functions like htobe64.

Suggested change
uint64_t seq_be = 0;
const auto* in = reinterpret_cast<const unsigned char*>(&seq);
auto* out = reinterpret_cast<unsigned char*>(&seq_be);
for (int i = 0; i < 8; ++i) {
out[i] = in[7 - i];
}
const uint64_t seq = next_zmq_sequence_.fetch_add(1);
uint64_t seq_be = htobe64(seq);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed -- htobe64 is cleaner and handles both endiannesses correctly. Will replace the manual loop. Since Mooncake targets Linux, htobe64 from <endian.h> is always available.

…er vs register)

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
@stmatengss
stmatengss requested a review from yejj710 May 26, 2026 15:23

mutable std::mutex queue_mutex_;
std::deque<PendingEvent> queue_;
std::condition_variable queue_cv_;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing include <condition_variable>

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No require

else()
message(FATAL_ERROR
"libzmq not found. Install libzmq3-dev to build mooncake_store.")
endif()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it more appropriate to strongly depend only when the publisher is enabled?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fix

// Emit legacy vLLM/SGLang field names alongside RFC #1527 fields.
bool emit_legacy_compat_fields{true};
// Bounded queue capacity; additional events are dropped.
uint32_t queue_capacity{65536};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The segment managed by the store is determined once the client is connected, so the queue size is controllable and does not grow indefinitely. Adding size constraints and eviction here may lead to consistency issues.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

Comment thread mooncake-store/src/master_service.cpp Outdated
}

if (metadata.IsValid() == false) {
PublishKvRemoved(key, metadata);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The store_event will only be published after putEnd succeeds. In putRevoke, there is no need to publish remove_event because the KV has not yet been stored in the mooncake-store, right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix

cursoragent and others added 3 commits June 2, 2026 17:07
- Fix msgpack map sizes with ComputeEventMapSize helper
- Close ZMQ message parts on partial send failure
- Emit per-medium removed events when memory replicas are evicted
- Remove spurious removed event on PutRevoke before PutEnd
- Use condition_variable::wait, drain full queue on shutdown, htobe64
- Drop bounded queue/drop-on-full to avoid indexer consistency gaps
- Make libzmq optional via ENABLE_KV_EVENTS (stub when disabled)
- Add missing condition_variable include and key_util for hash parsing

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Integrate upstream multi-tenant metadata and promotion_max_per_heartbeat
with KV events publisher changes. Eviction hooks use tenant-aware
try_evict_group_or_object; per-member removed events for grouped keys.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
@codecov-commenter

codecov-commenter commented Jun 5, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 67.98419% with 81 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mooncake-store/include/master_config.h 34.28% 23 Missing ⚠️
mooncake-store/src/master_service.cpp 73.86% 23 Missing ⚠️
mooncake-store/src/master_admin_service.cpp 5.88% 16 Missing ⚠️
mooncake-store/src/master.cpp 78.12% 14 Missing ⚠️
mooncake-store/src/rpc_service.cpp 0.00% 4 Missing ⚠️
mooncake-store/include/kv_event/key_util.h 94.11% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

cursoragent and others added 3 commits June 5, 2026 08:16
Propagate ENABLE_KV_EVENTS=OFF to the parent CMake scope when libzmq is
not found so kv_event_publisher_test uses the header stub consistently
with mooncake_store. Add libzmq3-dev to dependencies.sh for full builds.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
mooncake_store pulls in ZMQ symbols when libzmq is present; propagate
the optional -lzmq link in build.rs using the same has_library pattern
as uring/etcd.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Go integration tests link libmooncake_store.a statically and need -lzmq
when the KV events publisher is compiled in.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
@cursor
cursor Bot requested review from Ann-1024 and luketong777 as code owners June 6, 2026 17:20
cursoragent and others added 2 commits June 6, 2026 18:23
Previous build (3.12) failed during Codecov upload with 'No space left on
device' after all tests passed. No code changes.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
When libzmq was not found, ENABLE_KV_EVENTS was only cleared in the parent
CMake scope while the local value stayed ON. That defined
MOONCAKE_ENABLE_KV_EVENTS=1 without linking kv_event_publisher.cpp,
breaking mooncake_master on platforms like the Ascend CI image.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
…Revoke event, cmake guard

- PutRevoke: emit PublishKvRemoved before erasing an invalidated object
  so that downstream indexers are notified when a revoked key is removed
- cmake: upgrade ZMQ-not-found diagnostic from WARNING to FATAL_ERROR
  when ENABLE_KV_EVENTS is explicitly ON, since the user opted in
- Verified: msgpack map_size values (18/15/15/13) are already correct
  after ComputeEventMapSize refactor in ac17fbc
- Verified: ZMQ zmq_msg_close is already called on every send-failure
  path after the fix in ac17fbc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set stored base_block_idx=0 so at least one placement field is present
- Emit per-object tenant_id from master metadata, not only global config
- Stop spurious removed events on PutRevoke (abort before PutEnd)
- Publish removed on disk/NOF eviction paths with medium=disk
- Pass explicit medium to eviction removed helper

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation CI/Build labels Jun 15, 2026
@stmatengss

Copy link
Copy Markdown
Collaborator Author

@qoder review this PR according to #1527

@qoderai

qoderai Bot commented Jun 16, 2026 •

Copy link
Copy Markdown

Here’s a focused review of this PR against RFC #1527 (issue #1527):

Overall

  • Implements an opt-in, master-side RFC [RFC]: Kv Events API Standardization #1527 KV events publisher over ZMQ.
  • Design matches the RFC’s intent: master reports host/disk pool visibility, not GPU tiers, and uses standardized envelope fields plus optional vLLM/SGLang aliases.
  • Implementation addresses prior review from Gemini and human reviewers (map size, ZMQ lifetime, shutdown draining, endian handling).

API & Config (RFC alignment)

  • New config fields in MasterConfig/MasterServiceConfig/WrappedMasterServiceConfig (enable_kv_events, kv_events_bind_endpoint, kv_events_model_name, kv_events_backend_id, kv_events_tenant_id, kv_events_block_size, kv_events_dp_rank, kv_events_additional_salt, kv_events_lora_name, kv_events_emit_legacy_compat) line up with RFC [RFC]: Kv Events API Standardization #1527’s envelope.
  • Default JSON config in mooncake-store/conf/master.json keeps enable_kv_events=false, sane defaults for bind endpoint and tenant, and avoids accidental emission.
  • gflags in master.cpp are wired into config load paths and logged on startup; CLI vs config precedence behavior looks consistent with existing options.

Publisher implementation

  • KvEventConfig/KvEventPublisher split is clean; stub implementation when ENABLE_KV_EVENTS=OFF avoids hard dependency on libzmq.
  • CMake/Rust/Go build glue respects ENABLE_KV_EVENTS and only links zmq when present (KV_EVENTS_ZMQ_FOUND toggle, optional CGO/Rust linking). This addresses the “only strong-depend when enabled” feedback.
  • Queue is now unbounded in the publisher, with a separate, deprecated kv_events_queue_capacity config that is explicitly ignored, which matches the reviewer’s concern about consistency vs drops.
  • Worker loop uses a condition-variable wait with a stop predicate and drains remaining events on shutdown via DrainRemainingQueue, so we don’t silently drop queued events.
  • ZMQ multipart send path correctly:
    • Builds 3 frames: empty topic, big-endian sequence (htobe64), msgpack payload.
    • Checks each zmq_sendmsg call and closes unsent zmq_msg_t instances on failure, avoiding leaks.

Wire format & RFC #1527 details

  • Event payload structure:
    • Top-level array: [timestamp_ms, [events...], dp_rank] matches the RFC and vLLM/SGLang batch framing.
    • Each event map contains the RFC envelope fields: event_id, timestamp, event_type, model_name, block_size, additional_salt, lora_name, tenant_id, backend_id, medium, dp_rank, seq_hashes.
    • When kv_events_emit_legacy_compat is on, legacy aliases type and block_hashes are emitted as expected.
  • ComputeEventMapSize now matches the actual packed fields (18/15/15/13) so msgpack maps decode correctly.
  • Stored vs removed:
    • stored: emits base_block_idx=0, parent_hash=nil, token_ids=nil; this is in line with RFC [RFC]: Kv Events API Standardization #1527 for standalone pool blocks.
    • removed: omits extra fields beyond seq_hashes/base_block_idx as allowed by the RFC.
  • ParseSeqHashFromObjectKey enforces decimal or 0x-prefixed hex u64 keys and increments skipped_unparsed_keys when parsing fails, consistent with the RFC guidance to skip unparseable keys.

Master integration (event semantics)

  • MasterService wires publishing on:
    • PutEnd: calls PublishKvStored with medium chosen from ReplicaType/ObjectMetadata (cpu vs disk), consistent with the RFC’s medium semantics for master (no GPU).
    • Remove, EvictDiskReplica, BatchEvict, NoFBatchEvict: call PublishKvRemoved / PublishKvRemovedAfterEvict in the right places so disk and memory evictions generate removed events when replicas actually disappear.
  • MediumForReplicaType and MediumForMetadata produce "cpu" for memory and "disk" for on-disk/NOF replicas, matching the RFC’s “Master only reports host/disk” positioning.
  • PutRevoke path has been adjusted per review so we don’t emit removed for keys that never successfully stored in master.

Observability & control plane

  • HTTP endpoint /kv_events/status returns JSON status including enabled, published_batches, published_events, dropped_events, and skipped_unparsed_keys, which aligns with RFC [RFC]: Kv Events API Standardization #1527’s emphasis on observability.
  • Startup log prints KV events configuration, which is useful for diagnosing misconfigurations.

Docs

  • docs/source/design/conductor/indexer-api-design.md gets a dedicated “Mooncake Store master publisher” section plus a field provenance matrix and mermaid diagram. This directly documents how master events complement SGLang events and indexer registration, mirroring RFC [RFC]: Kv Events API Standardization #1527.

Tests

  • kv_event_publisher_test covers:
    • Hash parsing via ParseSeqHashFromObjectKey (decimal and hex variants, invalid cases).
    • Disabled publisher being a no-op and not emitting stats.
  • This gives basic correctness coverage for parsing and the stub behavior; deeper tests around actual ZMQ output would likely require integration tests and are reasonably out of scope here.

Alignment with issue #1527

  • The implemented design matches the RFC’s goals:
    • Optional, master-side publisher for host/disk tiers.
    • RFC [RFC]: Kv Events API Standardization #1527 envelope with optional legacy aliases.
    • Clear config surface for backend_id, model_name, tenant_id, block_size, dp_rank, and salt.
    • Indexer-facing docs explaining SGLang vs master vs registration provenance.

Conclusion

  • From an RFC [RFC]: Kv Events API Standardization #1527 perspective, this PR looks consistent and complete.
  • The previously raised critical issues (msgpack map sizing, ZMQ message lifetime, shutdown draining, endian conversion) have been addressed in the current diff.
  • I don’t see blocking problems relative to issue [RFC]: Kv Events API Standardization #1527; remaining refinements (e.g., per-medium removed granularity for tiered routing) can be handled as follow-up improvements.

Looks good to proceed from the RFC #1527 / KV events design standpoint. 🚀


🤖 Generated by Qoder • View workflow run

Comment on lines +26 to +28
"kv_events_model_name": "",
"kv_events_backend_id": "",
"kv_events_tenant_id": "default",

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need these configurations.

Comment thread mooncake-store/src/master_service.cpp Outdated
Comment on lines +2210 to +2230
if (replica_type != ReplicaType::DISK &&
replica_type != ReplicaType::LOCAL_DISK) {
LOG(ERROR) << "key=" << key
<< ", error=invalid_replica_type_for_eviction";
return tl::make_unexpected(ErrorCode::INVALID_PARAMS);
}

const size_t erased =
replica_type == ReplicaType::DISK
? metadata.EraseReplicas([](const Replica& replica) {
return replica.is_disk_replica();
})
: metadata.EraseReplicas([&client_id](const Replica& replica) {
return replica.is_local_disk_replica() &&
replica.get_descriptor()
.get_local_disk_descriptor()
.client_id == client_id;
});
if (replica_type == ReplicaType::DISK) {
MasterMetricManager::instance().dec_file_cache_nums();
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why need all these changes

Comment on lines +972 to +977
std::string kv_events_backend_id;
std::string kv_events_tenant_id = "default";
std::string kv_events_additional_salt;
std::string kv_events_lora_name;
uint32_t kv_events_block_size = 0;
uint32_t kv_events_dp_rank = 0;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mooncake store supports multiple tenants. So why fix all these configurations?

cursor Bot pushed a commit to stmatengss/Mooncake that referenced this pull request Jun 16, 2026
- Fix clang-format on kv_event_publisher.h (CI Check code format)
- Add optional object_key field (kv_events_emit_object_key, default on) for
  Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding
- Publish events with empty seq_hashes when object_key is emitted but hash
  cannot be parsed; update indexer API docs and field matrix
- Wire emit_object_key through master config, gflags, and master.json

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
- Fix clang-format on kv_event_publisher.h (CI Check code format)
- Add optional object_key field (kv_events_emit_object_key, default on) for
  Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding
- Publish events with empty seq_hashes when object_key is emitted but hash
  cannot be parsed; update indexer API docs and field matrix
- Wire emit_object_key through master config, gflags, and master.json

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
@cursor
cursor Bot force-pushed the cursor/kv-event-publisher-master-ca8e branch from ec3c14f to 7abfdca Compare June 16, 2026 14:06
cursoragent and others added 3 commits June 21, 2026 17:42
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Merge upstream main and re-apply KV event publisher integration after
upstream refactors (MasterAdminServer extraction, BatchEvict deferred
replicas, EraseReplicasWithCacheTotalAccounting).

- Restore KV hooks in master_service (PutEnd, Remove, evictions)
- Add /kv_events/status to master_admin_service
- Delegate KvEventsEnabled/GetKvEventStats from rpc_service
- Keep both kv_event_publisher_test and deadline_scheduler_test

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
@stmatengss

Copy link
Copy Markdown
Collaborator Author

@qoder Review this PR by line, and compare with spec (#1527) to verify it.

@qoderai

qoderai Bot commented Jun 22, 2026 •

Copy link
Copy Markdown

Done! ✅

I’ve completed a line-by-line review of PR #2214 against the RFC #1527 spec in issue #1527 and posted a review on the PR:

  • Confirmed the master KV events publisher behavior (ZMQ framing, msgpack layout, RFC field set, legacy aliases) matches the spec.
  • Verified config flags/JSON, hooks on PutEnd/Remove/eviction, /kv_events/status endpoint, and tests.
  • Left a summary review comment with two minor, non-blocking suggestions around kv_events_emit_legacy_compat JSON exposure and optional block_size.

Let me know if you’d like a stricter checklist (e.g., for downstream Dynamo integration) or follow-up tests. 🚀


🤖 Generated by Qoder • View workflow run

@qoderai qoderai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this PR against the RFC #1527 master KV events publisher spec from issue #1527. The implementation closely matches the spec: optional ZMQ publisher on mooncake_master, correct RFC field names plus legacy aliases, hooks on PutEnd/Remove/eviction, config surface + HTTP status endpoint, and basic tests. Left a summary review comment with two minor non-blocking suggestions (exposing kv_events_emit_legacy_compat in JSON and considering stricter handling of block_size for strict RFC consumers).


🤖 Generated by Qoder • View workflow run

@@ -0,0 +1,36 @@
#pragma once

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spec vs implementation review (RFC #1527 master KV publisher)

Overall the implementation matches the spec in #1527 quite closely:

  • Optional ZMQ PUB publisher on mooncake_master with RFC [RFC]: Kv Events API Standardization #1527 fields and vLLM-compatible aliases
  • Config surface on MasterConfig/flags/JSON, and Rust/Go build hooks for libzmq
  • Event hooks on PutEnd, Remove, and eviction paths, plus /kv_events/status HTTP endpoint
  • Tests for key parsing and disabled publisher path

Below are concrete line-level checks and a few small nits:

  1. Build/feature flagging and deps
  • mooncake-store/CMakeLists.txt + src/CMakeLists.txt + rust/build.rs + go/build.sh + dependencies.sh
    • ✅ Matches spec: ZMQ is optional, discovered via CMake/Rust/go build scripts, and ENABLE_KV_EVENTS defaults to OFF.
    • ✅ Fatal error when ENABLE_KV_EVENTS=ON but libzmq is missing is reasonable and matches “requires libzmq when ON”.
    • ✅ Rust and Go branches only link -lzmq when the library is present.
  1. Config surface (flags + JSON + structs)
  • mooncake-store/src/master.cpp (DEFINE_* for enable_kv_events, kv_events_*) and wiring into MasterConfig via InitMasterConf + LoadConfigFromCmdline
  • mooncake-store/conf/master.json and mooncake-store/include/master_config.h
    • ✅ Fields match the spec: enable_kv_events, kv_events_bind_endpoint, kv_events_model_name, kv_events_backend_id, kv_events_tenant_id, kv_events_block_size, kv_events_dp_rank, plus kv_events_additional_salt, kv_events_lora_name, kv_events_emit_legacy_compat, kv_events_emit_object_key.
    • ✅ Default tenant_id = "default" matches the spec.
    • 🔍 Minor: JSON only exposes kv_events_emit_object_key; kv_events_emit_legacy_compat is flag-only (no JSON default). That’s acceptable, but if you want full config parity, you might consider adding it to master.json as well.
  1. Key parsing and seq_hashes behavior
  • mooncake-store/include/kv_event/key_util.h
    • ✅ Implements decimal or 0x/0X hex u64 parsing from the entire object key, which aligns with the spec’s “object keys may encode rolling seq_hash as decimal or 0x-prefixed hex string; reject mixed strings like 123abc”.
    • ✅ Returns nullopt on partial or invalid parses.
  1. Publisher config and stub behavior
  • KvEventConfig (include/kv_event/kv_event_config.h) and stubbed KvEventPublisher when ZMQ is absent
    • ✅ Config fields and defaults match the RFC: enabled, bind_endpoint, model_name, backend_id, tenant_id, additional_salt, lora_name, block_size, dp_rank, emit_legacy_compat_fields, emit_object_key.
    • ✅ Stub implementation is a no-op but still exposes ParseSeqHashFromObjectKey and GetStats, which fits the “disabled emitter is safe noop” requirement.
  1. Wire format and field matrix
  • mooncake-store/src/kv_event/kv_event_publisher.cpp
    • ✅ ZMQ framing: three frames (topic="", big-endian sequence, msgpack payload) matches the spec and vLLM/SGLang format.
    • ✅ Top-level payload is [timestamp_ms, [events], dp_rank].
    • ✅ Each event map uses RFC field names plus optional legacy aliases:
      • Envelope: event_id, timestamp, event_type, model_name, block_size, additional_salt, lora_name, tenant_id, backend_id, medium, dp_rank, seq_hashes.
      • Optional: object_key (when emit_object_key), type + block_hashes (when emit_legacy_compat_fields).
      • Stored-specific: base_block_idx, parent_hash, token_ids, and parent_block_hash when legacy is enabled.
      • Removed-specific: base_block_idx.
    • ✅ event_type values stored/removed match the spec; legacy type BlockStored/BlockRemoved are present.
    • ✅ block_size and dp_rank are nullable (nil) when not set, which matches the optional envelope semantics.
    • ✅ seq_hashes behavior matches the matrix:
      • Single-element array when the object key parses as u64.
      • Empty array when parsing fails but emit_object_key=true (falling back to object_key only).
    • ✅ object_key is always emitted when emit_object_key is enabled, independently of parse result, which matches the text.
    • 🔍 Potential spec nuance: the spec notes that block_size is required for token↔block mapping. Here, you allow block_size=0 and pack nil. That’s consistent with a “best-effort” publisher, but for strict RFC consumers you might want to either (a) enforce nonzero block_size when enable_kv_events=true or (b) document that master publisher is not suitable for token-level queries without that set.
  1. Event hooks and semantics
  • mooncake-store/src/master_service.cpp and header declarations
    • ✅ PutEnd calls PublishKvStored, so successful writes emit stored events.
    • ✅ Remove calls PublishKvRemoved(key, metadata, tenant_id), so user-initiated deletes emit removed events.
    • ✅ Eviction paths:
      • EvictDiskReplica publishes removed when disk replicas are erased.
      • BatchEvict and NoFBatchEvict publish removed for CPU/disk evictions with the appropriate medium (cpu or disk).
      • PublishKvRemovedAfterEvict handles both freed_bytes>0 and complete-metadata removal cases, which covers the spec’s “eviction paths” requirement.
    • ✅ MediumForReplicaType/MediumForMetadata restrict master to "cpu"/"disk" mediums, matching the spec’s statement that master never emits GPU tiers.
    • ✅ BuildKvEventConfig pulls through all the expected kv_events_* fields from config.
    • ✅ KV publisher is initialized once in MasterService and guarded with enabled() checks, so it’s truly optional.
  1. Admin HTTP surface
  • mooncake-store/src/master_admin_service.cpp + include/master_admin_service.h
    • ✅ Adds GET /kv_events/status returning enabled, published_batches, published_events, dropped_events, and skipped_unparsed_keys via JSON, matching the spec.
    • ✅ MasterAdminServer::HandleKvEventsStatus delegates to WrappedMasterService::GetKvEventStats() / KvEventsEnabled() as expected.
  1. RPC layer and stats plumbing
  • mooncake-store/include/rpc_service.h and src/rpc_service.cpp
    • ✅ Exposes KvEventsEnabled() and GetKvEventStats() from WrappedMasterService down into MasterService, matching the admin handler usage.
  1. Tests
  • mooncake-store/tests/kv_event_publisher_test.cpp + tests/CMakeLists.txt
    • ✅ Confirms ParseSeqHashFromObjectKey decimal/hex behavior and invalid-input handling.
    • ✅ Confirms that a disabled KvEventPublisher publishes no events and doesn’t crash.
    • 🔍 Suggestion (non-blocking): adding a small ENABLE_KV_EVENTS test that constructs a publisher with a dummy inproc endpoint and validates the basic msgpack structure would more directly exercise the batch format, but that’s beyond the current spec.
  1. Docs
  • docs/source/design/conductor/indexer-api-design.md
    • ✅ New “Mooncake Store master publisher” section and field provenance matrix align with the RFC description and the actual implementation (field names, sources, and recommended split between SGLang/master/register).

Conclusion

  • From a spec-compliance standpoint, the PR looks correct and complete for the master-side KV events publisher.
  • The only minor gaps I see are:
    • kv_events_emit_legacy_compat is flag-only (not surfaced in master.json).
    • block_size is allowed to be unset even when events are enabled; if strict RFC consumers rely on it, you might consider enforcing it or clearly documenting that master events without block_size are for observability only.

Otherwise this aligns well with issue #1527. ✅


🤖 Generated by Qoder • Fix in Qoder

Align Mooncake master publisher with Dynamo KV events model: each event
describes a pooled block (seq_hash/object_key, medium, tenant_id), not
process-wide model/block_size/lora/dp_rank invariants.

Omit unknown envelope fields (nil) and supply stream dimensions via indexer
POST /register. Deprecate master flags that previously stamped every event.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built this revision with KV events enabled and exercised it end-to-end with SGLang HiCache and the Dynamo shared-cache router. Raw SGLang object keys work, but the removal contract and publisher failure behavior can make an event-driven consumer diverge from Mooncake. Requesting the four focused changes below before this stream is used as a presence index.

Comment thread mooncake-store/src/master_service.cpp Resolved
}

TEST(KvEventPublisherTest, DisabledPublisherIsNoop) {
KvEventConfig config;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only publisher test exercises the disabled no-op path, so CI never verifies the enabled ZMQ framing or the integration-critical SGLang key shape. Please add an enabled IPC/TCP test that receives the three frames, decodes MessagePack, and asserts a SHA256 _0_k object key is emitted with seq_hashes=[] and truthful stats.

cursoragent and others added 2 commits July 6, 2026 13:19
- Merge upstream main (parallel BatchEvict, tenant quota APIs, metadata cleanup)
- Emit removed events only when object metadata is fully invalid after eviction
- Bound publisher queue with dropped_events accounting and ZMQ sequence gaps
- Fix PublishBatch stats and always close ZMQ messages; count on send success only
- Add ZMQ IPC integration test for SGLang object_key shape and queue overflow

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

ishandhanani commented Jul 6, 2026 •

Copy link
Copy Markdown
Collaborator

I rebased the minimal ObjectMetadata.group_id event propagation onto the current PR head (e9fcf14) and opened a stacked PR against this PR branch: stmatengss#19 (6d670b5). This is a single focused commit; the queue/removal/counter fixes were dropped because the current PR head already includes them. Validation: rebuilt mooncake_master and passed kv_event_publisher_test.

…ixes

feat(store): include group ID in KV events

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-tested head 2f9df5bb with a source-matched Mooncake master/client, SGLang HiCache, and the Dynamo event-driven shared-cache branch. The original removal, queue/drop accounting, send accounting/cleanup, and wire-coverage blockers are resolved; the merged group_id field also works end to end. Live Qwen3-0.6B validation completed 50/50 requests with 1.0 shared-cache hit rate, 169 shared pages/request, 339 published events, zero drops/query errors, and 1.805 ms mean shared-cache lookup. Local kv_event_publisher_test, master_service_test, batch_remove_test, and offload_on_evict_test all passed. Approving. The current CI format failure in kv_event_config.h is mechanical and still needs to be fixed before merge.

@stmatengss
stmatengss merged commit 83ff47f into kvcache-ai:main Jul 8, 2026
16 of 18 checks passed
fy2462 pushed a commit to fy2462/Mooncake that referenced this pull request Jul 20, 2026
… master (kvcache-ai#2214)

* feat(store): add optional RFC kvcache-ai#1527 KV events publisher on master

Implement an opt-in ZMQ publisher on mooncake_master that emits
standardized KV cache events for Dynamo global KV indexer integration.

- Add KvEventPublisher with async bounded queue and background worker
- Publish stored/removed events on PutEnd, Remove, and eviction paths
- Wire RFC kvcache-ai#1527 envelope fields plus optional vLLM/SGLang compat aliases
- Expose gflags/config toggles (enable_kv_events, bind endpoint, backend_id)
- Add GET /kv_events/status on the master metrics HTTP server
- Document Mooncake master publisher usage in indexer API design

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* docs(conductor): add KV event field provenance matrix (SGLang vs master vs register)

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): address KV events PR review feedback

- Fix msgpack map sizes with ComputeEventMapSize helper
- Close ZMQ message parts on partial send failure
- Emit per-medium removed events when memory replicas are evicted
- Remove spurious removed event on PutRevoke before PutEnd
- Use condition_variable::wait, drain full queue on shutdown, htobe64
- Drop bounded queue/drop-on-full to avoid indexer consistency gaps
- Make libzmq optional via ENABLE_KV_EVENTS (stub when disabled)
- Add missing condition_variable include and key_util for hash parsing

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* style(store): apply clang-format-20 to KV events changes

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): fix KV events CI link failure when libzmq is absent

Propagate ENABLE_KV_EVENTS=OFF to the parent CMake scope when libzmq is
not found so kv_event_publisher_test uses the header stub consistently
with mooncake_store. Add libzmq3-dev to dependencies.sh for full builds.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): link libzmq in Rust bindings when KV events enabled

mooncake_store pulls in ZMQ symbols when libzmq is present; propagate
the optional -lzmq link in build.rs using the same has_library pattern
as uring/etcd.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): link libzmq in Go CGO flags for KV events

Go integration tests link libmooncake_store.a statically and need -lzmq
when the KV events publisher is compiled in.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* ci: re-trigger build after runner disk-space failure

Previous build (3.12) failed during Codecov upload with 'No space left on
device' after all tests passed. No code changes.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): disable KV events compile flag when libzmq is absent

When libzmq was not found, ENABLE_KV_EVENTS was only cleared in the parent
CMake scope while the local value stayed ON. That defined
MOONCAKE_ENABLE_KV_EVENTS=1 without linking kv_event_publisher.cpp,
breaking mooncake_master on platforms like the Ascend CI image.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): address review comments — msgpack map_size, ZMQ leak, PutRevoke event, cmake guard

- PutRevoke: emit PublishKvRemoved before erasing an invalidated object
  so that downstream indexers are notified when a revoked key is removed
- cmake: upgrade ZMQ-not-found diagnostic from WARNING to FATAL_ERROR
  when ENABLE_KV_EVENTS is explicitly ON, since the user opted in
- Verified: msgpack map_size values (18/15/15/13) are already correct
  after ComputeEventMapSize refactor in ac17fbc
- Verified: ZMQ zmq_msg_close is already called on every send-failure
  path after the fix in ac17fbc

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(cmake): default ENABLE_KV_EVENTS to OFF

KV events require libzmq which is not available on all CI runners
(e.g. Ascend). Default to OFF so builds succeed without it; users
who want the feature opt in with -DENABLE_KV_EVENTS=ON.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(store): align KV events publisher with RFC kvcache-ai#1527 spec

- Set stored base_block_idx=0 so at least one placement field is present
- Emit per-object tenant_id from master metadata, not only global config
- Stop spurious removed events on PutRevoke (abort before PutEnd)
- Publish removed on disk/NOF eviction paths with medium=disk
- Pass explicit medium to eviction removed helper

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): address PR kvcache-ai#2214 review and CI issues

- Fix clang-format on kv_event_publisher.h (CI Check code format)
- Add optional object_key field (kv_events_emit_object_key, default on) for
  Dynamo matching on Mooncake store keys without decimal/0x seq_hash encoding
- Publish events with empty seq_hashes when object_key is emitted but hash
  cannot be parsed; update indexer API docs and field matrix
- Wire emit_object_key through master config, gflags, and master.json

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* style: clang-format master_admin_service.cpp

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* fix(store): emit per-block KV events without global semantic fields

Align Mooncake master publisher with Dynamo KV events model: each event
describes a pooled block (seq_hash/object_key, medium, tenant_id), not
process-wide model/block_size/lora/dp_rank invariants.

Omit unknown envelope fields (nil) and supply stream dimensions via indexer
POST /register. Deprecate master flags that previously stamped every event.

Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>

* feat(store): include group ID in KV events

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

---------

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/Build documentation Improvements or additions to documentation run-ci Store

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants