Skip to content

Merge/dpsk4 mooncake store - #231

Merged
linhu-nv merged 33 commits into
mainfrom
merge/dpsk4-mooncake-store
Jul 24, 2026
Merged

linhu-nv merged 33 commits into
mainfrom
merge/dpsk4-mooncake-store

Conversation

@Luis-xu

@Luis-xu Luis-xu commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

Overview

This PR integrates mooncake-store as a distributed KV cache backend for FlexKV, wired in as the REMOTE tier alongside the existing PCFS / P2P remote paths. Unlike the file-addressed 3rd-party remote (block-id → file offset), mooncake-store is key-addressed: each KV / SWA block is keyed by its content hash, so the cluster pool is prefix-shareable across all FlexKV instances without a coordinated block-id allocator.

The integration is fully opt-in via FLEXKV_USE_MOONCAKE_STORE_BACKEND=1 (or UserConfig.use_mooncake_store_backend=True) and is orthogonal to the existing GPU / CPU / SSD tiers, layerwise transfer, EAGLE, DP-attention, and DSv4 SWA/state sidecars. When disabled the code paths remain unchanged.

Key design points:

  • Key semantics — <block_hash>_FlexKV[_pp_rank_i_of_N] for main KV, <tail_hash>_FlexKV_swa for SWA. PP-rank suffix is elided when the local node covers the full model layer range.
  • PUT/GET reuse the same H2REMOTE / REMOTE2H TransferOp types; only the payload carries mooncake_store_block_hashes (main) or mooncake_store_swa_block_hashes (SWA) so the batch-merge and graph-scheduler stay backend-agnostic.
  • Zero-copy — the CPU-tier pinned buffer is directly registered with mooncake for RDMA, so no extra staging buffer is introduced.
  • Match RPC — a single batch_is_exist(kv_keys + swa_keys) per sequence returns both the KV longest-prefix and the SWA joint hit, avoiding two round-trips.

Main Changes

1. New adapter module — flexkv/external/

File Purpose
mooncake_store_utils.py MooncakeStoreConfig (JSON loader), MooncakeStoreClient (thin MooncakeDistributedStore wrapper with batch_put / batch_get / batch_exists / register_buffer), and MooncakeStoreCacheEngine (drop-in replacement for CacheEngineAccel presenting match() / insert())
mooncake_store_keys.py Single source of truth for key construction (PoolKind.KV / PoolKind.SWA + build_key) shared by client, cache engine, and transfer worker
__init__.py Public re-exports with lazy import so unrelated FlexKV runs pay no mooncake import cost

2. MooncakeStoreTransferWorker — new remote I/O worker

flexkv/transfer/worker.py

  • Runs in the same spawn'd subprocess model as other workers; registers the CPU pinned buffer with mooncake and answers H2REMOTE / REMOTE2H ops
  • Two variants: main-KV pool (PoolKind.KV) and SWA pool (PoolKind.SWA, pure-client mode — override_global_segment_size=0 — so SWA workers do not contribute their own storage segment)
  • _preprocess_kv / _preprocess_swa compute (cpu_ptrs, sizes, keys) from the per-op block-id → CPU-slot mapping and the sequence hashes carried on the op

3. Cache-engine wiring — flexkv/cache/cache_engine.py

  • remote_cache_engine is instantiated as MooncakeStoreCacheEngine when the flag is on (else keeps the existing HierarchyLRCacheEngine / PCFS paths)
  • get_match/put_match now propagate the sequence hashes onto op_remote2h.mooncake_store_block_hashes / op_h2remote.mooncake_store_block_hashes, using remote_matched_result.kv_matched_blocks (added to MatchResultAccel) so PUT skips keys that already exist in the store
  • SWA path (_select_swa_read_source → _reserve_swa_read, PUT-side SWA chain in put_match) gains a mooncake branch: the tail hash of the SWA-covered prefix becomes the SWA key, and SWAReadSource carries mooncake_tail_hash in place of a slot / node handle (is_mooncake short-circuits the pin / unlock logic which does not apply on a key-addressed tier)

4. TransferOp / WorkerTransferOp — hash fields

flexkv/common/transfer.py, flexkv/transfer/worker_op.py

  • Two optional fields on TransferOp: mooncake_store_block_hashes (np.ndarray, main KV) and mooncake_store_swa_block_hashes (List[str], SWA tail hashes)
  • WorkerTransferOp forwards both to the worker process; when the fields are set, the op-buffer fast path is bypassed so block ids stay accessible

5. Batch merge — main and SWA lanes split

flexkv/common/transfer.py::merge_to_batch_graph

  • Refactored into _merge_ops (main-KV) + _merge_swa_ops (SWA), each concatenating their respective hash lists in-order and rejecting cross-mixing
  • supported_types extended with H2REMOTE / REMOTE2H; batch sink logic factored into _add_batch_sink (0/1/N terminals) and _pick_dp_client_id
  • Fused-layerwise GET path now keeps REMOTE2H (main + SWA) as standalone predecessors of the fused LAYERWISE op instead of folding them in — remote lanes remain SDK-driven while the local lanes stay layerwise-fused
  • Enforces "no mixed GET/PUT in a single batch" and "layerwise batches are GET-only" via asserts

6. Storage engine — no separate remote region when mooncake is on

flexkv/storage/storage_engine.py

  • When use_mooncake_store_backend is true, the hot CPU / SWA CPU buffers are registered with mooncake directly; the previous "allocate a separate REMOTE contiguous layout" path is skipped for both main and SWA pools

7. Transfer engine — worker map wiring

flexkv/transfer/transfer_engine.py

  • Creates a MooncakeStoreTransferWorker for main-KV H2REMOTE / REMOTE2H (mutually exclusive with the PCFS remote path)
  • Creates a SWA-pool variant when SWA is enabled (_swa_worker_map[H2REMOTE/REMOTE2H])

8. Config — new user-facing options

flexkv/common/config.py

  • UserConfig / CacheConfig: use_mooncake_store_backend, mooncake_store_config_path, plus five layer-range / PP fields (mooncake_store_pp_rank, mooncake_store_pp_size, mooncake_store_node_layer_start, mooncake_store_node_layer_end, mooncake_store_total_layers) auto-populated from rank_info
  • Env fallback: FLEXKV_USE_MOONCAKE_STORE_BACKEND / FLEXKV_MOONCAKE_STORE_CONFIG_PATH
  • enable_remote = enable_3rd_remote OR use_mooncake_store_backend; the "remote requires SSD" precondition is relaxed for mooncake (mooncake bypasses SSD)

Validation

I have tested the accuracy of DPSK-v4-Flash under TP8 and DP8 configurations. The specific methodology involved running the test dataset once on instance 1 and again on instance 2, then observing the evaluation accuracy for both runs. The results are as follows (each dataset was tested twice):

tp8:

数据集 instance1 instance2
AIME 0.6000 0.6333
HumanEval 0.915 0.915

dp8:

数据集 instance1 instance2
AIME 0.6667 0.6667
HumanEval 0.933 0.933

XingLiu and others added 30 commits July 15, 2026 20:31
Squash the final DeepSeek-V4 multi-group and SWA changes onto the latest origin/main. Represent indexer caches exclusively as LayerGroupSpec entries on the unified transfer path, without legacy indexer APIs or dedicated workers.
The four swa_* tensor params of layerwise_transfer are bound as
non-optional `const torch::Tensor&` with `torch::Tensor()` defaults,
which pybind renders as `= None`. Omitting them makes pybind try to
cast that None default back to a `torch.Tensor` param, which fails and
rejects the entire overload -- so even a byte-perfect 21-positional
call raises TypeError. The tensors cannot be defaulted; they must be
passed explicitly, exactly as production
LayerwiseTransfer._swa_transfer_kwargs() (layerwise.py:330) already
does. Pass four empty int64 tensors (swa ints keep their default 0,
which is a valid int -- only the Tensor() defaults are broken).

Fixes test_layerwise_h2d_notify_modes (4 cases); full
test_kv_transfer_correctness.py = 85 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ndefined

The `swa_h2d_src/swa_h2d_dst/swa_disk2h_src/swa_disk2h_dst` params of both
LayerwiseTransferGroup.layerwise_transfer and layerwise_transfer_multi_group
defaulted to `torch::Tensor()` (an *undefined* tensor). On torch 2.11/cu130
pybind bakes that into the Python signature as `= None`; when a caller omits
the args, pybind substitutes None and fails to cast it back to the
non-optional `torch.Tensor` param, so the ENTIRE overload is rejected with
`TypeError: incompatible function arguments` — even a byte-perfect all-required
positional call fails.

This is the same version-dependent pybind footgun that PR #222 fixed for the
LayerwiseTransferGroup *constructor*; that fix covered only the ctor, leaving
the two transfer methods broken. Default them to `torch::empty({0})` (a defined
empty tensor) so the omit path works. The C++ impl guards every SWA access with
`.numel() > 0` (layerwise.cpp:786/796/1077/1114), so the empty placeholder is
never dereferenced on the non-SWA path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The plain test_kvmanager cleaned up (shutdown_tp_client + kvmanager.shutdown)
only at the end of the function body, so an assertion failure or timeout in one
parametrized case skipped cleanup and leaked CUDA workers/contexts into the next
case — cascading into a full-file hang/timeout (ISSUE#4).

PR #222 hardened only the indexer path (_run_indexer_test); mirror it here:
- add the `request` fixture and register cleanup via request.addfinalizer so
  tp_client + kvmanager teardown runs even on failure/timeout;
- close the parent's copy of each child_conn after start (fd hygiene);
- bound the block-registration recv with INDEXER_STARTUP_TIMEOUT_S so a silently
  dead tp_client becomes a diagnostic TimeoutError instead of an indefinite hang.

Validated on 165: full-file run no longer cascades past failing cases, and the
per-process sweep is 54 pass / 30 fail with the 30 failures isolated entirely to
the SSD cache_config1 (H2DISK io_uring+O_DIRECT) environmental path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…chain

The dpskv4_refactor->main merge silently dropped feature commit 20c1fdd
("Add FlexKV transfer SEGV/D2H hang diagnostics across worker pipeline"):
merge flexkv/common/debug.py was byte-identical to origin/main (129 lines)
while the feature had 374, so the SGLang connector's
  from flexkv.common.debug import summarize_block_ids_from_slots
failed at runtime -> every offload logged "[FlexKV-SEGV-DEBUG] start_store_kv
stats failed ..." instead of "start_store_kv launch D2H ...", breaking the D2H
offload evidence chain (observed as baseline_d2h=0 in the accuracy harness even
though SWA D2H stores were happening).

Restore the debug helpers 20c1fdd added (summarize_block_ids_from_slots,
summarize_id_tensor, install_worker_crash_diagnostics, format_process_exit) and
20c1fdd's cache_engine.py D2H-op-creation diagnostics (which now import cleanly).
Verified on 165: `from flexkv.common.debug import summarize_block_ids_from_slots`
succeeds in a fresh interpreter and returns correct block stats.

Deliberately NOT re-grafted: 20c1fdd's transfer-worker internal SEGV logging in
worker.py/layerwise.py/transfer_engine.py. Those hunks conflict with main's
refactor (they reference `gi`/`_gpu_capacity_hint()` and a pre-nvcomp
launch_transfer structure that no longer exist) and are not required for the
connector D2H evidence chain. Left as a follow-up if the in-worker diagnostics
are wanted; the merged refactored transfer path is preserved intact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…to-main

# Conflicts:
#	flexkv/common/config.py
#	flexkv/common/transfer.py
#	flexkv/kvtask.py
#	flexkv/transfer/transfer_engine.py
#	flexkv/transfer/worker.py
#	tests/test_swa_state_sidecars.py
# Conflicts:
#	csrc/bindings.cpp
#	csrc/layerwise.cpp
#	flexkv/cache/cache_engine.py
#	flexkv/cache/swa_cache_engine.py
#	flexkv/common/config.py
#	flexkv/common/transfer.py
#	flexkv/kvtask.py
#	flexkv/storage/storage_engine.py
#	flexkv/transfer/host_buffer.py
#	flexkv/transfer/layerwise.py
#	flexkv/transfer/transfer_engine.py
#	flexkv/transfer/worker.py
#	tests/test_swa_state_sidecars.py
Comment thread flexkv/external/mooncake_store_utils.py Resolved
The multi-group transfer worker init methods recorded only the raw
data_ptr() of each imported CUDA-IPC KV tensor (via _get_layer_ptrs /
TPTransferThreadGroup) and let the torch tensors go out of scope. PyTorch
then released the underlying CUDA IPC mapping, leaving the stored pointers
dangling; the per-group transfer read/wrote freed device memory. This
surfaced as the DSA/NSA indexer group (the small, second layer group)
silently restoring zeros in test_kvmanager_with_indexer, while the large
main-KV group happened to survive GC.

Keep the imported tensors alive for the worker's lifetime via
self._multi_group_gpu_blocks_keepalive in all four multi-group init paths
(GPUCPU, tpGPUCPU, GDS, tpGDS), mirroring the retention the layerwise
worker already performs.

Verified: test_kvmanager_with_indexer[...cache_config0...] (cpu-only,
cr1/cr4) and test_kvmanager_with_indexer_layerwise (cr1/cr4) pass
deterministically across repeated runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread flexkv/cache/cache_engine.py Resolved
Comment thread flexkv/common/transfer.py Resolved
Comment thread flexkv/transfer/worker.py Resolved
@linhu-nv
linhu-nv self-requested a review July 24, 2026 01:06
@linhu-nv
linhu-nv merged commit 3ecd83f into main Jul 24, 2026
1 of 2 checks passed
zhuofan1123 added a commit to zhuofan1123/FlexKV that referenced this pull request Aug 5, 2026
…er layerwise

Merge taco-project#231 kept the swa_multi_group lineage (SWA gated via
_get_swa_multi_group_kwargs_tp*) and removed the _fuse_swa_into_layerwise
consumers from taco-project#225, but left the dead assignment referencing
CacheConfig.swa_multi_layer — a field that no longer exists. The variable
is never read, so the only effect was an AttributeError whenever
FLEXKV_ENABLE_LAYERWISE_TRANSFER=1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants