Repository navigation
Merge/dpsk4 mooncake store - #231
Merged
Merged
Conversation
Squash the final DeepSeek-V4 multi-group and SWA changes onto the latest origin/main. Represent indexer caches exclusively as LayerGroupSpec entries on the unified transfer path, without legacy indexer APIs or dedicated workers.
The four swa_* tensor params of layerwise_transfer are bound as non-optional `const torch::Tensor&` with `torch::Tensor()` defaults, which pybind renders as `= None`. Omitting them makes pybind try to cast that None default back to a `torch.Tensor` param, which fails and rejects the entire overload -- so even a byte-perfect 21-positional call raises TypeError. The tensors cannot be defaulted; they must be passed explicitly, exactly as production LayerwiseTransfer._swa_transfer_kwargs() (layerwise.py:330) already does. Pass four empty int64 tensors (swa ints keep their default 0, which is a valid int -- only the Tensor() defaults are broken). Fixes test_layerwise_h2d_notify_modes (4 cases); full test_kv_transfer_correctness.py = 85 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ndefined The `swa_h2d_src/swa_h2d_dst/swa_disk2h_src/swa_disk2h_dst` params of both LayerwiseTransferGroup.layerwise_transfer and layerwise_transfer_multi_group defaulted to `torch::Tensor()` (an *undefined* tensor). On torch 2.11/cu130 pybind bakes that into the Python signature as `= None`; when a caller omits the args, pybind substitutes None and fails to cast it back to the non-optional `torch.Tensor` param, so the ENTIRE overload is rejected with `TypeError: incompatible function arguments` — even a byte-perfect all-required positional call fails. This is the same version-dependent pybind footgun that PR #222 fixed for the LayerwiseTransferGroup *constructor*; that fix covered only the ctor, leaving the two transfer methods broken. Default them to `torch::empty({0})` (a defined empty tensor) so the omit path works. The C++ impl guards every SWA access with `.numel() > 0` (layerwise.cpp:786/796/1077/1114), so the empty placeholder is never dereferenced on the non-SWA path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The plain test_kvmanager cleaned up (shutdown_tp_client + kvmanager.shutdown) only at the end of the function body, so an assertion failure or timeout in one parametrized case skipped cleanup and leaked CUDA workers/contexts into the next case — cascading into a full-file hang/timeout (ISSUE#4). PR #222 hardened only the indexer path (_run_indexer_test); mirror it here: - add the `request` fixture and register cleanup via request.addfinalizer so tp_client + kvmanager teardown runs even on failure/timeout; - close the parent's copy of each child_conn after start (fd hygiene); - bound the block-registration recv with INDEXER_STARTUP_TIMEOUT_S so a silently dead tp_client becomes a diagnostic TimeoutError instead of an indefinite hang. Validated on 165: full-file run no longer cascades past failing cases, and the per-process sweep is 54 pass / 30 fail with the 30 failures isolated entirely to the SSD cache_config1 (H2DISK io_uring+O_DIRECT) environmental path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…chain The dpskv4_refactor->main merge silently dropped feature commit 20c1fdd ("Add FlexKV transfer SEGV/D2H hang diagnostics across worker pipeline"): merge flexkv/common/debug.py was byte-identical to origin/main (129 lines) while the feature had 374, so the SGLang connector's from flexkv.common.debug import summarize_block_ids_from_slots failed at runtime -> every offload logged "[FlexKV-SEGV-DEBUG] start_store_kv stats failed ..." instead of "start_store_kv launch D2H ...", breaking the D2H offload evidence chain (observed as baseline_d2h=0 in the accuracy harness even though SWA D2H stores were happening). Restore the debug helpers 20c1fdd added (summarize_block_ids_from_slots, summarize_id_tensor, install_worker_crash_diagnostics, format_process_exit) and 20c1fdd's cache_engine.py D2H-op-creation diagnostics (which now import cleanly). Verified on 165: `from flexkv.common.debug import summarize_block_ids_from_slots` succeeds in a fresh interpreter and returns correct block stats. Deliberately NOT re-grafted: 20c1fdd's transfer-worker internal SEGV logging in worker.py/layerwise.py/transfer_engine.py. Those hunks conflict with main's refactor (they reference `gi`/`_gpu_capacity_hint()` and a pre-nvcomp launch_transfer structure that no longer exist) and are not required for the connector D2H evidence chain. Left as a follow-up if the in-worker diagnostics are wanted; the merged refactored transfer path is preserved intact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t/FlexKV into merge/dpskv4-to-main
…com/taco-project/FlexKV into agent/dsv4-flexkv-compress-state-io
…to-main # Conflicts: # flexkv/common/config.py # flexkv/common/transfer.py # flexkv/kvtask.py # flexkv/transfer/transfer_engine.py # flexkv/transfer/worker.py # tests/test_swa_state_sidecars.py
…t/FlexKV into merge/dpskv4-to-main
…'swa_multi_layer' from config.
# Conflicts: # csrc/bindings.cpp # csrc/layerwise.cpp # flexkv/cache/cache_engine.py # flexkv/cache/swa_cache_engine.py # flexkv/common/config.py # flexkv/common/transfer.py # flexkv/kvtask.py # flexkv/storage/storage_engine.py # flexkv/transfer/host_buffer.py # flexkv/transfer/layerwise.py # flexkv/transfer/transfer_engine.py # flexkv/transfer/worker.py # tests/test_swa_state_sidecars.py
linhu-nv
reviewed
Jul 23, 2026
The multi-group transfer worker init methods recorded only the raw data_ptr() of each imported CUDA-IPC KV tensor (via _get_layer_ptrs / TPTransferThreadGroup) and let the torch tensors go out of scope. PyTorch then released the underlying CUDA IPC mapping, leaving the stored pointers dangling; the per-group transfer read/wrote freed device memory. This surfaced as the DSA/NSA indexer group (the small, second layer group) silently restoring zeros in test_kvmanager_with_indexer, while the large main-KV group happened to survive GC. Keep the imported tensors alive for the worker's lifetime via self._multi_group_gpu_blocks_keepalive in all four multi-group init paths (GPUCPU, tpGPUCPU, GDS, tpGDS), mirroring the retention the layerwise worker already performs. Verified: test_kvmanager_with_indexer[...cache_config0...] (cpu-only, cr1/cr4) and test_kvmanager_with_indexer_layerwise (cr1/cr4) pass deterministically across repeated runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
linhu-nv
reviewed
Jul 23, 2026
linhu-nv
reviewed
Jul 23, 2026
linhu-nv
reviewed
Jul 23, 2026
…fer partial complete.
…project/FlexKV into merge/dpsk4-mooncake-store
linhu-nv
self-requested a review
July 24, 2026 01:06
linhu-nv
approved these changes
Jul 24, 2026
zhuofan1123
added a commit
to zhuofan1123/FlexKV
that referenced
this pull request
Aug 5, 2026
…er layerwise Merge taco-project#231 kept the swa_multi_group lineage (SWA gated via _get_swa_multi_group_kwargs_tp*) and removed the _fuse_swa_into_layerwise consumers from taco-project#225, but left the dead assignment referencing CacheConfig.swa_multi_layer — a field that no longer exists. The variable is never read, so the only effect was an AttributeError whenever FLEXKV_ENABLE_LAYERWISE_TRANSFER=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR integrates mooncake-store as a distributed KV cache backend for FlexKV, wired in as the
REMOTEtier alongside the existing PCFS / P2P remote paths. Unlike the file-addressed 3rd-party remote (block-id → file offset), mooncake-store is key-addressed: each KV / SWA block is keyed by its content hash, so the cluster pool is prefix-shareable across all FlexKV instances without a coordinated block-id allocator.The integration is fully opt-in via
FLEXKV_USE_MOONCAKE_STORE_BACKEND=1(orUserConfig.use_mooncake_store_backend=True) and is orthogonal to the existing GPU / CPU / SSD tiers, layerwise transfer, EAGLE, DP-attention, and DSv4 SWA/state sidecars. When disabled the code paths remain unchanged.Key design points:
<block_hash>_FlexKV[_pp_rank_i_of_N]for main KV,<tail_hash>_FlexKV_swafor SWA. PP-rank suffix is elided when the local node covers the full model layer range.H2REMOTE/REMOTE2HTransferOptypes; only the payload carriesmooncake_store_block_hashes(main) ormooncake_store_swa_block_hashes(SWA) so the batch-merge and graph-scheduler stay backend-agnostic.batch_is_exist(kv_keys + swa_keys)per sequence returns both the KV longest-prefix and the SWA joint hit, avoiding two round-trips.Main Changes
1. New adapter module —
flexkv/external/mooncake_store_utils.pyMooncakeStoreConfig(JSON loader),MooncakeStoreClient(thinMooncakeDistributedStorewrapper withbatch_put/batch_get/batch_exists/register_buffer), andMooncakeStoreCacheEngine(drop-in replacement forCacheEngineAccelpresentingmatch()/insert())mooncake_store_keys.pyPoolKind.KV/PoolKind.SWA+build_key) shared by client, cache engine, and transfer worker__init__.py2.
MooncakeStoreTransferWorker— new remote I/O workerflexkv/transfer/worker.pyH2REMOTE/REMOTE2HopsPoolKind.KV) and SWA pool (PoolKind.SWA, pure-client mode —override_global_segment_size=0— so SWA workers do not contribute their own storage segment)_preprocess_kv/_preprocess_swacompute(cpu_ptrs, sizes, keys)from the per-op block-id → CPU-slot mapping and the sequence hashes carried on the op3. Cache-engine wiring —
flexkv/cache/cache_engine.pyremote_cache_engineis instantiated asMooncakeStoreCacheEnginewhen the flag is on (else keeps the existingHierarchyLRCacheEngine/ PCFS paths)get_match/put_matchnow propagate the sequence hashes ontoop_remote2h.mooncake_store_block_hashes/op_h2remote.mooncake_store_block_hashes, usingremote_matched_result.kv_matched_blocks(added toMatchResultAccel) so PUT skips keys that already exist in the store_select_swa_read_source→_reserve_swa_read, PUT-side SWA chain input_match) gains a mooncake branch: the tail hash of the SWA-covered prefix becomes the SWA key, andSWAReadSourcecarriesmooncake_tail_hashin place of a slot / node handle (is_mooncakeshort-circuits the pin / unlock logic which does not apply on a key-addressed tier)4.
TransferOp/WorkerTransferOp— hash fieldsflexkv/common/transfer.py,flexkv/transfer/worker_op.pyTransferOp:mooncake_store_block_hashes(np.ndarray, main KV) andmooncake_store_swa_block_hashes(List[str], SWA tail hashes)WorkerTransferOpforwards both to the worker process; when the fields are set, the op-buffer fast path is bypassed so block ids stay accessible5. Batch merge — main and SWA lanes split
flexkv/common/transfer.py::merge_to_batch_graph_merge_ops(main-KV) +_merge_swa_ops(SWA), each concatenating their respective hash lists in-order and rejecting cross-mixingsupported_typesextended withH2REMOTE/REMOTE2H; batch sink logic factored into_add_batch_sink(0/1/N terminals) and_pick_dp_client_id6. Storage engine — no separate remote region when mooncake is on
flexkv/storage/storage_engine.pyuse_mooncake_store_backendis true, the hot CPU / SWA CPU buffers are registered with mooncake directly; the previous "allocate a separate REMOTE contiguous layout" path is skipped for both main and SWA pools7. Transfer engine — worker map wiring
flexkv/transfer/transfer_engine.pyMooncakeStoreTransferWorkerfor main-KVH2REMOTE/REMOTE2H(mutually exclusive with the PCFS remote path)_swa_worker_map[H2REMOTE/REMOTE2H])8. Config — new user-facing options
flexkv/common/config.pyUserConfig/CacheConfig:use_mooncake_store_backend,mooncake_store_config_path, plus five layer-range / PP fields (mooncake_store_pp_rank,mooncake_store_pp_size,mooncake_store_node_layer_start,mooncake_store_node_layer_end,mooncake_store_total_layers) auto-populated fromrank_infoFLEXKV_USE_MOONCAKE_STORE_BACKEND/FLEXKV_MOONCAKE_STORE_CONFIG_PATHenable_remote = enable_3rd_remote OR use_mooncake_store_backend; the "remote requires SSD" precondition is relaxed for mooncake (mooncake bypasses SSD)Validation
I have tested the accuracy of DPSK-v4-Flash under TP8 and DP8 configurations. The specific methodology involved running the test dataset once on instance 1 and again on instance 2, then observing the evaluation accuracy for both runs. The results are as follows (each dataset was tested twice):
tp8:
dp8: