Repository navigation
Let decode ranks contribute passive Mooncake store capacity - #40153
weireweire wants to merge 1 commit into
Conversation
59daeb2 to
c19f59b
Compare
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] Review of 29ea9db: 2 inline comments (2 functional) and 1 PR-level finding below. Each comment states its severity and confidence.
[by Claude Code] Severity: style | Confidence: High
Issue
The GB300 DeepSeek-V4-Pro results do not have enough repro detail:
- There are no prefill, decode, or router launch commands. The HiCache flags, the Mooncake env for each role and arm, and the DP attention setting are not stated. The "48 ranks x 140 GB" total holds only when
attn_tp_sizeis 1, becauseMooncakeStoredivides the segment size bytp_size * pp_size. - There is no benchmark command. "Agentic coding trajectories" names no dataset, and the request count or duration settings are missing.
Fix
Add the exact launch commands for both arms, and the benchmark command with its dataset.
| ) -> bool: | ||
| # Must match default_radix_cache_factory and create_unified_radix_cache: | ||
| # a chunk cache ignores store flags, and HiCache takes precedence over a linker. | ||
| if disable_radix_cache and retraction_backup != "host_pool": |
There was a problem hiding this comment.
[by Claude Code] Severity: functional | Confidence: High
Issue
This check misses decode KV offload. With --disaggregation-decode-enable-offload-kvcache, DecodeKVCacheOffloadManager builds a HiCacheController with --hicache-storage-backend, and that controller creates its own MooncakeStore and segment. Offload forces cpu_tensor retraction and does not enable HiCache, so this function returns False for both the chunk cache and the decode radix cache. build_kv_cache then mounts a second segment through the contributor.
Example
Decode with offload, --hicache-storage-backend mooncake, MOONCAKE_MASTER, and MOONCAKE_GLOBAL_SEGMENT_SIZE=140gb: each rank now runs two Mooncake clients and pins twice the host memory. On CPU, the two helpers return False and 140 GiB for this config.
Fix
When disaggregation_decode_enable_offload_kvcache is set, return hicache_storage_backend is not None. Add this case to test_decode_cache_owns_store_matches_tree_cache_selection.
| ) | ||
|
|
||
| parallel = get_parallel() | ||
| storage_config = HiCacheStorageConfig( |
There was a problem hiding this comment.
[by Claude Code] Severity: functional | Confidence: Medium
Issue
This config leaves dp_rank at its default of 0. MooncakeStore.__init__ names each rank's SSD offload dir rank_{dp_rank}_{tp_rank}_{pp_rank}_{attn_cp_rank}. With DP attention, every decode rank also has attn_tp_rank=0, so all DP ranks on a node share one dir. The comment in MooncakeStore.__init__ says a shared dir silently truncates another rank's bucket files.
Example
Decode TP16 with DP attention, and MOONCAKE_ENABLE_SSD_OFFLOAD=1 plus MOONCAKE_OFFLOAD_FILE_STORAGE_PATH in the env shared with prefill: the 4 decode ranks on each node all use rank_0_0_0_0.
Fix
Pass dp_rank=parallel.attn_dp_rank, as HiCacheController._generate_storage_config does.
|
@ShangmingCai Could you help to review this? Thanks! |
| ) -> bool: | ||
| # Must match default_radix_cache_factory and create_unified_radix_cache: | ||
| # a chunk cache ignores store flags, and HiCache takes precedence over a linker. | ||
| if disable_radix_cache and retraction_backup != "host_pool": |
There was a problem hiding this comment.
This is not true for the linker. On decode the chunk cache still goes through create_unified_radix_cache (the ctx.disable_radix_cache branch in registry.py), and the elif enable_unified_cache_external_linker there still attaches the linker, which creates its own MooncakeStore(storage_config, mem_pool=None) in MooncakeDirectLinker.__init__. So decode + mooncake linker + cpu_tensor retraction will mount two segments per rank.
cpu_tensor is also the default retraction for MLA/DSV4 decode since host_pool only supports MHA/SWA pools, so anyone who launches decode with the same linker flags as prefill will hit this.
Together with the decode offload case in the other comment, I think it's simpler and safer to just skip the contributor when decode has offload or the linker enabled, instead of mirroring the tree cache selection logic here, which is easy to get out of sync later.
There was a problem hiding this comment.
Great catch! It was true when the code was written, but it became wrong after #43198 ~ #43201. Given this, enabling the external linker flag also makes decode contribute memory, so this PR only takes effect when the flag is off and the env is on. Not sure it's useful, let me just clean up the code as you said first.
| ) | ||
|
|
||
| parallel = get_parallel() | ||
| storage_config = HiCacheStorageConfig( |
There was a problem hiding this comment.
FWIW MooncakeDirectLinker has the same gap with a TODO, but there it's because fixing it changes the persisted storage keys. The contributor never writes any key, so there is no compat concern here, just set it the same way as _generate_storage_config.
| ("disable_radix", "hierarchical", "retraction", "storage", "linker", "owns"), | ||
| [ | ||
| # A chunk cache drops store flags copied from prefill; decode must still lend. | ||
| (True, False, "cpu_tensor", "mooncake", True, False), |
There was a problem hiding this comment.
This case actually asserts the double mount, see my comment on _decode_cache_owns_store. Also --hicache-storage-backend + linker is rejected by handle_hicache, so "mooncake" with linker=True is not a valid combination anyway. Please make it (True, False, "cpu_tensor", None, True, True) and add the decode offload case as well.
A disaggregated decode rank keeps no store-backed cache, so its host DRAM is idle while prefill ranks write KV into the Mooncake store. When MOONCAKE_GLOBAL_SEGMENT_SIZE is set explicitly and MOONCAKE_MASTER is available, create a MooncakeStore with no cache pool on each decode rank so it mounts a segment of that size and serves as passive capacity. The contributor is skipped when decode already mounts through its own path: HiCache or host-pool retraction, or the unified-cache external linker, which builds its own MooncakeStore. Standalone storage leaves the capacity to the local store service. Its storage config follows the one the HiCache cache controller builds.
29ea9db to
ce1c6e7
Compare
|
since the main already let decode node contribute memory when I feel that we can seen this as superseded by the main, so I'll close this. Feel free to open again if you think it's still needed, thanks. @ShangmingCai |
Motivation
Only ranks that link to the Mooncake store mount a segment into it. In a
disaggregated deployment the decode ranks default to a chunk cache, so their
host memory never joins the pool no matter how free it is, and the shared
prefix cache is bounded by prefill-node DRAM alone. Under a long-context
agentic workload the store then evicts prefixes that the decode nodes had room
to hold.
Modifications
A disaggregated decode rank now lends host memory to the Mooncake store when it
is configured the same way a prefill rank sizes its segment:
MOONCAKE_GLOBAL_SEGMENT_SIZEexplicitly set to a positive size, withMOONCAKE_MASTERset. This mirrors vLLM's embedded store mode, where everyworker that has the store configured contributes
global_segment_sizeand therole only decides whether it reads or writes. The decode rank mounts a segment
but attaches no cache pool, so it neither reads nor writes KV.
The contributor is created in
build_kv_cacheand held by the scheduler for theprocess lifetime, independent of the KV transfer backend (NIXL, MoRI or
Mooncake).
No capacity is lent when:
MOONCAKE_STANDALONE_STORAGE=1, where a local store service owns the node's capacity;MOONCAKE_MASTERis unset, in which case the size is ignored with a warning rather than failing startup.Behavior change: deployments that already export
MOONCAKE_GLOBAL_SEGMENT_SIZEandMOONCAKE_MASTERto decode workers (forexample one environment shared by all roles) will now mount decode capacity. To
opt out, leave the size unset on decode, set it to 0, or use standalone mode.
Accuracy Tests
No change to generated output: the contributing rank attaches no cache pool and
performs no cache I/O, so the KV path is untouched.
Benchmarking and Profiling
DeepSeek-V4-Pro, NVFP4, on GB300, disaggregated with 8 prefill nodes (TP8/EP8
per worker) and 4 decode nodes (TP16/EP16), agentic coding trajectories at
concurrency 1920, MTP with a fixed acceptance length, one hour of measurement
per run.
Three runs per arm. Run-to-run spread is 1.6% on the baseline and 0.3% with decode
capacity, well below the gap. Total tok/s counts cached prefill input, so it rises
faster than output tok/s: the extra capacity is exactly what stops those
prefixes from being evicted and recomputed.
The eviction count falls by less than the capacity grows because the store
evicts a fixed fraction of a now larger population each time it crosses its high
watermark. What changes is how long a prefix survives before its turn comes,
which is what the hit rate measures.
The two arms differ only in decode capacity and the branch under test; the image,
recipe, topology and acceptance length are identical.
Checklist
CI States
Latest PR Test (Base): ❌ Run #38532048324
Latest PR Test (Extra): ❌ Run #38532048250
Latest PR Test (AMD ROCm 10): ❌ Run #38532048308