Skip to content

Let decode ranks contribute passive Mooncake store capacity - #40153

Closed
weireweire wants to merge 1 commit into
sgl-project:mainfrom
weireweire:perf/mooncake-decode-store-contributor
Closed

weireweire wants to merge 1 commit into
sgl-project:mainfrom
weireweire:perf/mooncake-decode-store-contributor

Conversation

@weireweire

@weireweire weireweire commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Only ranks that link to the Mooncake store mount a segment into it. In a
disaggregated deployment the decode ranks default to a chunk cache, so their
host memory never joins the pool no matter how free it is, and the shared
prefix cache is bounded by prefill-node DRAM alone. Under a long-context
agentic workload the store then evicts prefixes that the decode nodes had room
to hold.

Modifications

A disaggregated decode rank now lends host memory to the Mooncake store when it
is configured the same way a prefill rank sizes its segment:
MOONCAKE_GLOBAL_SEGMENT_SIZE explicitly set to a positive size, with
MOONCAKE_MASTER set. This mirrors vLLM's embedded store mode, where every
worker that has the store configured contributes global_segment_size and the
role only decides whether it reads or writes. The decode rank mounts a segment
but attaches no cache pool, so it neither reads nor writes KV.

The contributor is created in build_kv_cache and held by the scheduler for the
process lifetime, independent of the KV transfer backend (NIXL, MoRI or
Mooncake).

No capacity is lent when:

  • the size is unset or 0;
  • MOONCAKE_STANDALONE_STORAGE=1, where a local store service owns the node's capacity;
  • the decode rank runs the external cache linker, which already mounts its own segment;
  • MOONCAKE_MASTER is unset, in which case the size is ignored with a warning rather than failing startup.

Behavior change: deployments that already export
MOONCAKE_GLOBAL_SEGMENT_SIZE and MOONCAKE_MASTER to decode workers (for
example one environment shared by all roles) will now mount decode capacity. To
opt out, leave the size unset on decode, set it to 0, or use standalone mode.

Accuracy Tests

No change to generated output: the contributing rank attaches no cache pool and
performs no cache I/O, so the KV path is untouched.

Benchmarking and Profiling

DeepSeek-V4-Pro, NVFP4, on GB300, disaggregated with 8 prefill nodes (TP8/EP8
per worker) and 4 decode nodes (TP16/EP16), agentic coding trajectories at
concurrency 1920, MTP with a fixed acceptance length, one hour of measurement
per run.

store capacity (host DRAM) objects evicted steady-state prefix hit output tok/s total tok/s total tok/s/GPU
baseline 4.5 TB (32 ranks x 140 GB) 12.6M / 12.9M / 12.3M 92.7-92.9% 33,739 / 34,737 / 33,919 4,397,386 / 4,536,507 / 4,434,760 91,612 / 94,511 / 92,391
decode sized 6.7 TB (48 ranks x 140 GB) 10.9M / 10.9M / 10.5M 95.2% 44,612 / 44,724 / 44,671 5,908,589 / 5,928,177 / 5,947,669 123,096 / 123,504 / 123,910
mean +50% -15% +2.4 pts 34,132 -> 44,669 (+30.9%) 4,456,218 -> 5,928,145 (+33.0%) 92,838 -> 123,503

Three runs per arm. Run-to-run spread is 1.6% on the baseline and 0.3% with decode
capacity, well below the gap. Total tok/s counts cached prefill input, so it rises
faster than output tok/s: the extra capacity is exactly what stops those
prefixes from being evicted and recomputed.

The eviction count falls by less than the capacity grows because the store
evicts a fixed fraction of a now larger population each time it crosses its high
watermark. What changes is how long a prefix survives before its turn comes,
which is what the hit rate measures.

The two arms differ only in decode capacity and the branch under test; the image,
recipe, topology and acceptance length are identical.

Checklist


CI States

Latest PR Test (Base): ❌ Run #38532048324
Latest PR Test (Extra): ❌ Run #38532048250
Latest PR Test (AMD ROCm 10): ❌ Run #38532048308

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Review of 29ea9db: 2 inline comments (2 functional) and 1 PR-level finding below. Each comment states its severity and confidence.


[by Claude Code] Severity: style | Confidence: High

Issue

The GB300 DeepSeek-V4-Pro results do not have enough repro detail:

  • There are no prefill, decode, or router launch commands. The HiCache flags, the Mooncake env for each role and arm, and the DP attention setting are not stated. The "48 ranks x 140 GB" total holds only when attn_tp_size is 1, because MooncakeStore divides the segment size by tp_size * pp_size.
  • There is no benchmark command. "Agentic coding trajectories" names no dataset, and the request count or duration settings are missing.

Fix

Add the exact launch commands for both arms, and the benchmark command with its dataset.

) -> bool:
# Must match default_radix_cache_factory and create_unified_radix_cache:
# a chunk cache ignores store flags, and HiCache takes precedence over a linker.
if disable_radix_cache and retraction_backup != "host_pool":

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: functional | Confidence: High

Issue

This check misses decode KV offload. With --disaggregation-decode-enable-offload-kvcache, DecodeKVCacheOffloadManager builds a HiCacheController with --hicache-storage-backend, and that controller creates its own MooncakeStore and segment. Offload forces cpu_tensor retraction and does not enable HiCache, so this function returns False for both the chunk cache and the decode radix cache. build_kv_cache then mounts a second segment through the contributor.

Example

Decode with offload, --hicache-storage-backend mooncake, MOONCAKE_MASTER, and MOONCAKE_GLOBAL_SEGMENT_SIZE=140gb: each rank now runs two Mooncake clients and pins twice the host memory. On CPU, the two helpers return False and 140 GiB for this config.

Fix

When disaggregation_decode_enable_offload_kvcache is set, return hicache_storage_backend is not None. Add this case to test_decode_cache_owns_store_matches_tree_cache_selection.

)

parallel = get_parallel()
storage_config = HiCacheStorageConfig(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: functional | Confidence: Medium

Issue

This config leaves dp_rank at its default of 0. MooncakeStore.__init__ names each rank's SSD offload dir rank_{dp_rank}_{tp_rank}_{pp_rank}_{attn_cp_rank}. With DP attention, every decode rank also has attn_tp_rank=0, so all DP ranks on a node share one dir. The comment in MooncakeStore.__init__ says a shared dir silently truncates another rank's bucket files.

Example

Decode TP16 with DP attention, and MOONCAKE_ENABLE_SSD_OFFLOAD=1 plus MOONCAKE_OFFLOAD_FILE_STORAGE_PATH in the env shared with prefill: the 4 decode ranks on each node all use rank_0_0_0_0.

Fix

Pass dp_rank=parallel.attn_dp_rank, as HiCacheController._generate_storage_config does.

@nvpohanh

Copy link
Copy Markdown
Collaborator

@ShangmingCai Could you help to review this? Thanks!

) -> bool:
# Must match default_radix_cache_factory and create_unified_radix_cache:
# a chunk cache ignores store flags, and HiCache takes precedence over a linker.
if disable_radix_cache and retraction_backup != "host_pool":

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not true for the linker. On decode the chunk cache still goes through create_unified_radix_cache (the ctx.disable_radix_cache branch in registry.py), and the elif enable_unified_cache_external_linker there still attaches the linker, which creates its own MooncakeStore(storage_config, mem_pool=None) in MooncakeDirectLinker.__init__. So decode + mooncake linker + cpu_tensor retraction will mount two segments per rank.

cpu_tensor is also the default retraction for MLA/DSV4 decode since host_pool only supports MHA/SWA pools, so anyone who launches decode with the same linker flags as prefill will hit this.

Together with the decode offload case in the other comment, I think it's simpler and safer to just skip the contributor when decode has offload or the linker enabled, instead of mirroring the tree cache selection logic here, which is easy to get out of sync later.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! It was true when the code was written, but it became wrong after #43198 ~ #43201. Given this, enabling the external linker flag also makes decode contribute memory, so this PR only takes effect when the flag is off and the env is on. Not sure it's useful, let me just clean up the code as you said first.

)

parallel = get_parallel()
storage_config = HiCacheStorageConfig(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW MooncakeDirectLinker has the same gap with a TODO, but there it's because fixing it changes the persisted storage keys. The contributor never writes any key, so there is no compat concern here, just set it the same way as _generate_storage_config.

("disable_radix", "hierarchical", "retraction", "storage", "linker", "owns"),
[
# A chunk cache drops store flags copied from prefill; decode must still lend.
(True, False, "cpu_tensor", "mooncake", True, False),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case actually asserts the double mount, see my comment on _decode_cache_owns_store. Also --hicache-storage-backend + linker is rejected by handle_hicache, so "mooncake" with linker=True is not a valid combination anyway. Please make it (True, False, "cpu_tensor", None, True, True) and add the decode offload case as well.

A disaggregated decode rank keeps no store-backed cache, so its host
DRAM is idle while prefill ranks write KV into the Mooncake store.
When MOONCAKE_GLOBAL_SEGMENT_SIZE is set explicitly and MOONCAKE_MASTER
is available, create a MooncakeStore with no cache pool on each decode
rank so it mounts a segment of that size and serves as passive capacity.

The contributor is skipped when decode already mounts through its own
path: HiCache or host-pool retraction, or the unified-cache external
linker, which builds its own MooncakeStore. Standalone storage leaves
the capacity to the local store service. Its storage config follows the
one the HiCache cache controller builds.
@weireweire
weireweire force-pushed the perf/mooncake-decode-store-contributor branch from 29ea9db to ce1c6e7 Compare October 14, 2026 07:59
@weireweire

Copy link
Copy Markdown
Contributor Author

since the main already let decode node contribute memory when --enable-unified-cache-external-linker , --unified-cache-external-linker-backend mooncake and env is set. This PR's only effect is make it contribute when flag is not set and use a light version.

I feel that we can seen this as superseded by the main, so I'll close this. Feel free to open again if you think it's still needed, thanks. @ShangmingCai

@weireweire weireweire closed this Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants