Skip to content

[HiCache] Add model-aware key isolation to Mooncake Store - #31920

Merged
huangtingwei9988 merged 2 commits into
sgl-project:mainfrom
antgroup:mooncake_model_isolation
Jul 23, 2026
Merged

huangtingwei9988 merged 2 commits into
sgl-project:mainfrom
antgroup:mooncake_model_isolation

Conversation

@huangtingwei9988

@huangtingwei9988 huangtingwei9988 commented Jul 21, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ✅ Run #29817867591
Latest PR Test (Extra): ❌ Run #29817867146

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces model name and backend tag prefixing to isolate tenants and models sharing a single Mooncake store. It replaces extra_backend_tag with config_prefix, which combines the backend tag and the model name (with slashes replaced by dashes) to tag keys. A unit test was also added to verify key isolation between different models. The review feedback suggests improving the model name formatting logic by filtering out empty segments when splitting and joining the model name to avoid consecutive or trailing dashes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

if extra_config and extra_config.get("extra_backend_tag") is not None:
config_prefix_parts.append(str(extra_config["extra_backend_tag"]))
if storage_config is not None and storage_config.model_name:
model_name = "-".join(storage_config.model_name.split("/"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent issues with leading, trailing, or consecutive slashes in the model_name (which would result in empty segments and double or trailing dashes in the prefix), filter out empty strings when splitting and joining.

Suggested change
model_name = "-".join(storage_config.model_name.split("/"))
model_name = "-".join(p for p in storage_config.model_name.split("/") if p)

@huangtingwei9988

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Jul 21, 2026
@huangtingwei9988
huangtingwei9988 merged commit 86e1bb5 into sgl-project:main Jul 23, 2026
235 of 268 checks passed
Zhylkaaa pushed a commit to Zhylkaaa/sglang that referenced this pull request Jul 29, 2026
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
joeltg added a commit to joeltg/dynamo that referenced this pull request Sep 16, 2026
…e shared cache

`--shared-cache-type hicache` makes the frontend re-derive Mooncake object
keys from the worker's `sglang_hicache_mooncake` runtime config and look them
up in the master's KV event index. SGLang's Mooncake backend tags every object
key and group id with `config_prefix` -- the configured `extra_backend_tag`,
then the served model name with `/` replaced by `-`, joined by `_` -- but
register.py published only the bare tag. `shared_cache.rs` (`maybe_prefix_key`,
`sglang_group_id`) therefore derived `<page_hash>_<rank>_k` while the master
announced `<model>_<page_hash>_<rank>_k`. The lookups never matched, so
`router_shared_cache_hit_rate` sat at zero and worker selection quietly fell
back to device-overlap-only scoring. No `extra_backend_tag` value fixes this
from the outside, because SGLang appends the model name after the tag.

SGLang added the model-name component in sgl-project/sglang#31920, first
released in v0.5.17; the bump to 0.5.19 in ai-dynamo#14151 crossed that boundary
without adapting the key derivation, so main currently skews.

Publish the prefix SGLang actually uses, replicating its construction exactly
-- including a present-but-empty tag contributing a leading separator and a
non-string tag being stringified. `has_same_layout` already compares this
field, so a fleet with mismatched prefixes is still detected.

Signed-off-by: Joel Gustafson <joel@reflection.ai>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
joeltg added a commit to joeltg/dynamo that referenced this pull request Sep 16, 2026
…e shared cache

`--shared-cache-type hicache` makes the frontend re-derive Mooncake object
keys from the worker's `sglang_hicache_mooncake` runtime config and look them
up in the master's KV event index. SGLang's Mooncake backend tags every object
key and group id with `config_prefix` -- the configured `extra_backend_tag`,
then the served model name with `/` replaced by `-`, joined by `_` -- but
register.py published only the bare tag. `shared_cache.rs` (`maybe_prefix_key`,
`sglang_group_id`) therefore derived `<page_hash>_<rank>_k` while the master
announced `<model>_<page_hash>_<rank>_k`. The lookups never matched, so
`router_shared_cache_hit_rate` sat at zero and worker selection quietly fell
back to device-overlap-only scoring. No `extra_backend_tag` value fixes this
from the outside, because SGLang appends the model name after the tag.

SGLang added the model-name component in sgl-project/sglang#31920, first
released in v0.5.17. ai-dynamo#13343 moved Dynamo from 0.5.16 to 0.5.17 on 2026-08-21
without adapting the derivation, and main has skewed since.

Publish the prefix SGLang actually uses, replicating its construction exactly
-- including a present-but-empty tag contributing a leading separator and a
non-string tag being stringified. `has_same_layout` already compares this
field, so a fleet with mismatched prefixes is still detected.

Signed-off-by: Joel Gustafson <joel@reflection.ai>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants