Skip to content

fix(sglang): publish SGLang's full Mooncake key prefix for the HiCache shared cache - #14952

Closed
joeltg wants to merge 1 commit into
ai-dynamo:mainfrom
joeltg:joeltg/sglang-mooncake-key-prefix
Closed

joeltg wants to merge 1 commit into
ai-dynamo:mainfrom
joeltg:joeltg/sglang-mooncake-key-prefix

Conversation

@joeltg

@joeltg joeltg commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Overview:

The router re-derives Mooncake object keys from the worker's sglang_hicache_mooncake runtime config, reimplementing SGLang's config_prefix. SGLang changed that derivation in 0.5.17 (sgl-project/sglang#31920) to prepend the served model name. Dynamo's copy was never updated. #13343 moved Dynamo from 0.5.16 to 0.5.17 on 2026-08-21, and main has been skewed since.

SGLang worker writes router derives
≤ 0.5.16 <hash>_<rank>_k <hash>_<rank>_k matches
≥ 0.5.17 <model>_<hash>_<rank>_k <hash>_<rank>_k never matches

Every lookup misses, with no error raised: router_shared_cache_hit_rate sits at zero and worker selection falls back to device-overlap-only scoring. --shared-cache-type hicache is inert on main. Releases on 0.5.16 or earlier are unaffected.

Details:

SGLang builds config_prefix from extra_backend_tag followed by the served model name with / replaced by -, joined by _ (mooncake_store.py:425-433). register.py published only the tag; _mooncake_config_prefix now publishes the full prefix.

  • No extra_backend_tag value works around this, because SGLang appends the model name after the tag.
  • The extra_backend_tag field keeps its name but now holds the full prefix. Renaming it, or adding a separate model_name field, would require a matching change to the Rust deserializer and a rollout window where workers and routers disagree about which field carries the prefix. has_same_layout already compares this field, so mismatched prefixes are still detected.
  • Two edge cases are replicated rather than normalized, since both derivations must agree exactly: a present-but-empty tag still contributes a part ({"extra_backend_tag": ""} gives _<model>), and a non-string tag is stringified — SGLang tests is not None and applies str(...).

args.py collapses --served-model-name to a single name and falls back to model_path; SGLang passes that same value as HiCacheStorageConfig.model_name (hiradix_cache.py:183), so the prefix is always populated on this path.

Tests cover the prefix cases and the published field. The helper was also differential-tested against SGLang v0.5.19's logic across tag and model-name combinations (absent, empty, non-string and None tags; /-bearing and bare model names; the model_path fallback) — all agree.

Draft because the author cannot run pytest locally: the new tests have been through that harness and the pinned pre-commit hooks only. The six W503 reports on register.py are pre-existing.

Where should the reviewer start?

_mooncake_config_prefix in components/src/dynamo/sglang/register.py, read against mooncake_store.py:425-433.

One question: should the field be renamed? extra_backend_tag now holds the full config_prefix rather than the configured tag. Renaming it means changing the SglangHicacheMooncakeConfig deserializer in shared_cache.rs and accepting both names for a release, since workers and routers upgrade independently. This PR keeps the name to stay within one file.

release/1.5.0 is on SGLang 0.5.18 with the old derivation, so 1.5.0 would ship with hicache shared-cache routing inert. Worth deciding whether this should be cherry-picked before the cut.

#13174 touches the same return dict in register.py; the conflict is textual, not semantic.

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

Summary by CodeRabbit

  • Bug Fixes

    • Improved Mooncake runtime metadata to consistently publish the complete configuration prefix, including model names and backend tags.
    • Preserved metadata for empty, null, and non-string backend tags instead of omitting them.
  • Tests

    • Added coverage for default model names, backend tags, missing or empty tags, non-string tags, and null values.

@copy-pr-bot

copy-pr-bot Bot commented Sep 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@joeltg
joeltg temporarily deployed to external_collaborator September 16, 2026 19:38 — with GitHub Actions Inactive
@joeltg
joeltg temporarily deployed to external_collaborator September 16, 2026 19:38 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi joeltg! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor backend::sglang Relates to the sglang backend labels Sep 16, 2026
@joeltg
joeltg force-pushed the joeltg/sglang-mooncake-key-prefix branch from d920766 to 97f1b2b Compare September 16, 2026 19:57
@joeltg
joeltg temporarily deployed to external_collaborator September 16, 2026 19:57 — with GitHub Actions Inactive
@joeltg
joeltg marked this pull request as ready for review September 16, 2026 20:03
@joeltg
joeltg requested a review from a team as a code owner September 16, 2026 20:03
…e shared cache

`--shared-cache-type hicache` makes the frontend re-derive Mooncake object
keys from the worker's `sglang_hicache_mooncake` runtime config and look them
up in the master's KV event index. SGLang's Mooncake backend tags every object
key and group id with `config_prefix` -- the configured `extra_backend_tag`,
then the served model name with `/` replaced by `-`, joined by `_` -- but
register.py published only the bare tag. `shared_cache.rs` (`maybe_prefix_key`,
`sglang_group_id`) therefore derived `<page_hash>_<rank>_k` while the master
announced `<model>_<page_hash>_<rank>_k`. The lookups never matched, so
`router_shared_cache_hit_rate` sat at zero and worker selection quietly fell
back to device-overlap-only scoring. No `extra_backend_tag` value fixes this
from the outside, because SGLang appends the model name after the tag.

SGLang added the model-name component in sgl-project/sglang#31920, first
released in v0.5.17. ai-dynamo#13343 moved Dynamo from 0.5.16 to 0.5.17 on 2026-08-21
without adapting the derivation, and main has skewed since.

Publish the prefix SGLang actually uses, replicating its construction exactly
-- including a present-but-empty tag contributing a leading separator and a
non-string tag being stringified. `has_same_layout` already compares this
field, so a fleet with mismatched prefixes is still detected.

Signed-off-by: Joel Gustafson <joel@reflection.ai>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@joeltg
joeltg force-pushed the joeltg/sglang-mooncake-key-prefix branch from 97f1b2b to 057a38d Compare September 16, 2026 20:07
@joeltg
joeltg temporarily deployed to external_collaborator September 16, 2026 20:07 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3c0f1572-2ecb-4980-89d9-02e27b13b0df

📥 Commits

Reviewing files that changed from the base of the PR and between 88c7770 and 057a38d.

📒 Files selected for processing (2)
  • components/src/dynamo/sglang/register.py
  • components/src/dynamo/sglang/tests/test_runtime_metadata.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The change adds Mooncake prefix derivation from the backend tag and served model name. Runtime metadata now publishes the complete prefix. Tests cover fallback names, tag variants, and metadata output.

Changes

Mooncake configuration prefix

Layer / File(s) Summary
Derive the Mooncake prefix
components/src/dynamo/sglang/register.py, components/src/dynamo/sglang/tests/test_runtime_metadata.py
Added _mooncake_config_prefix to combine the backend tag and served model name. Model-name slashes are replaced with hyphens. Tests cover missing, empty, non-string, and null tags.
Publish the derived metadata
components/src/dynamo/sglang/register.py, components/src/dynamo/sglang/tests/test_runtime_metadata.py
Mooncake runtime metadata now uses the complete computed prefix. Tests verify the published backend key prefix.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 057a3

The Mooncake metadata prefix remains compatible with worker-written shared-cache keys, including empty and non-string backend tags. No actionable merge risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely identifies the main change: publishing SGLang’s full Mooncake key prefix for HiCache shared-cache routing.
Description check ✅ Passed The description includes all required sections, explains the problem and implementation, identifies review files, and confirms that no related issue exists. It also documents compatibility, test cover…
  • Fix all pre-merge checks with AI

Comment @coderabbitai help to get the list of available commands.

@ishandhanani

Copy link
Copy Markdown
Contributor

Hi @joeltg - funny enough I was discussing this with someone else recently. See https://ai-dynamo.slack.com/archives/C0BKSDY8TC4/p1789453517934189?thread_ts=1789103071.576429&cid=C0BKSDY8TC4

--

TLDR - I think it makes sense for us to update sglang to give us the layout that would return the structure and then we leverage that instead of hardcoding things in our backend

@joeltg

joeltg commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

cool, makes sense! glad it's being tracked, will close this for now

@joeltg joeltg closed this Sep 17, 2026
littledgg added a commit to littledgg/dynamo that referenced this pull request Sep 21, 2026
SGLang expands Mooncake group ids from tagged keys, so they carry the
full config_prefix. The group verification fast-path in check_blocks
never matched when a layout advertised a key_prefix, since group ids
were still built from extra_backend_tag alone.

Inspired by ai-dynamo#14952 (the group-id path there is re-aligned by
construction, via overloading extra_backend_tag with the full
config_prefix).

Signed-off-by: littledgg <1658565283@qq.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 057a38d3 Deployed Sep 16, 2026 by joeltg via ok-to-test #18886
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend external-contribution Pull request is from an external contributor fix size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants