Repository navigation
Conversation
|
👋 Hi joeltg! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
d920766 to
97f1b2b
Compare
…e shared cache `--shared-cache-type hicache` makes the frontend re-derive Mooncake object keys from the worker's `sglang_hicache_mooncake` runtime config and look them up in the master's KV event index. SGLang's Mooncake backend tags every object key and group id with `config_prefix` -- the configured `extra_backend_tag`, then the served model name with `/` replaced by `-`, joined by `_` -- but register.py published only the bare tag. `shared_cache.rs` (`maybe_prefix_key`, `sglang_group_id`) therefore derived `<page_hash>_<rank>_k` while the master announced `<model>_<page_hash>_<rank>_k`. The lookups never matched, so `router_shared_cache_hit_rate` sat at zero and worker selection quietly fell back to device-overlap-only scoring. No `extra_backend_tag` value fixes this from the outside, because SGLang appends the model name after the tag. SGLang added the model-name component in sgl-project/sglang#31920, first released in v0.5.17. ai-dynamo#13343 moved Dynamo from 0.5.16 to 0.5.17 on 2026-08-21 without adapting the derivation, and main has skewed since. Publish the prefix SGLang actually uses, replicating its construction exactly -- including a present-but-empty tag contributing a leading separator and a non-string tag being stringified. `has_same_layout` already compares this field, so a fleet with mismatched prefixes is still detected. Signed-off-by: Joel Gustafson <joel@reflection.ai> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
97f1b2b to
057a38d
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. WalkthroughThe change adds Mooncake prefix derivation from the backend tag and served model name. Runtime metadata now publishes the complete prefix. Tests cover fallback names, tag variants, and metadata output. ChangesMooncake configuration prefix
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to The Mooncake metadata prefix remains compatible with worker-written shared-cache keys, including empty and non-string backend tags. No actionable merge risk remains. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
|
Hi @joeltg - funny enough I was discussing this with someone else recently. See https://ai-dynamo.slack.com/archives/C0BKSDY8TC4/p1789453517934189?thread_ts=1789103071.576429&cid=C0BKSDY8TC4 -- TLDR - I think it makes sense for us to update sglang to give us the layout that would return the structure and then we leverage that instead of hardcoding things in our backend |
|
cool, makes sense! glad it's being tracked, will close this for now |
SGLang expands Mooncake group ids from tagged keys, so they carry the full config_prefix. The group verification fast-path in check_blocks never matched when a layout advertised a key_prefix, since group ids were still built from extra_backend_tag alone. Inspired by ai-dynamo#14952 (the group-id path there is re-aligned by construction, via overloading extra_backend_tag with the full config_prefix). Signed-off-by: littledgg <1658565283@qq.com>
Overview:
The router re-derives Mooncake object keys from the worker's
sglang_hicache_mooncakeruntime config, reimplementing SGLang'sconfig_prefix. SGLang changed that derivation in 0.5.17 (sgl-project/sglang#31920) to prepend the served model name. Dynamo's copy was never updated. #13343 moved Dynamo from 0.5.16 to 0.5.17 on 2026-08-21, andmainhas been skewed since.<hash>_<rank>_k<hash>_<rank>_k<model>_<hash>_<rank>_k<hash>_<rank>_kEvery lookup misses, with no error raised:
router_shared_cache_hit_ratesits at zero and worker selection falls back to device-overlap-only scoring.--shared-cache-type hicacheis inert onmain. Releases on 0.5.16 or earlier are unaffected.Details:
SGLang builds
config_prefixfromextra_backend_tagfollowed by the served model name with/replaced by-, joined by_(mooncake_store.py:425-433).register.pypublished only the tag;_mooncake_config_prefixnow publishes the full prefix.extra_backend_tagvalue works around this, because SGLang appends the model name after the tag.extra_backend_tagfield keeps its name but now holds the full prefix. Renaming it, or adding a separatemodel_namefield, would require a matching change to the Rust deserializer and a rollout window where workers and routers disagree about which field carries the prefix.has_same_layoutalready compares this field, so mismatched prefixes are still detected.{"extra_backend_tag": ""}gives_<model>), and a non-string tag is stringified — SGLang testsis not Noneand appliesstr(...).args.pycollapses--served-model-nameto a single name and falls back tomodel_path; SGLang passes that same value asHiCacheStorageConfig.model_name(hiradix_cache.py:183), so the prefix is always populated on this path.Tests cover the prefix cases and the published field. The helper was also differential-tested against SGLang v0.5.19's logic across tag and model-name combinations (absent, empty, non-string and
Nonetags;/-bearing and bare model names; themodel_pathfallback) — all agree.Draft because the author cannot run
pytestlocally: the new tests have been through that harness and the pinned pre-commit hooks only. The sixW503reports onregister.pyare pre-existing.Where should the reviewer start?
_mooncake_config_prefixincomponents/src/dynamo/sglang/register.py, read againstmooncake_store.py:425-433.One question: should the field be renamed?
extra_backend_tagnow holds the fullconfig_prefixrather than the configured tag. Renaming it means changing theSglangHicacheMooncakeConfigdeserializer inshared_cache.rsand accepting both names for a release, since workers and routers upgrade independently. This PR keeps the name to stay within one file.release/1.5.0is on SGLang 0.5.18 with the old derivation, so 1.5.0 would ship with hicache shared-cache routing inert. Worth deciding whether this should be cherry-picked before the cut.#13174 touches the same return dict in
register.py; the conflict is textual, not semantic.Related Issues
🚫 This PR is NOT linked to an issue:
Summary by CodeRabbit
Bug Fixes
Tests