Repository navigation
[HiCache] Add model-aware key isolation to Mooncake Store - #31920
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces model name and backend tag prefixing to isolate tenants and models sharing a single Mooncake store. It replaces extra_backend_tag with config_prefix, which combines the backend tag and the model name (with slashes replaced by dashes) to tag keys. A unit test was also added to verify key isolation between different models. The review feedback suggests improving the model name formatting logic by filtering out empty segments when splitting and joining the model name to avoid consecutive or trailing dashes.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if extra_config and extra_config.get("extra_backend_tag") is not None: | ||
| config_prefix_parts.append(str(extra_config["extra_backend_tag"])) | ||
| if storage_config is not None and storage_config.model_name: | ||
| model_name = "-".join(storage_config.model_name.split("/")) |
There was a problem hiding this comment.
To prevent issues with leading, trailing, or consecutive slashes in the model_name (which would result in empty segments and double or trailing dashes in the prefix), filter out empty strings when splitting and joining.
| model_name = "-".join(storage_config.model_name.split("/")) | |
| model_name = "-".join(p for p in storage_config.model_name.split("/") if p) |
|
/tag-and-rerun-ci |
…e shared cache `--shared-cache-type hicache` makes the frontend re-derive Mooncake object keys from the worker's `sglang_hicache_mooncake` runtime config and look them up in the master's KV event index. SGLang's Mooncake backend tags every object key and group id with `config_prefix` -- the configured `extra_backend_tag`, then the served model name with `/` replaced by `-`, joined by `_` -- but register.py published only the bare tag. `shared_cache.rs` (`maybe_prefix_key`, `sglang_group_id`) therefore derived `<page_hash>_<rank>_k` while the master announced `<model>_<page_hash>_<rank>_k`. The lookups never matched, so `router_shared_cache_hit_rate` sat at zero and worker selection quietly fell back to device-overlap-only scoring. No `extra_backend_tag` value fixes this from the outside, because SGLang appends the model name after the tag. SGLang added the model-name component in sgl-project/sglang#31920, first released in v0.5.17; the bump to 0.5.19 in ai-dynamo#14151 crossed that boundary without adapting the key derivation, so main currently skews. Publish the prefix SGLang actually uses, replicating its construction exactly -- including a present-but-empty tag contributing a leading separator and a non-string tag being stringified. `has_same_layout` already compares this field, so a fleet with mismatched prefixes is still detected. Signed-off-by: Joel Gustafson <joel@reflection.ai> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e shared cache `--shared-cache-type hicache` makes the frontend re-derive Mooncake object keys from the worker's `sglang_hicache_mooncake` runtime config and look them up in the master's KV event index. SGLang's Mooncake backend tags every object key and group id with `config_prefix` -- the configured `extra_backend_tag`, then the served model name with `/` replaced by `-`, joined by `_` -- but register.py published only the bare tag. `shared_cache.rs` (`maybe_prefix_key`, `sglang_group_id`) therefore derived `<page_hash>_<rank>_k` while the master announced `<model>_<page_hash>_<rank>_k`. The lookups never matched, so `router_shared_cache_hit_rate` sat at zero and worker selection quietly fell back to device-overlap-only scoring. No `extra_backend_tag` value fixes this from the outside, because SGLang appends the model name after the tag. SGLang added the model-name component in sgl-project/sglang#31920, first released in v0.5.17. ai-dynamo#13343 moved Dynamo from 0.5.16 to 0.5.17 on 2026-08-21 without adapting the derivation, and main has skewed since. Publish the prefix SGLang actually uses, replicating its construction exactly -- including a present-but-empty tag contributing a leading separator and a non-string tag being stringified. `has_same_layout` already compares this field, so a fleet with mismatched prefixes is still detected. Signed-off-by: Joel Gustafson <joel@reflection.ai> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Motivation
Modifications
Accuracy Tests
Speed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ✅ Run #29817867591
Latest PR Test (Extra): ❌ Run #29817867146