Repository navigation
[Fix] [AMD] hicache: Honor UMBP anchor key namespaces - #49882
Open
matthewygf wants to merge 4 commits into
Open
matthewygf wants to merge 4 commits into
matthewygf wants to merge 4 commits into
Conversation
matthewygf
force-pushed
the
fix/umbp-anchor-namespace-20261111
branch
from
November 11, 2026 16:17
a6e4e12 to
9a78f33
Compare
matthewygf
marked this pull request as ready for review
November 11, 2026 16:19
matthewygf
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
ispobock,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
November 11, 2026 16:19
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A multi-tenant platform can serve customer fine-tunes of the same base model as
tenant-a-supportandtenant-b-support, with separate workers sharing along-lived UMBP cache across restarts. Common system/tool prefixes can produce
identical token-page hashes and equal-sized KV objects. UMBP omits the configured
model/backend prefix from primary KV keys, allowing a fresh B worker to restore
A's bytes despite their different served-model names.
For page hash
hat TP1/PP1, both models previously addressedh_0_k. With thefix and no extra backend tag, they address
tenant-a-support_h_0_kandtenant-b-support_h_0_k; V follows the same rule. Same-model replicas retainsharing. Isolation requires distinct resolved prefixes; existing model-name normalization
and backend-tag joining are unchanged.
Modifications
Apply existing
config_prefixin_anchor_keys(): +10/−1 production lines.Reads, writes and existence checks use that helper. Reject complete anchor keys
above 255 UTF-8 bytes before I/O so MoRI's SPDK proxy cannot truncate the
distinguishing rank/component suffix. Rank/component suffixes and pointer order are unchanged. Add six CPU unit tests and update two existing physical-key expectations.
Configured instances deliberately miss legacy unprefixed objects and recompute;
valid empty-prefix keys retain their existing format.
Related: #31920 added
model-aware key isolation to Mooncake Store. This change applies UMBP's existing
prefix to its anchor keys.
Accuracy Tests
Native correctness and latency comparisons used:
b174f4bb4fe0; after07d36b1bc8f5, mounted read-only.lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031,verified against the tested digest.
CPU/native byte-level cache correctness is measured. Native runs
use real CPU host pools and MoRI standalone DRAM, with one model-only and one
backend-tag-only separation case. Each synthetic page is 32 bytes:
A's K/V are
[100..103]/[120..123];B's are
[200..203]/[220..223]. Fresh processes perform seed, probe/write andindependent reads.
Six CPU unit tests cover model/tag isolation, legacy keys, depth-aware writes,
component ordering and key-length rejection. The added boundary tests exercise
186/187-byte prefixes with ASCII and multibyte text, the longer envelope suffix,
and rejection before any client call for a mixed valid/invalid anchor batch.
The two boundary methods fail on the previous PR revision.
170 passed initially; 15 passed after correcting the test environment
(memory budget and the existing
parameterizedtest dependency). Those 15 also pass on theprevious PR revision. GPU-only cases in that CPU suite remain skipped;
GPU serving checks are reported below.
Run the CPU unit tests
In a CPU-capable SGLang environment, from the candidate repository root:
The tests use real CPU KV buffers with a substituted MoRI client. They require
no GPU, model weights or standalone server. The native results above come from
a separate run with the real MoRI backend. To verify the failure before the fix,
run this same test file with the previous revision's
pythondirectory onPYTHONPATH.Test runtime: Python 3.12.3, Torch 2.11.0+rocm10.0.0 and a read-only
blake3==1.0.11package mount satisfying SGLang's existing dependency.This PR adds no dependency. The tests are registered in
base-a-test-cpuCI.GPU accuracy and cache checks (2026-11-12)
Ran on AMD Instinct MI355X**, using Qwen/Qwen3-8B BF16, TP1. Both revisions use Triton
attention, decode graphs, 64-token cache pages, a 4 GB host cache, and a fresh
native MoRI standalone DRAM service with 8 GiB capacity; SSD and hugepages are
disabled. The local runtime image ID is
sha256:e7a2b52f2fcfe9b0275ffe6bee00f423d576fecd428980f6511860e51e9b4ab6,with Python 3.12.3 and Torch
2.11.0+rocm10.0.0; native packages are unchanged.Ran the contribution-guide GSM8K sanity check with
sgl-eval0.1.3 on the samefrozen first 200 official test examples: temperature 0, seed 0, 16 threads,
2,048 maximum output tokens, and Qwen thinking disabled.
Independent replay verified all 200 IDs, labels, extracted answers and scores
per revision.
Native GPU cache checks restore 960 tokens after flushing L1/L2 and after a
same-namespace worker restart on both revisions. A fresh worker with a different
served-model namespace incorrectly restores 960 tokens before the fix and
correctly misses after it. Three natural-language fixtures (1,153 input tokens
each) return the correct answers and identical greedy token IDs across cold,
device-cache and UMBP-restored states on both revisions: 9/9 responses per revision.
Speed Tests and Profiling
This is a correctness fix with measurable key-processing cost. CPU/native
storage-call latency and GPU serving results are reported below. Timing compares
valid same-namespace warm reads and unique writes. A parent's incorrect
cross-model cache hit is not a valid speed baseline for the candidate's correct
miss/recomputation.
Measured on a shared host in the same LMSYS container:
All 5,600 raw samples/blocks and source/native hashes were independently audited.
Byte/canary checks are outside timing. The helper is measured separately in 50
blocks of 200 calls after ten warmup blocks.
GPU serving benchmark
Using the GPU/model/UMBP setup above, followed the contribution-guide
sglang.benchmark.servingworkload: 80 requests, concurrency 16, exactly256 input / 32 output tokens, random dataset backed by a pinned ShareGPT
corpus.
Three measured repetitions per cache state use seeds 42, 43 and 44; two full
warmup runs per revision are excluded. Cold runs flush L1/L2 and use unseen
inputs. Restore runs repeat the same inputs after a 15-second write drain and
another L1/L2 flush. There are no additional per-run warmup requests.
Values are medians of three run summaries; latency rows use each run's mean.
960/960 measured requests succeeded, with no errors and exactly 20,480 input /
2,560 output tokens per run. Cold runs report no cached tokens. Each restore
run reports 15,360 storage-cached tokens (75%), zero device/host cached tokens,
and
storage_backend=UMBPStore.Output-throughput ranges are 1892.6–1913.3 versus 1859.7–1896.5 tokens/s cold,
and 1414.0–1741.6 versus 1492.6–1739.2 tokens/s on restore. No clear serving-speed
change is established for this workload. The host was shared, revisions ran
sequentially, CPU/NUMA affinity was not pinned, and the workload is small.
Native restore is slower than recomputation for these short prefixes.
Checklist
CI States
Latest PR Test (Base): ❌ Run #43494411547
Latest PR Test (Extra): ❌ Run #43494411365
Latest PR Test (AMD ROCm 10): ❌ Run #43494411526
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: