Repository navigation
Remove swa and mamba radix cache - #40313
Merged
Merged
Conversation
ispobock
requested review from
Ying1123,
alphabetc1,
hanming-lu,
hnyls2002,
huangtingwei9988,
hzh0425,
merrymercy,
xiezhq-hermann and
yizhang2077
as code owners
September 19, 2026 08:37
hzh0425
approved these changes
Sep 19, 2026
Collaborator
Author
|
/rerun-failed-ci |
Nakanokensetsu
added a commit
to Nakanokensetsu/sglang
that referenced
this pull request
Sep 19, 2026
Three 66 K-token chains on a 27B hybrid GDN model (2x 12 GB, TP 2) went from
0 % prefix reuse on every revisit to 99.9 %, and from 57 s to 0.3 s, by moving
three flags together:
--mem-fraction-static 0.78 -> 0.85 static budget +856 MiB; KV pool
max_total_num_tokens 148,058 -> 210,000
--max-total-tokens 150000 -> 210000 lifts the cap; on its own it does nothing
--chunked-prefill-size 4096 -> 8192 halves mamba checkpoint demand so the
widened pool still fits 32 slots
No metric regressed: cold prefill 970 -> 1010 tok/s, decode identical at
51.9 tok/s, short-prompt TTFT during a long prefill 0.135 -> 0.123 s,
concurrency 10 at 0 errors and 175 W (up from 169 W, i.e. more real work),
discriminative long-context test at 96 K 17/20 -> 18/20.
The diagnosis was wrong first. Three mamba-side levers - PR sgl-project#38000's coverage
thinning, an int8 checkpoint pool, and the larger chunk - moved the hit rate not
at all; only reducing the chain count did. 3 x 66,000 = 198,000 tokens never fit
a 148,058-token KV pool. mamba_evictable sitting at 24/32 was three chains'
worth of state, not pressure.
Also documented: how to rebase this onto a moving upstream (check the class is
actually constructed - RadixCache and MambaRadixCache are dead on 0.5.19 and
deleted by sgl-project#40313; a clean three-way merge still silently carries removed APIs;
import success is not name resolution; measured v0.5.19 -> v0.5.20 churn, where
every retargeted API survives so only the merge is redone). A configuration that
starts cleanly and dies on the first long prefill, because Triton JITs kernels
onto the device after the startup VRAM numbers print. Measured slot costs (fp16
mamba 38 MB, int8 checkpoint 23 MB, not half). The cliff presenting as 100 %
utilisation at falling power. Two measurement traps: --allow-auto-truncate
silently answering from a truncated document, and comparing a warm server
against a cold one.
Includes the full launch script with placeholders, and the measurement scripts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
56 of 66 tasks
This was referenced Sep 23, 2026
3 of 5 tasks
This was referenced Oct 12, 2026
Klaud-Cold
pushed a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 19, 2026
…2-cu130 Pin lmsysorg/sglang:v0.5.22-cu130 by digest for dsv4-fp4-gb200-dynamo-sglang-agentic-disagg and drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, which upstream removed with the legacy SWA radix cache (sgl-project/sglang#40313). 将 dsv4-fp4-gb200-dynamo-sglang-agentic-disagg 的镜像更新为按摘要固定的 lmsysorg/sglang:v0.5.22-cu130,并删除上游随旧版 SWA radix cache 一并移除的 SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT 环境变量(sgl-project/sglang#40313)。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adibarra
added a commit
to SemiAnalysisAI/InferenceX
that referenced
this pull request
Oct 21, 2026
…agg image to v0.5.21-cu130 / [Klaud Cold] 将 DeepSeek-V4-Pro FP4 B200 Dynamo-SGLang AgentX 分离式镜像更新至 v0.5.21-cu130 (#4176) * chore(dsv4): update B200 Dynamo-SGLang AgentX disagg image to v0.5.21-cu130 Pin lmsysorg/sglang:v0.5.21-cu130 by digest for dsv4-fp4-b200-dynamo-sglang-agentic-disagg and drop SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, whose only reader was removed upstream with the legacy SWA radix cache (sgl-project/sglang#40313). 将 dsv4-fp4-b200-dynamo-sglang-agentic-disagg 的镜像按摘要固定为 lmsysorg/sglang:v0.5.21-cu130,并移除 SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT; 上游已随旧版 SWA radix cache 一并删除其唯一读取方(sgl-project/sglang#40313)。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(perf-changelog): record dsv4 B200 Dynamo-SGLang disagg image update Append the perf-changelog entry for dsv4-fp4-b200-dynamo-sglang-agentic-disagg. 为 dsv4-fp4-b200-dynamo-sglang-agentic-disagg 追加 perf-changelog 条目。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: refresh PR #4176 for sweep reuse [skip-sweep] --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Alec Ibarra <alec@semianalysis.com> Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
SWARadixCacheandMambaRadixCacheare no longer reachable. Hybrid SWA and hybrid SSM models route toUnifiedRadixCachesince #30468, and no registered backend constructs either class, so the two implementations only cost maintenance: every base class change has to update them, most recently #40075.Part of #20415.
RadixCachestays, it is still selected for non-hybrid models andPureSWARadixCachederives from it.The
legacy-swaandlegacy-mambaconfigs of the manualtest_unified_radix_cache_bench.py --benchCLI go with them.CI States
Latest PR Test (Base): ✅ Run #35441390417
Latest PR Test (Extra): ❌ Run #35441390265
Latest PR Test (AMD ROCm 10): ❌ Run #35441390346