Skip to content

Remove swa and mamba radix cache - #40313

Merged
ispobock merged 7 commits into
mainfrom
rm-legacy-radix-trees
Sep 20, 2026
Merged

ispobock merged 7 commits into
mainfrom
rm-legacy-radix-trees

Conversation

@ispobock

@ispobock ispobock commented Sep 19, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

SWARadixCache and MambaRadixCache are no longer reachable. Hybrid SWA and hybrid SSM models route to UnifiedRadixCache since #30468, and no registered backend constructs either class, so the two implementations only cost maintenance: every base class change has to update them, most recently #40075.

Part of #20415. RadixCache stays, it is still selected for non-hybrid models and PureSWARadixCache derives from it.

The legacy-swa and legacy-mamba configs of the manual test_unified_radix_cache_bench.py --bench CLI go with them.


CI States

Latest PR Test (Base): ✅ Run #35441390417
Latest PR Test (Extra): ❌ Run #35441390265
Latest PR Test (AMD ROCm 10): ❌ Run #35441390346

@github-actions github-actions Bot added documentation Improvements or additions to documentation unified-radix-cache labels Sep 19, 2026
@ispobock ispobock added run-ci CI: run the baseline test suite on this PR bypass-fastfail run-ci-extra CI: also run the extra suite (requires run-ci) labels Sep 19, 2026
@ispobock

Copy link
Copy Markdown
Collaborator Author

/rerun-failed-ci

Nakanokensetsu added a commit to Nakanokensetsu/sglang that referenced this pull request Sep 19, 2026
Three 66 K-token chains on a 27B hybrid GDN model (2x 12 GB, TP 2) went from
0 % prefix reuse on every revisit to 99.9 %, and from 57 s to 0.3 s, by moving
three flags together:

  --mem-fraction-static 0.78 -> 0.85   static budget +856 MiB; KV pool
                                       max_total_num_tokens 148,058 -> 210,000
  --max-total-tokens 150000 -> 210000  lifts the cap; on its own it does nothing
  --chunked-prefill-size 4096 -> 8192  halves mamba checkpoint demand so the
                                       widened pool still fits 32 slots

No metric regressed: cold prefill 970 -> 1010 tok/s, decode identical at
51.9 tok/s, short-prompt TTFT during a long prefill 0.135 -> 0.123 s,
concurrency 10 at 0 errors and 175 W (up from 169 W, i.e. more real work),
discriminative long-context test at 96 K 17/20 -> 18/20.

The diagnosis was wrong first. Three mamba-side levers - PR sgl-project#38000's coverage
thinning, an int8 checkpoint pool, and the larger chunk - moved the hit rate not
at all; only reducing the chain count did. 3 x 66,000 = 198,000 tokens never fit
a 148,058-token KV pool. mamba_evictable sitting at 24/32 was three chains'
worth of state, not pressure.

Also documented: how to rebase this onto a moving upstream (check the class is
actually constructed - RadixCache and MambaRadixCache are dead on 0.5.19 and
deleted by sgl-project#40313; a clean three-way merge still silently carries removed APIs;
import success is not name resolution; measured v0.5.19 -> v0.5.20 churn, where
every retargeted API survives so only the merge is redone). A configuration that
starts cleanly and dies on the first long prefill, because Triton JITs kernels
onto the device after the startup VRAM numbers print. Measured slot costs (fp16
mamba 38 MB, int8 checkpoint 23 MB, not half). The cliff presenting as 100 %
utilisation at falling power. Two measurement traps: --allow-auto-truncate
silently answering from a truncated document, and comparing a warm server
against a cold one.

Includes the full launch script with placeholders, and the measurement scripts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ispobock
ispobock merged commit a8a4d86 into main Sep 20, 2026
272 of 307 checks passed
@ispobock
ispobock deleted the rm-legacy-radix-trees branch September 20, 2026 08:16
Klaud-Cold pushed a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 19, 2026
…2-cu130

Pin lmsysorg/sglang:v0.5.22-cu130 by digest for
dsv4-fp4-gb200-dynamo-sglang-agentic-disagg and drop
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, which upstream removed with the
legacy SWA radix cache (sgl-project/sglang#40313).

将 dsv4-fp4-gb200-dynamo-sglang-agentic-disagg 的镜像更新为按摘要固定的
lmsysorg/sglang:v0.5.22-cu130,并删除上游随旧版 SWA radix cache 一并移除的
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT 环境变量(sgl-project/sglang#40313)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adibarra added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 21, 2026
…agg image to v0.5.21-cu130 / [Klaud Cold] 将 DeepSeek-V4-Pro FP4 B200 Dynamo-SGLang AgentX 分离式镜像更新至 v0.5.21-cu130 (#4176)

* chore(dsv4): update B200 Dynamo-SGLang AgentX disagg image to v0.5.21-cu130

Pin lmsysorg/sglang:v0.5.21-cu130 by digest for
dsv4-fp4-b200-dynamo-sglang-agentic-disagg and drop
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT, whose only reader was removed
upstream with the legacy SWA radix cache (sgl-project/sglang#40313).

将 dsv4-fp4-b200-dynamo-sglang-agentic-disagg 的镜像按摘要固定为
lmsysorg/sglang:v0.5.21-cu130,并移除 SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT;
上游已随旧版 SWA radix cache 一并删除其唯一读取方(sgl-project/sglang#40313)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(perf-changelog): record dsv4 B200 Dynamo-SGLang disagg image update

Append the perf-changelog entry for
dsv4-fp4-b200-dynamo-sglang-agentic-disagg.

为 dsv4-fp4-b200-dynamo-sglang-agentic-disagg 追加 perf-changelog 条目。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: refresh PR #4176 for sweep reuse [skip-sweep]

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alec Ibarra <alec@semianalysis.com>
Co-authored-by: adibarra <93070681+adibarra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail documentation Improvements or additions to documentation run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants