Skip to content

[Mamba] Add opt-in prefill-only radix caching (--radix-cache-skip-decode-insert) - #42660

Draft
YAMY1234 wants to merge 2 commits into
sgl-project:mainfrom
YAMY1234:hecate-q38fn-radix-skip-decode-insert-upstream
Draft

YAMY1234 wants to merge 2 commits into
sgl-project:mainfrom
YAMY1234:hecate-q38fn-radix-skip-decode-insert-upstream

Conversation

@YAMY1234

@YAMY1234 YAMY1234 commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

For hybrid (Mamba + attention) models, each decode step inserts the generated prefix into the radix tree and checkpoints Mamba state for it. A running request therefore pins 3–4 Mamba slots (~59 MB each for Qwen3.8-Flash-Next) instead of 2, plus KV pages for the generated segment. At high concurrency this state crowds out the prompt-boundary snapshots that the next turn actually hits, and recomputing the short generated segment is cheaper than keeping it.

This PR adds an opt-in flag, --radix-cache-skip-decode-insert, that keeps the prompt-end snapshot and skips decode-time insertion and Mamba checkpoints. Flag-off behaviour is unchanged.

When does it help?

The gain depends on whether later turns reuse the generated tokens. On an agentic replay workload that never reuses them, the flag gives +8.6% per-user decode speed and +6.0% throughput (3 runs per arm). On a multi-turn chat that does reuse them, it only helps once cache capacity binds: at equal memory budget it loses below ~64 concurrent requests (Mamba snapshot headroom is the limit) and wins above ~96 (KV is the limit; +13.8% / +20.6% throughput at 96 / 128). When enabling it, keep Mamba snapshot headroom beyond the (slots - 4) / 2 admission floor.

Cache hit, throughput and per-user speed vs concurrency, flag off vs on

Modifications

  • --radix-cache-skip-decode-insert (default False) in the memory arg group; mamba_hook.py rejects unsupported combinations instead of silently ignoring the flag.
  • With the flag on, checkpoint_kv_cache publishes the prompt snapshot at the end of prefill and releases the request-owned tracking slots (HybridReqToTokenPool.free_mamba_track_cache), keeping the live state and any slot already donated to the tree.
  • release_kv_cache does not insert generated segments; a request that finishes in prefill still publishes its prompt checkpoint; release is idempotent.
  • spec_utils skips Mamba step tracking for verify/decode when the flag is on; the live Mamba state is still updated from the accepted tokens.
  • Capacity accounting: with the flag on, each running request retains the live state plus one protected prefix (ratio 2) with a small reserve for chunked prefill and prefix donation; PrefillAdder re-checks Mamba headroom after prefix pinning.
  • Two CPU unit cases added to the existing test/registered/unit/mem_cache/test_mamba_donated_alloc_ratio.py (ownership across disabled / prefill / chunked / retracted, and capacity + admission).

Accuracy and Performance

  • Qwen3.8-Flash-Next (NVFP4, FP8 KV, MTP3) on a single VR200, AgentX multi-turn trace, concurrency 100, 3660 s, 3 runs vs 3 runs on the same stack and node pool:
    • per-user speed (1000 / ITL p90): 29.8 → 32.4 (+8.6%, ranges 29.70–30.46 vs 32.28–33.01, non-overlapping)
    • total token throughput: 273.6k → 290.1k tok/s (+6.0%, non-overlapping)
    • prefix cache hit rate: 92.6–92.8% → 93.4%
    • TTFT p50: 1,786 → 1,670 ms (ranges overlap; not claimed)
  • Flag off: no behaviour change (unit tests cover the flag-off path).
  • CPU unit tests: 14 passed / 12 subtests in the nightly container.

Checklist

  • Format your code according to the Code Formatting with Pre-Commit.
  • Add unit tests as outlined in the Running Unit Tests.
  • Update documentation / docstrings / example tutorials as needed.
  • Provide throughput / latency benchmark results and accuracy evaluation results as needed.
  • For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging.
  • Please feel free to join our Slack channel if you need help.

CI States

Latest PR Test (Base): ❌ Run #37507296748
Latest PR Test (Extra): ❌ Run #37507296187
Latest PR Test (AMD ROCm 10): ❌ Run #37507296689

@YAMY1234
YAMY1234 force-pushed the hecate-q38fn-radix-skip-decode-insert-upstream branch from 882cdde to 7bba33d Compare October 5, 2026 22:50
…nting

Skip generated-prefix checkpoints while retaining prompt-boundary snapshots.
Release request-owned tracking slots without freeing live or donated state.
Reserve transient capacity and recheck headroom after prefix pinning.
Keep flag-off behavior unchanged; exclude host-pool fraction controls.
Check that prefill releases the prompt snapshot slot but keeps the live state,
that finished requests publish the prompt prefix once with idempotent release,
and that the opt-in capacity cap and prefill admission guard hold.
@YAMY1234
YAMY1234 force-pushed the hecate-q38fn-radix-skip-decode-insert-upstream branch from 7bba33d to 511bc4e Compare October 6, 2026 17:55
detain added a commit to detain/sglang that referenced this pull request Oct 30, 2026
… C closure -- the post-merge port queue is now decided

ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE +
head-sha comparison for EVERY deferred item. Verdicts:
  RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B
    cherry is now redundant; behavior note rides the restart).
  CLOSED (C-items whose PRs arrived and whose integration this stack
    performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight
    precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/
    sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/
    sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard),
    C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle),
    C0a (arrival shas already recorded at entry 310 -- Section C now
    carries NO open code-integration items: C6/C10 retain only their
    explicitly-named docs/boot-measurement remainders, re-stated).
  STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to
    #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale
    bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617
    (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in
    range).
  HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/
    sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the
    sequence may be re-visited ONLY with the post-merge PCIe-IPC
    measurement at an operator sitting. sgl-project#44732: the 'one purge if done
    WITH the merge' premise LAPSED when merge sgl-project#5 completed without it
    -- later execution costs the SAME single purge once; no penalty,
    no haste. Recorded for the operator, not decided by B6.
YAMY1234 added a commit that referenced this pull request Nov 5, 2026
…de-insert)

Port of #42660 to this branch. With the flag on, a
hybrid (Mamba + attention) request publishes its prompt-end snapshot at
the end of prefill and skips decode-time radix insertion and Mamba
checkpoints, so a running request pins two Mamba slots instead of three
to four. Default off; flag-off behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant