Repository navigation
Conversation
YAMY1234
force-pushed
the
hecate-q38fn-radix-skip-decode-insert-upstream
branch
from
October 5, 2026 22:50
882cdde to
7bba33d
Compare
…nting Skip generated-prefix checkpoints while retaining prompt-boundary snapshots. Release request-owned tracking slots without freeing live or donated state. Reserve transient capacity and recheck headroom after prefix pinning. Keep flag-off behavior unchanged; exclude host-pool fraction controls.
Check that prefill releases the prompt snapshot slot but keeps the live state, that finished requests publish the prompt prefix once with idempotent release, and that the opt-in capacity cap and prefill admission guard hold.
YAMY1234
force-pushed
the
hecate-q38fn-radix-skip-decode-insert-upstream
branch
from
October 6, 2026 17:55
7bba33d to
511bc4e
Compare
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 30, 2026
… C closure -- the post-merge port queue is now decided ARRIVAL SCAN (git log 12dc623..2a6c204 --grep) + LIVE gh STATE + head-sha comparison for EVERY deferred item. Verdicts: RELEASED-BY-MERGE: sgl-project#42689 (@ 7a632cf in-range -- the SECTION-B cherry is now redundant; behavior note rides the restart). CLOSED (C-items whose PRs arrived and whose integration this stack performed): C1 (the dedicated merge week WAS merge sgl-project#5; all eight precursor PRs in-range), C2 (sgl-project#44993/sgl-project#44994 -- B5), C3 (sgl-project#45232/ sgl-project#45236/sgl-project#45334 -- B2), C4 (sgl-project#45238/sgl-project#45241/sgl-project#45242 -- B3), C5 (sgl-project#44912/ sgl-project#44913 -- arrived; entry-230 canary re-planted as its guard), C9 (sgl-project#39727 -- verified OPT-IN, no purge; BOOT_DELTA anti-needle), C0a (arrival shas already recorded at entry 310 -- Section C now carries NO open code-integration items: C6/C10 retain only their explicitly-named docs/boot-measurement remainders, re-stated). STILL-OPEN: sgl-project#44342 (head unchanged; auto-merge comfort LOST to #46970x), sgl-project#46153/sgl-project#37106/sgl-project#32479/sgl-project#45391 (gh CONFLICTING -- stale bases, rebase before any port), sgl-project#41624/sgl-project#42525/sgl-project#42660/sgl-project#44462/sgl-project#35617 (open, heads unchanged), sgl-project#46191/sgl-project#39845-class (watch-list, none in range). HOLD STANDS (operator decision, binding): sgl-project#43356/sgl-project#46452/sgl-project#43240/ sgl-project#43471 KV route -- step-1 'MERGE sgl-project#5 lands first' is NOW MET; the sequence may be re-visited ONLY with the post-merge PCIe-IPC measurement at an operator sitting. sgl-project#44732: the 'one purge if done WITH the merge' premise LAPSED when merge sgl-project#5 completed without it -- later execution costs the SAME single purge once; no penalty, no haste. Recorded for the operator, not decided by B6.
YAMY1234
added a commit
that referenced
this pull request
Nov 5, 2026
…de-insert) Port of #42660 to this branch. With the flag on, a hybrid (Mamba + attention) request publishes its prompt-end snapshot at the end of prefill and skips decode-time radix insertion and Mamba checkpoints, so a running request pins two Mamba slots instead of three to four. Default off; flag-off behaviour is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
For hybrid (Mamba + attention) models, each decode step inserts the generated prefix into the radix tree and checkpoints Mamba state for it. A running request therefore pins 3–4 Mamba slots (~59 MB each for Qwen3.8-Flash-Next) instead of 2, plus KV pages for the generated segment. At high concurrency this state crowds out the prompt-boundary snapshots that the next turn actually hits, and recomputing the short generated segment is cheaper than keeping it.
This PR adds an opt-in flag,
--radix-cache-skip-decode-insert, that keeps the prompt-end snapshot and skips decode-time insertion and Mamba checkpoints. Flag-off behaviour is unchanged.When does it help?
The gain depends on whether later turns reuse the generated tokens. On an agentic replay workload that never reuses them, the flag gives +8.6% per-user decode speed and +6.0% throughput (3 runs per arm). On a multi-turn chat that does reuse them, it only helps once cache capacity binds: at equal memory budget it loses below ~64 concurrent requests (Mamba snapshot headroom is the limit) and wins above ~96 (KV is the limit; +13.8% / +20.6% throughput at 96 / 128). When enabling it, keep Mamba snapshot headroom beyond the
(slots - 4) / 2admission floor.Modifications
--radix-cache-skip-decode-insert(defaultFalse) in the memory arg group;mamba_hook.pyrejects unsupported combinations instead of silently ignoring the flag.checkpoint_kv_cachepublishes the prompt snapshot at the end of prefill and releases the request-owned tracking slots (HybridReqToTokenPool.free_mamba_track_cache), keeping the live state and any slot already donated to the tree.release_kv_cachedoes not insert generated segments; a request that finishes in prefill still publishes its prompt checkpoint; release is idempotent.spec_utilsskips Mamba step tracking for verify/decode when the flag is on; the live Mamba state is still updated from the accepted tokens.PrefillAdderre-checks Mamba headroom after prefix pinning.test/registered/unit/mem_cache/test_mamba_donated_alloc_ratio.py(ownership across disabled / prefill / chunked / retracted, and capacity + admission).Accuracy and Performance
Checklist
CI States
Latest PR Test (Base): ❌ Run #37507296748
Latest PR Test (Extra): ❌ Run #37507296187
Latest PR Test (AMD ROCm 10): ❌ Run #37507296689