Repository navigation
[Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation - #43387
Merged
ispobock merged 25 commits intoOct 17, 2026
Merged
Conversation
alphabetc1
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
October 9, 2026 14:34
…t' into fix/mamba-spec-finished-checkpoint
5 tasks done
ispobock
approved these changes
Oct 12, 2026
kpham-sgl
approved these changes
Oct 12, 2026
kpham-sgl
left a comment
Collaborator
There was a problem hiding this comment.
nit: can we remove the single-line AI gen comments / docstrings in test. Only keep the critical ones
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 15, 2026
…ation guard (schedule_batch fix already in-tree via sgl-project#43567/0923bcd258) [batch-apply-e 2026-10-14] (cherry picked from commit 9aceb5d1a78dfe807a75875df3fc571348a53668)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 15, 2026
…ba_can_cache_finished_req (8cdf20c; donation logic preserved, stricter exact-commit+no-discard form) [batch-apply-e 2026-10-14] (cherry picked from commit 6025bc7a7fd61a99671b689779100ffe9d9c62bd)
detain
added a commit
to detain/sglang
that referenced
this pull request
Oct 15, 2026
… [batch-apply-e 2026-10-14] (cherry picked from commit 57519d339f3a4acabaac1b9d82f460b3cad20851)
nanjiangwill
pushed a commit
to modal-projects/sglang
that referenced
this pull request
Oct 29, 2026
…lative truncation (sgl-project#43387) Conflict resolved by hand: test/registered/unit/managers/test_batch_result_processor_spec_grammar.py, imports only; sglang-miles [41/42] (sgl-project#44164) added the KvWeightVersionTracker import where this commit adds release_kv_cache and get_context, and all three are kept. (cherry picked from commit 86e67f7) Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
nanjiangwill
pushed a commit
to modal-projects/sglang
that referenced
this pull request
Oct 29, 2026
…lative truncation (sgl-project#43387) Conflict resolved by hand: test/registered/unit/managers/test_batch_result_processor_spec_grammar.py, imports only; sglang-miles [41/42] (sgl-project#44164) added the KvWeightVersionTracker import where this commit adds release_kv_cache and get_context, and all three are kept. (cherry picked from commit 86e67f7) Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
nanjiangwill
pushed a commit
to modal-projects/sglang
that referenced
this pull request
Nov 1, 2026
…lative truncation (sgl-project#43387) Conflict resolved by hand: test/registered/unit/managers/test_batch_result_processor_spec_grammar.py, imports only; sglang-miles [41/42] (sgl-project#44164) added the KvWeightVersionTracker import where this commit adds release_kv_cache and get_context, and all three are kept. (cherry picked from commit 86e67f7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With Mamba
no_buffer+ speculative decoding, verify advances the live state before CPU grammar/finish truncation. Donating that state under a shorter cache key corrupts subsequent prefix reuse.Changes
Require the live state length to match
owned_kv_len()before donation, accounting for grammar-retained tokens and each algorithm's non-draft count. Otherwise skip the finished request's insertion, preserving existing prefixes and normal resource cleanup. The decision survives deferred release and adds no GPU synchronization.Extend
test/registered/radix_cache/test_mamba_cache_kl.pywithTestMambaSpecTruncationKL. NEXTN count/story requests end at short length limits; the test compares full-vocabulary next-token distributions for identical probe token IDs using warm cache versus a cold recompute. It also checks that speculative verify accepted drafts and that cache reuse actually occurred. Shared HTTP/distribution helpers retain the existing lazy checkpoint test.This branch includes #43567's shared test file and should merge after #43567. The no_buffer fix itself is independent of the lazy extra-buffer fix.
Validation
GB300, TP1, Qwen3.5-0.8B pinned at
2fc06364715b967f1860aea9cf38778875588b17, NEXTN (3 steps, topk 1, 4 draft tokens), no_buffer, page size 1, overlap/CUDA graphs disabled. The negative control retains #43567 and removes only #43387's finished-state guard.KL is
D_KL(cold || warm)over all 248,320 vocabulary entries at the same next-token position. The threshold is 0.005: valid prompt-prefix reuse versus cold prefill has numerical differences, while the corrupted-state cases are much larger. Unsafe finished suffixes now fall back to the cached prompt; this can reduce cache hits for truncated requests.extra-a,1-gpu-large; the new GPU test covers NEXTN length truncation. Grammar, EOS/stop, zero-commit, deferred release, and dedicated-slot controls remain covered by CPU tests.CI States
Latest PR Test (Base): ✅ Run #38998787342
Latest PR Test (Extra): ✅ Run #38998787385
Latest PR Test (AMD ROCm 10): ❌ Run #38998787326