Skip to content

[Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation - #43387

Merged
ispobock merged 25 commits into
sgl-project:mainfrom
alphabetc1:fix/mamba-spec-finished-checkpoint
Oct 17, 2026
Merged

ispobock merged 25 commits into
sgl-project:mainfrom
alphabetc1:fix/mamba-spec-finished-checkpoint

Conversation

@alphabetc1

@alphabetc1 alphabetc1 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With Mamba no_buffer + speculative decoding, verify advances the live state before CPU grammar/finish truncation. Donating that state under a shorter cache key corrupts subsequent prefix reuse.

Changes

Require the live state length to match owned_kv_len() before donation, accounting for grammar-retained tokens and each algorithm's non-draft count. Otherwise skip the finished request's insertion, preserving existing prefixes and normal resource cleanup. The decision survives deferred release and adds no GPU synchronization.

Extend test/registered/radix_cache/test_mamba_cache_kl.py with TestMambaSpecTruncationKL. NEXTN count/story requests end at short length limits; the test compares full-vocabulary next-token distributions for identical probe token IDs using warm cache versus a cold recompute. It also checks that speculative verify accepted drafts and that cache reuse actually occurred. Shared HTTP/distribution helpers retain the existing lazy checkpoint test.

This branch includes #43567's shared test file and should merge after #43567. The no_buffer fix itself is independent of the lazy extra-buffer fix.

Validation

GB300, TP1, Qwen3.5-0.8B pinned at 2fc06364715b967f1860aea9cf38778875588b17, NEXTN (3 steps, topk 1, 4 draft tokens), no_buffer, page size 1, overlap/CUDA graphs disabled. The negative control retains #43567 and removes only #43387's finished-state guard.

Scenario KL without guard KL with guard Warm cache length without / with guard
Count, max_new_tokens=9 0.876971 0.000235300 51 / 42
Story, max_new_tokens=5 4.107031 0.000941570 27 / 22

KL is D_KL(cold || warm) over all 248,320 vocabulary entries at the same next-token position. The threshold is 0.005: valid prompt-prefix reuse versus cold prefill has numerical differences, while the corrupted-state cases are much larger. Unsafe finished suffixes now fall back to the cached prompt; this can reduce cache hits for truncated requests.

  • Registered in extra-a, 1-gpu-large; the new GPU test covers NEXTN length truncation. Grammar, EOS/stop, zero-commit, deferred release, and dedicated-slot controls remain covered by CPU tests.
  • GPU: each NEXTN case repeated three times with identical KL values; all six negative controls failed and all six fixed cases passed. The existing lazy in-place KL test also passed with KL=0.
  • CPU: 8 tests and 11 subtests passed after integrating [Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking #43567.
  • Pre-commit passed. GPU validation is limited to GB300; no throughput benchmark.
python test/registered/radix_cache/test_mamba_cache_kl.py TestMambaSpecTruncationKL
python -m pytest -q test/registered/unit/managers/test_batch_result_processor_spec_grammar.py

CI States

Latest PR Test (Base): ✅ Run #38998787342
Latest PR Test (Extra): ✅ Run #38998787385
Latest PR Test (AMD ROCm 10): ❌ Run #38998787326

@alphabetc1 alphabetc1 added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) labels Oct 11, 2026
@alphabetc1 alphabetc1 changed the title [Bugfix] Prevent misaligned Mamba checkpoints after speculative truncation [Unified Radix] Prevent misaligned Mamba checkpoints after speculative truncation Oct 12, 2026
@alphabetc1 alphabetc1 changed the title [Unified Radix] Prevent misaligned Mamba checkpoints after speculative truncation [Unified Cache] Prevent misaligned Mamba checkpoints after speculative truncation Oct 12, 2026

@kpham-sgl kpham-sgl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we remove the single-line AI gen comments / docstrings in test. Only keep the critical ones

@alphabetc1 alphabetc1 changed the title [Unified Cache] Prevent misaligned Mamba checkpoints after speculative truncation [Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation Oct 14, 2026
detain added a commit to detain/sglang that referenced this pull request Oct 15, 2026
…ation guard (schedule_batch fix already in-tree via sgl-project#43567/0923bcd258) [batch-apply-e 2026-10-14]

(cherry picked from commit 9aceb5d1a78dfe807a75875df3fc571348a53668)
detain added a commit to detain/sglang that referenced this pull request Oct 15, 2026
…ba_can_cache_finished_req (8cdf20c; donation logic preserved, stricter exact-commit+no-discard form) [batch-apply-e 2026-10-14]

(cherry picked from commit 6025bc7a7fd61a99671b689779100ffe9d9c62bd)
detain added a commit to detain/sglang that referenced this pull request Oct 15, 2026
… [batch-apply-e 2026-10-14]

(cherry picked from commit 57519d339f3a4acabaac1b9d82f460b3cad20851)
@ispobock
ispobock merged commit 86e67f7 into sgl-project:main Oct 17, 2026
195 of 214 checks passed
@alphabetc1
alphabetc1 deleted the fix/mamba-spec-finished-checkpoint branch October 17, 2026 12:22
nanjiangwill pushed a commit to modal-projects/sglang that referenced this pull request Oct 29, 2026
…lative truncation (sgl-project#43387)

Conflict resolved by hand: test/registered/unit/managers/test_batch_result_processor_spec_grammar.py, imports only; sglang-miles [41/42]
(sgl-project#44164) added the KvWeightVersionTracker import where this commit adds
release_kv_cache and get_context, and all three are kept.

(cherry picked from commit 86e67f7)
Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
nanjiangwill pushed a commit to modal-projects/sglang that referenced this pull request Oct 29, 2026
…lative truncation (sgl-project#43387)

Conflict resolved by hand: test/registered/unit/managers/test_batch_result_processor_spec_grammar.py, imports only; sglang-miles [41/42]
(sgl-project#44164) added the KvWeightVersionTracker import where this commit adds
release_kv_cache and get_context, and all three are kept.

(cherry picked from commit 86e67f7)
Claude-Session: https://claude.ai/code/session_01VDwTQn4gGFgVhzGXFs4aJx
nanjiangwill pushed a commit to modal-projects/sglang that referenced this pull request Nov 1, 2026
…lative truncation (sgl-project#43387)

Conflict resolved by hand: test/registered/unit/managers/test_batch_result_processor_spec_grammar.py, imports only; sglang-miles [41/42]
(sgl-project#44164) added the KvWeightVersionTracker import where this commit adds
release_kv_cache and get_context, and all three are kept.

(cherry picked from commit 86e67f7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants