Repository navigation
[Mamba] Unify FI ReplaySSM STP/MTP/Prefix Caching Lifecycle #55688
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 44 commits
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
06b4023
[Mamba] Rebase unified FlashInfer ReplaySSM lifecycle
3431299
[Mamba] Pack ReplaySSM state and reuse serving metadata
2dff1f6
[Mamba] Separate ReplaySSM optimizations from prefix-cache integration
30e701a
[Mamba] Share ReplaySSM ring storage across cache groups
ca64a51
[Mamba] Enable ReplaySSM for Nemotron Nano VL
9ac27de
[Mamba] Skip ReplaySSM prefill self-copies
b6bdfa5
[Mamba] Simplify ReplaySSM setup and tests
d174268
[Mamba] Reduce ReplaySSM helpers and test duplication
6a86ade
[Mamba] Merge current vLLM main into unified ReplaySSM
6b7c7fa
[Mamba] Merge current vLLM main into unified ReplaySSM
a74677f
[Mamba] Update unified ReplaySSM for current vLLM main
d9c439c
[Mamba] Merge latest vLLM main after ReplaySSM update
00a1690
[Mamba] Keep FlashInfer ReplaySSM rings in cache pages
28c24a8
[Mamba] Keep completed shared prefixes reusable within a step
fe1b574
Merge upstream main and preserve ReplaySSM validation
8d8d1c7
[test] Match SSD initial-state and cache dtypes
6d3cfcc
[test] Cover a full packed ReplaySSM prefix block
5a514c4
[Mamba] Reuse ReplaySSM request positions and add lifecycle E2E coverage
b28bb6e
[Test] Enable worker geometry inspection in ReplaySSM E2E
26f9d88
[Test] Queue matched ReplaySSM stress batches and retain quality fail…
dd7951d
[Test] Queue ReplaySSM quality requests before scheduling
3fddf93
Merge latest vLLM main into ReplaySSM prefix caching
edc05d8
[Test] Compare ReplaySSM cached state on identical accepted history
731c81d
[Test] Preserve ReplaySSM acceptance across scheduling pauses
9cd15b9
[Test] Keep ReplaySSM lifecycle coverage on real execution paths
10a4e4c
[Test] Retain decoded checkpoints for ReplaySSM history comparison
974b49d
[Test] Separate ReplaySSM numerical checks from aggregate quality sco…
0fc1f42
[Mamba] Publish synchronous checkpoints before block retirement
459884f
[Tests] Remove FP32 ReplaySSM MTP comparisons
1c72376
[ReplaySSM] Simplify cache initialization and focus regression coverage
4678287
[ReplaySSM] Retain three reusable blocks in the MTP stress test
b94a058
[ReplaySSM] Clarify the cache specification flag
8d0ea53
[Mamba] Batch ReplaySSM lifecycle metadata across cache groups
37be80b
[Mamba] Preserve ReplaySSM stochastic seed progression
bc55647
[Mamba] Share ReplaySSM plans across cache groups
a99bc6e
Merge main into ReplaySSM prefix caching
9ad6c89
Fix ReplaySSM alignment and metadata review cases
d8eb082
Match FlashInfer materialization to the null block ID
ec928c9
Handle absent Mamba metadata during V2 warmup
00cf5cb
Exercise prefix reuse in ReplaySSM GSM8K evaluation
d9cd1fd
Document FlashInfer ReplaySSM prefix caching
f02c563
Narrow ReplaySSM CI triggers and isolate nightly evaluation
b112309
Merge latest vLLM main and preserve ReplaySSM platform CI coverage
bc10c65
Merge branch 'main' into feat/unify_replayssm_cycle
vllm-agent a778ee5
Fix CPU cache binding after ReplaySSM integration
db0179c
Update ReplaySSM test fixtures and prefix-cache wrapper
2d6aa97
Merge branch 'main' into feat/unify_replayssm_cycle
vllm-agent d3fcd07
Merge main and preserve ReplaySSM cache state through resizing
02daa60
Describe Mamba temporal copy layout without backend identity
81ac2f1
Merge main and keep Triton ReplaySSM decode planning
204d009
Resolve ReplaySSM GPU copy layout in the worker
0f29dd1
Merge branch 'main' into feat/unify_replayssm_cycle
vllm-agent a8877fa
Fix ReplaySSM prefill reset synchronization and test setup
50a9238
Merge main and preserve ReplaySSM cache rebinding
b5c933f
Merge branch 'main' into feat/unify_replayssm_cycle
mergify[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4-ReplaySSM-align.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" | ||
| accuracy_threshold: 0.93 | ||
| num_questions: 1319 | ||
| num_fewshot: 5 | ||
| startup_max_wait_seconds: 1200 | ||
| # Require draft steps and report natural acceptance without a performance bound. | ||
| min_acceptance_length: 1.0 | ||
| expect_prefix_cache_hits: true | ||
| env: | ||
| VLLM_USE_V2_MODEL_RUNNER: "1" | ||
| # 327 divides each 5232-token page and preserves the shared 654-token boundary. | ||
| server_args: >- | ||
| --enforce-eager | ||
| --max-model-len 4096 | ||
| --tensor-parallel-size 2 | ||
| --enable-expert-parallel | ||
| --dtype bfloat16 | ||
| --mamba-backend flashinfer | ||
| --mamba-ssm-cache-dtype float16 | ||
| --enable-mamba-cache-stochastic-rounding | ||
| --mamba-cache-philox-rounds 5 | ||
| --use-replayssm | ||
| --replayssm-buffer-len 16 | ||
| --enable-prefix-caching | ||
| --prefix-match-unit 327 | ||
| --mamba-cache-mode align | ||
| --speculative-config '{"method":"mtp","num_speculative_tokens":5}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Nemotron-3-Super-120B-A12B-NVFP4-ReplaySSM-align.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.