Skip to content
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
06b4023
[Mamba] Rebase unified FlashInfer ReplaySSM lifecycle
Oct 1, 2026
3431299
[Mamba] Pack ReplaySSM state and reuse serving metadata
Oct 1, 2026
2dff1f6
[Mamba] Separate ReplaySSM optimizations from prefix-cache integration
Oct 1, 2026
30e701a
[Mamba] Share ReplaySSM ring storage across cache groups
Oct 2, 2026
ca64a51
[Mamba] Enable ReplaySSM for Nemotron Nano VL
Oct 5, 2026
9ac27de
[Mamba] Skip ReplaySSM prefill self-copies
Oct 6, 2026
b6bdfa5
[Mamba] Simplify ReplaySSM setup and tests
Oct 6, 2026
d174268
[Mamba] Reduce ReplaySSM helpers and test duplication
Oct 7, 2026
6a86ade
[Mamba] Merge current vLLM main into unified ReplaySSM
Oct 8, 2026
6b7c7fa
[Mamba] Merge current vLLM main into unified ReplaySSM
Oct 12, 2026
a74677f
[Mamba] Update unified ReplaySSM for current vLLM main
Oct 14, 2026
d9c439c
[Mamba] Merge latest vLLM main after ReplaySSM update
Oct 14, 2026
00a1690
[Mamba] Keep FlashInfer ReplaySSM rings in cache pages
Oct 23, 2026
28c24a8
[Mamba] Keep completed shared prefixes reusable within a step
Oct 23, 2026
fe1b574
Merge upstream main and preserve ReplaySSM validation
Oct 23, 2026
8d8d1c7
[test] Match SSD initial-state and cache dtypes
Oct 23, 2026
6d3cfcc
[test] Cover a full packed ReplaySSM prefix block
Oct 23, 2026
5a514c4
[Mamba] Reuse ReplaySSM request positions and add lifecycle E2E coverage
Oct 24, 2026
b28bb6e
[Test] Enable worker geometry inspection in ReplaySSM E2E
Oct 24, 2026
26f9d88
[Test] Queue matched ReplaySSM stress batches and retain quality fail…
Oct 24, 2026
dd7951d
[Test] Queue ReplaySSM quality requests before scheduling
Oct 24, 2026
3fddf93
Merge latest vLLM main into ReplaySSM prefix caching
Oct 24, 2026
edc05d8
[Test] Compare ReplaySSM cached state on identical accepted history
Oct 24, 2026
731c81d
[Test] Preserve ReplaySSM acceptance across scheduling pauses
Oct 24, 2026
9cd15b9
[Test] Keep ReplaySSM lifecycle coverage on real execution paths
Oct 24, 2026
10a4e4c
[Test] Retain decoded checkpoints for ReplaySSM history comparison
Oct 24, 2026
974b49d
[Test] Separate ReplaySSM numerical checks from aggregate quality sco…
Oct 24, 2026
0fc1f42
[Mamba] Publish synchronous checkpoints before block retirement
Oct 24, 2026
459884f
[Tests] Remove FP32 ReplaySSM MTP comparisons
Oct 25, 2026
1c72376
[ReplaySSM] Simplify cache initialization and focus regression coverage
Oct 25, 2026
4678287
[ReplaySSM] Retain three reusable blocks in the MTP stress test
Oct 25, 2026
b94a058
[ReplaySSM] Clarify the cache specification flag
Oct 25, 2026
8d0ea53
[Mamba] Batch ReplaySSM lifecycle metadata across cache groups
Oct 25, 2026
37be80b
[Mamba] Preserve ReplaySSM stochastic seed progression
Oct 25, 2026
bc55647
[Mamba] Share ReplaySSM plans across cache groups
Oct 25, 2026
a99bc6e
Merge main into ReplaySSM prefix caching
Oct 26, 2026
9ad6c89
Fix ReplaySSM alignment and metadata review cases
Oct 26, 2026
d8eb082
Match FlashInfer materialization to the null block ID
Oct 26, 2026
ec928c9
Handle absent Mamba metadata during V2 warmup
Oct 26, 2026
00cf5cb
Exercise prefix reuse in ReplaySSM GSM8K evaluation
Oct 26, 2026
d9cd1fd
Document FlashInfer ReplaySSM prefix caching
Oct 26, 2026
f02c563
Narrow ReplaySSM CI triggers and isolate nightly evaluation
Oct 30, 2026
b112309
Merge latest vLLM main and preserve ReplaySSM platform CI coverage
Oct 30, 2026
bc10c65
Merge branch 'main' into feat/unify_replayssm_cycle
vllm-agent Nov 1, 2026
a778ee5
Fix CPU cache binding after ReplaySSM integration
Nov 1, 2026
db0179c
Update ReplaySSM test fixtures and prefix-cache wrapper
Nov 1, 2026
2d6aa97
Merge branch 'main' into feat/unify_replayssm_cycle
vllm-agent Nov 1, 2026
d3fcd07
Merge main and preserve ReplaySSM cache state through resizing
Nov 3, 2026
02daa60
Describe Mamba temporal copy layout without backend identity
Nov 3, 2026
81ac2f1
Merge main and keep Triton ReplaySSM decode planning
Nov 3, 2026
204d009
Resolve ReplaySSM GPU copy layout in the worker
Nov 3, 2026
0f29dd1
Merge branch 'main' into feat/unify_replayssm_cycle
vllm-agent Nov 3, 2026
a8877fa
Fix ReplaySSM prefill reset synchronization and test setup
Nov 3, 2026
50a9238
Merge main and preserve ReplaySSM cache rebinding
Nov 3, 2026
b5c933f
Merge branch 'main' into feat/unify_replayssm_cycle
mergify[bot] Nov 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .buildkite/test_areas/engine.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -206,18 +206,26 @@ steps:
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- tests/v1/e2e/test_replayssm_decode.py
- tests/kernels/mamba/test_ssu_dispatch.py
Comment thread
askliar marked this conversation as resolved.
Resolved
- vllm/v1/core/sched/scheduler.py
- vllm/v1/worker/utils.py
- vllm/v1/kv_cache_interface.py
- vllm/config/cache.py
- vllm/config/vllm.py
- vllm/model_executor/layers/mamba/
- vllm/model_executor/models/nemotron_h.py
- vllm/model_executor/warmup/kernel_warmup.py
- vllm/model_executor/warmup/replayssm_warmup.py
- vllm/v1/attention/backends/mamba_attn.py
- vllm/v1/worker/mamba_utils.py
- vllm/v1/worker/replayssm_utils.py
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/
- vllm/v1/worker/gpu_model_runner.py
- vllm/v1/worker/gpu_worker.py
Comment thread
askliar marked this conversation as resolved.
Resolved
commands:
- pytest -v -s v1/e2e/test_replayssm_decode.py
- VLLM_TEST_REQUIRE_REPLAYSSM=1 pytest -v -s v1/e2e/test_replayssm_decode.py
- VLLM_TEST_REQUIRE_REPLAYSSM=1 pytest -v kernels/mamba/test_ssu_dispatch.py::test_replayssm_materializes_only_accepted_boundary_with_compacted_rows kernels/mamba/test_ssu_dispatch.py::test_replayssm_postprocess_commits_staged_transition
mirror:
amd:
label: ":amd: (MI355) ReplaySSM E2E"
Expand All @@ -229,6 +237,8 @@ steps:
- image-build-amd
source_file_dependencies:
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/test_replayssm_decode.py

- label: ":nvidia: (L4) [Nightly] V1 E2E"
key: v1-e2e-2-gpus
Expand Down
26 changes: 26 additions & 0 deletions .buildkite/test_areas/lm_eval.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,32 @@ steps:
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-spec-decode-rocm.txt

- label: ":nvidia: (B200) [Nightly] LM Eval ReplaySSM"
key: lm-eval-replayssm-2xb200
timeout_in_minutes: 60
device: b200-k8s
optional: true
num_devices: 2
source_file_dependencies:
- .buildkite/test_areas/lm_eval.yaml
- tests/evals/gsm8k/configs/models-replayssm.txt
- tests/evals/gsm8k/configs/Nemotron-3-Super-120B-A12B-NVFP4-ReplaySSM-align.yaml
- tests/evals/gsm8k/test_gsm8k_correctness.py
- tests/evals/gsm8k/gsm8k_eval.py
- tests/evals/gsm8k/conftest.py
- vllm/model_executor/layers/mamba/abstract.py
- vllm/model_executor/layers/mamba/mamba_mixer2.py
- vllm/model_executor/layers/mamba/ops/ssu_dispatch.py
- vllm/model_executor/layers/mamba/ops/replayssm_config.py
- vllm/model_executor/warmup/kernel_warmup.py
- vllm/model_executor/warmup/replayssm_warmup.py
- vllm/v1/attention/backends/mamba_attn.py
- vllm/v1/worker/mamba_utils.py
- vllm/v1/worker/replayssm_utils.py
- vllm/v1/worker/gpu/model_states/mamba_hybrid.py
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-replayssm.txt

- label: ":nvidia: (B200) [Nightly] LM Eval Large Models EP"
key: lm-eval-large-models-ep-2xb200
automatic_shard: true
Expand Down
31 changes: 31 additions & 0 deletions docs/features/automatic_prefix_caching.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,37 @@ vllm serve <hybrid-model> \

Choose a value that divides the block size of every prefix-cacheable KV cache group, and that is a multiple of the per-state compression ratio for models that use one, such as sparse MLA. vLLM validates both at startup and names the offending sizes in the error. Read the served block size from the startup log. 64 is a reasonable starting point.

### FlashInfer ReplaySSM

FlashInfer ReplaySSM caches recent inputs to reduce full-state writes during Mamba2 decoding. It is disabled by default. Both Model Runners V1 and V2 support prefix caching with ReplaySSM, including MTP speculative decoding.

This example enables ReplaySSM and prefix caching with three MTP draft tokens:

```bash
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
--tensor-parallel-size 2 \
--max-model-len 12288 \
--dtype bfloat16 \
--mamba-backend flashinfer \
--mamba-ssm-cache-dtype float16 \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5 \
--use-replayssm \
--replayssm-buffer-len 16 \
--enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

`--enable-prefix-caching` selects Mamba cache mode `align`. To use ReplaySSM without prefix caching, replace this flag with `--no-enable-prefix-caching`. This selects mode `none`. To disable speculative decoding, remove `--speculative-config`.

FlashInfer ReplaySSM has these limits:

- `align` requires BF16 model inputs for its replay rings. The SSM state dtype is independent. The example uses FP16 state with stochastic rounding.
- The logical replay buffer cannot exceed 16 tokens. For speculative decoding, it must contain at least `1 + num_speculative_tokens` tokens.
- Pipeline parallelism, microbatching, and KV connectors are not supported.

Prefix reuse still requires a shared prefix with a saved Mamba checkpoint. The matching rules for hybrid Mamba models above also apply.

## Example workloads

We describe two example workloads, where APC can provide huge performance benefit:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4"
accuracy_threshold: 0.93
num_questions: 1319
num_fewshot: 5
startup_max_wait_seconds: 1200
# Require draft steps and report natural acceptance without a performance bound.
min_acceptance_length: 1.0
expect_prefix_cache_hits: true
env:
VLLM_USE_V2_MODEL_RUNNER: "1"
# 327 divides each 5232-token page and preserves the shared 654-token boundary.
server_args: >-
--enforce-eager
--max-model-len 4096
--tensor-parallel-size 2
--enable-expert-parallel
--dtype bfloat16
--mamba-backend flashinfer
--mamba-ssm-cache-dtype float16
--enable-mamba-cache-stochastic-rounding
--mamba-cache-philox-rounds 5
--use-replayssm
--replayssm-buffer-len 16
--enable-prefix-caching
--prefix-match-unit 327
--mamba-cache-mode align
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'
1 change: 1 addition & 0 deletions tests/evals/gsm8k/configs/models-replayssm.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Nemotron-3-Super-120B-A12B-NVFP4-ReplaySSM-align.yaml
24 changes: 24 additions & 0 deletions tests/evals/gsm8k/test_gsm8k_correctness.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,28 @@ def get_acceptance_length(server_url: str) -> float:
return 1.0 + num_accepted / num_drafts


def check_prefix_cache_hits(server_url: str, expected: bool) -> None:
"""Check that the evaluated requests exercised the configured cache mode."""
response = requests.get(f"{server_url.rstrip('/').removesuffix('/v1')}/metrics")
response.raise_for_status()
hits = []
for line in response.text.splitlines():
name, _, value = line.rpartition(" ")
if name.split("{")[0] == "vllm:prefix_cache_hits_total":
hits.append(float(value))
assert hits, "prefix-cache hit metric is missing"
total_hits = sum(hits)
print(f" Prefix-cache hit tokens: {total_hits:g}")
if expected:
assert total_hits > 0, (
"prefix caching was enabled but no cached tokens were reused"
)
else:
assert total_hits == 0, (
"prefix caching was disabled but cached tokens were reused"
)


def test_gsm8k_correctness(config_filename):
"""Test GSM8K correctness for a given model configuration."""
eval_config = yaml.safe_load(config_filename.read_text(encoding="utf-8"))
Expand Down Expand Up @@ -189,6 +211,8 @@ def test_gsm8k_correctness(config_filename):
assert results and not any(results), f"{method} failed: {results}"

results = run_gsm8k_eval(eval_config, server_url)
if "expect_prefix_cache_hits" in eval_config:
check_prefix_cache_hits(server_url, eval_config["expect_prefix_cache_hits"])

measured_metric = results["accuracy"]
expected_metric = eval_config["accuracy_threshold"]
Expand Down
1 change: 1 addition & 0 deletions tests/kernels/mamba/test_precopy_mamba_align.py
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,7 @@ def __init__(self, n):
self.mamba_state_idx_buf = _FakeCpuGpuBuffer(n)
self.precopy_src_col_buf = _FakeCpuGpuBuffer(n)
self.precopy_token_bias_buf = _FakeCpuGpuBuffer(n)
self.replayssm = None
self.calls = []

def initialize_from_forward_context(self, *args, **kwargs):
Expand Down
Loading
Loading