Skip to content
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .buildkite/test_areas/engine.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,11 @@ steps:
- vllm/v1/worker/gpu_worker.py
commands:
- VLLM_TEST_REQUIRE_REPLAYSSM=1 pytest -v -s v1/e2e/test_replayssm_decode.py
- VLLM_TEST_REQUIRE_REPLAYSSM=1 pytest -v kernels/mamba/test_ssu_dispatch.py::test_replayssm_materializes_only_accepted_boundary_with_compacted_rows kernels/mamba/test_ssu_dispatch.py::test_replayssm_postprocess_commits_staged_transition
- >-
VLLM_TEST_REQUIRE_REPLAYSSM=1 pytest -v -s
kernels/mamba/test_ssu_dispatch.py::test_replayssm_materializes_only_accepted_boundary_with_compacted_rows
kernels/mamba/test_ssu_dispatch.py::test_replayssm_postprocess_commits_staged_transition
kernels/mamba/test_ssu_dispatch.py::test_replayssm_skips_canonical_self_copy_but_resets_prefill_trackers
mirror:
amd:
automatic_shard: true
Expand Down
4 changes: 4 additions & 0 deletions .buildkite/test_areas/kernels.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -755,6 +755,8 @@ steps:
- vllm/platforms/cuda.py
- vllm/cute_utils/
- vllm/model_executor/layers/mamba/ops/gdn_chunk_cutedsl/
- vllm/model_executor/layers/mamba/gdn/
- vllm/model_executor/layers/mamba/mamba_utils.py
- vllm/model_executor/layers/fused_moe/router/bf16x3_router_gemm_sm100.py
- cmake/external_projects/fmha_sm100.cmake
- vllm/models/minimax_m3/common/sparse_attention.py
Expand All @@ -771,6 +773,7 @@ steps:
- tests/kernels/quantization/nvfp4_utils.py
- tests/kernels/test_fused_minimax_m3_qknorm_rope_kv_insert.py
- tests/kernels/mamba/test_gdn_prefill_cutedsl.py
- tests/kernels/mamba/test_gdn_replayssm.py
- tests/kernels/test_bf16x3_router_gemm_cutedsl.py
- tests/kernels/test_top_k_per_row.py
- csrc/fused_mqa_topk/
Expand Down Expand Up @@ -810,6 +813,7 @@ steps:
- pytest -v -s tests/kernels/moe/test_flashinfer.py
- pytest -v -s tests/kernels/moe/test_trtllm_nvfp4_moe.py
- pytest -v -s tests/kernels/mamba/test_gdn_prefill_cutedsl.py
- pytest -v -s tests/kernels/mamba/test_gdn_replayssm.py
mirror:
amd:
automatic_shard: true
Expand Down
22 changes: 22 additions & 0 deletions docs/features/automatic_prefix_caching.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,28 @@ FlashInfer ReplaySSM has these limits:

Prefix reuse still requires a shared prefix with a saved Mamba checkpoint. The matching rules for hybrid Mamba models above also apply.

#### Qwen3.5 Gated DeltaNet

On Qwen3.5 models, `--use-replayssm` applies ReplaySSM to the Gated DeltaNet layers. It runs FlashInfer's UMMA kernel, which requires FlashInfer 0.7.3 or later, an SM100-class GPU, and FP16 recurrent state:

```bash
vllm serve nvidia/Qwen3.5-122B-A10B-NVFP4 \
--dtype bfloat16 \
--mamba-ssm-cache-dtype float16 \
--use-replayssm \
--enable-prefix-caching \
--speculative-config '{"method":"mtp","num_speculative_tokens":7}'
```

Remove `--speculative-config` for standard decoding. Qwen3.5 Gated DeltaNet ReplaySSM has these limits:

- Model activations and the convolution state must be BF16, with 128-element key and value heads and at most four value heads per key head.
- Speculative decoding supports up to seven draft tokens. Adaptive verification is not supported; set `enable_adaptive_verification` to `false`.
- `--replayssm-buffer-len` must be 16, the default.
- Mamba cache stochastic rounding, pipeline parallelism, microbatching, and KV connectors are not supported.

With prefix caching, vLLM sets `GDN_UCACHE_STATE_DTYPE=fp16` before starting its workers, because FlashInfer reads it when first imported. If your own code imports FlashInfer before creating the engine, set this variable before that import.

## Example workloads

We describe two example workloads, where APC can provide huge performance benefit:
Expand Down
Loading