[askliar]

Overview

Adds FlashInfer ReplaySSM to Qwen3.5 and Qwen3.8-Flash-Next (Qwen4Exp) Gated DeltaNet (GDN), extending the shared lifecycle from #55688.
ReplaySSM stores a recurrent checkpoint and compact replay buffers instead of a full recurrent-state copy for each speculative token.

This PR uses the released FlashInfer kernels. Related Triton integrations are #62004, #64699, and #56466.

Claims

• Supports standard decoding and MTP with up to seven draft tokens through --use-replayssm.
• Supports prefix caching with Mamba cache mode align, plus decoding without prefix caching in mode none.
• Requires FlashInfer 0.7.3, SM100-class GPUs, and FP16 recurrent state. Model activations remain BF16.

Validation

Direct comparison with native Triton ReplaySSM (November 9–10, 2026).
Both backends use Qwen3.5-122B-A10B-NVFP4 on one GB300, TP1, a 256-request limit, and CUDA graphs.
Both use FP16 checkpoint state, BF16 activations, Triton prefill, and FlashInfer 0.7.3.
Prefix caching and FlashInfer autotuning are disabled.
Each workload ran in Triton / FlashInfer / FlashInfer / Triton order. The table gives the mean of two rates.

Workload	Metric	Triton ReplaySSM	FlashInfer ReplaySSM	Change
Standard decoding, 1,000 output tokens	Decode tokens/s/request	58.94	59.76	+1.4%
Standard decoding, 1,000 output tokens	Whole-request output tokens/s	14,922.93	15,128.77	+1.4%
MTP, 7 drafts, GSM8K, 256 output tokens	Decode tokens/s/request	90.26	108.86	+20.6%
MTP, 7 drafts, GSM8K, 256 output tokens	Whole-request output tokens/s	17,086.71	19,774.07	+15.7%

Decode rate excludes each request's time to first token. Its interval can overlap another request's prefill.
Whole-request throughput includes prefill. All eight runs completed with zero preemptions.
Mean acceptance length, including the bonus token, was 5.5717 for Triton and 5.5836 for FlashInfer.

Cache measure	Triton ReplaySSM	FlashInfer ReplaySSM
Standard decoding: assigned MB/request	122.683	138.412
Standard decoding: GDN assigned MB/request	92.013	103.809
Standard decoding: derived request limit	1,479	1,311
MTP7: assigned MB/request	156.762	156.762
MTP7: GDN assigned MB/request	117.572	117.572
MTP7: derived request limit	1,109	1,109

MB means 1,000,000 bytes. Assigned memory includes attention and GDN cache pages.
Each request uses four blocks. Derived limits reserve the null block and use floor((pool_blocks - 1) / 4).
These are cache allocation calculations, not measured admission limits under overload. The timed scheduler limit remains 256.
FlashInfer STP assigns 12.8% more cache per request and has an 11.4% lower derived request limit.
Native STP uses a 16-slot replay ring. FlashInfer uses 32 slots. The resulting automatic page sizes also differ.
Both MTP7 implementations use 32 slots and assign equal memory per request.

Direct comparison: source, controls, correctness, and STP profile

The isolated benchmark checkout 9c8c398196a1f01f7439d989ba5f58f349e46f9a adds a native Triton adapter to PR head ef69cf2738.
That adapter is not part of this PR. Both measured backends use the same benchmark checkout.
The native STP and MTP kernel files are unchanged from Johnny-Liou/ReplaySSM at 73a9592e9d8a.
Native d/k replay rings use FP16. FlashInfer u/k rings use BF16. Both backends use FP32 g rings and FP16 checkpoint state.
This comparison does not reproduce the paper's FP32-state results.

• Model revision: 98915d837c4e7c87ac8296d02e89de19b3207e6d.
• Model Runner V2, Torch 2.14.0+cu130, FP8 E4M3 attention KV, greedy sampling, and seed zero.
• MTP chat uses enable_thinking=False. STP uses a raw prompt without a chat template.
• STP uses a fixed 13-token prompt and max_model_len=1256.
• MTP7 uses the first 256 GSM8K test prompts at revision 740312add88f781978c0658806c59bc2815b9866 and max_model_len=2048.
• Timing ignores EOS. Factual checks use normal EOS.
• STP decode samples: Triton 58.9420 / 58.9424, FlashInfer 59.7661 / 59.7611 tokens/s/request.
• MTP7 decode samples: Triton 90.2492 / 90.2711, FlashInfer 108.8484 / 108.8627 tokens/s/request.

Five native GPU cases passed recurrence, accepted state, ragged inputs, padding, page strides, flushes, ring wrap, slot reuse, and graph replay.
Four model cases covered both backends in STP and MTP7. All four factual answers matched exactly between backends in each mode.
The 192-token structured MTP7 outputs matched exactly. Structured STP answers differed in correct wording.
These prompt checks do not establish full accuracy parity.

A separate STP profile captured 16 early decode steps at batch 256.
Mean GDN core time fell from 5.975 ms to 3.865 ms.
Surrounding GPU operations increased from 0.420 ms to 1.812 ms. Gaps on the core stream added another 0.433 ms.
The complete GDN path fell from 6.495 ms to 6.210 ms.
Source and event order suggest input copies and cursor preparation as optimization targets. This attribution is inferred.
All 256 profiled requests matched through output token 32. None matched across the complete 1,000-token continuation.
These captures cover early decode only. Instrumented rates are excluded from the performance table.

Qwen3.8-Flash-Next-FP8 correctness: pinned model revision 7fb48d708ca6cd8eeed8f4db0433be18af96984f, four GB300 GPUs, TP4, and CUDA graphs.
Replay runs use source ef69cf2738. No-replay controls use d642d839e7, before the PLE-only metadata correction and usage documentation.
Both arms use BF16 activations, FP16 recurrent state, and FlashInfer prefill. The no-replay decoder uses Triton.

Mode	No replay	ReplaySSM	Exact token matches
Standard decoding	4/4 correct, normal completion	4/4 correct, normal completion	3/4
MTP, 7 drafts	4/4 correct, normal completion	4/4 correct, normal completion	3/4

The STP difference is “stored in glucose” versus “stored as glucose.”
The MTP difference is “This consistent addition of 2” versus “This constant addition.” All ten requested values remain correct.
MTP accepted/rejected draft-token counts were 103/86 without replay and 98/98 with replay. These counts exclude warmup.
All four replay workers report 36 GDN replay layers. Two kernel cases passed at the checkpoint's per-rank head counts, Hk=4, Hv=12.
Those cases cover recurrence, accepted state, ring wrap, flushes, and prefix materialization.
Prefix caching passed 4/4 cached/reset comparisons with identical token IDs in each mode.
STP follow-ups reused 576 tokens each. MTP7 follow-ups reused 1,152 tokens each. Every reset control recorded zero reuse.
Producer prompts crossed physical checkpoint boundaries during decode. Runtime logs confirm GDN prefix materializer execution.
These prompt checks do not replace a model quality evaluation.

Historical Qwen3.5 performance: measurements at b95a484d72ba: Qwen3.5-122B-A10B-NVFP4 on one GB300, TP1, with CUDA graphs.
Each configuration ran twice in OFF/ON/ON/OFF order, with 256 submitted requests.

Decode rate per request. These rates exclude each request's time to first token.

Workload	No replay, tokens/s/request	ReplaySSM, tokens/s/request	Change
Standard decoding, 1,000 output tokens	50.29	61.38	+22.04%
MTP, 7 drafts, GSM8K prompts, 256 output tokens	68.43	106.78	+56.06%

Per-request cache allocation at the historical revision. MB means 1,000,000 bytes.

Workload	ReplaySSM	Attention assigned, MB	SSM assigned, MB	Total assigned, MB	SSM tensor payload, MB
Standard decoding	OFF	26.149	78.447	104.595	78.152
Standard decoding	ON	34.013	102.040	136.053	102.040
MTP, 7 drafts	OFF	30.458	730.989	761.446	674.759
MTP, 7 drafts	ON	39.191	117.572	156.762	108.233

SSM assignment includes GDN convolution state, recurrent checkpoints, replay buffers, and allocator padding.
Tensor payload excludes that padding. Attention assignment includes the MTP draft layer when present.
STP increases total assigned cache by 31.457 MB/request (+30.1%). Its SSM portion increases by 23.593 MB.
MTP7 reduces total assigned cache by 604.684 MB/request (−79.4%). Its SSM portion decreases by 613.417 MB.
Attention pages also grow because the allocator aligns their size with the larger GDN pages.

These values were recovered from the original runs' recorded tensor shapes, byte strides, and startup logs at b95a484d72ba.
The exact allocator rules supply group padding and request block demand. Both repetitions agree in all four configurations.
This is reconstruction from recorded runtime layouts, without a new GPU run.
Assigned cache bytes exclude model weights, activation scratch, and CUDA allocation granularity.
These historical values remain separate from the later direct Triton-ReplaySSM comparison above.

Cache-based admission capacity. The engine reports the following maximum concurrency at the configured sequence length.

Workload	Maximum sequence length	No replay	ReplaySSM	Capacity change
Standard decoding	1,256 tokens	1,735.75×	1,334.00×	−23.15%
MTP, 7 drafts	2,048 tokens	228.92×	1,109.75×	4.85×

These are cache-capacity estimates, not observed running-request counts or requests/s.
Both configurations set max_num_seqs=256. No capacity-matched comparison exists.
The baseline uses Triton because the fused CUDA decoder does not support FP16 state.

• Prefix caching: 4/4 cached/reset output pairs matched in each mode. Follow-ups reused 2,816 tokens for standard decoding and 2,944 for MTP7.
• Output checks: all 32 factual answers were correct. Timed continuations matched exactly for 0/256 standard-decoding requests and 29/256 MTP7 requests.
• CPU startup checks at aefb6c963e: Mamba2, GDN, wrapped GDN, and incompatible-ring checks passed. Both startup failures were reproduced before their fixes.

Still pending: Ray startup at the current revision, full-model quality evaluation, and FP16 long-context measurements.
The historical Qwen3.5 measurements do not establish current-revision output parity or Qwen4 performance.

Metric definitions, benchmark configuration, and run ranges

• Model revision: 98915d837c4e7c87ac8296d02e89de19b3207e6d.
• Model Runner V2, FlashInfer 0.7.3, FP16 recurrent state, BF16 activations, FP8 E4M3 attention KV, and Triton prefill.
• Prefix caching and FlashInfer autotuning disabled. Greedy sampling, seed zero, thinking disabled, and EOS ignored during timing.
• MTP uses the first 256 GSM8K questions at revision 740312add88f781978c0658806c59bc2815b9866.
• Standard-decoding ranges: 50.2892–50.2988 tokens/s/request without replay, 61.1484–61.6142 with ReplaySSM.
• MTP7 ranges: 68.3036–68.5476 tokens/s/request without replay, 106.7666–106.7987 with ReplaySSM.
• Mean acceptance length, including the verifier token: 5.581 without replay, 5.559 with ReplaySSM.

For each run, decode rate is 1 / mean((last_token_time - first_token_time) / (output_tokens - 1)).
The table averages the two run rates. These engine-core intervals include scheduling delays and interference from other requests.
The saved data does not isolate aggregate GPU decode throughput.

Cache capacity uses total cache blocks divided by maximum-context blocks per request.
The block counts are 6,943 → 5,336 for standard decoding and 5,723 → 4,439 for MTP7.
Per-request block demand is 4 → 4 and 25 → 4, respectively.
STP adds replay buffers without reducing checkpoint copies. MTP7 replaces eight checkpoint copies per GDN layer with one checkpoint and replay buffers.
The recorded per-layer page strides are 2,179,072 / 2,834,432 bytes for STP OFF/ON and 2,342,912 / 3,014,656 for MTP7 OFF/ON.
STP groups contain 12 layers. MTP's attention group contains 13 layers, including the draft layer.
Its three 12-layer GDN groups each reserve one padding position.
Let U = page_stride * group_width, where group_width is 12 for STP and 13 for MTP7.
Let c = 8 for MTP7 OFF and c = 1 for the other configurations.
Then attention = U, SSM = 3*c*U, and total = (1+3*c)*U.
The attention token blocks exceed the configured maximum sequence lengths, so each valid request needs one attention block.

The engine's reported estimate includes its reserved null block.
After that reservation, calculated whole-request budgets are 1,735 → 1,333 for standard decoding and 228 → 1,109 for MTP7.
These calculated budgets exclude the scheduler limit. They are not measured saturation limits.

Factual prompts used normal EOS handling. Their output tokens matched across configurations within each mode.
Timed MTP input arrays were not saved. Dataset revision, fingerprint, script, and prompt lengths matched.
Prefix reset controls recorded zero reuse. These checks establish prefix reuse, not prefix-cache performance.

Archived offline benchmark commands

Both performance tables use offline vllm.LLM.generate. No HTTP server or client was involved.
The Bash below factors repeated paths and arguments from the saved Python invocations.
Argument values, source revisions, and first-run labels match the completed measurements.
The paths identify the archived environment. The saved harness snapshots and benchmark-only native adapter are not yet publicly published.
This records the executed configurations. It is not a standalone public reproduction package.

python=/workspace/replayssm-page-rings-20261021/review-fixes/.venv/bin/python
task=/workspace/gdn-review-20261105
data=/workspace/gdn-paper-20261104
model="$data/cache/huggingface/hub/models--nvidia--Qwen3.5-122B-A10B-NVFP4/snapshots/98915d837c4e7c87ac8296d02e89de19b3207e6d"

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_USE_FLASHINFER_SAMPLER=0
export VLLM_ALLOW_INSECURE_SERIALIZATION=1
export VLLM_NO_USAGE_STATS=1
export SGLANG_GDN_WY_STRIDED_QKV=1
export GDN_SMOKE_STATE_DTYPE=float16
export CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=8
export HF_HOME="$data/cache/huggingface"
export HF_HUB_CACHE="$HF_HOME/hub" HF_DATASETS_CACHE="$data/cache/datasets"
export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1
export TMPDIR="$task/tmp" XDG_CACHE_HOME="$task/cache/xdg"
export UV_CACHE_DIR="$task/cache/uv" TORCH_HOME="$task/cache/torch"
export TRITON_CACHE_DIR="$task/cache/triton" CUDA_CACHE_PATH="$task/cache/cuda"
export FLASHINFER_WORKSPACE_BASE="$task/cache/flashinfer" VLLM_CACHE_ROOT="$task/cache/vllm"
unset GDN_UCACHE_STATE_DTYPE

common=(
  --model-id "$model" --batch-size 256
  --dataset-revision 740312add88f781978c0658806c59bc2815b9866
  --runner v2 --gdn-prefill-backend triton
  --flashinfer-autotune off --baseline-kernel triton --state-dtype float16
)

run_saved() {
  local label="$1"
  shift
  uv run --offline --no-project --python "$python" "$python" \
    "$results/$label/executed-workload.py" \
    "${common[@]}" --source-revision "$revision" --comparison-axis "$axis" \
    "$@" --output "$results/$label/result.json"
}

# Historical ReplaySSM OFF/ON.
revision=b95a484d72ba1bc1b75a18a8c1e17560f243f812
axis=replay
results="$task/results"
export PYTHONPATH="$task/umma-prefix:$task/released-0.7.3:$task/overlay:$data/deps"
run_saved scoped-stp-off-1-fp16-r1 \
  --workload stp --replay off --abba-position 1 --language-model-only
run_saved scoped-stp-on-2-fp16-r1 \
  --workload stp --replay on --abba-position 2 --language-model-only
run_saved scoped-mtp7-off-1-fp16-r1 \
  --workload mtp --replay off --abba-position 1 --num-speculative-tokens 7
run_saved scoped-mtp7-on-2-fp16-r1 \
  --workload mtp --replay on --abba-position 2 --num-speculative-tokens 7

# Direct Triton ReplaySSM / FlashInfer ReplaySSM.
revision=9c8c398196a1f01f7439d989ba5f58f349e46f9a
axis=replay_backend
results="$task/direct-triton-20261109/results"
export PYTHONPATH="$task/direct-triton-source:$task/released-0.7.3:$task/overlay:$data/deps"
native=(--native-source-revision 73a9592e9d8a2268620e8883a82f5cec7442ed36)
run_saved direct-stp-triton-1-run1 \
  --workload stp --replay on --abba-position 1 --language-model-only \
  --replay-backend triton "${native[@]}"
run_saved direct-stp-flashinfer-2-run1 \
  --workload stp --replay on --abba-position 2 --language-model-only \
  --replay-backend flashinfer "${native[@]}"
run_saved direct-mtp7-triton-1-run1 \
  --workload mtp --replay on --abba-position 1 --num-speculative-tokens 7 \
  --replay-backend triton "${native[@]}"
run_saved direct-mtp7-flashinfer-2-run1 \
  --workload mtp --replay on --abba-position 2 --num-speculative-tokens 7 \
  --replay-backend flashinfer "${native[@]}"

Each workload has two measurements per configuration. Historical order was OFF/ON/ON/OFF, and direct-comparison order was Triton/FlashInfer/FlashInfer/Triton.
The second measurements use positions 3 and 4 in their arguments and saved directory labels.
The harness sets VLLM_GDN_DECODE_KERNEL=triton for historical OFF and flashinfer_replayssm for historical ON.
The direct harness sets that variable to triton for both arms, then selects VLLM_BENCH_GDN_REPLAYSSM_BACKEND from --replay-backend.
The direct harness also sets VLLM_ENABLE_V1_MULTIPROCESSING=0 for its local inspection callbacks.

STP warms up with 128 output tokens before generating 1,000 measured output tokens per request.
MTP7 repeats the workload for at least 30 seconds before generating 256 measured output tokens per request.
Both use greedy sampling and ignore EOS during timing. Factual output checks use normal EOS.

CPU checks and startup regressions

The Qwen4 update passed four configuration checks, seven worker checks, and ten existing warmup checks.
Configuration checks cover both target architectures, the checkpoint's head counts, unchanged PLE state, and the attention-only MTP draft.
Worker checks cover canonical convolution state beside ReplaySSM and reject unsupported canonical temporal state.

GPU validation exposed PLE metadata reading absent replay states. The shared builder now selects replay metadata from each layer specification.
Two new PLE metadata cases and six existing cases passed. The corrected GPU run passed initialization and graph capture.
The earlier GDN state-copy and wrapped-spec startup regressions remain covered.

All changed-file commit hooks passed, including mypy. An independent reviewer checked the implementation, and the original implementer returned its response.
Frontend configuration checks ran on macOS without selecting its unsupported speculative CPU runner.
These CPU checks do not establish GPU correctness. The model results above supply separate runtime evidence.

Published Triton ReplaySSM reference

ReplaySSM: Cache SSM Inputs, Not State, Appendix E, Tables 6 and 9, reports the following Triton results at batch 256:

Workload	Published speedup over conventional vLLM
Standard decoding	1.20×
MTP, 7 drafts (T=8)	2.09×

The paper uses Qwen3.5-122B-A10B-NVFP4, one B300, TP1, FP32 recurrent state, BF16 activations, L16, and CUDA graphs.
These are whole-model measurements. This PR's FlashInfer measurements report per-request decode rates on GB300 with FP16 recurrent state.
The paper and historical on/off figures provide separate references. The direct comparison above supplies matched FP16-state measurements.

The paper's serving implementation uses Triton GDN kernels.
Its standalone CuTe precision experiment is separate from serving dispatch.
Ze-Wei Liou's open #62004 replaces closed #48792 for standard decoding. Closed #49887 contains the earlier GDN speculative implementation.

Qwen4 correctness configuration and reproduction

The six configurations cover STP OFF/ON, MTP7 OFF/ON, and replay prefix caching in each mode.
Each configuration uses TP4, max_model_len=8192, max_num_seqs=4, max_num_batched_tokens=512, seed 42, and greedy sampling.
The runs disable NUMA binding and use Model Runner V2 with FULL/PIECEWISE CUDA graphs.
MTP uses seven drafts. Both MTP arms request block_size=192, which resolves to 576 tokens and satisfies QSA ring alignment.
The default no-replay MTP setting failed because its 448-token block was not divisible by the 12-token QSA ring capacity.

The prefix producer starts with 560 tokens. STP generates 48 tokens, and MTP generates 624 tokens before the follow-up.
Physical GDN blocks contain 576 tokens. MTP hash alignment is separately 192 tokens because of its draft cache group.
The follow-up includes the exact generated token IDs. Each cached run has a reset control with the same input IDs.
The STP and MTP runtime logs contain GdnPrefixMaterializeKernel launches.

Use the Qwen4 settings below with --max-model-len 8192 --max-num-seqs 4 --max-num-batched-tokens 512.
Set numa_bind=False in the engine configuration.
Run OFF/ON for standard decoding and MTP7, then compare cached/reset follow-ups with --enable-prefix-caching.
The saved correctness harness records the source revision, settings, worker backend selection, output IDs, acceptance counts, and reuse counters.

Details

The adapter packs variable-length rows into kernel widths 1, 4, or 8 and returns only real-token outputs.
The worker commits accepted replay cursors after all layers execute. At aligned prefix boundaries, it materializes canonical state into existing Mamba cache pages.
Uniform decode supports FULL graphs. Canonical prefill uses PIECEWISE graphs or eager execution.
Changes to the cache-pool layout can still require materializer compilation.

Shared changes affect Mamba2 zero-length-row postprocessing, final-prefill metadata, Ray environment propagation, and the NVIDIA Qwen3.5/Qwen4 cache schemas.
Qwen4 keeps PLE convolution state in its existing canonical format. Its attention-only MTP draft disables ReplaySSM independently of the target.

Example with seven MTP drafts:

vllm serve nvidia/Qwen3.5-122B-A10B-NVFP4 \
    --dtype bfloat16 --mamba-ssm-cache-dtype float16 \
    --gdn-prefill-backend triton --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":7,"enable_adaptive_verification":false}' \
    --use-replayssm

Replace --no-enable-prefix-caching with --enable-prefix-caching to use align.
Remove --speculative-config for standard decoding.
For Qwen4, use Qwen/Qwen3.8-Flash-Next-FP8 with --tensor-parallel-size 4 --block-size 192.
Use --gdn-prefill-backend flashinfer to match the Qwen4 correctness runs.
The explicit block size satisfies the current QSA ring alignment for seven drafts.
For a no-replay control, remove --use-replayssm and set VLLM_GDN_DECODE_KERNEL=triton.

Unsupported configurations include BF16 recurrent state, cache mode all, adaptive verification, stochastic rounding, pipeline parallelism, microbatching, and KV connectors.
GDN heads must have dimension 128, with at most four value heads per key head.
Usage documentation is in docs/features/automatic_prefix_caching.md.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill.
• AI assistance was used: OpenAI Codex and Claude Code. The human submitter remains responsible for the change.
• Design Fit: Reuses the shared ReplaySSM lifecycle and released FlashInfer APIs.
• Testing and Validation: Qwen4 GPU kernel, STP/MTP7 output, and prefix-cache checks passed. Broader quality limits are stated above.
• Code Quality and Style: Focused checks and independent review completed. Usage documentation is included.
• Pull Request Contents: Includes usage, results, comparison conditions, and remaining limitations.

[mergify]

Documentation preview: https://vllm--65657.org.readthedocs.build/en/65657/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[askliar]

@ZJY0516 @vadiklyutiy take a look please! Let's get it in ASAP!

[vadiklyutiy]

Thanks for the contribution! Could you please add a few things to help with validation and reproducibility?

1. Include the exact server and client commands used for each configuration in the performance table, directly in the PR description.
2. Add a few accuracy tests comparing ReplaySSM enabled vs. disabled, covering both standard decoding and MTP, and report the results.
3. Include per-request cache memory figures in MB for both configurations alongside the cache-capacity estimates (how much SSM adds to the total cache size for a single request and how much memory the SSM cache alone occupies for that request).

[vadiklyutiy]

Is there any reason why we don't support it in Qwen4?

[vadiklyutiy]

--gdn-prefill-backend triton and --use-replayssm together looks a bit confusing.

In your implementation --use-replayssm use FI, so I think it should fail with --gdn-prefill-backend triton

[vadiklyutiy]

Is there a chance to compare performance of FI ReplaySSM vs early triton implementation of replayssm?

[askliar]

Re: Validation, commands and memory

Added the commands and memory breakdown to the description, including SSM separately. These were offline LLM.generate runs, so no separate server/client.

STP/MTP7 on/off output checks are there too. The answers are correct, but it’s still a few prompts rather than a full accuracy eval.

The old logs had enough shape/stride information to recover the memory numbers. The benchmark scripts themselves aren’t public yet.

[askliar]

Re: Qwen4

Added it! Used Qwen/Qwen3.8-Flash-Next-FP8 for validation.

Ran STP + MTP7 with replay on/off, plus prefix caching. All four prompts gave correct answers in each configuration. Three matched exactly between on/off, with wording differences in the fourth. Prefix cached/reset outputs matched 4/4 in both modes.

[askliar]

Re: Triton prefill + FlashInfer ReplaySSM

I don’t think this should fail. These control different phases: Triton does prefill, FI ReplaySSM does decode.

Prefill writes the recurrent state, then the worker resets the replay cursors so FI can continue from there. That’s also the combination I used for the Qwen3.5 benchmarks. Added a clarification to the docs.

[askliar]

Re: Direct Triton comparison

Yep, added that comparison too. Used the original Triton kernels on the same V2 base, one GB300, batch 256 and FP16 state.

FI gets +1.4% for STP and +20.6% for MTP7 in per-request decode rate. Whole-request throughput is +1.4% and +15.7%.

STP still uses 12.8% more cache because FI has a 32-slot ring vs Triton’s 16. MTP7 memory is the same. Full numbers are in the description.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @askliar.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork