[Manfredss]

Overview

Fix #61319. LFM2 models (Lfm2ForCausalLM, Lfm2MoeForCausalLM) can't use batch-invariant mode: with VLLM_BATCH_INVARIANT=1 the engine refuses to start with RuntimeError: VLLM batch_invariant mode is not supported for SHORT_CONV_ATTN. That leaves them without the documented way to get reproducible greedy outputs. This PR lets SHORT_CONV_ATTN declare batch invariance. PLE_SHORT_CONV_ATTN stays opted out.

Claims

• LFM2 models start and are batch-invariant under VLLM_BATCH_INVARIANT=1. LiquidAI/LFM2-8B-A1B passes the full tests/v1/determinism/test_batch_invariance.py suite (22/22) on an A40.
• The symptom in [Bug]: Greedy (temperature=0) decoding is not reproducible across repeated calls on the same engine for LiquidAI/LFM2-8B-A1B (hybrid mamba) #61319 (round 0 of repeated generate() calls differs from later rounds) goes away in batch-invariant mode, with prefix caching on.
• No change when VLLM_BATCH_INVARIANT is unset.

Validation

A40 (SM86), official image vllm/vllm-openai:nightly-39983a5e (no upstream changes to the touched files between that commit and this branch's base). Probe: 8 prompts × 128 greedy tokens, 4 generate() rounds on one engine, then each prompt alone. Counts are prompts (out of 8) whose token ids differ. The enforce_eager / no-prefix-cache rows and the 22-test suite ran with an env-gated copy of the same one-line opt-in. The rest ran with this branch's file installed.

model	config	round 0 vs rounds 1–3	rounds 1–3	batched vs alone
LFM2-8B-A1B	main, default	8	0	8
LFM2-8B-A1B	main, enable_prefix_caching=False	0	0	6
LFM2-8B-A1B	main, VLLM_BATCH_INVARIANT=1	engine fails to start
LFM2-8B-A1B	this PR, VLLM_BATCH_INVARIANT=1	0	0	0
LFM2-8B-A1B	this PR, BI + enforce_eager	0	0	0
LFM2-8B-A1B	this PR, BI + enable_prefix_caching=False	0	0	0
LFM2-1.2B	main, default	2	0	3
LFM2-1.2B	this PR, VLLM_BATCH_INVARIANT=1	0	0	0

Upstream BI suite with this change:

VLLM_TEST_MODEL=LiquidAI/LFM2-8B-A1B pytest -v tests/v1/determinism/test_batch_invariance.py
# 22 passed in 1057s (FLASH_ATTN / FLEX_ATTENTION / TRITON_ATTN)
VLLM_TEST_MODEL=LiquidAI/LFM2-1.2B pytest -v "tests/v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle[default-FLASH_ATTN]"
# 1 passed in 35s

Details

Why #61319 happens. LFM2 runs with prefix caching on by default (mamba_cache_mode=align, block size 16). From round 1 on, any prompt with a full block gets a cache hit, so the first prefill step shrinks (92 → 76 tokens in my probe, where only the 17-token prompt hits). Without batch invariance, kernel selection depends on batch shape. Every sequence in that step then moves by a few ULPs, and near-tied argmaxes flip later. In my runs, two warm rounds never disagreed. With prefix caching on but no prompt long enough to hit, all rounds match. This is the documented "no bitwise reproducibility by default" behavior, and the documented fix is batch-invariant mode, which LFM2 couldn't use. Setting CUBLAS_WORKSPACE_CONFIG=:16:8 CUBLASLT_WORKSPACE_SIZE=1, as #61440 does, still leaves 2/8 prompts differing, because this isn't run-to-run atomics.

Why ShortConv qualifies. causal_conv1d_fn (prefill) and causal_conv1d_update (decode) compute each output token from that sequence's own L_cache window plus its conv-state slot. There is no reduction across sequences and no split depending on batch size. The rest of the layer is in_proj/out_proj, which already go through the batch-invariant linear path, plus elementwise gating.

Scope and limitations.

• PLE_SHORT_CONV_ATTN (Qwen4Exp's dilated conv) inherits from ShortConvAttentionBackend. It is explicitly kept at False: it uses its own packing path, and I couldn't validate it.
• All runs used Model Runner V2 (the default). The check is shared with V1 through get_mamba_attn_backend, but V1 wasn't run.
• Spec decoding under batch invariance and LFM2-VL weren't tested.
• Only Ampere (SM86) was tested.
• I didn't add an end-to-end LFM2 line to CI. If you want one, VLLM_TEST_MODEL=LiquidAI/LFM2-1.2B ... -k FLASH_ATTN on the needle test costs ~35 s per variant, and I'm happy to add it.

Not a duplicate. The only PR linked to #61319 is #61440. It forces deterministic cuBLAS/cuDNN settings globally, which doesn't address this (see above), and its diff currently spans 891 files. I searched open PRs for supports_batch_invariance, SHORT_CONV_ATTN/ShortConv, and LFM2 batch invariance. #45819 (GDN) and the Mamba2 BI PRs touch other backends only.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance (Claude Code) was used to reproduce and root-cause the issue, write the change, and run the GPU validation above. I reviewed every changed line.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[mergify]

Documentation preview: https://vllm--62367.org.readthedocs.build/en/62367/

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Manfredss.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Manfredss]

Merged current main; the only conflict was the tested-models list in docs/features/batch_invariance.md (#61254 appended to the same list).

E2E runs in the format requested on #60013/#60163, so they're here before review. One RTX 3090 (sm86, driver 580.65.06, CUDA 13.0) on Runpod, image vllm/vllm-openai:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f. "main" is the image as shipped; "PR" is the same image with only this PR's changed files copied over the installed package (sha256 of every swapped file is logged per run).

Reproduce.

$ env VLLM_BATCH_INVARIANT=1 vllm serve LiquidAI/LFM2-8B-A1B --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9

# main
(EngineCore pid=14778) ERROR 10-20 07:01:06 [core.py:1543] RuntimeError: VLLM batch_invariant mode is not supported for SHORT_CONV_ATTN.

# PR
(EngineCore pid=15303) INFO 10-20 07:01:46 [cuda.py:508] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'TRITON_ATTN', 'FLEX_ATTENTION'].
INFO:     Application startup complete.

Batch invariance:

# main, `vllm serve LiquidAI/LFM2-8B-A1B --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9`
$ python3 repro_bi.py --model LiquidAI/LFM2-8B-A1B --rounds 3
round 0: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at 4, max |dlogprob| on agreeing positions 0.1060
round 1: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.1735
round 2: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.1735

# PR, `env VLLM_BATCH_INVARIANT=1 vllm serve LiquidAI/LFM2-8B-A1B --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.9`
$ python3 repro_bi.py --model LiquidAI/LFM2-8B-A1B --rounds 3
round 0: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 1: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000
round 2: A=954 tokens, B=1915 tokens | A alone vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000 | A batched with B vs A alone: first token diff at None, max |dlogprob| on agreeing positions 0.0000

repro_bi.py (greedy; prompt A alone twice, then A in the same batch as a longer prompt B)

import argparse, random, threading, time
import httpx


def prompt(seed, n_words):
    rng = random.Random(seed)
    return " ".join(f"w{rng.randrange(10000)}" for _ in range(n_words))


def gen(base, model, text, n):
    r = httpx.post(base + "/v1/completions", json={"model": model, "prompt": text, "max_tokens": n,
                                                    "temperature": 0.0, "logprobs": 1}, timeout=600)
    r.raise_for_status()
    ch = r.json()["choices"][0]
    return ch["logprobs"]["tokens"], ch["logprobs"]["token_logprobs"], r.json()["usage"]["prompt_tokens"]


def diff(x, y):
    n = min(len(x[0]), len(y[0]))
    first = next((i for i in range(n) if x[0][i] != y[0][i]), None)
    agree = n if first is None else first
    gap = max((abs(x[1][i] - y[1][i]) for i in range(agree)), default=0.0)
    return f"first token diff at {first}, max |dlogprob| on agreeing positions {gap:.4f}"


ap = argparse.ArgumentParser()
ap.add_argument("--base-url", default="http://127.0.0.1:8000")
ap.add_argument("--model", default="Qwen/Qwen3-8B-AWQ")
ap.add_argument("--rounds", type=int, default=3)
a = ap.parse_args()
A, B = prompt(1, 330), prompt(2, 660)
for r in range(a.rounds):
    solo = gen(a.base_url, a.model, A, 48)
    solo2 = gen(a.base_url, a.model, A, 48)
    out = {}
    tb = threading.Thread(target=lambda: out.__setitem__("B", gen(a.base_url, a.model, B, 48)))
    ta = threading.Thread(target=lambda: out.__setitem__("A", gen(a.base_url, a.model, A, 48)))
    tb.start(); time.sleep(0.005); ta.start(); tb.join(); ta.join()
    print(f"round {r}: A={solo[2]} tokens, B={out['B'][2]} tokens | A alone vs A alone: {diff(solo2, solo)} | "
          f"A batched with B vs A alone: {diff(out['A'], solo)}")

Unit test.

$ cd tests && python3 -m pytest -v -p no:cacheprovider v1/determinism/test_short_conv_batch_invariant.py
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_prefill_is_batch_invariant[False-other_lens0] PASSED [ 11%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_prefill_is_batch_invariant[False-other_lens1] PASSED [ 22%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_prefill_is_batch_invariant[False-other_lens2] PASSED [ 33%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_prefill_is_batch_invariant[True-other_lens0] PASSED [ 44%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_prefill_is_batch_invariant[True-other_lens1] PASSED [ 55%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_prefill_is_batch_invariant[True-other_lens2] PASSED [ 66%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_decode_is_batch_invariant[2] PASSED [ 77%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_decode_is_batch_invariant[8] PASSED [ 88%]
v1/determinism/test_short_conv_batch_invariant.py::test_short_conv_decode_is_batch_invariant[15] PASSED [100%]
============================== 9 passed in 4.08s ===============================

lm_eval (gsm8k). main can't start with VLLM_BATCH_INVARIANT=1, so main is shown in its default mode.

$ lm_eval --model local-completions --model_args base_url=http://127.0.0.1:8000/v1/completions,model=LiquidAI/LFM2-8B-A1B,num_concurrent=64,max_retries=3,tokenized_requests=False --tasks gsm8k

# main, default
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'LiquidAI/LFM2-8B-A1B', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  | 0.84|±  |0.0101|
|     |       |strict-match    |     5|exact_match|↑  | 0.84|±  |0.0101|

# PR, VLLM_BATCH_INVARIANT=1
local-completions ({'base_url': 'http://127.0.0.1:8000/v1/completions', 'model': 'LiquidAI/LFM2-8B-A1B', 'num_concurrent': 64, 'max_retries': 3, 'tokenized_requests': False}), gen_kwargs: ({}), limit: None, num_fewshot: None, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8378|±  |0.0102|
|     |       |strict-match    |     5|exact_match|↑  |0.8385|±  |0.0101|

vllm bench serve.

$ vllm bench serve --model LiquidAI/LFM2-8B-A1B --host 127.0.0.1 --port 8000 --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 200 --request-rate inf --num-warmups 16

# main, default
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  17.72     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              11.29     
Output token throughput (tok/s):         1444.94   
Peak output token throughput (tok/s):    5640.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          13004.48  
---------------Time to First Token----------------
Mean TTFT (ms):                          7092.61   
Median TTFT (ms):                        6914.11   
P99 TTFT (ms):                           14328.10  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          74.42     
Median TPOT (ms):                        76.28     
P99 TPOT (ms):                           116.38    
---------------Inter-token Latency----------------
Mean ITL (ms):                           74.42     
Median ITL (ms):                         32.91     
P99 ITL (ms):                            151.65    
==================================================

# PR, VLLM_BATCH_INVARIANT=1
============ Serving Benchmark Result ============
Successful requests:                     200       
Failed requests:                         0         
Benchmark duration (s):                  17.29     
Total input tokens:                      204800    
Total generated tokens:                  25600     
Request throughput (req/s):              11.57     
Output token throughput (tok/s):         1480.64   
Peak output token throughput (tok/s):    5049.00   
Peak concurrent requests:                200.00    
Total token throughput (tok/s):          13325.73  
---------------Time to First Token----------------
Mean TTFT (ms):                          6642.99   
Median TTFT (ms):                        6496.35   
P99 TTFT (ms):                           13292.56  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          72.00     
Median TPOT (ms):                        73.44     
P99 TPOT (ms):                           108.56    
---------------Inter-token Latency----------------
Mean ITL (ms):                           72.00     
Median ITL (ms):                         36.84     
P99 ITL (ms):                            137.35    
==================================================

[Monishver11]

Hi @Manfredss, thanks for this work, just reviewed this as an outside contributor (not a maintainer).
LGTM; causal_conv1d sums each token's taps in a fixed order so the opt-in looks right. @yewentao256 one question: since the new kernel test passes on main too, should we add the ~35s LFM2-1.2B needle line to the H100 BI step to guard the opt-in?

[Manfredss]

Thanks for taking a look, @Monishver11. Agreed: the kernel test only pins the conv property, so CI wouldn't notice if LFM2 stopped starting or stopped being batch invariant under VLLM_BATCH_INVARIANT=1. The line I'd add to the H100 step, matching the existing ones:

- VLLM_TEST_MODEL=LiquidAI/LFM2-1.2B pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k FLASH_ATTN

On the A40 the default RMSNorm variant took 35 s. -k FLASH_ATTN also selects the vllm_c variant, which I'd run before pushing. @yewentao256, happy to add it to this PR if you want it.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Manfredss.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Manfredss]

Merged main again to clear the conflict. #63609 commented out the Batch Invariance steps until the pytest plugin installs are pinned, so the new test line now sits inside the disabled H100 block, and tests/v1/determinism/test_short_conv_batch_invariant.py is in test_tethering_allowlist.txt with the same reason as the other determinism tests. It will run again when that step is re-enabled.

[yewentao256]

Thanks for the work! I don't think we need these added tests as they can pass in main.

[Manfredss]

Thanks, agreed. f37ca2b removes it, along with its line in the disabled H100 block and its tethering allowlist entry. The PR is now just the backend opt-in plus the docs line.

[yewentao256]

LGTM, thanks for the work!

[github-actions]

✅ @Manfredss, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[mergify]

Hi @Manfredss, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[Manfredss]

/ci run

[github-actions]

✅ Triggered Buildkite CI #98485 for commit 5b42efce15cc.

[Manfredss]

/ci run

[github-actions]

✅ Triggered Buildkite CI #98893 for commit f45ac607dda0.

[Manfredss]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99341 for commit 42183ada5a3d.

ℹ️ This PR is 12 commits behind upstream main. Merged upstream main into this branch as 42183ada5a3d before starting CI. Pull the branch before pushing more changes.

[Manfredss]

Hi @yewentao256, all CI checks have passed. Could you take a look when you get a chance? Thanks!