Skip to content

[Model] Add Mistral Large 4 - #64534

Merged
NickLucche merged 7 commits into
vllm-project:mainfrom
NickLucche:mistral-large-4
Nov 13, 2026
Merged

NickLucche merged 7 commits into
vllm-project:mainfrom
NickLucche:mistral-large-4

Conversation

@NickLucche

Copy link
Copy Markdown
Member

Mistral Large 4 support for vLLM, shared as an open working branch. This PR is not meant to be merged as is; generic pieces are being upstreamed in separate PRs.

  • Mistral-Large-4-1T-A52B (FP8) and -NVFP4 on model runner V2
  • Sparse MLA with the DSA indexer, cross-layer KV sharing, SSMax
  • DSpark speculative decoding
  • DCP/PCP long context, NIXL P/D, HiSparse host KV offload
  • NVIDIA (Blackwell, Hopper) and AMD ROCm (MI350 Series) kernels

Recipe: https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B

AI assistance was used for this change.

@mergify

mergify Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--64534.org.readthedocs.build/en/64534/

@mergify mergify Bot added documentation Improvements or additions to documentation ci/build deepseek Related to DeepSeek models frontend mistral Related to Mistral models new-model Requests to new models quantization nvidia rocm Related to AMD ROCm labels Oct 29, 2026
@mergify mergify Bot added the cpu Related to CPU backends label Oct 29, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Oct 29, 2026
@mergify mergify Bot added dflash mrv2 Model Runner V2 specific labels Oct 29, 2026
@mergify mergify Bot added the tool-calling label Oct 29, 2026
@mergify mergify Bot added the kv-connector label Oct 29, 2026
@mergify mergify Bot added the hisparse Related to HiSparse label Oct 29, 2026
@github-project-automation github-project-automation Bot moved this to Unsorted queue in Tool Calling Oct 29, 2026
@github-project-automation github-project-automation Bot moved this to Unsorted queue in Tool Calling (arpera) Oct 29, 2026

@LucasWilkinson LucasWilkinson left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines +1334 to +1336

if self.is_aiter_triton_fp4_bmm_enabled:
mqa_ql_nope = rocm_aiter_ops.batched_gemm_a16wfp4(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current merge conflict is with #64560 and it looks like moving the fused_q_heads inlines from #64560 around this if branch makes sense.

@NickLucche

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102097 for commit 9ef6499d932d.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 9ef6499d932d before starting CI. Pull the branch before pushing more changes.

@khluu khluu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI stamp

@mergify

mergify Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@NickLucche

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102470 for commit 5e3a68ac9014.

@NickLucche

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ CI is already running for this commit: https://buildkite.com/vllm/ci/builds/102470

NickLucche and others added 7 commits November 13, 2026 13:07
Add Mistral Large 4 (mistralai/Mistral-Large-4-1T-A52B and the NVFP4
checkpoint) on model runner V2, with sparse MLA attention, cross-layer KV
sharing, the DSpark speculative decoding drafter, DCP/PCP long-context
support, NIXL P/D, HiSparse host KV offload, and NVIDIA and ROCm kernels.

Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Felix Marty <Felix.Marty@amd.com>
Co-authored-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Andy Luo <andy.luo@amd.com>
Co-authored-by: Julien Denize <julien.denize@mistral.ai>
Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
Co-authored-by: Julien Debache <jdebache@nvidia.com>
Co-authored-by: vanshil shah <49670736+vanshilshah97@users.noreply.github.com>
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Move vllm/models/mistral_large4 to vllm/models/mistral/mistral_large4 so the
Mistral Large 4 model sits next to the shared Mistral building blocks
(vllm/models/mistral/{common,nvidia,amd}). The inner layout (platform
dispatch __init__, nvidia/, amd/) is unchanged.

- Update absolute imports, the two model registry entries
  (MistralLarge4ForCausalLM, MistralLarge4DSparkModel), the ruff per-file
  ignore for specialized_decode/mono, and the README test path.
- Move tests/models/mistral_large4/amd_specialized_decode to
  tests/models/mistral/mistral_large4/amd_specialized_decode to mirror the
  package, and fix imports in the ML4 DSpark, modelopt and quark tests.
- No compatibility shim at the old path: nothing in the repo or the
  recipes loads the model through it.

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
The ROCm specialized decode kernel was adapted from ROCm/ATOM's GLM-5.2
mono kernel and kept its identifiers and docstrings, although every
geometry value is ML4's. Rename GLM5_*/glm5_*/Glm5MonoKernel to ML4 names
and reword the comments; keep the provenance lines in the README, the
vendored mono/__init__.py header and the FlyDSL port note. No functional
change.

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
- HY-V4: accept and forward the new topk_length argument in its
  _fp8_flash_mla_kernel override (the DCP decode path now passes it).
- ROCm specialized decode: vLLM SPDX headers (MIT code from ROCm/ATOM
  keeps its attribution, as for the DeepSeek-V4 AMD mono code),
  regex instead of re, torch.accelerator instead of torch.cuda, typo
  fixes and targeted type-ignores for mypy in the vendored kernel code.
- Buildkite: run tests/models/mistral on B200 and MI355.

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
test_hf_registry_coverage and test_registry_imports require an example
entry for every registered architecture.

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Use file-level ruff noqa comments in the vendored FlyDSL mono kernel
instead of a pyproject.toml per-file ignore, so the change no longer
touches pyproject.toml.

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
fused_indexer_q_rope_quant asserts a CUDA platform, so the use_rope=True
cases of test_indexer_q failed on MI355.

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
@NickLucche

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #102482 for commit 0794c2135e96.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends deepseek Related to DeepSeek models dflash documentation Improvements or additions to documentation frontend hisparse Related to HiSparse kv-cache-manager kv-connector mistral Related to Mistral models mrv2 Model Runner V2 specific new-model Requests to new models nixl NIXL KV connector / multi-tiered KV offloading (KVCR) nvidia quantization ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm speculative-decoding structured-output tool-calling

Projects

Status: Done
Status: Done
Status: Done
Status: Done
Status: Unsorted queue

Development

Successfully merging this pull request may close these issues.

8 participants