Skip to content

[Bugfix][Model] Port Mistral Large 4 NVIDIA indexer cache to get_kv_cache_config - #68103

Open
kainoj wants to merge 1 commit into
vllm-project:mainfrom
kainoj:mistral-nvidia-indexer-kv-cache-config
Open

kainoj wants to merge 1 commit into
vllm-project:mainfrom
kainoj:mistral-nvidia-indexer-kv-cache-config

Conversation

@kainoj

@kainoj kainoj commented Nov 17, 2026 •

Copy link
Copy Markdown

Overview

Mistral Large 4 fails to start on NVIDIA GPUs on main: every worker crashes at model init because the NVIDIA MistralIndexerCache is still abstract. #67984 fixed the AMD copy of this class. This PR makes the same change to the NVIDIA copy.

Claims

  • Fixes the NVIDIA startup crash TypeError: Can't instantiate abstract class MistralIndexerCache without an implementation for abstract method 'get_kv_cache_config' for Mistral Large 4.
  • No other change in behavior: the indexer cache spec is the same as before, now wrapped in a KVCacheLayerConfig under FullAttentionManager, as DeepseekV32IndexerCache and the AMD MistralIndexerCache do.
  • test_indexer_k_cache_write now builds the real MistralIndexerCache on CUDA too, so the :nvidia: (B200) Mistral Large 4 CI step catches this kind of break.

Validation

Regression test: test_indexer_k_cache_write in tests/models/mistral/test_fused_ops.py now checks on CUDA, as it already did on ROCm, that the page the model's MistralIndexerCache.get_kv_cache_config() requests matches the cache the kernels are tested on. We had no free NVIDIA GPU to run the test itself. We ran its new check (the same calls, without the CUDA kernel part) on CPU in the vllm/vllm-openai:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d image:

Without fix
TypeError: Can't instantiate abstract class MistralIndexerCache without an implementation for abstract method 'get_kv_cache_config'
With fix
block_size=1: page_size_bytes=132 ok
block_size=16: page_size_bytes=2112 ok
block_size=64: page_size_bytes=8448 ok
The CPU check
import torch
from vllm.config import (CacheConfig, CompilationConfig, CompilationMode, DeviceConfig,
                         VllmConfig, set_current_vllm_config)
from vllm.models.mistral.nvidia.indexer import MistralIndexerCache

head_dim = 128
for block_size in (1, 16, 64):
    vllm_config = VllmConfig(device_config=DeviceConfig(device="cpu"), compilation_config=CompilationConfig(mode=CompilationMode.NONE))
    with set_current_vllm_config(vllm_config):
        k_cache = MistralIndexerCache(
            head_dim=head_dim + 4,
            dtype=torch.uint8,
            prefix="indexer.k_cache",
            cache_config=CacheConfig(block_size=block_size),
        )
    spec = k_cache.get_kv_cache_config(vllm_config).spec
    cache_page = torch.zeros(block_size, head_dim + 4, dtype=torch.uint8)
    assert spec.page_size_bytes == cache_page.numel() * cache_page.element_size()
    print(f"block_size={block_size}: page_size_bytes={spec.page_size_bytes} ok")

ruff check, ruff format --check (v0.17.2) and typos (v1.51.1) pass on both changed files.

End-to-end on 8x NVIDIA B300, TP=8, image nightly-60bd6dd2 (both runs also carry the parser-only #63161):

vllm serve mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --tensor-parallel-size 8 --max-model-len 262144 \
  --enable-auto-tool-choice --tool-call-parser mistral --reasoning-parser mistral \
  --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'

Our command also sets sampling defaults, batching limits and KV offloading. None of them touch this code.

Without fix: all 8 workers fail at model init
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091]   File ".../vllm/models/mistral/mistral_large4/nvidia/model.py", line 86, in __init__
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091]     self.self_attn = MistralMLAAttention(
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091]   File ".../vllm/models/mistral/nvidia/attention.py", line 129, in __init__
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091]     MistralIndexer(
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091]   File ".../vllm/models/mistral/nvidia/indexer.py", line 146, in __init__
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091]     self.k_cache = MistralIndexerCache(
(Worker_TP1 pid=531) ERROR [multiproc_executor.py:1091] TypeError: Can't instantiate abstract class MistralIndexerCache without an implementation for abstract method 'get_kv_cache_config'
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
With fix: the server starts and serves requests
(Worker_TP0 pid=923) INFO [model_runner.py:449] Model loading took 76.79 GiB memory and 662.359565 seconds
(EngineCore pid=709) INFO [kv_cache_utils.py:1134] GPU KV cache size: 3,336,320 tokens, Maximum concurrency for 262,144 tokens per request: 12.73x
(EngineCore pid=709) INFO [core.py:443] init engine (profile, create kv cache, warmup model) took 784.09 s
INFO:     Application startup complete.
(APIServer pid=1) INFO [loggers.py:339] Engine 000: Avg prompt throughput: 2.3 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, Finished: 1 reqs, ...
(APIServer pid=1) INFO [metrics.py:135] SpecDecoding metrics: Mean acceptance length: 3.39, ...

Details

Root cause: a merge race. #61452 (merged 2026-11-13 04:47 UTC) made get_kv_cache_config the abstract KV-cache hook on AttentionLayerBase and moved every existing layer to it. #64534 (merged 2026-11-13 15:05 UTC) was written against the old interface. Nothing in vLLM calls get_kv_cache_spec any more, so the old method was dead code and the class could not be created.

Fix: rename the method in vllm/models/mistral/nvidia/indexer.py and return FullAttentionManager.layer_config(spec), as DeepseekV32IndexerCache and #67984 do.

Test: the page-size check in test_indexer_k_cache_write moves out of the ROCm-only branch, so it runs on both platforms. The cache readers check stays ROCm-only.

CI: the test runs in the existing :nvidia: (B200) Mistral Large 4 step (pytest -v -s models/mistral). CI cannot run the full 1T-parameter model, so end-to-end coverage is the manual run above.

Not a duplicate: no open issue or PR covers the NVIDIA file. #67984 changed only the AMD file, and the open #68061 does not touch the NVIDIA indexer file.

AI assistance (Claude Code) was used to find the cause, write the change and write this description. I reviewed every changed line.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@mergify mergify Bot added mistral Related to Mistral models nvidia bug Something isn't working labels Nov 17, 2026
@kainoj
kainoj marked this pull request as ready for review November 17, 2026 07:18

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 17, 2026
@DarkLight1337

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ @kainoj, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103406 for commit 97e5a0d0492d.

ℹ️ This PR is 7 commits behind upstream main. Merged upstream main into this branch as 97e5a0d0492d before starting CI. Pull the branch before pushing more changes.

@kainoj

kainoj commented Nov 17, 2026

Copy link
Copy Markdown
Author

/ci run

@github-actions

Copy link
Copy Markdown

❌ CI already ran on this commit: https://buildkite.com/vllm/ci/builds/103406. No new CI build was started. Push new changes to run CI again, use /ci retry to rerun failed jobs, or ask a reviewer with write access to rerun it.

@kainoj

kainoj commented Nov 17, 2026

Copy link
Copy Markdown
Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103514 for commit a8f48a27a24f.

@kainoj

kainoj commented Nov 17, 2026

Copy link
Copy Markdown
Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 1 failed job(s) for retry in Buildkite CI #103514.

…ache_config

vllm-project#61452 replaced AttentionLayerBase.get_kv_cache_spec with the abstract
get_kv_cache_config. vllm-project#64534 (Mistral Large 4) merged after it and
implements only get_kv_cache_spec, so on NVIDIA GPUs every worker fails at
model init. vllm-project#67984 fixed the AMD copy; this makes the same change to the
NVIDIA one, and test_indexer_k_cache_write now builds the real
MistralIndexerCache on CUDA too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Przemyslaw <kainoj@users.noreply.github.com>
@kainoj
kainoj force-pushed the mistral-nvidia-indexer-kv-cache-config branch from a8f48a2 to 449ba45 Compare November 18, 2026 01:55
@kainoj

kainoj commented Nov 18, 2026

Copy link
Copy Markdown
Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #103682 for commit 449ba454de10, running 1 failed step(s) from Buildkite CI #103514.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working mistral Related to Mistral models nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants