Repository navigation
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
|
/ci run |
|
✅ @kainoj, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #103406 for commit ℹ️ This PR is 7 commits behind upstream |
|
/ci run |
|
❌ CI already ran on this commit: https://buildkite.com/vllm/ci/builds/103406. No new CI build was started. Push new changes to run CI again, use |
|
/ci run |
|
✅ Triggered Buildkite CI #103514 for commit |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #103514. |
…ache_config vllm-project#61452 replaced AttentionLayerBase.get_kv_cache_spec with the abstract get_kv_cache_config. vllm-project#64534 (Mistral Large 4) merged after it and implements only get_kv_cache_spec, so on NVIDIA GPUs every worker fails at model init. vllm-project#67984 fixed the AMD copy; this makes the same change to the NVIDIA one, and test_indexer_k_cache_write now builds the real MistralIndexerCache on CUDA too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Przemyslaw <kainoj@users.noreply.github.com>
a8f48a2 to
449ba45
Compare
|
/ci retry |
|
✅ Triggered Buildkite CI #103682 for commit |
Overview
Mistral Large 4 fails to start on NVIDIA GPUs on
main: every worker crashes at model init because the NVIDIAMistralIndexerCacheis still abstract. #67984 fixed the AMD copy of this class. This PR makes the same change to the NVIDIA copy.Claims
TypeError: Can't instantiate abstract class MistralIndexerCache without an implementation for abstract method 'get_kv_cache_config'for Mistral Large 4.KVCacheLayerConfigunderFullAttentionManager, asDeepseekV32IndexerCacheand the AMDMistralIndexerCachedo.test_indexer_k_cache_writenow builds the realMistralIndexerCacheon CUDA too, so the:nvidia: (B200) Mistral Large 4CI step catches this kind of break.Validation
Regression test:
test_indexer_k_cache_writeintests/models/mistral/test_fused_ops.pynow checks on CUDA, as it already did on ROCm, that the page the model'sMistralIndexerCache.get_kv_cache_config()requests matches the cache the kernels are tested on. We had no free NVIDIA GPU to run the test itself. We ran its new check (the same calls, without the CUDA kernel part) on CPU in thevllm/vllm-openai:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27dimage:Without fix
With fix
The CPU check
ruff check,ruff format --check(v0.17.2) andtypos(v1.51.1) pass on both changed files.End-to-end on 8x NVIDIA B300, TP=8, image
nightly-60bd6dd2(both runs also carry the parser-only #63161):vllm serve mistralai/Mistral-Large-4-1T-A52B-NVFP4 \ --tokenizer-mode mistral --config-format mistral --load-format mistral \ --tensor-parallel-size 8 --max-model-len 262144 \ --enable-auto-tool-choice --tool-call-parser mistral --reasoning-parser mistral \ --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}'Our command also sets sampling defaults, batching limits and KV offloading. None of them touch this code.
Without fix: all 8 workers fail at model init
With fix: the server starts and serves requests
Details
Root cause: a merge race. #61452 (merged 2026-11-13 04:47 UTC) made
get_kv_cache_configthe abstract KV-cache hook onAttentionLayerBaseand moved every existing layer to it. #64534 (merged 2026-11-13 15:05 UTC) was written against the old interface. Nothing in vLLM callsget_kv_cache_specany more, so the old method was dead code and the class could not be created.Fix: rename the method in
vllm/models/mistral/nvidia/indexer.pyand returnFullAttentionManager.layer_config(spec), asDeepseekV32IndexerCacheand #67984 do.Test: the page-size check in
test_indexer_k_cache_writemoves out of the ROCm-only branch, so it runs on both platforms. The cache readers check stays ROCm-only.CI: the test runs in the existing
:nvidia: (B200) Mistral Large 4step (pytest -v -s models/mistral). CI cannot run the full 1T-parameter model, so end-to-end coverage is the manual run above.Not a duplicate: no open issue or PR covers the NVIDIA file. #67984 changed only the AMD file, and the open #68061 does not touch the NVIDIA indexer file.
AI assistance (Claude Code) was used to find the cause, write the change and write this description. I reviewed every changed line.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.