Repository navigation
chore(vllm): bump vLLM to 0.27.1 - #13059
Conversation
Update the Python dependency, runtime images, examples, compatibility surfaces, and XPU baseline SBOM for vLLM 0.27.1. Keep vLLM-Omni pinned to v0.26.0rc1 for a later upstream-aligned bump. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Fall back from the removed nixl_connector module to v1.nixl without hiding dependency import failures. Add regression coverage for scratch and real KV-cache registration paths. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
|
🎯 Code Coverage (details) 🔗 Commit SHA: bac9646 | Docs | Datadog PR Page | Give us feedback! |
Adapt Dynamo to vLLM's engine-backed Mistral parser and explicit multimodal hashing API. Repair XPU SYCL header discovery, the emulated multi-node topology, optional GMS test collection, and generated API references. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
vLLM 0.27 warms additional MRV2 scheduler and kernel specializations before serving. XPU CI measured 92-105 seconds of startup warmup, exceeding the previous 110-second whole-test limit. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…7.1__DYN-3836 Signed-off-by: Julien Darve <jdarve@NVIDIA.com> # Conflicts: # docs/fern/pages/reference/api/python/frontend.mdx
…7.1__DYN-3836 Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
vLLM 0.27.1 release images may resolve Transformers 5.15.0, whose heterogeneous Gemma 4 configs require vLLM changes that missed the release. Pin 5.14.1 after optional runtime layering and assert the final installed version until the upstream fix ships in a stable release. Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…7.1__DYN-3836 Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
WalkthroughThe PR updates vLLM to 0.27.1, adds stream interval validation, improves streaming parser finalization, supports relocated NIXL connectors, updates container builds and deployment references, and expands compatibility tests. ChangesvLLM integration
Estimated code review effort: 3 (Moderate) | ~30 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (1)
815-815: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMove
nullcontextto the module imports.Line 815 imports a standard-library dependency inside the test. Import
nullcontextwith the module-level standard-library imports. Keep the vLLM import local only if collection must remain resilient when vLLM is absent. As per coding guidelines, “Keep imports at the top of the file” and “always flagimportstatements inside function bodies.” As per path instructions, “keep imports at module scope except for optional-dependency test skips.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py` at line 815, Move the standard-library nullcontext import from the test body to the module-level imports in test_vllm_processor_unit.py, while keeping any local vLLM import only if needed for optional-dependency handling.Sources: Coding guidelines, Path instructions
lib/gpu_memory_service/integrations/vllm/patches.py (1)
134-144: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftMake the vLLM import boundary explicit.
The
NixlConnectorimports execute only whenpatch_register_kv_caches()runs. This hides the vLLM dependency and makes import behavior depend on call order. Move the version-selection adapter to module scope with optional-dependency handling, or document and test a deliberate lazy-import exception if CPU-only collection requires it.As per coding guidelines, Python imports must remain at module scope. As per path instructions,
.ai/python-guidelines.mdapplies this rule to vLLM compatibility and GMS connector code except optional-dependency test skips.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/gpu_memory_service/integrations/vllm/patches.py` around lines 134 - 144, Move the vLLM NixlConnector version-selection imports out of patch_register_kv_caches() and into module scope, preserving the existing legacy-module fallback and re-raising dependency ImportErrors. Expose one module-level adapter used by patch_register_kv_caches(), while retaining optional-dependency behavior without call-order-dependent imports.Sources: Coding guidelines, Path instructions
🔇 Additional comments (40)
components/src/dynamo/frontend/prepost.py (1)
646-646: LGTM!Also applies to: 665-675, 789-812, 853-853, 1012-1012, 1027-1041, 1111-1123
tests/frontend/test_prepost_mistral.py (1)
24-31: LGTM!Also applies to: 106-130
docs/fern/pages/reference/api/python/frontend.mdx (1)
23-23: LGTM!Also applies to: 35-35, 45-45, 356-356, 370-370, 382-382, 392-392, 657-657
lib/llm/src/protocols/openai/common_ext.rs (1)
90-94: LGTM!Also applies to: 152-152, 169-169, 193-193, 229-229, 244-244
lib/llm/src/protocols/openai/validate.rs (1)
221-227: LGTM!lib/llm/src/protocols/openai/chat_completions.rs (1)
539-539: LGTM!lib/llm/src/protocols/openai/completions.rs (1)
453-453: LGTM!lib/llm/tests/test_common_ext.rs (1)
4-4: LGTM!Also applies to: 32-61
components/src/dynamo/frontend/vllm_processor.py (1)
20-20: LGTM!Also applies to: 41-41, 477-485
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (2)
740-745: LGTM!Also applies to: 760-776, 811-814, 817-843
746-746: 📐 Maintainability & Code QualityConfirm
RUF012before changingmodel_fields.If Ruff flags this mutable class attribute, annotate it as
ClassVar[dict[str, object]]and add the required import.lib/gpu_memory_service/tests/test_vllm_patches.py (3)
1-30: LGTM!
33-44: LGTM!Also applies to: 54-58
61-76: LGTM!Also applies to: 78-94, 96-109
lib/gpu_memory_service/tests/test_v1_checkpoint.py (1)
11-18: LGTM!lib/gpu_memory_service/tests/test_v1_memory_manager.py (1)
7-13: LGTM!lib/gpu_memory_service/tests/test_v1_parameter_storage.py (1)
7-13: LGTM!lib/gpu_memory_service/tests/test_v1_protocol.py (1)
12-19: LGTM!lib/gpu_memory_service/tests/test_v1_server.py (1)
13-20: LGTM!lib/gpu_memory_service/tests/test_v1_weight_artifact.py (1)
13-20: LGTM!container/compliance/base_sboms/manifest.json (2)
141-150: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
⚠️ Unverified finding
Sandbox verification was unavailable.Update the CUDA compliance entries with the new
FROMimage.This change adds the XPU
v0.27.1entry, but the existingvllm/vllm-openaientries still recordv0.22.1-ubuntu2404and its old digest.container/context.yamlnow selectsv0.27.1-ubuntu2404for the CUDA runtime. The manifest does not describe the image used by the current Dockerfile. This can produce incorrect compliance and SBOM attribution. Regenerate both CUDA architecture entries and their captured digests and SBOM metadata.
165-165: LGTM!container/context.yaml (1)
69-69: LGTM!Also applies to: 81-95
container/templates/args.Dockerfile (1)
98-98: LGTM!container/templates/vllm_runtime.Dockerfile (2)
22-22: LGTM!
45-48: 🩺 Stability & AvailabilityVerify the complete XPU compiler path after removing the environment overrides.
The new comments remove
ONEAPI_ROOT,CMPLR_ROOT, and related path variables. The added check at Lines 232-236 only proves that Triton can seesycl/sycl.hppin its include roots. It does not prove that first-request JIT compilation can find the compiler executable and oneAPI shared libraries. Run an XPU image smoke test that performs a Triton compile and a representative model request.docs/fern/pages/reference/general/releases-machine-readable.mdx (1)
18-18: LGTM!components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.py (1)
421-421: LGTM!components/src/dynamo/vllm/tests/test_vllm_api_contract.py (1)
97-112: LGTM!tests/serve/launch/multi_node_tp_headless.sh (1)
9-10: LGTM!Also applies to: 40-40, 51-51
tests/serve/test_vllm_xpu.py (1)
541-543: 🚀 Performance & ScalabilityConfirm that this long-running XPU test belongs in pre-merge.
The new comment reports 92–105 seconds of startup warmup, but
guided_decodingremainspytest.mark.pre_mergeat Line 544. Keep the pre-merge marker only if guided decoding is an explicitly critical pre-merge path. Otherwise, move this configuration topost_merge. The 360-second timeout provides sufficient headroom.As per path instructions, “Only use
pre_mergescheduling marker for absolutely critical tests; preferpost_mergefor tests averaging over 60 seconds unless they guard a critical path.”Source: Path instructions
pyproject.toml (1)
63-63: LGTM!lib/sidecar/vllm/deploy/agg.yaml (1)
42-45: LGTM!lib/sidecar/vllm/deploy/disagg.yaml (1)
62-62: LGTM!Also applies to: 158-158
deploy/inference-gateway/ext-proc/examples/onramp/README.md (1)
20-20: LGTM!deploy/inference-gateway/ext-proc/examples/onramp/agg.yaml (1)
79-79: LGTM!docs/fern/assets/releases.json (1)
15-15: LGTM!docs/fern/components/releases.data.ts (1)
76-76: LGTM!docs/fern/pages/kubernetes/kv-aware-routing/vanilla-vllm-onramp.mdx (1)
247-247: LGTM!Also applies to: 263-263
docs/fern/pages/reference/general/compatibility.mdx (1)
159-159: LGTM!
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@container/templates/vllm_runtime.Dockerfile`:
- Line 236: Replace the assert-based invariant checks in the Dockerfile Python
commands, including the check around COMPILATION_HELPER.include_dir, with
explicit conditional validation that raises RuntimeError containing the existing
diagnostic details. Preserve the current conditions and failure messages, and
apply the same change to the checks at the later referenced lines so validation
remains active under Python optimization.
In `@lib/gpu_memory_service/integrations/vllm/patches.py`:
- Around line 137-148: Update the connector import fallback around the legacy
and canonical NixlConnector imports to catch only ModuleNotFoundError, allowing
other ImportError exceptions to propagate. Preserve the existing module-specific
fallback behavior and deferred optional vLLM imports so register_kv_caches is
patched only when a connector module is genuinely absent.
In `@lib/gpu_memory_service/tests/test_vllm_patches.py`:
- Around line 49-53: Update the test’s module-unavailability setup around the
monkeypatch.delitem call to assign the legacy vLLM module entry in sys.modules
to None, rather than merely removing it, so imports consistently raise
ModuleNotFoundError even when the module exists on disk.
---
Nitpick comments:
In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py`:
- Line 815: Move the standard-library nullcontext import from the test body to
the module-level imports in test_vllm_processor_unit.py, while keeping any local
vLLM import only if needed for optional-dependency handling.
In `@lib/gpu_memory_service/integrations/vllm/patches.py`:
- Around line 134-144: Move the vLLM NixlConnector version-selection imports out
of patch_register_kv_caches() and into module scope, preserving the existing
legacy-module fallback and re-raising dependency ImportErrors. Expose one
module-level adapter used by patch_register_kv_caches(), while retaining
optional-dependency behavior without call-order-dependent imports.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 28308612-59fd-41b5-9cf8-4709c4379050
📒 Files selected for processing (37)
components/src/dynamo/frontend/prepost.pycomponents/src/dynamo/frontend/tests/test_vllm_processor_unit.pycomponents/src/dynamo/frontend/vllm_processor.pycomponents/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.pycomponents/src/dynamo/vllm/tests/test_vllm_api_contract.pycontainer/compliance/base_sboms/manifest.jsoncontainer/compliance/base_sboms/vllm-openai-xpu@3309c2e1-amd64.cdx.jsoncontainer/context.yamlcontainer/templates/args.Dockerfilecontainer/templates/vllm_runtime.Dockerfiledeploy/inference-gateway/ext-proc/examples/onramp/README.mddeploy/inference-gateway/ext-proc/examples/onramp/agg.yamldocs/fern/assets/releases.jsondocs/fern/components/releases.data.tsdocs/fern/pages/kubernetes/kv-aware-routing/vanilla-vllm-onramp.mdxdocs/fern/pages/reference/api/python/frontend.mdxdocs/fern/pages/reference/general/compatibility.mdxdocs/fern/pages/reference/general/releases-machine-readable.mdxlib/gpu_memory_service/integrations/vllm/patches.pylib/gpu_memory_service/tests/test_v1_checkpoint.pylib/gpu_memory_service/tests/test_v1_memory_manager.pylib/gpu_memory_service/tests/test_v1_parameter_storage.pylib/gpu_memory_service/tests/test_v1_protocol.pylib/gpu_memory_service/tests/test_v1_server.pylib/gpu_memory_service/tests/test_v1_weight_artifact.pylib/gpu_memory_service/tests/test_vllm_patches.pylib/llm/src/protocols/openai/chat_completions.rslib/llm/src/protocols/openai/common_ext.rslib/llm/src/protocols/openai/completions.rslib/llm/src/protocols/openai/validate.rslib/llm/tests/test_common_ext.rslib/sidecar/vllm/deploy/agg.yamllib/sidecar/vllm/deploy/disagg.yamlpyproject.tomltests/frontend/test_prepost_mistral.pytests/serve/launch/multi_node_tp_headless.shtests/serve/test_vllm_xpu.py
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
Approving
All 12 findings from my earlier review are addressed, and I verified each against the code at 3dfd1cc3 rather than from the commit messages. Thanks for the thoroughness here — several of these were fixed more completely than I asked for.
On the sidecar, which was the blocking item and the one I got wrong: you took the harder option, and it is the right one. I independently recomputed both hashes:
inference.proto vendored=4e6c90467ea308bb… v0.27.1=4e6c90467ea308bb… MATCH
control.proto vendored=390c88e94f1b6842… v0.27.1=390c88e94f1b6842… MATCH
service Inference now agrees on both sides, all three manifests are on v0.27.1, and proto/README.md pins tag v0.27.1 with commit 6e448d0ea9bf3d88d898b65449ca6dc2aec170ac — I confirmed that is genuinely the v0.27.1 tag commit, so the vendored copy is now checkable against a published artifact instead of a diverged dev commit. convert.rs:374 raises invalid_argument before any RPC rather than dropping fields silently, tests.rs asserts the rejection happens before RPC submission, and the README no longer advertises media inputs. That closes the silent-multimodal-loss path completely.
Some fixes came out better than my suggestion:
- scoping
transformers<5.15.0to theai-dynamo[vllm]extra instead of the base dependency, so SGLang and TRT-LLM installs stay unconstrained; - removing
stream_intervaloutright rather than keeping a parameter that no-ops below the engine-level interval; - relying on the Omni constraints snapshot for resolver-side protection instead of a global
uv pip check— which, as your logs showed, was tripping on pre-existing conflicts in the upstream image. My "true positive" call in that thread was wrong.
Also picked up along the way: the stale manifest.json from_tag, and the flashinf_ref ARG removed together with its args.Dockerfile declaration.
CI is green at this head — 86 passing, 0 failures. xpu-ci-vllm / build-xpu is still queued; it passed on 9bbda48961 and nothing since touches XPU, so I do not consider it a risk, but worth a glance before merge.
One follow-up worth tracking separately, not blocking: sidecar multimodal is now explicitly rejected, so whatever #12735 was building toward with session_id/media needs an upstream vLLM release that carries those fields. Given nothing in CI exercises these manifests, a proto-hash check in the sidecar build would also keep the vendored copy and the deployed tag from drifting apart again — that drift is what made both the v0.26.0 service-name mismatch and this one invisible.
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
connorcarpenter15
left a comment
There was a problem hiding this comment.
Please remove the changes to the lib/sidecar folder. These changes remove functionality that I just added that will land in vLLM 0.28.0. It will also interfere with other sidecar functionality I am currently adding. I will bump the sidecar version in a separate PR.
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
connorcarpenter15
left a comment
There was a problem hiding this comment.
Thanks @JulienDarve
rmccorm4
left a comment
There was a problem hiding this comment.
Frontend finding noted inline.
rmccorm4
left a comment
There was a problem hiding this comment.
Approving for frontend, with one request for follow-up: #13059 (comment)
Signed-off-by: Julien Darve <jdarve@NVIDIA.com> Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
vllm.exceptions.VLLMClientError was introduced alongside the 0.27.1 bump (ai-dynamo#13059). Dynamo's worker error path imports it unconditionally, so any build pinned to an earlier vLLM fails at request time with ImportError and returns HTTP 500 for every generation. Fall back to a local base class when the symbol is absent, and route handlers through .errors so there is a single import site. Signed-off-by: Shubham <shugandhi@nvidia.com>
Overview
Bump Dynamo's base vLLM integration and CUDA, XPU, and CPU runtime images from 0.26.0 to 0.27.1, with the compatibility changes required by the vLLM 0.27 release line. This also fixes the GPU Memory Service NIXL connector patch that could silently land on the pull-only alias instead of the shared connector base.
vLLM-Omni is upgraded from
v0.26.0rc1tov0.27.0rc1, which rebases Omni onto the vLLM 0.27 release line. The native vLLM sidecar remains unchanged frommain; its engine image and vendored protocol will be upgraded separately so this PR does not remove its existing multimodal support.Details
v0.27.0rc1and forward tensor, pipeline, and data parallel sizes into its realDiffusionParallelConfig.5.14.1before the Omni dependency solve because vLLM 0.27.1 does not include support for the Transformers 5.15 heterogeneous Gemma 4 configuration format (upstream vLLM #49797). Constrain Omni's dependency solve to the installed protected-package versions, assert that the completed image still contains Transformers 5.14.1, and cap theai-dynamo[vllm]extra below Transformers 5.15.stream_intervalunsupported because it was not consistently propagated across Dynamo preprocessing paths. The engine-level vLLM setting remains available.SchedulerOutputKV-connector fields.NixlBaseConnectorso GMS scratch-KV deferral covers both pull and push connectors, plus normal and cross-layer KV registration. Missing optional vLLM modules remain skippable; failures inside an installed connector remain visible.mainbranch NIXL 1.3.2 pin.flashinf_ref/FLASHINF_REFmetadata, which had no installer consumer and did not change the effective FlashInfer package.lib/sidecar/vllmunchanged and defer its image/protocol upgrade to a separate PR.The upstream audit covered Dynamo's imported and duck-typed scheduler, multimodal preprocessing, KV events, ZMQ framing, request validation, parser contracts, and integration-test surfaces. The vLLM 0.27.1 patch introduces no additional Dynamo API drift beyond the 0.27.0 compatibility work. The separate Omni audit verified the published
v0.27.0rc1constructor and configuration surfaces used by Dynamo.XPU scope and baseline SBOM
The XPU image is intentionally upgraded in this PR so Dynamo's CUDA, CPU, and XPU vLLM runtimes remain on the same release. Earlier core vLLM bumps generally left XPU unchanged and updated it in a follow-up; the like-for-like precedent is #12190, whose vLLM XPU 0.26.0 baseline replacement was larger (+36,472/-68,026 lines) than this one.
Upstream vLLM 0.27.1 substantially changed the XPU image: it removed the apt-installed oneAPI 2025.3 compiler/MKL/DNNL toolkits and offline oneCCL installation, moved runtime libraries to
/opt/venv/lib, and adopted PyTorch 2.13 plus Intel 2026 runtime wheels. Correspondingly, the baseline falls from 22,477 to 16,597 components; 5,594 removed file records are under/opt/intel/oneapi/. The textual diff is further inflated by Syft 1.49-to-1.51bom-refchurn: 15,736 of 15,744 common component identities are otherwise unchanged. The new digest and layer-prefix invariant pass the live baseline drift check.Where should the reviewer start?
components/src/dynamo/frontend/prepost.py,components/src/dynamo/frontend/vllm_processor.py, andcomponents/src/dynamo/vllm/handlers.pyfor parser and request-error compatibility.components/src/dynamo/vllm/omni/base_handler.pyand its focused tests for Omni topology forwarding.lib/gpu_memory_service/integrations/vllm/patches.pyandworker.pyfor the NIXL base-connector fix and wake replay.container/templates/vllm_runtime.Dockerfile,container/context.yaml,pyproject.toml, and the compliance manifest for dependency, image, and SBOM changes.Related Issues
This PR is not linked to a GitHub issue:
Validation
cargo fmt --all -- --checkcargo test -p dynamo-llm --test test_common_ext— 13 passedv0.26.0rc1tov0.27.0rc1) and published-wheel metadata verificationcargo test -p dynamo-vllm-sidecar— 20 passed; Clippy passed for all targets with warnings denied;lib/sidecar/vllmhas zero net diff from currentmaingit diff --checkmainmerged; replacement pre-merge CI is running for the current headRepository-wide container and hardware validation remains delegated to replacement pre-merge CI.