Skip to content

chore(vllm): bump vLLM to 0.27.1 - #13059

Merged
JulienDarve merged 31 commits into
mainfrom
jdarve/vllm-bump-v0.27.1__DYN-3836
Aug 18, 2026
Merged

JulienDarve merged 31 commits into
mainfrom
jdarve/vllm-bump-v0.27.1__DYN-3836

Conversation

@JulienDarve

@JulienDarve JulienDarve commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Bump Dynamo's base vLLM integration and CUDA, XPU, and CPU runtime images from 0.26.0 to 0.27.1, with the compatibility changes required by the vLLM 0.27 release line. This also fixes the GPU Memory Service NIXL connector patch that could silently land on the pull-only alias instead of the shared connector base.

vLLM-Omni is upgraded from v0.26.0rc1 to v0.27.0rc1, which rebases Omni onto the vLLM 0.27 release line. The native vLLM sidecar remains unchanged from main; its engine image and vendored protocol will be upgraded separately so this PR does not remove its existing multimodal support.

Details

  • Update the base vLLM dependency and CUDA, XPU, and CPU runtime images to 0.27.1.
  • Upgrade vLLM-Omni to v0.27.0rc1 and forward tensor, pipeline, and data parallel sizes into its real DiffusionParallelConfig.
  • Pin Transformers to 5.14.1 before the Omni dependency solve because vLLM 0.27.1 does not include support for the Transformers 5.15 heterogeneous Gemma 4 configuration format (upstream vLLM #49797). Constrain Omni's dependency solve to the installed protected-package versions, assert that the completed image still contains Transformers 5.14.1, and cap the ai-dynamo[vllm] extra below Transformers 5.15.
  • Preserve vLLM's HTTP 400/404/422 client-error distinction across frontend preprocessing, frontend streaming, and decode/prefill worker paths.
  • Align reasoning and tool-parser lifecycle behavior with vLLM 0.27, including independent mutable parsers for parallel choices, prompt-derived reasoning state, and safe engine-parser finalization.
  • Keep per-request stream_interval unsupported because it was not consistently propagated across Dynamo preprocessing paths. The engine-level vLLM setting remains available.
  • Guard the new optional SchedulerOutput KV-connector fields.
  • Patch NixlBaseConnector so GMS scratch-KV deferral covers both pull and push connectors, plus normal and cross-layer KV registration. Missing optional vLLM modules remain skippable; failures inside an installed connector remain visible.
  • Preserve the current main branch NIXL 1.3.2 pin.
  • Remove the inert flashinf_ref/FLASHINF_REF metadata, which had no installer consumer and did not change the effective FlashInfer package.
  • Leave lib/sidecar/vllm unchanged and defer its image/protocol upgrade to a separate PR.
  • Update deployment examples, compatibility metadata, and the XPU baseline SBOM for the new runtime.

The upstream audit covered Dynamo's imported and duck-typed scheduler, multimodal preprocessing, KV events, ZMQ framing, request validation, parser contracts, and integration-test surfaces. The vLLM 0.27.1 patch introduces no additional Dynamo API drift beyond the 0.27.0 compatibility work. The separate Omni audit verified the published v0.27.0rc1 constructor and configuration surfaces used by Dynamo.

XPU scope and baseline SBOM

The XPU image is intentionally upgraded in this PR so Dynamo's CUDA, CPU, and XPU vLLM runtimes remain on the same release. Earlier core vLLM bumps generally left XPU unchanged and updated it in a follow-up; the like-for-like precedent is #12190, whose vLLM XPU 0.26.0 baseline replacement was larger (+36,472/-68,026 lines) than this one.

Upstream vLLM 0.27.1 substantially changed the XPU image: it removed the apt-installed oneAPI 2025.3 compiler/MKL/DNNL toolkits and offline oneCCL installation, moved runtime libraries to /opt/venv/lib, and adopted PyTorch 2.13 plus Intel 2026 runtime wheels. Correspondingly, the baseline falls from 22,477 to 16,597 components; 5,594 removed file records are under /opt/intel/oneapi/. The textual diff is further inflated by Syft 1.49-to-1.51 bom-ref churn: 15,736 of 15,744 common component identities are otherwise unchanged. The new digest and layer-prefix invariant pass the live baseline drift check.

Where should the reviewer start?

  • components/src/dynamo/frontend/prepost.py, components/src/dynamo/frontend/vllm_processor.py, and components/src/dynamo/vllm/handlers.py for parser and request-error compatibility.
  • components/src/dynamo/vllm/omni/base_handler.py and its focused tests for Omni topology forwarding.
  • lib/gpu_memory_service/integrations/vllm/patches.py and worker.py for the NIXL base-connector fix and wake replay.
  • container/templates/vllm_runtime.Dockerfile, container/context.yaml, pyproject.toml, and the compliance manifest for dependency, image, and SBOM changes.

Related Issues

This PR is not linked to a GitHub issue:

  • Confirmed — no related GitHub issue
  • Internal tracking: DYN-3836. DYN-3837, which originally tracked the GMS import bug separately, is marked as a duplicate of DYN-3836.

Validation

  • Dynamo pre-commit and commit-message/DCO hooks passed for each review-fix commit and the merge commit.
  • cargo fmt --all -- --check
  • cargo test -p dynamo-llm --test test_common_ext — 13 passed
  • Focused GMS NIXL base/pull/push/cross-layer regression test — 1 passed
  • vLLM 0.27.1 API-contract tests in the upstream XPU image — 16 passed
  • CUDA 13.0, CUDA-EFA, CPU, and XPU runtime/development Dockerfile rendering, including Transformers-before-Omni ordering and the final target-specific Transformers version assertion
  • BuildKit no-build validation for the rendered CUDA and XPU runtime Dockerfiles
  • Real dependency-layer simulation in the upstream XPU image: vLLM 0.27.1+xpu, vLLM-Omni 0.27.0rc1, and final Transformers 5.14.1
  • Fresh-source vLLM-Omni API audit (v0.26.0rc1 to v0.27.0rc1) and published-wheel metadata verification
  • Affected Python vLLM compatibility tests in an isolated vLLM 0.27.1 / vLLM-Omni 0.27.0rc1 environment — 223 passed
  • Generated API-reference test suite — 205 passed; generated-reference freshness and Fern local validation passed
  • Sidecar rollback verification: cargo test -p dynamo-vllm-sidecar — 20 passed; Clippy passed for all targets with warnings denied; lib/sidecar/vllm has zero net diff from current main
  • git diff --check
  • Commit GPG signatures and DCO signoffs verified
  • Current main merged; replacement pre-merge CI is running for the current head

Repository-wide container and hardware validation remains delegated to replacement pre-merge CI.

Update the Python dependency, runtime images, examples, compatibility surfaces, and XPU baseline SBOM for vLLM 0.27.1.

Keep vLLM-Omni pinned to v0.26.0rc1 for a later upstream-aligned bump.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Fall back from the removed nixl_connector module to v1.nixl without hiding dependency import failures.

Add regression coverage for scratch and real KV-cache registration paths.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
@github-actions github-actions Bot added chore documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` container xpu labels Aug 11, 2026
@datadog-official

datadog-official Bot commented Aug 11, 2026 •

Copy link
Copy Markdown

Pipelines

🎯 Code Coverage (details)
• Patch Coverage: 89.11%
• Overall Coverage: 53.18% (+0.78%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bac9646 | Docs | Datadog PR Page | Give us feedback!

@JulienDarve
JulienDarve requested review from a team August 11, 2026 23:23
Adapt Dynamo to vLLM's engine-backed Mistral parser and explicit multimodal hashing API. Repair XPU SYCL header discovery, the emulated multi-node topology, optional GMS test collection, and generated API references.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
@github-actions

github-actions Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

vLLM 0.27 warms additional MRV2 scheduler and kernel specializations before serving. XPU CI measured 92-105 seconds of startup warmup, exceeding the previous 110-second whole-test limit.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…7.1__DYN-3836

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>

# Conflicts:
#	docs/fern/pages/reference/api/python/frontend.mdx
…7.1__DYN-3836

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
vLLM 0.27.1 release images may resolve Transformers 5.15.0, whose heterogeneous Gemma 4 configs require vLLM changes that missed the release. Pin 5.14.1 after optional runtime layering and assert the final installed version until the upstream fix ships in a stable release.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
…7.1__DYN-3836

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
@JulienDarve
JulienDarve marked this pull request as ready for review August 12, 2026 21:24
@JulienDarve
JulienDarve requested review from a team as code owners August 12, 2026 21:24

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 5 potential issues.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The PR updates vLLM to 0.27.1, adds stream interval validation, improves streaming parser finalization, supports relocated NIXL connectors, updates container builds and deployment references, and expands compatibility tests.

Changes

vLLM integration

Layer / File(s) Summary
Streaming parser lifecycle
components/src/dynamo/frontend/prepost.py, tests/frontend/test_prepost_mistral.py, docs/fern/pages/reference/api/python/frontend.mdx
Streaming postprocessing tracks parser state, merges parser deltas, detects reasoning completion, and flushes active parsers when output ends. Mistral tests use ReasoningParserManager and a tokenizer decoder.
Stream interval contract and error handling
lib/llm/src/protocols/openai/*, lib/llm/tests/test_common_ext.rs, components/src/dynamo/frontend/vllm_processor.py, components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
Requests support optional stream_interval values and reject zero. vLLM client validation errors become Dynamo InvalidArgument errors. Tests verify propagation and error messages.
vLLM 0.27.1 runtime and release pins
container/*, pyproject.toml, lib/sidecar/vllm/deploy/*, deploy/inference-gateway/ext-proc/examples/onramp/*, docs/fern/assets/releases.json, docs/fern/components/releases.data.ts, docs/fern/pages/kubernetes/kv-aware-routing/vanilla-vllm-onramp.mdx, docs/fern/pages/reference/general/*
Runtime images, optional dependencies, SBOM metadata, deployment examples, and release references now use vLLM 0.27.1. The runtime pins and verifies the configured Transformers version and checks XPU SYCL headers.
GPU memory service connector compatibility
lib/gpu_memory_service/integrations/vllm/patches.py, lib/gpu_memory_service/tests/*
NIXL connector loading supports legacy and relocated modules. Tests skip when the dependency is unavailable and cover scratch, real-backed, and failed KV-cache registration.
vLLM compatibility and environment validation
components/src/dynamo/vllm/tests/*, tests/serve/launch/multi_node_tp_headless.sh, tests/serve/test_vllm_xpu.py
Compatibility tests use explicit BLAKE3 hashing and preserve optional scheduler fields. Multi-node tests disable custom all-reduce, and the XPU test allows longer vLLM 0.27 startup warmup.

Estimated code review effort: 3 (Moderate) | ~30 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.17% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The description explicitly confirms that this pull request has no related GitHub issue and identifies internal tracking.
Out of Scope Changes check ✅ Passed The listed changes support the vLLM 0.27.1 upgrade, compatibility work, runtime updates, documentation, and regression testing.
Title check ✅ Passed The title clearly summarizes the primary change: upgrading the vLLM integration to version 0.27.1.
Description check ✅ Passed The description covers all required template sections and provides detailed scope, reviewer guidance, issue status, and validation results.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (1)

815-815: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Move nullcontext to the module imports.

Line 815 imports a standard-library dependency inside the test. Import nullcontext with the module-level standard-library imports. Keep the vLLM import local only if collection must remain resilient when vLLM is absent. As per coding guidelines, “Keep imports at the top of the file” and “always flag import statements inside function bodies.” As per path instructions, “keep imports at module scope except for optional-dependency test skips.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py` at line
815, Move the standard-library nullcontext import from the test body to the
module-level imports in test_vllm_processor_unit.py, while keeping any local
vLLM import only if needed for optional-dependency handling.

Sources: Coding guidelines, Path instructions

lib/gpu_memory_service/integrations/vllm/patches.py (1)

134-144: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Make the vLLM import boundary explicit.

The NixlConnector imports execute only when patch_register_kv_caches() runs. This hides the vLLM dependency and makes import behavior depend on call order. Move the version-selection adapter to module scope with optional-dependency handling, or document and test a deliberate lazy-import exception if CPU-only collection requires it.

As per coding guidelines, Python imports must remain at module scope. As per path instructions, .ai/python-guidelines.md applies this rule to vLLM compatibility and GMS connector code except optional-dependency test skips.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/gpu_memory_service/integrations/vllm/patches.py` around lines 134 - 144,
Move the vLLM NixlConnector version-selection imports out of
patch_register_kv_caches() and into module scope, preserving the existing
legacy-module fallback and re-raising dependency ImportErrors. Expose one
module-level adapter used by patch_register_kv_caches(), while retaining
optional-dependency behavior without call-order-dependent imports.

Sources: Coding guidelines, Path instructions

🔇 Additional comments (40)
components/src/dynamo/frontend/prepost.py (1)

646-646: LGTM!

Also applies to: 665-675, 789-812, 853-853, 1012-1012, 1027-1041, 1111-1123

tests/frontend/test_prepost_mistral.py (1)

24-31: LGTM!

Also applies to: 106-130

docs/fern/pages/reference/api/python/frontend.mdx (1)

23-23: LGTM!

Also applies to: 35-35, 45-45, 356-356, 370-370, 382-382, 392-392, 657-657

lib/llm/src/protocols/openai/common_ext.rs (1)

90-94: LGTM!

Also applies to: 152-152, 169-169, 193-193, 229-229, 244-244

lib/llm/src/protocols/openai/validate.rs (1)

221-227: LGTM!

lib/llm/src/protocols/openai/chat_completions.rs (1)

539-539: LGTM!

lib/llm/src/protocols/openai/completions.rs (1)

453-453: LGTM!

lib/llm/tests/test_common_ext.rs (1)

4-4: LGTM!

Also applies to: 32-61

components/src/dynamo/frontend/vllm_processor.py (1)

20-20: LGTM!

Also applies to: 41-41, 477-485

components/src/dynamo/frontend/tests/test_vllm_processor_unit.py (2)

740-745: LGTM!

Also applies to: 760-776, 811-814, 817-843


746-746: 📐 Maintainability & Code Quality

Confirm RUF012 before changing model_fields.

If Ruff flags this mutable class attribute, annotate it as ClassVar[dict[str, object]] and add the required import.

lib/gpu_memory_service/tests/test_vllm_patches.py (3)

1-30: LGTM!


33-44: LGTM!

Also applies to: 54-58


61-76: LGTM!

Also applies to: 78-94, 96-109

lib/gpu_memory_service/tests/test_v1_checkpoint.py (1)

11-18: LGTM!

lib/gpu_memory_service/tests/test_v1_memory_manager.py (1)

7-13: LGTM!

lib/gpu_memory_service/tests/test_v1_parameter_storage.py (1)

7-13: LGTM!

lib/gpu_memory_service/tests/test_v1_protocol.py (1)

12-19: LGTM!

lib/gpu_memory_service/tests/test_v1_server.py (1)

13-20: LGTM!

lib/gpu_memory_service/tests/test_v1_weight_artifact.py (1)

13-20: LGTM!

container/compliance/base_sboms/manifest.json (2)

141-150: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

⚠️ Unverified finding
Sandbox verification was unavailable.

Update the CUDA compliance entries with the new FROM image.

This change adds the XPU v0.27.1 entry, but the existing vllm/vllm-openai entries still record v0.22.1-ubuntu2404 and its old digest. container/context.yaml now selects v0.27.1-ubuntu2404 for the CUDA runtime. The manifest does not describe the image used by the current Dockerfile. This can produce incorrect compliance and SBOM attribution. Regenerate both CUDA architecture entries and their captured digests and SBOM metadata.


165-165: LGTM!

container/context.yaml (1)

69-69: LGTM!

Also applies to: 81-95

container/templates/args.Dockerfile (1)

98-98: LGTM!

container/templates/vllm_runtime.Dockerfile (2)

22-22: LGTM!


45-48: 🩺 Stability & Availability

Verify the complete XPU compiler path after removing the environment overrides.

The new comments remove ONEAPI_ROOT, CMPLR_ROOT, and related path variables. The added check at Lines 232-236 only proves that Triton can see sycl/sycl.hpp in its include roots. It does not prove that first-request JIT compilation can find the compiler executable and oneAPI shared libraries. Run an XPU image smoke test that performs a Triton compile and a representative model request.

docs/fern/pages/reference/general/releases-machine-readable.mdx (1)

18-18: LGTM!

components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.py (1)

421-421: LGTM!

components/src/dynamo/vllm/tests/test_vllm_api_contract.py (1)

97-112: LGTM!

tests/serve/launch/multi_node_tp_headless.sh (1)

9-10: LGTM!

Also applies to: 40-40, 51-51

tests/serve/test_vllm_xpu.py (1)

541-543: 🚀 Performance & Scalability

Confirm that this long-running XPU test belongs in pre-merge.

The new comment reports 92–105 seconds of startup warmup, but guided_decoding remains pytest.mark.pre_merge at Line 544. Keep the pre-merge marker only if guided decoding is an explicitly critical pre-merge path. Otherwise, move this configuration to post_merge. The 360-second timeout provides sufficient headroom.

As per path instructions, “Only use pre_merge scheduling marker for absolutely critical tests; prefer post_merge for tests averaging over 60 seconds unless they guard a critical path.”

Source: Path instructions

pyproject.toml (1)

63-63: LGTM!

lib/sidecar/vllm/deploy/agg.yaml (1)

42-45: LGTM!

lib/sidecar/vllm/deploy/disagg.yaml (1)

62-62: LGTM!

Also applies to: 158-158

deploy/inference-gateway/ext-proc/examples/onramp/README.md (1)

20-20: LGTM!

deploy/inference-gateway/ext-proc/examples/onramp/agg.yaml (1)

79-79: LGTM!

docs/fern/assets/releases.json (1)

15-15: LGTM!

docs/fern/components/releases.data.ts (1)

76-76: LGTM!

docs/fern/pages/kubernetes/kv-aware-routing/vanilla-vllm-onramp.mdx (1)

247-247: LGTM!

Also applies to: 263-263

docs/fern/pages/reference/general/compatibility.mdx (1)

159-159: LGTM!

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@container/templates/vllm_runtime.Dockerfile`:
- Line 236: Replace the assert-based invariant checks in the Dockerfile Python
commands, including the check around COMPILATION_HELPER.include_dir, with
explicit conditional validation that raises RuntimeError containing the existing
diagnostic details. Preserve the current conditions and failure messages, and
apply the same change to the checks at the later referenced lines so validation
remains active under Python optimization.

In `@lib/gpu_memory_service/integrations/vllm/patches.py`:
- Around line 137-148: Update the connector import fallback around the legacy
and canonical NixlConnector imports to catch only ModuleNotFoundError, allowing
other ImportError exceptions to propagate. Preserve the existing module-specific
fallback behavior and deferred optional vLLM imports so register_kv_caches is
patched only when a connector module is genuinely absent.

In `@lib/gpu_memory_service/tests/test_vllm_patches.py`:
- Around line 49-53: Update the test’s module-unavailability setup around the
monkeypatch.delitem call to assign the legacy vLLM module entry in sys.modules
to None, rather than merely removing it, so imports consistently raise
ModuleNotFoundError even when the module exists on disk.

---

Nitpick comments:
In `@components/src/dynamo/frontend/tests/test_vllm_processor_unit.py`:
- Line 815: Move the standard-library nullcontext import from the test body to
the module-level imports in test_vllm_processor_unit.py, while keeping any local
vLLM import only if needed for optional-dependency handling.

In `@lib/gpu_memory_service/integrations/vllm/patches.py`:
- Around line 134-144: Move the vLLM NixlConnector version-selection imports out
of patch_register_kv_caches() and into module scope, preserving the existing
legacy-module fallback and re-raising dependency ImportErrors. Expose one
module-level adapter used by patch_register_kv_caches(), while retaining
optional-dependency behavior without call-order-dependent imports.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 28308612-59fd-41b5-9cf8-4709c4379050

📥 Commits

Reviewing files that changed from the base of the PR and between 975d6a1 and ad5ae03.

📒 Files selected for processing (37)
  • components/src/dynamo/frontend/prepost.py
  • components/src/dynamo/frontend/tests/test_vllm_processor_unit.py
  • components/src/dynamo/frontend/vllm_processor.py
  • components/src/dynamo/vllm/tests/multimodal_utils/test_vllm_request_processor.py
  • components/src/dynamo/vllm/tests/test_vllm_api_contract.py
  • container/compliance/base_sboms/manifest.json
  • container/compliance/base_sboms/vllm-openai-xpu@3309c2e1-amd64.cdx.json
  • container/context.yaml
  • container/templates/args.Dockerfile
  • container/templates/vllm_runtime.Dockerfile
  • deploy/inference-gateway/ext-proc/examples/onramp/README.md
  • deploy/inference-gateway/ext-proc/examples/onramp/agg.yaml
  • docs/fern/assets/releases.json
  • docs/fern/components/releases.data.ts
  • docs/fern/pages/kubernetes/kv-aware-routing/vanilla-vllm-onramp.mdx
  • docs/fern/pages/reference/api/python/frontend.mdx
  • docs/fern/pages/reference/general/compatibility.mdx
  • docs/fern/pages/reference/general/releases-machine-readable.mdx
  • lib/gpu_memory_service/integrations/vllm/patches.py
  • lib/gpu_memory_service/tests/test_v1_checkpoint.py
  • lib/gpu_memory_service/tests/test_v1_memory_manager.py
  • lib/gpu_memory_service/tests/test_v1_parameter_storage.py
  • lib/gpu_memory_service/tests/test_v1_protocol.py
  • lib/gpu_memory_service/tests/test_v1_server.py
  • lib/gpu_memory_service/tests/test_v1_weight_artifact.py
  • lib/gpu_memory_service/tests/test_vllm_patches.py
  • lib/llm/src/protocols/openai/chat_completions.rs
  • lib/llm/src/protocols/openai/common_ext.rs
  • lib/llm/src/protocols/openai/completions.rs
  • lib/llm/src/protocols/openai/validate.rs
  • lib/llm/tests/test_common_ext.rs
  • lib/sidecar/vllm/deploy/agg.yaml
  • lib/sidecar/vllm/deploy/disagg.yaml
  • pyproject.toml
  • tests/frontend/test_prepost_mistral.py
  • tests/serve/launch/multi_node_tp_headless.sh
  • tests/serve/test_vllm_xpu.py

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving

All 12 findings from my earlier review are addressed, and I verified each against the code at 3dfd1cc3 rather than from the commit messages. Thanks for the thoroughness here — several of these were fixed more completely than I asked for.

On the sidecar, which was the blocking item and the one I got wrong: you took the harder option, and it is the right one. I independently recomputed both hashes:

inference.proto  vendored=4e6c90467ea308bb…  v0.27.1=4e6c90467ea308bb…  MATCH
control.proto    vendored=390c88e94f1b6842…  v0.27.1=390c88e94f1b6842…  MATCH

service Inference now agrees on both sides, all three manifests are on v0.27.1, and proto/README.md pins tag v0.27.1 with commit 6e448d0ea9bf3d88d898b65449ca6dc2aec170ac — I confirmed that is genuinely the v0.27.1 tag commit, so the vendored copy is now checkable against a published artifact instead of a diverged dev commit. convert.rs:374 raises invalid_argument before any RPC rather than dropping fields silently, tests.rs asserts the rejection happens before RPC submission, and the README no longer advertises media inputs. That closes the silent-multimodal-loss path completely.

Some fixes came out better than my suggestion:

  • scoping transformers<5.15.0 to the ai-dynamo[vllm] extra instead of the base dependency, so SGLang and TRT-LLM installs stay unconstrained;
  • removing stream_interval outright rather than keeping a parameter that no-ops below the engine-level interval;
  • relying on the Omni constraints snapshot for resolver-side protection instead of a global uv pip check — which, as your logs showed, was tripping on pre-existing conflicts in the upstream image. My "true positive" call in that thread was wrong.

Also picked up along the way: the stale manifest.json from_tag, and the flashinf_ref ARG removed together with its args.Dockerfile declaration.

CI is green at this head — 86 passing, 0 failures. xpu-ci-vllm / build-xpu is still queued; it passed on 9bbda48961 and nothing since touches XPU, so I do not consider it a risk, but worth a glance before merge.

One follow-up worth tracking separately, not blocking: sidecar multimodal is now explicitly rejected, so whatever #12735 was building toward with session_id/media needs an upstream vLLM release that carries those fields. Given nothing in CI exercises these manifests, a proto-hash check in the sidecar build would also keep the vendored copy and the deployed tag from drifting apart again — that drift is what made both the v0.26.0 service-name mismatch and this one invisible.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rubber stamping

@Aphoh Aphoh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Comment thread lib/sidecar/vllm/proto/inference.proto Resolved

@connorcarpenter15 connorcarpenter15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the changes to the lib/sidecar folder. These changes remove functionality that I just added that will land in vLLM 0.28.0. It will also interfere with other sidecar functionality I am currently adding. I will bump the sidecar version in a separate PR.

Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>

@connorcarpenter15 connorcarpenter15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JulienDarve

@rmccorm4 rmccorm4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frontend finding noted inline.

@rmccorm4 rmccorm4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving for frontend, with one request for follow-up: #13059 (comment)

@JulienDarve
JulienDarve merged commit 08f12f7 into main Aug 18, 2026
203 of 204 checks passed
@JulienDarve
JulienDarve deleted the jdarve/vllm-bump-v0.27.1__DYN-3836 branch August 18, 2026 17:37
ynpandey-nv pushed a commit that referenced this pull request Aug 25, 2026
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
shugandhi added a commit to shugandhi/dynamo that referenced this pull request Aug 26, 2026
vllm.exceptions.VLLMClientError was introduced alongside the 0.27.1 bump
(ai-dynamo#13059). Dynamo's worker error path imports it unconditionally, so any
build pinned to an earlier vLLM fails at request time with ImportError
and returns HTTP 500 for every generation.

Fall back to a local base class when the symbol is absent, and route
handlers through .errors so there is a single import site.

Signed-off-by: Shubham <shugandhi@nvidia.com>
@rmccorm4 rmccorm4 mentioned this pull request Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend chore container documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XXL xpu

Projects

None yet

Development

Successfully merging this pull request may close these issues.