Skip to content

chore(release): backport Qwen3.8-2.4T with vLLM 0.27.1 - #13504

Merged
ynpandey-nv merged 11 commits into
release/1.4.0-qwen-3.8-2.4t-dev.1from
ynpandey-nv/cherrypick-13105-13059-qwen-3.8-2.4t
Aug 27, 2026
Merged

ynpandey-nv merged 11 commits into
release/1.4.0-qwen-3.8-2.4t-dev.1from
ynpandey-nv/cherrypick-13105-13059-qwen-3.8-2.4t

Conversation

@ynpandey-nv

@ynpandey-nv ynpandey-nv commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Rebuilds the release backport as true delta cherry-picks on the existing release/1.4.0-qwen-3.8-2.4t-dev.1 base (f4625cde5c16c976fc09626b67674bff6a788446). The release branch itself was not rebased or moved.

Cherry-picks, in validated order:

Additional release fixes:

  • Publishes coherent 1.4.0.dev1 Python and Rust-backed wheels
  • Keeps branch-native Go/Python SBOM baselines while updating vLLM CUDA/XPU metadata
  • Removes unsupported vLLM 0.27.1 MTP flags for qwen3_5_moe_text
  • Documents that SGLang profiles require a patched image or SGLang v0.5.17+

Verification

  • dynamo.vllm.handlers keeps the branch-local DP topology helper (no missing dp_topology module)
  • vllm_runtime.Dockerfile does not install a nonexistent aisimulate*.whl
  • Wheel versions are consistently 1.4.0.dev1; workspace crates remain 1.4.0
  • Every SBOM manifest reference exists
  • Fern generated tables and Python API reference are fresh
  • Recipe catalog validation passes
  • Changed Python modules compile
  • git diff --check passes

@ynpandey-nv
ynpandey-nv requested review from a team as code owners August 19, 2026 00:25
@copy-pr-bot

copy-pr-bot Bot commented Aug 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added chore documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal container xpu labels Aug 19, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 8 potential issues.

Open in Devin Review

Comment thread components/src/dynamo/vllm/handlers.py Resolved
Comment thread pyproject.toml Resolved
Comment thread components/src/dynamo/vllm/handlers.py Resolved
@ynpandey-nv ynpandey-nv changed the title chore(vllm): cherry-pick Qwen3.8-2.4T support and vLLM 0.27.1 (#13105, #13059) chore(recipes): cherry-pick Qwen3.8-2.4T, vLLM 0.27.1, and recipe fixes (#13105, #13059, #13265, #13324) Aug 21, 2026
@KrishnanPrash

Copy link
Copy Markdown
Contributor

We built this PR's tree locally (CI-shape render + docker build of the vllm-runtime image, amd64) and validated a corrected variant. Summary of what the PR still needs before it can merge.

1. Convert the two file-copy commits into real cherry-picks

15abb534dfbb (#13105) and 6660393d3388 (#13059) copy each touched file's final state from main. The branch base is 200+ commits behind main, so the copies carry fragments of interim main commits whose sibling files never came along. Concretely, on this PR's tree:

  • components/src/dynamo/vllm/handlers.py:94 imports .dp_topology, which does not exist on the branch. Every vLLM worker fails at import. Reproduced in-container: ModuleNotFoundError: No module named 'dynamo.vllm.dp_topology'.
  • container/templates/vllm_runtime.Dockerfile installs aisimulate*.whl, but the branch wheel_builder.Dockerfile builds that wheel only for target == "planner". The image build fails at that step. Reproduced: error: The wheel filename "aisimulate*.whl" is invalid: Must have a version.
  • pyproject.toml carries main's 1.5.0 pins (version = "1.5.0", ai-dynamo-runtime==1.5.0, aisimulate==0.1.0.dev1); the branch builds 1.4.0 wheels.
  • container/compliance/base_sboms/manifest.json references go@66dcc593-* and python@ec133be9-* baselines that are not in the tree.

We also verified this cannot be repaired incrementally. Picking main's aisimulate fix (#12525) moves the failure to the missing root-workspace crate registration. Syncing the root Cargo.toml/Cargo.lock on top of that fails again on main's 1.5.0 version bump against the branch crates pinned at 1.4.0 (failed to select a version for kvbm-logical = "^1.5.0", candidate: 1.4.0). Real cherry-picks avoid all of this because each pick carries only its own delta.

2. The chain that works

Validated end to end: with the chain below, the vllm-runtime image builds on the first attempt (~12 min with warm caches), dynamo.vllm.handlers imports, the wheel pair is a coherent 1.4.0, and vllm.__version__ is 0.27.1.

git cherry-pick -m1 c8a33bf20d   # 13105 qwen support           (currently a file copy)
git cherry-pick -m1 1d68231d95   # 13265 FP8 rename + HF id     (already picked)
git cherry-pick -m1 26c02ecdbc   # 13232 recipe_specific_images (MISSING from this PR)
git cherry-pick -m1 08f12f739d   # 13059 vLLM 0.27.1            (currently a file copy)
git cherry-pick -m1 cfb9622361   # 13324 frontend args fix      (already picked)

Expected conflicts and the resolutions we used:

  • fix(recipes): correct model ID to Qwen/Qwen3.8-2.4T-A95B-FP8 #13265, recipes/README.md: keep the renamed Qwen3.8-2.4T-A95B-FP8 row; drop the two Qwen3.5-122B rows that ride in from an interim commit.
  • chore(vllm): bump vLLM to 0.27.1 #13059, components/src/dynamo/frontend/prepost.py: keep self.reasoning_is_done = bool(reasoning_ended); drop the self._uses_dynamo_json_tool_call_fallback = ... line. Its constructor parameter comes from an interim commit that is not on this branch.
  • chore(vllm): bump vLLM to 0.27.1 #13059, container/compliance/base_sboms/manifest.json: take only the vllm-openai-xpu@3309c2e1 entry (its .cdx.json file comes with the pick as a rename); keep the branch's existing go@e86c88d0 and python@043085f9 entries. Do not take the whole file from main; that reintroduces the dangling-baseline problem.
  • chore(vllm): bump vLLM to 0.27.1 #13059, compatibility.mdx and releases-machine-readable.mdx: take the incoming rows, then run docs/fern/scripts/gen_llms_tables.py. A pre-commit hook rejects the commit while these generated tables are stale.
  • chore(vllm): bump vLLM to 0.27.1 #13059, docs/fern/pages/reference/api/python/frontend.mdx: generated line anchors; regenerate.

3. Add the pieces no pick provides

  • A wheels chore commit, same shape as the Kimi-K3 release branch precedent (chore(release): set ... wheels to ...dev1), so the branch publishes correctly-versioned dev wheels.
  • An engine decision for the recipes: vllm/agg-gb200-agentic, vllm/agg-gb300-chat, and vllm/agg-gb300-agentic still set --speculative-config MTP, but stock vLLM 0.27.1 has no MTP mapping for qwen3_5_moe_text (see vllm/config/speculative.py), and the stock lmsysorg/sglang:v0.5.16 base cannot load the text-only checkpoint (needs sgl#32401, first released in v0.5.17). Either remove the MTP flags and document the sglang gap, or move the bases to patched/newer engines.
  • Optional, for behavior parity with main on the code paths this PR copies: fix(vllm): preserve prompt embedding errors #13381 (error contract), fix(vllm): keep response reasoning parser active #13487 (response reasoning parser), fix(frontend): standardize resolving thinking controls #11739 (thinking controls).

Happy to share the build logs or the validated branch.

ynpandey-nv and others added 6 commits August 25, 2026 13:32
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: chw001 <52851226+chw001@users.noreply.github.com>
Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>
Co-authored-by: chw001 <52851226+chw001@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Shreyas Misra <shreyasm@nvidia.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: Cheng Wang <chengwa@nvidia.com>
Signed-off-by: chw001 <chengwa@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: Dan Gil <dagil@nvidia.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
…ix parser flag (#13324)

Signed-off-by: Cheng Wang <chengwa@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Yogendra Pandey <20666257+ynpandey-nv@users.noreply.github.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
@ynpandey-nv ynpandey-nv changed the title chore(recipes): cherry-pick Qwen3.8-2.4T, vLLM 0.27.1, and recipe fixes (#13105, #13059, #13265, #13324) chore(release): backport Qwen3.8-2.4T with vLLM 0.27.1 Aug 25, 2026
@ynpandey-nv
ynpandey-nv force-pushed the ynpandey-nv/cherrypick-13105-13059-qwen-3.8-2.4t branch from 3978519 to 55de8f1 Compare August 25, 2026 21:13
@ynpandey-nv
ynpandey-nv requested a review from a team as a code owner August 25, 2026 21:13
The isolated pytest-marker-report hook env does not install project deps, so collection fails on tqdm imports from AI Simulate.

Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
The composition replay targets main and tag publishes; release backports should not fail on docs-website drift.

Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>

@dagil-nvidia dagil-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 9c40629 against base release/1.4.0-qwen-3.8-2.4t-dev.1: fidelity-clean backport, 1 must-fix, 5 consider. Cherry-pick verification was done by content, never ancestry: 3 of the 5 picks are byte-identical to their main commits, the other 2 are change-line-identical with explainable context drift, and the 90,500-line SBOM swap is exactly main's tool-produced artifact with no additions beyond the version swap. Every release-only commit was individually assessed; all repo gates plus the branch's own generators are green in a head worktree.

Must-fix

  1. docs/fern/pages/recipes/model-recipes/qwen-3-8-2-4t-a95b-fp8.mdx:84,92 - broken relative link ../kubernetes/quickstart.mdx (resolves to a nonexistent path; every sibling recipe page uses the deeper form). Inherited verbatim from main's copy of the same file, so fix on main first and re-pick, or fix in both - fixing only here gets reverted by the next CP wave.
    See the [Kubernetes Deployment Guide](../../kubernetes/getting-started/quickstart.mdx).
    

Consider

  1. .github/workflows/release.yml:183 - with BASE_VERSION=1.4.0.dev1, the nightly path would produce 1.4.0.dev1.dev<date> (invalid PEP 440) and a 4-part Helm SemVer; latent, since the RC path this branch uses emits valid versions, but strip a trailing .devN before appending nightly suffixes.
  2. recipes/qwen3.8-2.4t-a95b-fp8/README.md:227 plus the 4 SGLang deploy manifests - the README says the branch's pinned stock SGLang base cannot load the checkpoint and v0.5.17-or-patched is required, yet the manifests pin the sglang-runtime tag this branch builds. Confirm that tag is built patched outside this branch's container definition, or the SGLang recipes ship inoperable.
  3. Upstream candidate: this branch removes --speculative-config mtp from three vLLM profiles because stock vLLM 0.27.1 has no MTP mapping for this model, but main still carries the flag in the same three profiles against the same image tag - if the stock-engine claim holds, main's profiles fail at boot. The removal and the two Known-issues bullets belong on main too.
  4. container/compliance/base_sboms/manifest.json:166 - generated_at is hand-set to main's regeneration timestamp on a hand-edited manifest; a borrowed timestamp on a compliance artifact can mislead an audit. Regenerate or stamp the edit date.
  5. The fla-core and flash-linear-attention MIT license overrides exist nowhere in main's history (they target runtime-image compliance scans); if main's scans hit the same packages, upstream them.

Verified sound

Per-commit fidelity as summarized above, including the manifest split that reassembles main's exact change and the on-branch regeneration of the generated docs (freshness checks green); wheel-vs-crate version split matches the PR description; all changed modules compile and every added import resolves on the branch; catalog validation, CODEOWNERS gates, and the PR's own new artifact test (17/17) pass. Pre-existing on the branch base, not this PR: the catalog README frontmatter warning.

Comment-only review; severity calls stay with the human reviewers.

Signed-off-by: Yogendra Pandey <ypandey@nvidia.com>
@ynpandey-nv
ynpandey-nv force-pushed the ynpandey-nv/cherrypick-13105-13059-qwen-3.8-2.4t branch from 472d3cd to 5faa57f Compare August 27, 2026 21:59
@ynpandey-nv

Copy link
Copy Markdown
Contributor Author

Note on the failing lychee check: it is a pre-existing failure on the branch base, not introduced by this PR.

  • The same check failed identically on head 9c40629 (the head reviewed and approved above), run 32908514297.
  • None of the reported 404s originate in files this PR adds or modifies. They are absolute URLs such as github.com/ai-dynamo/dynamo/blob/main/recipes/llama-3-70b/..., .../recipes/deepseek-v32-fp4/... and .../recipes/nemotron-3-ultra/vllm/agg-h200-chat-mtp/deploy.yaml (recipes renamed or removed on main after this branch forked), plus dead third-party links (xgrammar.mlc.ai/docs/structural_tag/...) and docs.nvidia.com/dynamo/latest/kubernetes-deployment/start-here/installation-guide.
  • Those links already exist on release/1.4.0-qwen-3.8-2.4t-dev.1 in six files untouched here: docs/fern/pages/kubernetes/disaggregated-serving/overview.md, docs/fern/pages/recipes/feature-benchmarks/deepseek-v3-2-wideep-routing.mdx, docs/fern/pages/recipes/feature-benchmarks/llama-3-3-70b-topology.mdx, docs/fern/pages/recipes/model-recipes/deepseek-v3-2-nvfp4.mdx, docs/fern/pages/recipes/model-recipes/llama-3-3-70b-fp8.mdx, docs/fern/pages/use-cases/tool-calling-and-reasoning/structural-tag.md.
  • main passes this check because its docs no longer reference those recipe paths; this branch is behind and still does. Fixing them here would touch unrelated docs and be re-diverged by the next backport wave, so leaving them for a main-side cleanup.

The must-fix relative link in docs/fern/pages/recipes/model-recipes/qwen-3-8-2-4t-a95b-fp8.mdx is addressed in 5faa57f, and the checks that validate relative links (Check for broken markdown links, Fern Broken Links Check) both pass. lychee excludes local file links, so it does not cover that fix.

@ynpandey-nv
ynpandey-nv merged commit 65c4c57 into release/1.4.0-qwen-3.8-2.4t-dev.1 Aug 27, 2026
29 of 30 checks passed
@ynpandey-nv
ynpandey-nv deleted the ynpandey-nv/cherrypick-13105-13059-qwen-3.8-2.4t branch August 27, 2026 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::vllm Relates to the vllm backend chore container documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XXL xpu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants