Skip to content

build(container): bump EFA installer to 1.51.0, Mooncake 0.3.14, and NIXL refs - #18003

Merged
dagil-nvidia merged 4 commits into
ai-dynamo:mainfrom
erezzarum:efa_mooncake
Nov 6, 2026
Merged

dagil-nvidia merged 4 commits into
ai-dynamo:mainfrom
erezzarum:efa_mooncake

Conversation

@erezzarum

@erezzarum erezzarum commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Closes #18280

Bumps the AWS EFA installer to 1.51.0 (libfabric 2.7.0amzn2.0, aws-ofi-nccl 1.22.0), the Mooncake transfer engine floor to 0.3.14, and aligns the per-framework NIXL refs.

  • This PR introduce a new Mooncake version (0.3.14) that includes the EFA transport in a unified wheel file, removes the need for a special Mooncake EFA variant wheel file.
  • TensorRT-LLM - for EFA, override container image upstream NIXL to 1.5.2 to resolve CUDA context issues

Key Changes

Validation

  • Built vLLM, SGLang, TensorRT-LLM EFA variants.

Issues and Ongoing Work

Pull Request Checklist
  • I used Dynamo's /pr-checklist skill.

Summary by CodeRabbit

  • Updates
    • Updated EFA and NIXL versions used in container images.
    • Updated the minimum Mooncake Transfer Engine version for vLLM and SGLang.
    • Adjusted EFA installer options for AWS images.

…NIXL refs.

Signed-off-by: Erez Zarum <erezz@amazon.com>
@erezzarum
erezzarum requested review from a team as code owners November 4, 2026 01:27
@copy-pr-bot

copy-pr-bot Bot commented Nov 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@erezzarum
erezzarum temporarily deployed to external_collaborator November 4, 2026 01:27 — with GitHub Actions Inactive
@erezzarum
erezzarum temporarily deployed to external_collaborator November 4, 2026 01:27 — with GitHub Actions Inactive
@github-actions github-actions Bot added the build label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

👋 Hi erezzarum! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor backend::vllm Relates to the vllm backend container labels Nov 4, 2026
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

Container configuration updates EFA package versions and installer options, refreshes NIXL references for three inference backends, and raises the minimum Mooncake transfer-engine version for vLLM and SGLang.

Changes

Container updates

Layer / File(s) Summary
EFA versions and installer options
container/context.yaml, container/compliance/native_packages.yaml, container/templates/aws.Dockerfile
The default EFA version and recorded libfabric and aws-ofi-nccl versions change. The AWS EFA installer command uses --skip-mpi --skip-rdma-core --build-ngc; comments describe these options.
Inference backend dependency versions
container/context.yaml, container/deps/requirements.vllm.txt, container/deps/requirements.sglang.txt
The NIXL references for vLLM, SGLang, and TensorRT-LLM change. The minimum mooncake-transfer-engine-cuda13 version increases for vLLM and SGLang; the <0.4 upper bound remains unchanged.

Estimated code review effort: 2 (Simple) | ~8 minutes

Merge Risk: Minimal · up to 42836

This PR updates container dependency versions and the EFA installer options. No behavioral risk was found in the changed files. The PR still needs an issue link in its description to satisfy the CI check.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly summarizes the main changes: the EFA installer, Mooncake, and NIXL references are updated.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description includes all required sections, validation, issue references, and the checklist. It states that the TensorRT-LLM NIXL reference is v1.5.2, but the change summary reports a default of v…
Full details: Description check

Explanation

The description includes all required sections, validation, issue references, and the checklist. It states that the TensorRT-LLM NIXL reference is v1.5.2, but the change summary reports a default of v1.5.0. Clarify whether an EFA-specific override sets v1.5.2.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 54ee241 and 42836c8.

Files selected for processing: 5

  • container/compliance/native_packages.yaml
  • container/context.yaml
  • container/deps/requirements.sglang.txt
  • container/deps/requirements.vllm.txt
  • container/templates/aws.Dockerfile

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@nv-anants

Copy link
Copy Markdown
Member

/ok to test 42836c8

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0 or P1 found. This review read head 42836c868d1146724e281d3e6ab14bd59066b81d.

Findings, each inline:

  • [P2] container/templates/aws.Dockerfile:35: --skip-rdma-core leaves rdma-core 50.0 under libfabric 2.7.0amzn2.0.
  • [P2] container/templates/aws.Dockerfile:25: --build-ngc removes libnccl-net.so from the vLLM EFA image.
  • [P3] container/context.yaml:93: the nixl_libfabric_ref comment still names libfabric 2.6.0.

The two suggestions on aws.Dockerfile are one fix. Please apply both.

Open items, with no severity:

  • #17227 sets trtllm.nixl_ref to v1.4.0, and #17031 removes the CUDA nixl_ref lines. Both conflict with this PR in container/context.yaml. The PR that merges second must pick the ref.
  • mooncake-transfer-engine-cuda13 0.3.14 came out on 2026-11-03. Please say in the description what this PR needs from it.
  • The description says that the vLLM, SGLang and TensorRT-LLM EFA images build. On this head, sglang-runtime-efa and trtllm-runtime-efa built. vllm-runtime-efa stopped on an apt-get update mirror error before the installer ran. A build does not load libfabric, so it does not test the EFA path at run time.
Other checks on this head.
  • NIXL refs: vLLM v1.5.0 matches vLLM v0.32.0 (nixl == 1.5.0). TensorRT-LLM v1.5.0 matches TensorRT-LLM v1.4.0rc2 (NIXL_VERSION="v1.5.0"), so the TensorRT-LLM EFA image now preloads the NIXL version that TensorRT-LLM builds. The SGLang build fetched NIXL v1.5.2 and built it with UCX v1.22.0. nixl-sys stays at 1.3.2, which is not newer than any of these.
  • native_packages.yaml matches the EFA 1.51.0 release notes (libfabric 2.7.0amzn2.0, OFI NCCL plugin 1.22.0) and the build logs of this head. The Open MPI packages that --skip-mpi removes are Debian packages, and the dpkg scan follows them. So no other compliance file needs a change.
  • --skip-mpi is safe. Nothing in this repository uses /opt/amazon/openmpi, and the TensorRT-LLM image runs its own Open MPI from /opt/dynamo/mpi.
  • Package signature checks stay off, as on main. The installer checks signatures only with --check-signatures. --no-verify skips the EFA device test, not a signature check.
  • This PR merges with main at 5a26da671a without conflicts.

Comment thread container/templates/aws.Dockerfile Resolved
Comment thread container/templates/aws.Dockerfile Resolved
Comment thread container/context.yaml Resolved
Signed-off-by: Erez Zarum <erezz@amazon.com>
@erezzarum
erezzarum temporarily deployed to external_collaborator November 4, 2026 18:25 — with GitHub Actions Inactive
@dmitry-tokarev-nv

Copy link
Copy Markdown
Contributor

/ok to test 7bb0fc2

@devin-ai-integration

devin-ai-integration Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 42142204305 (attempt 1) on 47912733fa

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 42142204305. Updated on every full-CI run of this PR.

Signed-off-by: Erez Zarum <erezz@amazon.com>
@erezzarum
erezzarum temporarily deployed to external_collaborator November 5, 2026 22:39 — with GitHub Actions Inactive
@dmitry-tokarev-nv
dmitry-tokarev-nv dismissed their stale review November 5, 2026 22:43

Withdrawing my approval because new commits landed after it. I will review the new commits before I approve again.

@dmitry-tokarev-nv dmitry-tokarev-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No P0 or P1 found. This review read head 07c6a2abba0b371d63d0161a895c98194f2d8475. It covers the new commit 07c6a2abba. An earlier review on 2026-11-04 read the other commits.

The new commit has no finding.

Open items:

  • [P2] container/templates/aws.Dockerfile:27: the --build-ngc thread from 2026-11-04 is still open. This commit does not change it, because the vLLM EFA Dockerfile renders the same at 7bb0fc2d09 and at this head.
  • The PR workflow did not run on this head. No CI job built a TensorRT-LLM image with NIXL v1.5.2 yet.
What this review read and measured for commit 07c6a2a.
  • Scope: the commit changes one line, container/context.yaml:187. Its parent is 7bb0fc2d09. With container/render.py, the TensorRT-LLM runtime and EFA Dockerfiles for amd64 and arm64 differ from 7bb0fc2d09 in one line, ARG NIXL_REF=v1.5.2.
  • Effect: wheel_builder fetches and builds NIXL v1.5.2. Each TensorRT-LLM runtime image installs nixl==1.5.2 and nixl-cu13==1.5.2 from PyPI. The EFA image also replaces /opt/nvidia/nvda_nixl with the wheel_builder build, preloads its libnixl.so and sets NIXL_PLUGIN_DIR to its plugins. The other runtime image keeps the libnixl.so of TensorRT-LLM in LD_PRELOAD.
  • Source: the v1.5.2 tag points to e72c52c784, the tip of release/1.5.2. GitHub (web-flow) signed that commit. The release is not a pre-release. PyPI has nixl-cu13 1.5.2 wheels for CPython 3.12 on x86_64 and aarch64, and none is yanked.
  • API: from the v1.5.0 tag to v1.5.2, nothing changes in src/api/python, in src/bindings/python, or in the C API in src/bindings/rust. In src/api/cpp, two constructors lose explicit, the inline nixlBackendEngine constructor loses its null check on customParams, and comments change. No struct layout or function signature changes.
  • Lineage: TensorRT-LLM v1.4.0rc2 builds the v1.5.0 tag, which is on NIXL main. v1.5.2 is on the release branch. So for the EFA image, this commit adds 17 release-branch commits to NIXL and removes 47 main commits.
  • libfabric: v1.5.2 adds ai-dynamo/nixl#2470 (the CUDA context fallback that the description names), ai-dynamo/nixl#2529 and ai-dynamo/nixl#2542. It does not have ai-dynamo/nixl#2107. That change tells the target about a batch write that the initiator cannot post. ai-dynamo/nixl#2529 sets its EFA option by value, so it also works with libfabric headers older than 2.5. On 7bb0fc2d09, the SGLang build compiled NIXL v1.5.2 against libfabric v2.6.0 (nixl_libfabric_ref) and UCX v1.22.0, on CUDA 13.0.
  • UCX: src/plugins/ucx/config.cpp is the same file in both tags. With NVIDIA GPUs present, it rejects a UCX_TLS allow list that lacks cuda_copy, cuda and all. No TensorRT-LLM EFA path in this repository sets UCX_TLS.
  • Upstream: TensorRT-LLM main also moved to NIXL v1.5.2, in NVIDIA/TensorRT-LLM#19783.
  • CI: on 7bb0fc2d09, the TensorRT-LLM runtime and EFA builds and the three TensorRT-LLM test jobs passed with NIXL v1.5.0.
  • #18237 changes the same two Mooncake lines to the same text. If it merges first, this PR still merges without a conflict, and the two requirement files leave its net change.
  • This head merges with main at 118f8cc201 without conflicts.
  • Not tested: a TensorRT-LLM build with NIXL v1.5.2 on CUDA 13.1, and NIXL over EFA at run time.

@dmitry-tokarev-nv

Copy link
Copy Markdown
Contributor

/ok to test 07c6a2a

@dagil-nvidia
dagil-nvidia temporarily deployed to external_collaborator November 6, 2026 01:11 — with GitHub Actions Inactive
@dagil-nvidia

Copy link
Copy Markdown
Collaborator

/ok to test 4791273

@erezzarum

Copy link
Copy Markdown
Contributor Author

Are there any blockers to prevent merging this?

@dagil-nvidia
dagil-nvidia merged commit ef8d3b1 into ai-dynamo:main Nov 6, 2026
121 of 122 checks passed
ishandhanani added a commit to ishandhanani/dynamo that referenced this pull request Nov 6, 2026
Absorbs three upstream commits; two import-only conflicts in ext-proc:

- ai-dynamo#18072 feat(llm): add DYN_LORA_S3_TLS_INSECURE for S3 LoRA downloads:
  lib/lora/src/source.rs (S3 builder TLS knob plus three unit tests),
  lib/runtime/src/config/environment_names.rs, two LoRA docs pages.
  Outside the router; auto-merged.
- ai-dynamo#17250 perf(epp): move tokenization off async workers and add stage
  metrics: deploy/inference-gateway/ext-proc (new tokio_runtime.rs and
  log_budget.rs, stage and stall metrics, envoy_helpers body rewrite,
  server.rs stream trace, epp.rs tokenize slots, Cargo.toml tokio
  `schedule-latency` feature), gateway and observability docs. Upstream's
  changes taken whole; the two conflicts were adjacent import lines only:
  epp.rs keeps the stack's `dynamo_scheduler::host::{RouterConfigOverride,
  RoutingConstraints}` next to upstream's `tokio::sync::Semaphore`, and
  server.rs keeps the stack's `dynamo_kv_events::zmq_wire::
  DYNAMO_CACHE_SALT_PREFIX` next to upstream's `crate::tokio_runtime`.
- ai-dynamo#18003 build(container): bump EFA installer to 1.51.0, Mooncake 0.3.14,
  and NIXL refs: container/ only. Outside the router; auto-merged.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
nv-nmailhot pushed a commit that referenced this pull request Nov 7, 2026
…ooncake 0.3.14, and NIXL refs (#18003) (#18348)

Signed-off-by: Dan Gil <dagil@nvidia.com>
devivasudevan pushed a commit to devivasudevan/dynamo that referenced this pull request Nov 7, 2026
…NIXL refs (ai-dynamo#18003)

Signed-off-by: Erez Zarum <erezz@amazon.com>
Co-authored-by: Dan Gil <dagil@nvidia.com>

This branch was previously deployed

1 inactive deployment
external_collaborator — 47912733 Deployed Nov 6, 2026 by dagil-nvidia via ok-to-test #47163
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend build container external-contribution Pull request is from an external contributor size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bump EFA installer to 1.51.0, Mooncake to 0.3.14 and NIXL refs for EFA images

5 participants