Repository navigation
build(container): bump EFA installer to 1.51.0, Mooncake 0.3.14, and NIXL refs - #18003
Conversation
…NIXL refs. Signed-off-by: Erez Zarum <erezz@amazon.com>
|
👋 Hi erezzarum! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
|
No actionable comments were generated in the recent review. WalkthroughContainer configuration updates EFA package versions and installer options, refreshes NIXL references for three inference backends, and raises the minimum Mooncake transfer-engine version for vLLM and SGLang. ChangesContainer updates
Estimated code review effort: 2 (Simple) | ~8 minutes Merge Risk:
|
|
/ok to test 42836c8 |
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. This review read head 42836c868d1146724e281d3e6ab14bd59066b81d.
Findings, each inline:
- [P2]
container/templates/aws.Dockerfile:35:--skip-rdma-coreleaves rdma-core 50.0 under libfabric 2.7.0amzn2.0. - [P2]
container/templates/aws.Dockerfile:25:--build-ngcremoveslibnccl-net.sofrom the vLLM EFA image. - [P3]
container/context.yaml:93: thenixl_libfabric_refcomment still names libfabric 2.6.0.
The two suggestions on aws.Dockerfile are one fix. Please apply both.
Open items, with no severity:
- #17227 sets
trtllm.nixl_reftov1.4.0, and #17031 removes the CUDAnixl_reflines. Both conflict with this PR incontainer/context.yaml. The PR that merges second must pick the ref. mooncake-transfer-engine-cuda130.3.14 came out on 2026-11-03. Please say in the description what this PR needs from it.- The description says that the vLLM, SGLang and TensorRT-LLM EFA images build. On this head,
sglang-runtime-efaandtrtllm-runtime-efabuilt.vllm-runtime-efastopped on anapt-get updatemirror error before the installer ran. A build does not load libfabric, so it does not test the EFA path at run time.
Other checks on this head.
- NIXL refs: vLLM
v1.5.0matches vLLM v0.32.0 (nixl == 1.5.0). TensorRT-LLMv1.5.0matches TensorRT-LLM v1.4.0rc2 (NIXL_VERSION="v1.5.0"), so the TensorRT-LLM EFA image now preloads the NIXL version that TensorRT-LLM builds. The SGLang build fetched NIXLv1.5.2and built it with UCXv1.22.0.nixl-sysstays at 1.3.2, which is not newer than any of these. native_packages.yamlmatches the EFA 1.51.0 release notes (libfabric 2.7.0amzn2.0, OFI NCCL plugin 1.22.0) and the build logs of this head. The Open MPI packages that--skip-mpiremoves are Debian packages, and the dpkg scan follows them. So no other compliance file needs a change.--skip-mpiis safe. Nothing in this repository uses/opt/amazon/openmpi, and the TensorRT-LLM image runs its own Open MPI from/opt/dynamo/mpi.- Package signature checks stay off, as on
main. The installer checks signatures only with--check-signatures.--no-verifyskips the EFA device test, not a signature check. - This PR merges with
mainat5a26da671awithout conflicts.
Signed-off-by: Erez Zarum <erezz@amazon.com>
|
/ok to test 7bb0fc2 |
✅ Dynamo PR CI passed — run 42142204305 (attempt 1) on
|
Signed-off-by: Erez Zarum <erezz@amazon.com>
Withdrawing my approval because new commits landed after it. I will review the new commits before I approve again.
dmitry-tokarev-nv
left a comment
There was a problem hiding this comment.
No P0 or P1 found. This review read head 07c6a2abba0b371d63d0161a895c98194f2d8475. It covers the new commit 07c6a2abba. An earlier review on 2026-11-04 read the other commits.
The new commit has no finding.
Open items:
- [P2]
container/templates/aws.Dockerfile:27: the--build-ngcthread from 2026-11-04 is still open. This commit does not change it, because the vLLM EFA Dockerfile renders the same at7bb0fc2d09and at this head. - The
PRworkflow did not run on this head. No CI job built a TensorRT-LLM image with NIXLv1.5.2yet.
What this review read and measured for commit 07c6a2a.
- Scope: the commit changes one line,
container/context.yaml:187. Its parent is7bb0fc2d09. Withcontainer/render.py, the TensorRT-LLM runtime and EFA Dockerfiles for amd64 and arm64 differ from7bb0fc2d09in one line,ARG NIXL_REF=v1.5.2. - Effect:
wheel_builderfetches and builds NIXLv1.5.2. Each TensorRT-LLM runtime image installsnixl==1.5.2andnixl-cu13==1.5.2from PyPI. The EFA image also replaces/opt/nvidia/nvda_nixlwith thewheel_builderbuild, preloads itslibnixl.soand setsNIXL_PLUGIN_DIRto its plugins. The other runtime image keeps thelibnixl.soof TensorRT-LLM inLD_PRELOAD. - Source: the
v1.5.2tag points toe72c52c784, the tip ofrelease/1.5.2. GitHub (web-flow) signed that commit. The release is not a pre-release. PyPI hasnixl-cu131.5.2 wheels for CPython 3.12 on x86_64 and aarch64, and none is yanked. - API: from the
v1.5.0tag tov1.5.2, nothing changes insrc/api/python, insrc/bindings/python, or in the C API insrc/bindings/rust. Insrc/api/cpp, two constructors loseexplicit, the inlinenixlBackendEngineconstructor loses its null check oncustomParams, and comments change. No struct layout or function signature changes. - Lineage: TensorRT-LLM v1.4.0rc2 builds the
v1.5.0tag, which is on NIXLmain.v1.5.2is on the release branch. So for the EFA image, this commit adds 17 release-branch commits to NIXL and removes 47maincommits. - libfabric:
v1.5.2addsai-dynamo/nixl#2470(the CUDA context fallback that the description names),ai-dynamo/nixl#2529andai-dynamo/nixl#2542. It does not haveai-dynamo/nixl#2107. That change tells the target about a batch write that the initiator cannot post.ai-dynamo/nixl#2529sets its EFA option by value, so it also works with libfabric headers older than 2.5. On7bb0fc2d09, the SGLang build compiled NIXLv1.5.2against libfabric v2.6.0 (nixl_libfabric_ref) and UCX v1.22.0, on CUDA 13.0. - UCX:
src/plugins/ucx/config.cppis the same file in both tags. With NVIDIA GPUs present, it rejects aUCX_TLSallow list that lackscuda_copy,cudaandall. No TensorRT-LLM EFA path in this repository setsUCX_TLS. - Upstream: TensorRT-LLM
mainalso moved to NIXLv1.5.2, inNVIDIA/TensorRT-LLM#19783. - CI: on
7bb0fc2d09, the TensorRT-LLM runtime and EFA builds and the three TensorRT-LLM test jobs passed with NIXLv1.5.0. - #18237 changes the same two Mooncake lines to the same text. If it merges first, this PR still merges without a conflict, and the two requirement files leave its net change.
- This head merges with
mainat118f8cc201without conflicts. - Not tested: a TensorRT-LLM build with NIXL
v1.5.2on CUDA 13.1, and NIXL over EFA at run time.
|
/ok to test 07c6a2a |
|
/ok to test 4791273 |
|
Are there any blockers to prevent merging this? |
Absorbs three upstream commits; two import-only conflicts in ext-proc: - ai-dynamo#18072 feat(llm): add DYN_LORA_S3_TLS_INSECURE for S3 LoRA downloads: lib/lora/src/source.rs (S3 builder TLS knob plus three unit tests), lib/runtime/src/config/environment_names.rs, two LoRA docs pages. Outside the router; auto-merged. - ai-dynamo#17250 perf(epp): move tokenization off async workers and add stage metrics: deploy/inference-gateway/ext-proc (new tokio_runtime.rs and log_budget.rs, stage and stall metrics, envoy_helpers body rewrite, server.rs stream trace, epp.rs tokenize slots, Cargo.toml tokio `schedule-latency` feature), gateway and observability docs. Upstream's changes taken whole; the two conflicts were adjacent import lines only: epp.rs keeps the stack's `dynamo_scheduler::host::{RouterConfigOverride, RoutingConstraints}` next to upstream's `tokio::sync::Semaphore`, and server.rs keeps the stack's `dynamo_kv_events::zmq_wire:: DYNAMO_CACHE_SALT_PREFIX` next to upstream's `crate::tokio_runtime`. - ai-dynamo#18003 build(container): bump EFA installer to 1.51.0, Mooncake 0.3.14, and NIXL refs: container/ only. Outside the router; auto-merged. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…NIXL refs (ai-dynamo#18003) Signed-off-by: Erez Zarum <erezz@amazon.com> Co-authored-by: Dan Gil <dagil@nvidia.com>
Overview
Closes #18280
Bumps the AWS EFA installer to 1.51.0 (libfabric 2.7.0amzn2.0, aws-ofi-nccl 1.22.0), the Mooncake transfer engine floor to 0.3.14, and aligns the per-framework NIXL refs.
Key Changes
--build-ngc --skip-mpi --skip-rdma-coreefa_version1.51.0; NIXL refs vllm v1.5.0, sglang v1.5.2, trtllm v1.5.2.mooncake-transfer-engine-cuda13>=0.3.14,<0.4.Validation
Issues and Ongoing Work
Pull Request Checklist
/pr-checklistskill.Summary by CodeRabbit