Repository navigation
[Feat] docker: Add AWS EFA image variants built on published images - #41006
Conversation
|
built sglang docker with AWS EFA user space libraries installed and tested the above built docker image for P/D disagg inference between two AWS p5en.48xlarge instances with NIXL LIBFABRIC (efa provider) backend. |
4081f03 to
51ae62e
Compare
|
Tested the newly built image on 2 nodes of AWS p5en.48xlarge instances with EFA for
|
|
relates to this issue #40995 |
Thanks @snadampal ! I was wondering if u can try doing Mooncake store P2P EFA for L3 DRAM between 2 nodes too |
| # Prefer the AWS libfabric over the distro libfabric1 that libopenmpi3t64 pulls | ||
| # in, and expose fi_info/fi_pingpong for debugging EFA from inside the image. | ||
| ENV PATH="/opt/amazon/efa/bin:${PATH}" \ | ||
| LD_LIBRARY_PATH="/opt/amazon/efa/lib:${LD_LIBRARY_PATH}" |
There was a problem hiding this comment.
I think it's safe to remove this, as the EFA Installer updates the dynamic linker to prioritize this.
# cat /etc/ld.so.conf.d/000_efa.conf
/opt/amazon/efa/lib
|
tested Mooncake store with EFA backend, did KV offloading from Inference node HBM to remote node DRAM.
|
|
hi @snadampal since the new proposed approach is to have an separate efa specific build for vllm/sglang, we might not need to wait for nccl-EP to get well integrated into sglang/vllm to get multi node WideEP (>=EP16) working out of the box for EFA since you can reinstall the efa fork'ed DeepEP in the dockerfile i.e. |
|
pls rebase |
|
Hey team, Can we consider an approach where we make a separate thin Dockerfile built on the published runtime image The repo already does this: docker/sagemaker.Dockerfile is just FROM lmsysorg/sglang:latest plus a few lines. We can the same with a new docker/efa.Dockerfile (we can separate the workflows out from main workflow as well) |
So for nightlies, you first produce a container and then build the EFA on top of that, overriding BASE_IMAGE? |
2 things
|
we would highly prefer if the upstream build pipeline built & publishes official upstream EFA nightly too |
| # architecture, so keep the explicit `:+` then `:-` pair. | ||
| ARG ARCH_IF_NOT_AMD64=${TARGETARCH#amd64} | ||
| ARG PROTOCOL_IF_NOT_AMD64=${ARCH_IF_NOT_AMD64:+rdma} | ||
| ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa} |
There was a problem hiding this comment.
Suggest also setting SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC in this image. SGLang defaults it to UCX (srt/environ.py), so right now --disaggregation-transfer-backend nixl in the runtime-efa image still creates a UCX backend unless the user knows to override it. The EFA path in this image is the LIBFABRIC one.
| ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa} | |
| ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa} \ | |
| SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC |
Tested on 2x p5en.48xlarge with this stage applied on lmsysorg/sglang:latest-runtime (0.5.20): P/D disaggregation with --disaggregation-transfer-backend nixl + SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC works. Both sides log NIXL KVManager initialized with backend: LIBFABRIC, and outputs are correct. This is unconditional since the EFA installer and NIXL's LIBFABRIC plugin are there on both amd64 and arm64.
|
[by Claude Code] @snadampal could this PR also publish the EFA variant as a nightly from The current diff adds Concrete case: the B300 DeepSeek-V4 Dynamo+SGLang disaggregated recipes on AWS EFA use With The nightly image has only Ubuntu's |
ffc3d6a to
e86f54b
Compare
|
@alexnails, @ishandhanani I have moved the EFA build to a new dockerfile as an overlay on top of the base images, and also created a new workflow file. please review and provide feedback. |
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] Severity: style | Confidence: High
Issue
The title and description still describe the old design: a runtime-efa target in docker/Dockerfile and edits to the existing CI workflows. The head commit adds docker/efa.Dockerfile and release-docker-efa.yml instead, leaves docker/Dockerfile unchanged, and also publishes EFA variants of the full release image and of nightly-dev-*. All test results in the description come from the old runtime-efa target.
Fix
Update the title and description to match the head commit. Add the build command and result for docker/efa.Dockerfile with BASE_IMAGE=lmsysorg/sglang:nightly-dev-<date>-<sha> (amd64 is enough), plus a short smoke check on that image, for example fi_info -p efa and NIXL starting with the LIBFABRIC backend. The nightly image has more packages than the runtime image tested so far.
|
@snadampal could you address these comments? |
|
Thanks for the review, @nvpohanh , will address them. |
Add docker/efa.Dockerfile, a thin layer over any published CUDA 13
lmsysorg/sglang image. It installs the AWS EFA userspace stack
(libfabric, rdma-core, aws-ofi-nccl) and, on amd64, swaps in the EFA
Mooncake wheel pinned to the base image's Mooncake version. It also sets
EFA defaults for Mooncake, NIXL (LIBFABRIC) and NCCL (ofi net/tuner).
LD_LIBRARY_PATH puts the AWS libfabric first.
Add release-docker-efa.yml, which runs after the base workflows and
publishes:
- v{version}-efa, latest-efa (from the release image)
- v{version}-runtime-efa, latest-runtime-efa (from the runtime image)
- nightly-dev-efa-{date}-{sha}, dev-efa (from the nightly dev image)
docker/Dockerfile and the existing workflows are unchanged.
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
e86f54b to
47e1d53
Compare
|
Hi @nvpohanh, I've updated the PR, please give it a try. Hi @ishandhanani , @alexnails , appreciate if you could trigger CI run for these new workflows. |
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] Severity: style | Confidence: High
Issue
The Check PR title job fails on the current title, [docker] Add AWS EFA image variants built on published images (log). .github/workflows/pr-title.yml requires [Type] scope: title, where Type is one of Feat, Fix, Refactor, Perf, Doc, Test, CI, or Chore.
Fix
Rename the PR, for example to [Feat] docker: Add AWS EFA image variants built on published images.
| ENVIRONMENT="prod" | ||
| ;; | ||
| "Build and Push Development Docker Images/schedule"|"Build and Push Development Docker Images/workflow_dispatch") | ||
| # The base nightly tag is nightly-dev-{date}-{short_sha}, where |
There was a problem hiding this comment.
[by Claude Code] Severity: style | Confidence: High
Issue
This comment block (lines 143-153) is 11 lines, and the concurrency comment on lines 47-51 is 5 lines. Both are long prose that is hard to scan. .claude/rules/comment-style.md asks for one or two lines.
Fix
Keep one or two lines of rationale per block, for example:
# Find the tag this run pushed (same commit, pushed after the run started),
# not today's date; an older nightly of this commit must not move dev-efa.On line 47: # One group per base run, so only a re-run cancels an in-progress EFA build.
|
Hi @nvpohanh , thanks for approving the PR! I will address the comment wording along with any other changes we may need after the CI runs. Please let me know the next steps to get it merged. |
|
/tag-and-rerun-ci |
|
Thanks for merging the PR! |
…261019 Move all seven B300 DeepSeek-V4-Pro-0813 Dynamo+SGLang AgentX recipes and both master-config keys to lmsysorg/sglang:nightly-dev-efa-20261019-3c560089. - The EFA variant (sgl-project/sglang#41006) adds the AWS EFA installer's libfabric. NIXL 1.5.0's LIBFABRIC plugin needs its FABRIC_1.7 symbols, which the regular image's Ubuntu libfabric 1.17 lacks, so the plugin failed to load and disaggregated workers stopped with "unsupported backend 'LIBFABRIC'". - Use the image's bundled Dynamo 1.6.0.dev20261018 (install: false) in place of the b208989fbe source build. It includes ai-dynamo/dynamo#16005, which counts attention-DP ranks from attn_dp_size. - No SGLang arg or env var used by these recipes was renamed, removed or given a new default between 7224fce1 and 3c560089, and no Dynamo flag or env var they set changed between b208989fbe and the bundled build. 将 B300 DeepSeek-V4-Pro-0813 Dynamo+SGLang AgentX 的七个配方及两个主配置项 更新为 lmsysorg/sglang:nightly-dev-efa-20261019-3c560089。 - EFA 镜像变体(sgl-project/sglang#41006)加入 AWS EFA 安装包中的 libfabric。 NIXL 1.5.0 的 LIBFABRIC 插件需要其 FABRIC_1.7 符号,而常规镜像的 Ubuntu libfabric 1.17 不具备,导致插件无法加载,分离式 worker 报错 "unsupported backend 'LIBFABRIC'"。 - 改用镜像自带的 Dynamo 1.6.0.dev20261018(install: false),取代 b208989fbe 源码构建。该版本已包含 ai-dynamo/dynamo#16005,按 attn_dp_size 计算注意力 DP rank。 - 在 7224fce1 与 3c560089 之间,这些配方使用的 SGLang 参数与环境变量均未重命名、 删除或改变默认值;在 b208989fbe 与镜像自带版本之间,配方设置的 Dynamo 参数与 环境变量也均无变化。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Motivation
To improve sglang out-of-box experience on AWS GPU clusters.
Modifications
Adds AWS EFA variants of the published SGLang CUDA 13 images. Each one is a thin layer over an image that's already published, so
docker/Dockerfileand the existing release workflows are unchanged.docker/efa.Dockerfile:FROM ${BASE_IMAGE}, following the pattern ofdocker/sagemaker.Dockerfile. It adds:MOONCAKE_PROTOCOL=efa(tcpon arm64, where the EFA wheel isn't published),SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC, andNCCL_NET_PLUGIN=ofi/NCCL_TUNER_PLUGIN=ofi.LD_LIBRARY_PATH=/opt/amazon/efa/lib. With--gpus, the NVIDIA container toolkit rerunsldconfig, and the rebuilt cache picks the distro libfabric 1.17.0. That copy lacksFABRIC_1.7, so NIXL fails withcreateBackend: unsupported backend 'LIBFABRIC', the same error as onnightly-dev-20261013-7224fce1..github/workflows/release-docker-efa.yml: runs after each base workflow succeeds and publishes amd64 + arm64 images:v{version}v{version}-efa,latest-efa,v{version}-cu130-efa,latest-cu130-efav{version}-runtimev{version}-runtime-efa,latest-runtime-efa,v{version}-cu130-runtime-efa,latest-cu130-runtime-efanightly-dev-{date}-{sha}nightly-dev-efa-{date}-{sha},dev-efaNightly EFA tags reuse the base tag's exact
{date}-{sha}suffix, so each one matches anightly-dev-*image on the same SGLang commit. Aworkflow_dispatchcan layer EFA onto any published tag, and oldnightly-dev-efa-*tags are pruned.mooncake_store.py: when"protocol"is missing from the config file or extra config, the store now falls back toMOONCAKE_PROTOCOL.get()instead of the hard-coded default, so it honors the image'sMOONCAKE_PROTOCOL=efa.Docs:
install.mdxlists the EFA tags.pd_disaggregation.mdxdescribes what the images contain, the amd64 limits, adocker runexample with the EFA devices, and how to buildefa.Dockerfileon a custom base.Build of
docker/efa.Dockerfileon a published nightly image (amd64):Build of
docker/efa.Dockerfileon a published latest (amd64):Build of
docker/efa.Dockerfileon a published latest-runtime (amd64):Accuracy Tests
fi_infoon AWS p5en.48xlarge instanceSpeed Tests and Profiling
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #38592779615
Latest PR Test (Extra): ❌ Run #38592779623
Latest PR Test (AMD ROCm 10): ⏳ Run #38592779704