Skip to content

[Feat] docker: Add AWS EFA image variants built on published images - #41006

Merged
Fridge003 merged 1 commit into
sgl-project:mainfrom
snadampal:efa_install
Oct 15, 2026
Merged

Fridge003 merged 1 commit into
sgl-project:mainfrom
snadampal:efa_install

Conversation

@snadampal

@snadampal snadampal commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

To improve sglang out-of-box experience on AWS GPU clusters.

Modifications

Adds AWS EFA variants of the published SGLang CUDA 13 images. Each one is a thin layer over an image that's already published, so docker/Dockerfile and the existing release workflows are unchanged.

  • docker/efa.Dockerfile: FROM ${BASE_IMAGE}, following the pattern of docker/sagemaker.Dockerfile. It adds:

    • The AWS EFA userspace stack: libfabric with the EFA provider, rdma-core and aws-ofi-nccl.
    • On amd64, the EFA Mooncake wheel, pinned to the Mooncake version already in the base image.
    • EFA defaults: MOONCAKE_PROTOCOL=efa (tcp on arm64, where the EFA wheel isn't published), SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC, and NCCL_NET_PLUGIN=ofi / NCCL_TUNER_PLUGIN=ofi.
    • LD_LIBRARY_PATH=/opt/amazon/efa/lib. With --gpus, the NVIDIA container toolkit reruns ldconfig, and the rebuilt cache picks the distro libfabric 1.17.0. That copy lacks FABRIC_1.7, so NIXL fails with createBackend: unsupported backend 'LIBFABRIC', the same error as on nightly-dev-20261013-7224fce1.
    • Build-time checks: the EFA provider is present, every NIXL LIBFABRIC plugin resolves to the AWS libfabric, the ofi-nccl plugins are in the loader cache, and the Mooncake wheel swap leaves exactly one Mooncake package at the expected version.
  • .github/workflows/release-docker-efa.yml: runs after each base workflow succeeds and publishes amd64 + arm64 images:

    Base EFA tags
    v{version} v{version}-efa, latest-efa, v{version}-cu130-efa, latest-cu130-efa
    v{version}-runtime v{version}-runtime-efa, latest-runtime-efa, v{version}-cu130-runtime-efa, latest-cu130-runtime-efa
    nightly-dev-{date}-{sha} nightly-dev-efa-{date}-{sha}, dev-efa

    Nightly EFA tags reuse the base tag's exact {date}-{sha} suffix, so each one matches a nightly-dev-* image on the same SGLang commit. A workflow_dispatch can layer EFA onto any published tag, and old nightly-dev-efa-* tags are pruned.

  • mooncake_store.py: when "protocol" is missing from the config file or extra config, the store now falls back to MOONCAKE_PROTOCOL.get() instead of the hard-coded default, so it honors the image's MOONCAKE_PROTOCOL=efa.

  • Docs: install.mdx lists the EFA tags. pd_disaggregation.mdx describes what the images contain, the amd64 limits, a docker run example with the EFA devices, and how to build efa.Dockerfile on a custom base.

Build of docker/efa.Dockerfile on a published nightly image (amd64):

docker buildx build \
  --platform linux/amd64 \
  --load \
  -f docker/efa.Dockerfile \
  --build-arg BASE_IMAGE=lmsysorg/sglang:nightly-dev-<date>-<sha> \
  --build-arg EFA_IMAGE_TAG=lmsysorg/sglang:nightly-dev-efa-<date>-<sha> \
  -t lmsysorg/sglang:nightly-dev-efa-<date>-<sha> \
  docker/

Build of docker/efa.Dockerfile on a published latest (amd64):

docker buildx build \
  --platform linux/amd64 \
  --load \
  -f docker/efa.Dockerfile \
  --build-arg BASE_IMAGE=lmsysorg/sglang:latest \
  --build-arg EFA_IMAGE_TAG=lmsysorg/sglang:latest-efa \
  -t lmsysorg/sglang:latest-efa \
  docker/

Build of docker/efa.Dockerfile on a published latest-runtime (amd64):

docker buildx build \
  --platform linux/amd64 \
  --load \
  -f docker/efa.Dockerfile \
  --build-arg BASE_IMAGE=lmsysorg/sglang:latest-runtime \
  --build-arg EFA_IMAGE_TAG=lmsysorg/sglang:latest-runtime-efa \
  -t lmsysorg/sglang:latest-runtime-efa \
  docker/

Accuracy Tests

  1. Tested P/D disagg inference between two AWS p5en.48xlarge nodes with NIXL EFA backend for kv xfers. Used Qwen3-32B model for testing.
libfabric:256:1791987714::core:core:fi_fabric_():1590<info> Opened fabric: efa-direct
libfabric:256:1791987714::core:core:fi_fabric_():1597<info> Using efa provider 2.6, path:/opt/amazon/efa/lib/libfabric.so.1
libfabric:263:1791987715::core:core:fi_fabric_():1590<info> Opened fabric: efa-direct
libfabric:263:1791987715::core:core:fi_fabric_():1597<info> Using efa provider 2.6, 
.....
libfabric:256:1791988037::efa:cq:efa_rdm_pke_handle_handshake_recv():137<info> Received peer host id: i-0feb9a31f26d23262
libfabric:256:1791988037::efa:cq:efa_rdm_pke_handle_handshake_recv():141<info> Received peer EFA device version: 0xefa2
libfabric:256:1791988037::efa:cq:efa_rdm_pke_handle_handshake_recv():148<info> Received peer HMEM caps: p2p_supported=1
libfabric:259:1791988037::efa:ep_data:efa_rdm_ep_get_peer_explicit():92<info> Creating peer for addr 20
libfabric:255:1791988037::efa:cq:efa_rdm_pke_handle_handshake_recv():104<info> HANDSHAKE received from peer with explicit fi_addr 20 implicit fi_addr 18446744073709551615

[2026-10-14 14:27:17 TP0] Decode batch, #running-req: 1, #token: 25, token usage: 0.00, pre-allocated usage: 0.00, #prealloc-req: 0, #transfer-req: 0, #retracted-req: 0, cuda graph: True, gen throughput (token/s): 1.51, #queue-req: 0
[2026-10-14 14:27:18 TP0] Decode batch, #running-req: 1, #token: 65, token usage: 0.00, pre-allocated usage: 0.00, #prealloc-req: 0, #transfer-req: 0, #retracted-req: 0, cuda graph: True, gen throughput (token/s): 214.95, #queue-req: 0
[2026-10-14 14:27:18] INFO:     10.0.51.229:34648 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1. Tested P-D disagg with Mooncake TransferEngine P2P
I1014 15:23:55.708606  2823 efa_transport.cpp:275] Auto-split params: page_size=4096, max_pte_entries=23068672, pte_limit=94489280512, max_mr_size=206158430208, chunk_limit=94489280512
I1014 15:23:55.708631  2612 efa_transport.cpp:275] Auto-split params: page_size=4096, max_pte_entries=23068672, pte_limit=94489280512, max_mr_size=206158430208, chunk_limit=94489280512
I1014 15:23:55.708680  3122 efa_transport.cpp:275] Auto-split params: page_size=4096, max_pte_entries=23068672, pte_limit=94489280512, max_mr_size=206158430208, chunk_limit=94489280512
I1014 15:23:55.718873   584 efa_transport.cpp:790] EfaTransport: registered 138 buffers on 138 threads in 14170ms
[2026-10-14 15:23:56] Engine startup timings (s): load_weight=4.64, kv_cache_allocation=0.71, scheduler_e2e=118.77, cuda_graph={prefill=0.00, decode=44.02, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=130.52
[2026-10-14 15:23:56] INFO:     Started server process [1]
[2026-10-14 15:23:56] INFO:     Waiting for application startup.
[2026-10-14 15:23:56] Using default chat sampling params from model generation config: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-10-14 15:23:56] INFO:     Application startup complete.
[2026-10-14 15:23:56] INFO:     Uvicorn running on http://0.0.0.0:30001 (Press CTRL+C to quit)
[2026-10-14 15:23:57] INFO:     127.0.0.1:57740 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-14 15:23:57] Start of pd disaggregation warmup ...
[2026-10-14 15:24:03] INFO:     127.0.0.1:57756 - "POST /generate HTTP/1.1" 200 OK
  1. tested a basic sanity by querying fi_info on AWS p5en.48xlarge instance
docker run --rm $(for d in /dev/infiniband/uverbs*; do echo -n "--device $d "; done) \                                            
    --entrypoint fi_info lmsysorg/sglang:nightly-dev-efa-20261013-7224fce1 -p efa

provider: efa
    fabric: efa-direct
    domain: rdmap85s0-rdm
    version: 206.0
    type: FI_EP_RDM
    protocol: FI_PROTO_EFA
provider: efa
    fabric: efa-direct
    domain: rdmap86s0-rdm
    version: 206.0
    type: FI_EP_RDM
    protocol: FI_PROTO_EFA

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #38592779615
Latest PR Test (Extra): ❌ Run #38592779623
Latest PR Test (AMD ROCm 10): ⏳ Run #38592779704

@snadampal

snadampal commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor Author

built sglang docker with AWS EFA user space libraries installed

cd ~/SGLang_setup/sglang

docker build \
    --target runtime-efa \
    -f docker/Dockerfile \
    -t sglang:efa \
    --build-arg CUDA_VERSION=13.0.3 \
    --build-arg BUILD_TYPE=all \
    --build-arg BRANCH_TYPE=local \
    --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
    .

and tested the above built docker image for P/D disagg inference between two AWS p5en.48xlarge instances with NIXL LIBFABRIC (efa provider) backend.

@snadampal snadampal changed the title [docker] Install AWS EFA userspace stack (libfabric plus its EFA provider) [docker] add runtime-efa target to build images with AWS EFA libs and mooncake-transfer-engine-efa Sep 25, 2026
@snadampal

Copy link
Copy Markdown
Contributor Author

Tested the newly built image on 2 nodes of AWS p5en.48xlarge instances with EFA for

  1. P-D disagg inference with NIXL+LIBFABRIC (efa provider) backend for KV xfers
  2. P-D disagg inference with Mooncake Transfer Engine + efa transport for KV xfers

@functionstackx

Copy link
Copy Markdown
Collaborator

relates to this issue #40995

@functionstackx

Copy link
Copy Markdown
Collaborator

Tested the newly built image on 2 nodes of AWS p5en.48xlarge instances with EFA for

  1. P-D disagg inference with NIXL+LIBFABRIC (efa provider) backend for KV xfers
  2. P-D disagg inference with Mooncake Transfer Engine + efa transport for KV xfers

Thanks @snadampal ! I was wondering if u can try doing Mooncake store P2P EFA for L3 DRAM between 2 nodes too

Comment thread docker/Dockerfile Outdated
# Prefer the AWS libfabric over the distro libfabric1 that libopenmpi3t64 pulls
# in, and expose fi_info/fi_pingpong for debugging EFA from inside the image.
ENV PATH="/opt/amazon/efa/bin:${PATH}" \
LD_LIBRARY_PATH="/opt/amazon/efa/lib:${LD_LIBRARY_PATH}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's safe to remove this, as the EFA Installer updates the dynamic linker to prioritize this.

# cat /etc/ld.so.conf.d/000_efa.conf 
/opt/amazon/efa/lib

@snadampal

Copy link
Copy Markdown
Contributor Author

tested Mooncake store with EFA backend, did KV offloading from Inference node HBM to remote node DRAM.

node1: SGLang inference with moocake service
node2: mooncake-master and mooncake-store with DRAM registered as the segments
node1: offloaded its KV cache from HBM to node2 DRAM via mooncake TE/efa transport.

@functionstackx

Copy link
Copy Markdown
Collaborator

hi @snadampal

since the new proposed approach is to have an separate efa specific build for vllm/sglang, we might not need to wait for nccl-EP to get well integrated into  sglang/vllm to get multi node WideEP (>=EP16) working out of the box for EFA since  you can reinstall the efa fork'ed DeepEP in the dockerfile i.e.

@ishandhanani

Copy link
Copy Markdown
Collaborator

pls rebase

@alexnails

alexnails commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Hey team,

Can we consider an approach where we make a separate thin Dockerfile built on the published runtime image

The repo already does this: docker/sagemaker.Dockerfile is just FROM lmsysorg/sglang:latest plus a few lines. We can the same with a new docker/efa.Dockerfile (we can separate the workflows out from main workflow as well)

ARG BASE_IMAGE=lmsysorg/sglang:latest-runtime
FROM ${BASE_IMAGE}

ARG TARGETARCH
ARG EFA_INSTALLER_VERSION=1.50.0
ARG EFA_INSTALLER_SHA256=fa6dff8593d866866c13cb4640d9059835cd4efa427971f100ab40c97bef2841
ARG MOONCAKE_EFA_VERSION=0.3.13.post1

RUN set -eux; \
    curl -fsSL --retry 3 -o /tmp/efa.tgz \
        https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz; \
    echo "${EFA_INSTALLER_SHA256}  /tmp/efa.tgz" | sha256sum -c -; \
    tar -xzf /tmp/efa.tgz -C /tmp; \
    apt-get update; \
    cd /tmp/aws-efa-installer && ./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify; \
    /opt/amazon/efa/bin/fi_info -l | grep -q '^efa:'; \
    rm -rf /tmp/efa.tgz /tmp/aws-efa-installer /var/lib/apt/lists/*

RUN --mount=type=cache,target=/root/.cache/pip \
    if [ "${TARGETARCH}" = "amd64" ]; then \
        pip uninstall -y mooncake-transfer-engine-cuda13 && \
        pip install --no-deps mooncake-transfer-engine-efa-cuda13==${MOONCAKE_EFA_VERSION}; \
    fi

ENV PATH="/opt/amazon/efa/bin:${PATH}"

@erezzarum

Copy link
Copy Markdown

Hey team,

Can we consider an approach where we make a separate thin Dockerfile built on the published runtime image

The repo already does this: docker/sagemaker.Dockerfile is just FROM lmsysorg/sglang:latest plus a few lines. We can the same with a new docker/efa.Dockerfile (we can separate the workflows out from main workflow as well)

ARG BASE_IMAGE=lmsysorg/sglang:latest-runtime
FROM ${BASE_IMAGE}

ARG TARGETARCH
ARG EFA_INSTALLER_VERSION=1.50.0
ARG EFA_INSTALLER_SHA256=fa6dff8593d866866c13cb4640d9059835cd4efa427971f100ab40c97bef2841
ARG MOONCAKE_EFA_VERSION=0.3.13.post1

RUN set -eux; \
    curl -fsSL --retry 3 -o /tmp/efa.tgz \
        https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz; \
    echo "${EFA_INSTALLER_SHA256}  /tmp/efa.tgz" | sha256sum -c -; \
    tar -xzf /tmp/efa.tgz -C /tmp; \
    apt-get update; \
    cd /tmp/aws-efa-installer && ./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify; \
    /opt/amazon/efa/bin/fi_info -l | grep -q '^efa:'; \
    rm -rf /tmp/efa.tgz /tmp/aws-efa-installer /var/lib/apt/lists/*

RUN --mount=type=cache,target=/root/.cache/pip \
    if [ "${TARGETARCH}" = "amd64" ]; then \
        pip uninstall -y mooncake-transfer-engine-cuda13 && \
        pip install --no-deps mooncake-transfer-engine-efa-cuda13==${MOONCAKE_EFA_VERSION}; \
    fi

ENV PATH="/opt/amazon/efa/bin:${PATH}"

So for nightlies, you first produce a container and then build the EFA on top of that, overriding BASE_IMAGE?

@ishandhanani

Copy link
Copy Markdown
Collaborator

Hey team,
Can we consider an approach where we make a separate thin Dockerfile built on the published runtime image
The repo already does this: docker/sagemaker.Dockerfile is just FROM lmsysorg/sglang:latest plus a few lines. We can the same with a new docker/efa.Dockerfile (we can separate the workflows out from main workflow as well)

ARG BASE_IMAGE=lmsysorg/sglang:latest-runtime
FROM ${BASE_IMAGE}

ARG TARGETARCH
ARG EFA_INSTALLER_VERSION=1.50.0
ARG EFA_INSTALLER_SHA256=fa6dff8593d866866c13cb4640d9059835cd4efa427971f100ab40c97bef2841
ARG MOONCAKE_EFA_VERSION=0.3.13.post1

RUN set -eux; \
    curl -fsSL --retry 3 -o /tmp/efa.tgz \
        https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz; \
    echo "${EFA_INSTALLER_SHA256}  /tmp/efa.tgz" | sha256sum -c -; \
    tar -xzf /tmp/efa.tgz -C /tmp; \
    apt-get update; \
    cd /tmp/aws-efa-installer && ./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify; \
    /opt/amazon/efa/bin/fi_info -l | grep -q '^efa:'; \
    rm -rf /tmp/efa.tgz /tmp/aws-efa-installer /var/lib/apt/lists/*

RUN --mount=type=cache,target=/root/.cache/pip \
    if [ "${TARGETARCH}" = "amd64" ]; then \
        pip uninstall -y mooncake-transfer-engine-cuda13 && \
        pip install --no-deps mooncake-transfer-engine-efa-cuda13==${MOONCAKE_EFA_VERSION}; \
    fi

ENV PATH="/opt/amazon/efa/bin:${PATH}"

So for nightlies, you first produce a container and then build the EFA on top of that, overriding BASE_IMAGE?

2 things

  1. Yea I agree with @alexnails - it'll be easier to just overlay the EFA bits on top of the container that we publish for official release images
  2. A thought - do we need to publish nightlies for EFA images? Is that not something that can be built in external pipelines by doing something similar to the release where you just build a few layers on top of the existing shipped nightly?

@functionstackx

Copy link
Copy Markdown
Collaborator

Yea I agree with @alexnails - it'll be easier to just overlay the EFA bits on top of the container that we publish for official release images
A thought - do we need to publish nightlies for EFA images? Is that not something that can be built in external pipelines by doing something similar to the release where you just build a few layers on top of the existing shipped nightly?

we would highly prefer if the upstream build pipeline built & publishes official upstream EFA nightly too

Comment thread docker/Dockerfile Outdated
# architecture, so keep the explicit `:+` then `:-` pair.
ARG ARCH_IF_NOT_AMD64=${TARGETARCH#amd64}
ARG PROTOCOL_IF_NOT_AMD64=${ARCH_IF_NOT_AMD64:+rdma}
ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest also setting SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC in this image. SGLang defaults it to UCX (srt/environ.py), so right now --disaggregation-transfer-backend nixl in the runtime-efa image still creates a UCX backend unless the user knows to override it. The EFA path in this image is the LIBFABRIC one.

Suggested change
ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa}
ENV MOONCAKE_PROTOCOL=${PROTOCOL_IF_NOT_AMD64:-efa} \
SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC

Tested on 2x p5en.48xlarge with this stage applied on lmsysorg/sglang:latest-runtime (0.5.20): P/D disaggregation with --disaggregation-transfer-backend nixl + SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC works. Both sides log NIXL KVManager initialized with backend: LIBFABRIC, and outputs are correct. This is unconditional since the EFA installer and NIXL's LIBFABRIC plugin are there on both amd64 and arm64.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code]

@snadampal could this PR also publish the EFA variant as a nightly from release-docker-dev.yml, for example lmsysorg/sglang:nightly-dev-efa-{date}-{short_sha} (plus a moving dev-efa) next to the existing nightly-dev-{date}-{short_sha} tags?

The current diff adds runtime-efa only to release-docker-runtime.yml, so EFA images would appear only with releases. SemiAnalysis InferenceX AgentX submissions pin dated nightly-dev-* images. A release-only EFA tag can't unblock them until the next release, and it would also move them to a different SGLang commit.

Concrete case: the B300 DeepSeek-V4 Dynamo+SGLang disaggregated recipes on AWS EFA use SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC. On nightly-dev-20261013-7224fce1, every prefill rank fails at KV-manager init:

E1013 05:03:12 nixl_agent.cpp:334] createBackend: unsupported backend 'LIBFABRIC'
nixl_cu13._bindings.nixlNotFoundError: NIXL_ERR_NOT_FOUND

With NIXL_LOG_LEVEL=DEBUG, the cause that NIXL otherwise logs only at INFO is:

Failed to load plugin from .../.nixl_cu13.mesonpy.libs/plugins/libplugin_LIBFABRIC.so: /usr/lib/x86_64-linux-gnu/libfabric.so.1: version `FABRIC_1.7' not found

The nightly image has only Ubuntu's libfabric1 1.17.0, whose newest symbol version is FABRIC_1.6. libplugin_LIBFABRIC.so in both nixl-cu13 1.4.1 and 1.5.0 imports fi_getinfo@FABRIC_1.7. The AWS libfabric that this PR's runtime-efa stage installs provides it, so a nightly build of that stage would fix this case.

@github-actions github-actions Bot added documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang labels Oct 14, 2026
@snadampal

Copy link
Copy Markdown
Contributor Author

@alexnails, @ishandhanani I have moved the EFA build to a new dockerfile as an overlay on top of the base images, and also created a new workflow file. please review and provide feedback.

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: style | Confidence: High

Issue

The title and description still describe the old design: a runtime-efa target in docker/Dockerfile and edits to the existing CI workflows. The head commit adds docker/efa.Dockerfile and release-docker-efa.yml instead, leaves docker/Dockerfile unchanged, and also publishes EFA variants of the full release image and of nightly-dev-*. All test results in the description come from the old runtime-efa target.

Fix

Update the title and description to match the head commit. Add the build command and result for docker/efa.Dockerfile with BASE_IMAGE=lmsysorg/sglang:nightly-dev-<date>-<sha> (amd64 is enough), plus a short smoke check on that image, for example fi_info -p efa and NIXL starting with the LIBFABRIC backend. The nightly image has more packages than the runtime image tested so far.

Comment thread docker/efa.Dockerfile Resolved
@nvpohanh

Copy link
Copy Markdown
Collaborator

@snadampal could you address these comments?

@snadampal

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @nvpohanh , will address them.

@snadampal snadampal changed the title [docker] add runtime-efa target to build images with AWS EFA libs and mooncake-transfer-engine-efa [docker] Add AWS EFA image variants built on published images Oct 14, 2026
Add docker/efa.Dockerfile, a thin layer over any published CUDA 13
lmsysorg/sglang image. It installs the AWS EFA userspace stack
(libfabric, rdma-core, aws-ofi-nccl) and, on amd64, swaps in the EFA
Mooncake wheel pinned to the base image's Mooncake version. It also sets
EFA defaults for Mooncake, NIXL (LIBFABRIC) and NCCL (ofi net/tuner).
LD_LIBRARY_PATH puts the AWS libfabric first.

Add release-docker-efa.yml, which runs after the base workflows and
publishes:
  - v{version}-efa, latest-efa (from the release image)
  - v{version}-runtime-efa, latest-runtime-efa (from the runtime image)
  - nightly-dev-efa-{date}-{sha}, dev-efa (from the nightly dev image)

docker/Dockerfile and the existing workflows are unchanged.

Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@snadampal

snadampal commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Hi @nvpohanh, I've updated the PR, please give it a try.

Hi @ishandhanani , @alexnails , appreciate if you could trigger CI run for these new workflows.

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: style | Confidence: High

Issue

The Check PR title job fails on the current title, [docker] Add AWS EFA image variants built on published images (log). .github/workflows/pr-title.yml requires [Type] scope: title, where Type is one of Feat, Fix, Refactor, Perf, Doc, Test, CI, or Chore.

Fix

Rename the PR, for example to [Feat] docker: Add AWS EFA image variants built on published images.

ENVIRONMENT="prod"
;;
"Build and Push Development Docker Images/schedule"|"Build and Push Development Docker Images/workflow_dispatch")
# The base nightly tag is nightly-dev-{date}-{short_sha}, where

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: style | Confidence: High

Issue

This comment block (lines 143-153) is 11 lines, and the concurrency comment on lines 47-51 is 5 lines. Both are long prose that is hard to scan. .claude/rules/comment-style.md asks for one or two lines.

Fix

Keep one or two lines of rationale per block, for example:

# Find the tag this run pushed (same commit, pushed after the run started),
# not today's date; an older nightly of this commit must not move dev-efa.

On line 47: # One group per base run, so only a re-run cancels an in-progress EFA build.

@snadampal snadampal changed the title [docker] Add AWS EFA image variants built on published images [Feat] docker: Add AWS EFA image variants built on published images Oct 15, 2026
@snadampal

Copy link
Copy Markdown
Contributor Author

Hi @nvpohanh , thanks for approving the PR! I will address the comment wording along with any other changes we may need after the CI runs. Please let me know the next steps to get it merged.

@ShangmingCai

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 15, 2026
@Fridge003
Fridge003 merged commit aa5c7e8 into sgl-project:main Oct 15, 2026
229 of 266 checks passed
@snadampal

Copy link
Copy Markdown
Contributor Author

Thanks for merging the PR!

nvpohanh added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Oct 19, 2026
…261019

Move all seven B300 DeepSeek-V4-Pro-0813 Dynamo+SGLang AgentX recipes and
both master-config keys to lmsysorg/sglang:nightly-dev-efa-20261019-3c560089.

- The EFA variant (sgl-project/sglang#41006) adds the AWS EFA installer's
  libfabric. NIXL 1.5.0's LIBFABRIC plugin needs its FABRIC_1.7 symbols,
  which the regular image's Ubuntu libfabric 1.17 lacks, so the plugin
  failed to load and disaggregated workers stopped with
  "unsupported backend 'LIBFABRIC'".
- Use the image's bundled Dynamo 1.6.0.dev20261018 (install: false) in
  place of the b208989fbe source build. It includes ai-dynamo/dynamo#16005,
  which counts attention-DP ranks from attn_dp_size.
- No SGLang arg or env var used by these recipes was renamed, removed or
  given a new default between 7224fce1 and 3c560089, and no Dynamo flag or
  env var they set changed between b208989fbe and the bundled build.

将 B300 DeepSeek-V4-Pro-0813 Dynamo+SGLang AgentX 的七个配方及两个主配置项
更新为 lmsysorg/sglang:nightly-dev-efa-20261019-3c560089。

- EFA 镜像变体(sgl-project/sglang#41006)加入 AWS EFA 安装包中的 libfabric。
  NIXL 1.5.0 的 LIBFABRIC 插件需要其 FABRIC_1.7 符号,而常规镜像的 Ubuntu
  libfabric 1.17 不具备,导致插件无法加载,分离式 worker 报错
  "unsupported backend 'LIBFABRIC'"。
- 改用镜像自带的 Dynamo 1.6.0.dev20261018(install: false),取代 b208989fbe
  源码构建。该版本已包含 ai-dynamo/dynamo#16005,按 attn_dp_size 计算注意力 DP rank。
- 在 7224fce1 与 3c560089 之间,这些配方使用的 SGLang 参数与环境变量均未重命名、
  删除或改变默认值;在 b208989fbe 与镜像自带版本之间,配方设置的 Dynamo 参数与
  环境变量也均无变化。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-maintenance documentation Improvements or additions to documentation hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants