Repository navigation
[MORI] Run MORI EPv2 across hosts (opt-in) - #66735
Open
ChengYM5685 wants to merge 3 commits into
Open
ChengYM5685 wants to merge 3 commits into
ChengYM5685 wants to merge 3 commits into
Conversation
Contributor
|
Documentation preview: https://vllm--66735.org.readthedocs.build/en/66735/ |
ChengYM5685
marked this pull request as ready for review
November 11, 2026 01:51
ChengYM5685
requested review from
ProExpertProg,
WoosukKwon,
bnellnm,
houseroad,
mgoin,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256 and
youkaichao
as code owners
November 11, 2026 01:51
6 tasks done
okorzh-amd
reviewed
Nov 11, 2026
okorzh-amd
reviewed
Nov 11, 2026
MORI EPv2 required every expert-parallel rank in one CCO-LSA team, so EP across hosts fell back to EPv1, and VLLM_MORI_EP_V2=1 failed at startup. MORI's hip backend has an internode path: ranks in the same LSA team are reached over LSA, the others over GDA RDMA; it is selected by `gpu_per_node < world_size`. MoriEPv2All2AllManager now accepts equal LSA teams that tile the EP group and builds its ops with `gpu_per_node` and the hip backend, sizing their arena with MORI's `internode_regions`. The prepare/finalize is unchanged: the internode path dispatches unquantized BF16 and combines by gather, as EPv2 does on one host. Across hosts EPv1 stays the default; VLLM_MORI_EP_V2=1 selects EPv2 when the installed MoRI has the internode path. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…renas
Running MORI EPv2 across hosts with GLM-5.3 FP8 (DP16 + EP16) showed three
issues in the manager:
- An op whose arena outgrows the first window opens another one after
weight loading, when hosts arrive tens of seconds apart; cco's P2P handle
exchange then fails ("failed to connect to peer 0"). Barrier the EP group
before Communicator.init.
- An op that sets kernel_backend itself (AsyncLL) collided with the
internode defaults; let the op's kwargs win.
- An AsyncLL op across hosts was sized with internode_regions, whose send
buffers are per node; take MORI's asyncll_regions as its hip backend does.
Also drop the claim that a window only reserves address space: across
hosts it takes device memory.
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
The selection before startup only guesses whether EP spans hosts: with external load balancing or Ray and ep_size <= 8 over two hosts, or on gfx1250 when the UALink team does not cover every rank, it picks EPv2 without VLLM_MORI_EP_V2=1 and the manager then ran the internode path by default. The manager, which sees the actual CCO-LSA team, now raises with the opt-in hint unless VLLM_MORI_EP_V2=1 is set, as main fails there. Drop mori_epv2_internode_unavailable_reason: every MoRI that has EPv2 (1.2.4 and later) ships internode_regions, so it could not fire. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
ChengYM5685
force-pushed
the
mori-epv2-internode
branch
from
November 11, 2026 08:32
5f0f846 to
86fb7d1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
MORI EPv2 (
--all2all-backend mori, #61361) needs every expert-parallel rank in one CCO-LSA team: across hosts it falls back to MORI EPv1, andVLLM_MORI_EP_V2=1fails at startup. MoRI's EPv2 has an internode path (LSA within a host, GDA RDMA across hosts) in its hip backend; this PR wires it in, opt-in.Claims
VLLM_MORI_EP_V2=1, MORI EPv2 runs across hosts: DeepSeek-V4.1-Flash DEP16, Kimi-K3.1-Pro DP4 x TP4 + EP16 and GLM-5.3 FP8 DEP16 on 2x 8 MI355X, with GSM8K matching one host.Validation
2x 8 MI355X (AMD Pollara AINIC, RoCE),
VLLM_MORI_EP_V2=1. Functional checks (chat, tool call, multi-token decode, a ~349k-token needle), GSM8K (200 questions), short agentic runs:All three logs show the internode path taken (16 ranks as 2 teams of 8). MoRI's own internode test (
tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_internode.py, hidden 7168, top-k 8) passed on the same hosts at 128 (v2_ll kernel) and 1024 (v2 kernel) tokens per rank. The runs used this manager change on an earlier base (vLLM a2bcb56). The GLM-5.3 row needs the second commit: before it, its arena outgrew the first window and the window opened after weight loading failed (ccoWindowRegister: P2P FD exchange failed). That run also carried #65128 and #66734, and with them the same setup passed with DBO (0.985), AsyncLL (0.995) and AsyncLL + DBO (0.98), which also exercises the other two fixes of the second commit.Server command, abridged (DeepSeek-V4.1-Flash, host 0; host 1 adds --headless --data-parallel-start-rank 8)
Unit tests (CPU):
pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.pypasses, with new cases for the manager (one LSA team; two teams of 8 for 16 ranks, whose ops getgpu_per_node=8and the hip backend, withVLLM_MORI_EP_V2=1; the same teams without it, which fail with the opt-in hint; teams that do not tile the group) and for the selection across hosts (with and withoutVLLM_MORI_EP_V2=1), plus the barrier before every window, an op-setkernel_backendand the AsyncLL arena table (each fails without its fix). On a CPU-only host the sharedtests/conftest.pyfixture raisesCannot access accelerator devicearound the first test; that test passes when run after another.CI does not run MORI EPv2 across hosts.
Details
MoriEPv2All2AllManageraccepts equal LSA teams that tile the EP group. Its ops getgpu_per_node(the team size) andkernel_backend="hip", the only MoRI backend with the internode path, and their arena is sized with MoRI'sinternode_regions. A one-time log reports the arena per rank: the window is registered with the NIC as one memory region, and AMD AINIC firmware older than 1.117.5-a-190 silently aliases a region past 2^18 page entries (feat(umbp): warn when a NIC memory region outgrows what an AINIC translates ROCm/mori#896).kernel_backenditself (AsyncLL, [MORI][DBO] Dual batch overlap with MoRI EPv2 (native and AsyncLL), and DBO fixes #65128) does not collide with them, and an AsyncLL op across hosts is sized with MoRI'sasyncll_regions(per-rank send buffers), as the hip backend does, not withinternode_regions.docs/serving/expert_parallel_deployment.mddescribes the opt-in.AI assistance (Claude) was used to write this change and this description. The submitter reviewed the changes and ran the validation above.
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.