Skip to content

[MORI] Run MORI EPv2 across hosts (opt-in) - #66735

Open
ChengYM5685 wants to merge 3 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-internode
Open

ChengYM5685 wants to merge 3 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-internode

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Overview

MORI EPv2 (--all2all-backend mori, #61361) needs every expert-parallel rank in one CCO-LSA team: across hosts it falls back to MORI EPv1, and VLLM_MORI_EP_V2=1 fails at startup. MoRI's EPv2 has an internode path (LSA within a host, GDA RDMA across hosts) in its hip backend; this PR wires it in, opt-in.

Claims

  • With VLLM_MORI_EP_V2=1, MORI EPv2 runs across hosts: DeepSeek-V4.1-Flash DEP16, Kimi-K3.1-Pro DP4 x TP4 + EP16 and GLM-5.3 FP8 DEP16 on 2x 8 MI355X, with GSM8K matching one host.
  • A cco window opened after weight loading (when an op's arena outgrows the first window) no longer fails across hosts.
  • The default across hosts stays EPv1; the fallback says how to opt in. Where vLLM cannot tell before startup that EP spans hosts (external load balancing, Ray, gfx1250), the manager still requires the opt-in and fails at startup without it, as on main.
  • No change on one host, and no new flags or environment variables.

Validation

2x 8 MI355X (AMD Pollara AINIC, RoCE), VLLM_MORI_EP_V2=1. Functional checks (chat, tool call, multi-token decode, a ~349k-token needle), GSM8K (200 questions), short agentic runs:

Model Layout MoRI GSM8K Agentic run
DeepSeek-V4.1-Flash TP1 x DP16 + EP16 a66704c8 0.97 (one host, DP8 + EP8: 0.97) c1 / c64, 0 errors
Kimi-K3.1-Pro DP4 x TP4 + EP16 (+ DCP4) 1.2.5.dev94 (765653da) 0.995 c1 / c16, 0 errors
GLM-5.3 (FP8) TP1 x DP16 + EP16, MTP 1 a66704c8 0.99 (MTP acceptance 1.945) not run

All three logs show the internode path taken (16 ranks as 2 teams of 8). MoRI's own internode test (tests/python/ops/dispatch_combine_v2/test_dispatch_combine_v2_internode.py, hidden 7168, top-k 8) passed on the same hosts at 128 (v2_ll kernel) and 1024 (v2 kernel) tokens per rank. The runs used this manager change on an earlier base (vLLM a2bcb56). The GLM-5.3 row needs the second commit: before it, its arena outgrew the first window and the window opened after weight loading failed (ccoWindowRegister: P2P FD exchange failed). That run also carried #65128 and #66734, and with them the same setup passed with DBO (0.985), AsyncLL (0.995) and AsyncLL + DBO (0.98), which also exercises the other two fixes of the second commit.

Server command, abridged (DeepSeek-V4.1-Flash, host 0; host 1 adds --headless --data-parallel-start-rank 8)
VLLM_MORI_EP_V2=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \
vllm serve deepseek-ai/DeepSeek-V4.1-Flash --all2all-backend mori \
  --tensor-parallel-size 1 --enable-expert-parallel \
  --data-parallel-size 16 --data-parallel-size-local 8 \
  --data-parallel-address <host0> --data-parallel-rpc-port 13345 \
  --block-size 128 --engram-config '{"cpu_offload":true}' --gpu-memory-utilization 0.9 \
  --max-model-len 1048576 --max-num-batched-tokens 8192 --max-num-seqs 8 \
  --moe-backend aiter --tokenizer-mode deepseek_v41 --reasoning-parser deepseek_v41 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5}'

Unit tests (CPU): pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.py passes, with new cases for the manager (one LSA team; two teams of 8 for 16 ranks, whose ops get gpu_per_node=8 and the hip backend, with VLLM_MORI_EP_V2=1; the same teams without it, which fail with the opt-in hint; teams that do not tile the group) and for the selection across hosts (with and without VLLM_MORI_EP_V2=1), plus the barrier before every window, an op-set kernel_backend and the AsyncLL arena table (each fails without its fix). On a CPU-only host the shared tests/conftest.py fixture raises Cannot access accelerator device around the first test; that test passes when run after another.

CI does not run MORI EPv2 across hosts.

Details

  • MoriEPv2All2AllManager accepts equal LSA teams that tile the EP group. Its ops get gpu_per_node (the team size) and kernel_backend="hip", the only MoRI backend with the internode path, and their arena is sized with MoRI's internode_regions. A one-time log reports the arena per rank: the window is registered with the NIC as one memory region, and AMD AINIC firmware older than 1.117.5-a-190 silently aliases a region past 2^18 page entries (feat(umbp): warn when a NIC memory region outgrows what an AINIC translates ROCm/mori#896).
  • Each window opens after a barrier of the EP group: cco's init exchanges P2P handles with the node's other ranks, and a window opened after weight loading, where hosts finish tens of seconds apart, failed across hosts without it. Across hosts the window takes device memory, so it stays sized to the ops.
  • The op's own kwargs take precedence over the internode defaults, so an op that sets kernel_backend itself (AsyncLL, [MORI][DBO] Dual batch overlap with MoRI EPv2 (native and AsyncLL), and DBO fixes #65128) does not collide with them, and an AsyncLL op across hosts is sized with MoRI's asyncll_regions (per-rank send buffers), as the hip backend does, not with internode_regions.
  • The prepare/finalize is unchanged: the internode path dispatches unquantized BF16 and combines by gather, as EPv2 does on one host.
  • Opt-in: only two hosts were tested and performance is not compared with EPv1 here, so EPv1 stays the default across hosts. Making EPv2 the default can follow with that data. The selection before startup only guesses whether EP spans hosts, so the manager, which sees the actual CCO-LSA team, enforces the opt-in too. No separate check for MoRI's internode path: every MoRI that has EPv2 (1.2.4 and later) has it.
  • Docs: docs/serving/expert_parallel_deployment.md describes the opt-in.

AI assistance (Claude) was used to write this change and this description. The submitter reviewed the changes and ran the validation above.


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@mergify

mergify Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--66735.org.readthedocs.build/en/66735/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Nov 10, 2026
@ChengYM5685
ChengYM5685 marked this pull request as ready for review November 11, 2026 01:51

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Comment thread vllm/utils/import_utils.py Resolved
MORI EPv2 required every expert-parallel rank in one CCO-LSA team, so EP
across hosts fell back to EPv1, and VLLM_MORI_EP_V2=1 failed at startup.
MORI's hip backend has an internode path: ranks in the same LSA team are
reached over LSA, the others over GDA RDMA; it is selected by
`gpu_per_node < world_size`.

MoriEPv2All2AllManager now accepts equal LSA teams that tile the EP group
and builds its ops with `gpu_per_node` and the hip backend, sizing their
arena with MORI's `internode_regions`. The prepare/finalize is unchanged:
the internode path dispatches unquantized BF16 and combines by gather, as
EPv2 does on one host.

Across hosts EPv1 stays the default; VLLM_MORI_EP_V2=1 selects EPv2 when
the installed MoRI has the internode path.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…renas

Running MORI EPv2 across hosts with GLM-5.3 FP8 (DP16 + EP16) showed three
issues in the manager:

- An op whose arena outgrows the first window opens another one after
  weight loading, when hosts arrive tens of seconds apart; cco's P2P handle
  exchange then fails ("failed to connect to peer 0"). Barrier the EP group
  before Communicator.init.
- An op that sets kernel_backend itself (AsyncLL) collided with the
  internode defaults; let the op's kwargs win.
- An AsyncLL op across hosts was sized with internode_regions, whose send
  buffers are per node; take MORI's asyncll_regions as its hip backend does.

Also drop the claim that a window only reserves address space: across
hosts it takes device memory.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
The selection before startup only guesses whether EP spans hosts: with
external load balancing or Ray and ep_size <= 8 over two hosts, or on
gfx1250 when the UALink team does not cover every rank, it picks EPv2
without VLLM_MORI_EP_V2=1 and the manager then ran the internode path by
default. The manager, which sees the actual CCO-LSA team, now raises
with the opt-in hint unless VLLM_MORI_EP_V2=1 is set, as main fails
there.

Drop mori_epv2_internode_unavailable_reason: every MoRI that has EPv2
(1.2.4 and later) ships internode_regions, so it could not fire.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants