Skip to content

[MORI] Add MORI EP V2 support - #61361

Merged
shen-shanshan merged 50 commits into
vllm-project:mainfrom
ChengYM5685:feat/mori-epv2
Oct 31, 2026
Merged

shen-shanshan merged 50 commits into
vllm-project:mainfrom
ChengYM5685:feat/mori-epv2

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Adds --all2all-backend mori, an expert-parallel all2all backend that runs MoRI EPv2 (mori.ops.dispatch_combine_v2) where it is supported and MoRI EPv1 otherwise. EPv2 dispatches and combines tokens through a MoRI CCO symmetric-memory window, so all EP ranks must sit in one load/store-accessible (LSA) team, e.g. one node.

VLLM_MORI_EP_V2 selects the version. When it is unset, mori prefers EPv2 and falls back to EPv1 with a warning in any of these cases:

  • the EP size exceeds 8 (except on gfx1250, whose UALink team can span hosts);
  • MORI_ENABLE_SDMA=1 is set (except on gfx1250);
  • --moe-backend aiter_mega_moe is used;
  • the model is not BF16.

Setting it to 1 forces EPv2, and 0 selects EPv1. The choice is made once in the ROCm platform's config hook. On EPv1 across nodes, mori uses InterNodeV1LL when the per-rank token budget is at most VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD (default 256), and InterNodeV1 otherwise.

The existing MoRI backends (mori_high_throughput, mori_low_latency) are unchanged and always run EPv1. docs/serving/expert_parallel_deployment.md explains how to choose among them.

The MoRI EPv1 flag use_mori_kernels is renamed to use_mori_epv1_kernels, next to use_mori_epv2_kernels. This touches its existing users in rocm_aiter_moe.py, deepseek_v32 and the MiniMax-M3 test.

No other open PR adds a MoRI EPv2 backend.

Claims

  • On DeepSeek-V4-Pro with DP8+EP8 on 8× MI355X, mori on EPv2 matches TP8 and the existing MoRI EP backend on GSM8K.
  • Under the InferenceX MI355X serving recipe (8k/1k), mori on EPv2 gives about 13% more output throughput and about 11% lower TPOT than mori_high_throughput.

Validation

DeepSeek-V4-Pro, 8× MI355X, --moe-backend aiter, CUDA graphs, MoRI main.

Accuracy

Backend Parallelism GSM8K (lm-eval, 1319 q, 5-shot, strict)
default TP8 0.950
mori_high_throughput DP8+EP8 0.958
mori (EPv2) DP8+EP8 0.955

GSM8K ran with 4096 max new tokens, so about 33 answers per backend were cut off before the final answer. The absolute scores are lower than with a longer budget, but the backends can be compared with each other.

Throughput. InferenceX vLLM MI355X recipe: DP8+EP8, --max-num-batched-tokens 8192, prefix caching, DSpark speculative decoding (6 tokens), and model runner V2. The load is random input/output of 8k/1k, with 640 requests at concurrency 64 and 1280 requests at concurrency 128. Both runs used the same image, on an earlier revision of this branch where the backend was still named mori_epv2.

Backend c64 output tok/s (TPOT ms) c128 output tok/s (TPOT ms) GSM8K (200 q)
mori_high_throughput 1835 (30.3) 2339 (47.0) 0.965
mori (EPv2) 2073 (26.9) 2644 (41.8) 0.960

The eager mode (--enforce-eager) start-up also passed. Unit tests: pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.py.

Follow-ups

  • Make mori the single MoRI backend: fold mori_high_throughput and mori_low_latency into it (deprecate first, then remove). VLLM_MORI_EP_V2 stays as the EPv2/EPv1 override.
  • Add EPv2 across nodes (RDMA), so multi-node EP no longer falls back to EPv1.
  • Use quantized (FP8/FP4) dispatch on EPv2. MoRI supports it, but this backend does not use it yet.

AI assistance (Claude, Codex) was used to write and review this change. The submitter reviewed the changes and ran the validation above.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Motivation: MOTIVATION_RULES.md: "Prefer the soundest design supported by evidence."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Initialize padding on the device before expert kernels consume reused
receive buffers.

Motivation: MOTIVATION_RULES.md: "Prefer the soundest design supported by evidence."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Negative routing IDs are omitted from sorting even without an expert map.
Clear their scratch slots before the second GEMM and final sum.

Motivation: MOTIVATION_RULES.md: "Prefer the soundest design supported by evidence."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Motivation: MOTIVATION_RULES.md: "Keep comments and tests near the host repository's density."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Skipped routes leave stale first-GEMM rows in the shared activation scale.
Clear that scratch only for dynamic per-tensor FP8/INT8 quantization.

Motivation: MOTIVATION_RULES.md: "Prefer the soundest design supported by evidence."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Motivation: MOTIVATION_RULES.md: "Keep comments and tests near the host repository's density."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Motivation: MOTIVATION_RULES.md: "Keep comments and tests near the host repository's density."

Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Codex
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
@mergify mergify Bot added deepseek Related to DeepSeek models gpt-oss Related to GPT-OSS models DSv4 labels Oct 13, 2026
@mergify mergify Bot added the nvidia label Oct 13, 2026
@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@ChengYM5685 ChengYM5685 changed the title Add mori epv2 support [MORI] Add MORI EP V2 support Oct 14, 2026
@mergify

mergify Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--61361.org.readthedocs.build/en/61361/

@mergify mergify Bot added the documentation Improvements or additions to documentation label Oct 14, 2026
- Slice the capacity-sized receive buffers to a host-side bound from the DP
  token counts before masking, experts and combine.
- Accept MORI main's dispatch, which has no clone_routing parameter.
- Let TRITON_UNFUSED accept gfx1250.
- Route DP padding tokens (expert id -1 under VLLM_MOE_SKIP_PADDING) to the
  rank's own experts; MORI EPv2 has no invalid-expert slot and its FlyDSL
  dispatch faults on -1, e.g. in the sampler warm-up dummy run.
- Check that the ops' arenas fit the fixed cco window and name
  VLLM_MORI_EPV2_PER_RANK_VMM_GB when they do not, instead of failing inside
  MORI's allocator.
- Clear the HIP last error after Communicator.init. cco's fabric-handle probe
  fails with hipErrorNotSupported on a single MI355X node and leaves it set,
  so the worker's next allocation raised "operation not supported" and every
  DP+EP start-up failed.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Drop VLLM_MORI_EPV2_PER_RANK_VMM_GB. The first 4 GiB window still opens at
start-up for the LSA-team check; an op whose arena does not fit the remaining
windows gets its own window, rounded to MORI's 4 GiB reservation granularity.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
- Return MORI's combine-input view from allocate_fused_expert_output for
  experts that reduce top-k themselves, so combine skips its copy-in.
- Skip clearing stale activation rows for AITER experts, which only read
  expert_num_tokens rows, unless a dynamic per-tensor scale spans every row.
- Use the full receive buffer when the DP token counts do not cover this
  rank's own batch.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
MORI EPv2 dispatches unquantized BF16 activations. Fail at kernel set-up for
a non-BF16 model dtype or for experts that expect activations quantized
before dispatch (e.g. FP8/INT8 Triton experts), and register the backend as
BF16-only in the modular-kernel test sweep.

Also:
- Require the experts to split evenly over the EP ranks: MORI EPv2 routes
  expert e to rank e // experts_per_rank.
- Free the cached ops before their CCO windows on destroy.
- Drop the hipGetLastError workaround; MORI clears the fabric-probe error
  itself since ROCm/mori#761.
- Drop the clone_routing and combine_in_view compatibility paths, and stop
  passing indices to combine, which takes the mapping from the routing.
- Log the backend's own start-up and fallback messages at debug level.
- Triton MoE: let moe_sum skip unrouted (-1) slots whenever topk_ids is
  given, instead of zeroing intermediate_cache3 on every call; compare the
  FP8 path of the no-mul test by mismatch ratio, since one FP8 rounding flip
  can move an output past the element-wise tolerance.
- Trim the MORI EPv2 unit tests and document the backend.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Replace --all2all-backend mori_epv2 with mori. It runs MORI EPv2 unless the
deployment rules it out, and MORI EPv1 otherwise; mori_high_throughput and
mori_low_latency keep running EPv1 as before.

- VLLM_MORI_EP_V2: unset prefers EPv2 and falls back to EPv1 with a warning
  when the EP size exceeds 8 (except on gfx1250, whose UALink team can span
  hosts), MORI_ENABLE_SDMA=1, moe_backend=aiter_mega_moe or a non-BF16
  model; 1 forces EPv2 and 0 selects EPv1. The choice is made once in
  VllmConfig and carried to the workers in ParallelConfig.mori_ep_v2; the
  CCO-LSA team check at start-up still guards EPv2.
- On EPv1 across nodes, mori uses InterNodeV1LL when the per-rank token
  budget is at most VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD
  (default 256), and InterNodeV1 otherwise.
- Let AITER MegaMoE on DeepSeek-V4 run with mori on EPv1.
- Cover the selection and the inter-node kernel choice in the MORI EPv2 unit
  tests, and document the backend.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@github-actions

Copy link
Copy Markdown

✅ Queued 1 failed job(s) for retry in Buildkite CI #98704.

@ChengYM5685

Copy link
Copy Markdown
Contributor Author

/amd-ci run all

@github-actions

Copy link
Copy Markdown

❌ This PR is 50 commits behind upstream main. The bot could not merge upstream main into this branch (API returned 422: merge conflict between base and head). No new CI build was started. Either merge or rebase onto the latest main yourself and rerun /amd-ci run all, or use /amd-ci run all --allow-stale to test this branch as is.

Includes the set_sys_modules compatibility fix from vllm-project#60873 and resolves the
ModelConfig type import conflict while preserving the MoRI EPv2 changes.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: ChengYM <320424417+ChengYM5685@users.noreply.github.com>
@ChengYM5685

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98895 for commit ed7689a3683a, running 1 failed step(s) from Buildkite CI #98704.

@mergify mergify Bot removed the needs-rebase label Oct 31, 2026
@ChengYM5685

Copy link
Copy Markdown
Contributor Author

/amd-ci run all

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite AMD CI #15753 for commit 5782d8432bd5.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 5782d8432bd5 before starting CI. Pull the branch before pushing more changes.

@peymanr

peymanr commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

PR Pundit review

(Testing PR Pundit Code Review agent. Not all comments may be relevant. This is a one-time automated pass: please address what is useful, no need to request a re-review. A quick reply per point (useful or not, and why) helps us calibrate the agent. Reach out to @peymanr for suggestions and feedback.)

# Finding Where
1 🟠 vLLM logs 'MoRI 1.2.4.dev15 is older than 1.2.4' and runs EPv1: this PR adds a strict version gate vllm/utils/import_utils.py:657

1. 🟠 Should fix: vLLM logs 'MoRI 1.2.4.dev15 is older than 1.2.4' and runs EPv1: this PR adds a strict version gate

vllm/utils/import_utils.py:657

mori_epv2_unavailable_reason compares Version(mori.__version__) < Version("1.2.4"), so pre-release or source builds of 1.2.4 (1.2.4.dev0, 1.2.4rc1, 0.0.0+git) are treated as too old even if they ship mori.cco and dispatch_combine_v2. A missing or unparseable version is reported as MoRI unknown is older than 1.2.4 ... upgrade MoRI, which is misleading. Such builds fall back to EPv1 with a warning; VLLM_MORI_EP_V2=1 still forces EPv2. A broken MoRI import does not crash config resolution, because _has_module catches every import exception and returns False.

Who is affected: ROCm users running the mori all2all backend with a dev/rc/source MoRI build that already has the EPv2 modules but a version string below 1.2.4.

Checked against the code by an independent pass.

Related past reviews: #46419 on vllm/model_executor/layers/fused_moe/experts/rocm_aiter_moe.py, #47896 on vllm/model_executor/layers/fused_moe/experts/rocm_aiter_moe.py

Shared from a pilot of PR Pundit, an agentic code reviewer that builds and tests the change and reads the project's past reviews. Reviewed at 0503177c; findings checked and posted by @peymanr; points already raised on this PR are left out.

@ChengYM5685

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98913 for commit 94603b36fe77.

ℹ️ This PR is 4 commits behind upstream main. Merged upstream main into this branch as 94603b36fe77 before starting CI. Pull the branch before pushing more changes.

@shen-shanshan
shen-shanshan merged commit d5301a4 into vllm-project:main Oct 31, 2026
450 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Oct 31, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Oct 31, 2026
@ChengYM5685
ChengYM5685 deleted the feat/mori-epv2 branch October 31, 2026 09:17
ningding01 pushed a commit to ningding01/vllm that referenced this pull request Nov 17, 2026
…e blocking dispatch instead of before the router

Rebased onto vLLM be47189 (0.34.0rc1.dev302): upstream split
use_mori_kernels into use_mori_epv1_kernels / use_mori_epv2_kernels
(d5301a4 / vllm-project#61361); the deferred launch keys on EPv1 (mori_high_throughput
/ mori_low_latency / mori without EPv2), the path this was measured on.

(cherry picked from commit 06ecf24)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models documentation Improvements or additions to documentation DSv4 gpt-oss Related to GPT-OSS models minimax nvidia ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

Status: Done
Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

10 participants