Skip to content
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
285d899
add direct MORI EPv2 backend
inkcherry Sep 17, 2026
20d8842
skip attention preshuffle on gfx1250
inkcherry Sep 18, 2026
68c3359
clear unused MORI EPv2 receive rows
inkcherry Sep 18, 2026
f7499b1
clear unwritten Triton MoE output slots
inkcherry Sep 18, 2026
b6e3fa7
consolidate Triton MoE workspace shape checks
inkcherry Sep 18, 2026
ad66023
isolate MoE activation scales from unused scratch
inkcherry Sep 18, 2026
bb28fdb
trim redundant MoE test prose
inkcherry Sep 18, 2026
b0322eb
consolidate MoE workspace shape checks
inkcherry Sep 18, 2026
3afb93a
fix MORI EPv2 configuration for non-gated experts
inkcherry Sep 18, 2026
e8ebcaf
allow MORI EPv2 across one LSA fabric
inkcherry Sep 18, 2026
308fcfa
Bound MORI EPv2 receive rows and harden its start-up
ChengYM5685 Oct 12, 2026
59771d1
Size MORI EPv2 CCO windows from op arenas
ChengYM5685 Oct 13, 2026
2af0662
Let experts write MORI EPv2 combine staging and trim receive-row work
ChengYM5685 Oct 13, 2026
887c4fb
Reject experts MORI EPv2 cannot feed, and tidy the backend
ChengYM5685 Oct 13, 2026
bc75e08
Add a mori all2all backend that picks MORI EPv2 or EPv1
ChengYM5685 Oct 14, 2026
84d7e69
Merge branch 'main' into feat/mori-epv2
ChengYM5685 Oct 15, 2026
c3d2286
Resolve the mori all2all backend in the ROCm platform hook
ChengYM5685 Oct 15, 2026
3c2b208
Broadcast the CCO unique ID from the EP group's first rank
ChengYM5685 Oct 16, 2026
8c00ce7
Merge branch 'main' into feat/mori-epv2
ChengYM5685 Oct 17, 2026
a1c33c0
Keep MORI EPv2 under MORI_ENABLE_SDMA on gfx1250
ChengYM5685 Oct 19, 2026
7dbe5a3
Drop the Triton MoE unrouted-slot changes
ChengYM5685 Oct 20, 2026
9f07df7
Document how to choose among the MoRI backends
ChengYM5685 Oct 20, 2026
88efbfc
Point users from the legacy MoRI backends to mori
ChengYM5685 Oct 20, 2026
7283714
Merge branch 'main' into feat/mori-epv2
ChengYM5685 Oct 26, 2026
ca13162
Name the MoRI EPv1 kernel flag after EPv1
ChengYM5685 Oct 26, 2026
d238711
Point MORI EPv2 expert rejections at VLLM_MORI_EP_V2=0
ChengYM5685 Oct 26, 2026
9fef971
Correct the MoRI EPv1 fallback for quantized-input experts
ChengYM5685 Oct 26, 2026
d7b3208
Merge branch 'main' into feat/mori-epv2
ChengYM5685 Oct 27, 2026
921b876
Merge branch 'main' into feat/mori-epv2
ChengYM5685 Oct 27, 2026
94a08bd
Merge branch 'main' into feat/mori-epv2
vllm-agent Oct 28, 2026
e648839
[Test] Set all2all_backend in the ROCm mm prefix-LM config test
ChengYM5685 Oct 28, 2026
c233884
Merge branch 'main' into feat/mori-epv2
vllm-agent Oct 28, 2026
bb62ddb
Merge remote-tracking branch 'upstream/main' into feat/mori-epv2
ChengYM5685 Oct 29, 2026
a793d2c
Drop gfx1250 bring-up changes unrelated to MORI EPv2
ChengYM5685 Oct 29, 2026
2d65093
Merge branch 'main' into feat/mori-epv2
shen-shanshan Oct 29, 2026
62b99b0
Deprecate mori_high_throughput and mori_low_latency
ChengYM5685 Oct 29, 2026
2eab8b4
Merge remote-tracking branch 'upstream/main' into feat/mori-epv2
ChengYM5685 Oct 29, 2026
c383fe8
Fix DeepSeek-V3.2 on MoRI EPv2 and the host-span check
ChengYM5685 Oct 30, 2026
1ccc073
Merge branch 'main' into feat/mori-epv2
shen-shanshan Oct 30, 2026
4b73f2e
Fall back to MoRI EPv1 where EPv2 cannot run
ChengYM5685 Oct 30, 2026
2e26247
Merge remote-tracking branch 'chengym/feat/mori-epv2' into feat/mori-…
ChengYM5685 Oct 30, 2026
d35f5b9
Docs: list every MoRI EPv2 fallback and its FP8 support
ChengYM5685 Oct 30, 2026
fbdf2ae
Test: run the pinned-EPv2 mori backend without per-tensor FP8
ChengYM5685 Oct 30, 2026
a9ce493
Merge branch 'main' into feat/mori-epv2
vllm-agent Oct 30, 2026
921086e
Detect compressed-tensors and Quark per-tensor FP8; find HIP via Cuda…
ChengYM5685 Oct 30, 2026
715635b
Keep MoRI EPv2 ops alive in the manager; match the arena estimate to …
ChengYM5685 Oct 30, 2026
0503177
Merge branch 'main' into feat/mori-epv2
vllm-agent Oct 30, 2026
ed7689a
Merge upstream main to fix depyf fullgraph CI
ChengYM5685 Oct 31, 2026
5782d84
Merge branch 'main' into feat/mori-epv2
vllm-agent Oct 31, 2026
94603b3
Merge branch 'main' into feat/mori-epv2
vllm-agent Oct 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions docs/serving/expert_parallel_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ vLLM provides multiple communication backends for EP. Use `--all2all-backend` to
| `flashinfer_nvlink_one_sided` | MNNVL systems | FlashInfer's one-sided A2A strategy for multi-node NVLink | High-throughput workloads |
| `passthrough` | Any | Internal, not accepted on the CLI: bound automatically for MoE backends that dispatch and combine themselves, so no all-to-all runs in the framework | `--moe-backend flashinfer_moe_ep_*` (FlashInfer MoE-EP megakernels) |
| `flashinfer_nvlink_two_sided` | MNNVL systems | FlashInfer's two-sided A2A strategy for multi-node NVLink | Systems with NVLink across nodes |
| `mori` | ROCm | MoRI EP: EPv2 (dispatch/combine over a CCO symmetric-memory window) where supported, otherwise EPv1 | AMD Instinct; see [MoRI EP](#mori-ep-rocm) |
Comment thread
ChengYM5685 marked this conversation as resolved.
Resolved

## Single Node Deployment

Expand Down Expand Up @@ -80,6 +81,48 @@ vllm serve deepseek-ai/DeepSeek-V3-0324 \
--enable-expert-parallel # Enable expert parallelism
```

### MoRI EP (ROCm)

`--all2all-backend mori` moves MoE tokens with MoRI. It runs MoRI EPv2 (`mori.ops.dispatch_combine_v2`, which reads and writes peer GPU memory directly through a MoRI CCO symmetric-memory window) where EPv2 is supported, and MoRI EPv1 otherwise.
Comment thread
ChengYM5685 marked this conversation as resolved.
Resolved

```bash
VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0 \
vllm serve deepseek-ai/DeepSeek-V4-Pro \
--tensor-parallel-size 1 \
--data-parallel-size 8 \
--enable-expert-parallel \
--all2all-backend mori
```

`VLLM_MORI_EP_V2` selects the MoRI version:
Comment thread
ChengYM5685 marked this conversation as resolved.
Resolved

| `VLLM_MORI_EP_V2` | Behavior |
| ----------------- | -------- |
| unset | EPv2, falling back to EPv1 with a warning when the EP size exceeds 8 (except on gfx1250, whose UALink team can span hosts), `MORI_ENABLE_SDMA=1`, `--moe-backend aiter_mega_moe`, or a non-BF16 model |
| `1` | Always EPv2. The checks above only log a warning; startup fails if the EP ranks are not in one CCO-LSA team |
| `0` | EPv1 |

On EPv1, `mori` uses the IntraNode kernel within one node. Across nodes it uses InterNodeV1LL when the per-rank token budget (`--max-num-batched-tokens`) is at most `VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD` (default 256), and InterNodeV1 otherwise. `mori_high_throughput` and `mori_low_latency` always run EPv1 and pin the inter-node kernel to InterNodeV1 and InterNodeV1LL respectively.

Choosing a MoRI backend: use EPv2 wherever it applies, and EPv1 for deployments EPv2 does not support. EPv2 cuts its receive buffers to the rows a step can actually receive and lets AITER fused MoE write its output straight into the combine buffer, so it does less work per step than EPv1.

| Deployment | Backend |
| ---------- | ------- |
| EP within one node (or one gfx1250 UALink fabric), BF16 model, AITER fused MoE | `mori` (runs EPv2) |
| EP within one node, with experts that take activations quantized before dispatch (e.g. FP8 or INT8 Triton experts) | EPv2 rejects these experts at startup. Enable AITER fused MoE (`VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1`) so `mori` runs EPv2 with AITER experts; MoRI EPv1 (`VLLM_MORI_EP_V2=0`) also needs AITER fused MoE. To keep Triton experts without AITER, use a non-MoRI backend such as `allgather_reducescatter` |
| EP across nodes | `mori` (falls back to EPv1; picks InterNodeV1LL when the per-rank token budget is at most `VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD`, else InterNodeV1) |

Within one node, all EPv1 options run the same IntraNode kernel. To pin the inter-node kernel, set the threshold: `0` always picks InterNodeV1, and a value at least the per-rank token budget always picks InterNodeV1LL. `mori_high_throughput` and `mori_low_latency` do the same pinning and are kept for compatibility; they will be deprecated in favor of `mori`.

EPv2 limitations:

- All EP ranks must be in one MoRI CCO-LSA team, i.e. GPUs that can load and store each other's memory: one node, or one UALink fabric. Startup fails otherwise; there is no multi-node RDMA path yet.
- Activations are dispatched in BF16, so the model dtype must be BF16 and the experts must take unquantized inputs (AITER fused MoE, which quantizes after the all-to-all, or BF16 experts). Experts that expect activations quantized before dispatch, such as FP8 or INT8 Triton experts, are rejected at startup.
- AITER shared-expert fusion is not supported: set `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0`.
- Install a MoRI build that ships `mori.ops.dispatch_combine_v2`. On NICs that limit a registered memory region to 2 GiB, the window registration fails once the dispatch arena outgrows that limit (DeepSeek-V4-Pro at EP8 already does at the default 16k batched tokens).
- Older MoRI builds leave a stale HIP error behind when the fabric-handle probe fails (e.g. on a single MI355X node), and the next PyTorch allocation raises `operation not supported`.

## Multi-Node Deployment

For multi-node deployment, use the DeepEP communication kernel with one of two modes (see [Backend Selection Guide](#backend-selection-guide) above).
Expand Down
9 changes: 8 additions & 1 deletion tests/kernels/moe/modular_kernel_tools/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,9 @@ def make_env_data(self) -> tuple[VllmConfig, dict[Any, Any]]:
}

vllm_config.parallel_config.all2all_backend = self.all2all_backend()
# "mori" labels the MORI EPv2 prepare/finalize here.
if self.all2all_backend() == "mori":
vllm_config.parallel_config.mori_ep_v2 = True

return vllm_config, env_dict

Expand Down Expand Up @@ -353,7 +356,11 @@ def needs_aiter(self):

def needs_mori(self):
info = prepare_finalize_info(self.prepare_finalize_type)
return info.backend in ("mori_high_throughput", "mori_low_latency")
return info.backend in (
"mori_high_throughput",
"mori_low_latency",
"mori",
)

def all2all_backend(self):
info = prepare_finalize_info(self.prepare_finalize_type)
Expand Down
13 changes: 13 additions & 0 deletions tests/kernels/moe/modular_kernel_tools/mk_objects.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,19 @@ def expert_info(kind) -> ExpertInfo:
supports_apply_weight_on_input=False,
)

from vllm.model_executor.layers.fused_moe.prepare_finalize.mori_epv2 import (
MoriEPv2PrepareAndFinalize,
)

register_prepare_and_finalize(
MoriEPv2PrepareAndFinalize,
standard_format,
[torch.bfloat16],
blocked_quantization_support=False,
backend="mori",
supports_apply_weight_on_input=False,
)

if has_flashinfer_cutlass_fused_moe() and current_platform.has_device_capability(100):
from vllm.model_executor.layers.fused_moe.experts.flashinfer_cutlass_moe import (
FlashInferExperts,
Expand Down
8 changes: 6 additions & 2 deletions tests/kernels/moe/test_moe_layer.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ def on_gfx950() -> bool:
BACKENDS = ["allgather_reducescatter"]

if has_mori():
BACKENDS += ["mori_high_throughput", "mori_low_latency"]
BACKENDS += ["mori_high_throughput", "mori_low_latency", "mori"]

if has_flashinfer_nvlink_two_sided():
BACKENDS += ["flashinfer_nvlink_two_sided"]
Expand All @@ -135,7 +135,7 @@ def on_gfx950() -> bool:
# DeepEP forces IBGDA on, so also hide the NICs from it.
"NVSHMEM_HCA_LIST": "none",
}
MORI_BACKENDS = {"mori_high_throughput", "mori_low_latency"}
MORI_BACKENDS = {"mori_high_throughput", "mori_low_latency", "mori"}

QUANT_METHODS = [
None,
Expand All @@ -151,6 +151,7 @@ def on_gfx950() -> bool:
"allgather_reducescatter": {None, "fp8", "modelopt_fp8", "modelopt_fp4"}, # noqa: E501
"mori_high_throughput": {None, "fp8", "modelopt_fp8"},
"mori_low_latency": {None, "fp8", "modelopt_fp8"},
"mori": {None, "fp8", "modelopt_fp8"},
"flashinfer_nvlink_two_sided": {None, "fp8_blocked", "modelopt_fp4"}, # noqa: E501
"flashinfer_nvlink_one_sided": {None, "fp8_blocked", "modelopt_fp4"}, # noqa: E501
"deepep_low_latency": {None, "fp8_blocked", "modelopt_fp4"}, # noqa: E501
Expand All @@ -163,6 +164,7 @@ def on_gfx950() -> bool:
"allgather_reducescatter": (True, True, True, True),
"mori_high_throughput": (True, False, False, True),
"mori_low_latency": (True, False, False, True),
"mori": (True, False, False, True),
"flashinfer_nvlink_two_sided": (False, True, False, False),
"flashinfer_nvlink_one_sided": (True, True, False, False),
"deepep_low_latency": (True, False, False, True),
Expand Down Expand Up @@ -2189,6 +2191,8 @@ def test_moe_layer(
tensor_parallel_size=tp_size,
enable_expert_parallel=use_ep,
all2all_backend=backend,
# Pin "mori" to EPv2; mori_high_throughput/mori_low_latency cover EPv1.
mori_ep_v2=True if backend == "mori" else None,
enable_eplb=enable_eplb,
eplb_config=eplb_config,
)
Expand Down
Loading
Loading