Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/serving/expert_parallel_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,8 +112,8 @@ vllm serve deepseek-ai/DeepSeek-V4-Pro \

| `VLLM_MORI_EP_V2` | Behavior |
| ----------------- | -------- |
| unset | EPv2, falling back to EPv1 with a warning when MoRI is older than 1.2.4, the GPU is not gfx950 or gfx1250, EP spans hosts (except on gfx1250, whose UALink team can span hosts), `MORI_ENABLE_SDMA=1`, `--moe-backend aiter_mega_moe`, the model is not BF16, or MoE activations use per-tensor FP8 |
| `1` | Always EPv2. The checks above only log a warning; startup fails if the EP ranks are not in one CCO-LSA team |
| unset | EPv2, falling back to EPv1 with a warning when MoRI is older than 1.2.4, the GPU is not gfx950 or gfx1250, EP spans hosts (except on gfx1250, whose UALink team can span hosts; across hosts EPv2 is opt-in), `MORI_ENABLE_SDMA=1`, `--moe-backend aiter_mega_moe`, the model is not BF16, or MoE activations use per-tensor FP8 |
| `1` | Always EPv2. The checks above only log a warning. Across hosts EPv2 runs its internode path (see below); startup fails if the EP ranks do not split into equal CCO-LSA teams |
| `0` | EPv1 |

On EPv1, `mori` uses the IntraNode kernel within one node. Across nodes it uses InterNodeV1LL when the per-rank token budget (`--max-num-batched-tokens`) is at most `VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD` (default 256), and InterNodeV1 otherwise. `mori_high_throughput` and `mori_low_latency` always run EPv1 and pin the inter-node kernel to InterNodeV1 and InterNodeV1LL respectively.
Expand All @@ -124,13 +124,13 @@ Choosing a MoRI backend: use EPv2 wherever it applies, and EPv1 for deployments
| ---------- | ------- |
| EP within one node (or one gfx1250 UALink fabric), BF16 model, AITER fused MoE | `mori` (runs EPv2) |
| EP within one node, with experts that take activations quantized before dispatch (e.g. FP8 or INT8 Triton experts) | EPv2 rejects these experts at startup. Enable AITER fused MoE (`VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1`) so `mori` runs EPv2 with AITER experts; MoRI EPv1 (`VLLM_MORI_EP_V2=0`) also needs AITER fused MoE. To keep Triton experts without AITER, use a non-MoRI backend such as `allgather_reducescatter` |
| EP across nodes | `mori` (falls back to EPv1; picks InterNodeV1LL when the per-rank token budget is at most `VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD`, else InterNodeV1) |
| EP across nodes | `mori` (falls back to EPv1; picks InterNodeV1LL when the per-rank token budget is at most `VLLM_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD`, else InterNodeV1). `VLLM_MORI_EP_V2=1` runs EPv2's internode path instead |

Within one node, all EPv1 options run the same IntraNode kernel. To pin the inter-node kernel, set the threshold: `0` always picks InterNodeV1, and a value at least the per-rank token budget always picks InterNodeV1LL. `mori_high_throughput` and `mori_low_latency` do the same pinning but are deprecated and will be removed in a future release; use `mori` instead.

EPv2 limitations:

- All EP ranks must be in one MoRI CCO-LSA team, i.e. GPUs that can load and store each other's memory: one node, or one UALink fabric. Startup fails otherwise; there is no multi-node RDMA path yet.
- Within one MoRI CCO-LSA team (GPUs that can load and store each other's memory: one node, or one UALink fabric) EPv2 moves tokens through the shared window. Across nodes, with `VLLM_MORI_EP_V2=1`, it runs MoRI's internode path: ranks in the same team are reached as before and the others over GDA RDMA. The EP ranks must split into equal teams (e.g. 2 nodes x 8 GPUs). Where vLLM cannot tell before startup that EP spans hosts (external load balancing, Ray, or a gfx1250 UALink team that does not cover every rank), startup fails unless `VLLM_MORI_EP_V2=1` is set.
- Activations are dispatched in BF16, so the model dtype must be BF16 and the experts must take unquantized inputs (AITER fused MoE, which quantizes after the all-to-all, or BF16 experts). Experts that expect activations quantized before dispatch, such as FP8 or INT8 Triton experts, are rejected at startup. Block FP8 and MXFP4 weights work; per-tensor FP8 activations (`--quantization fp8` without blocks, ModelOpt FP8) do not, since AITER cannot quantize the received rows per tensor.
- AITER shared-expert fusion is not supported: set `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0`.
- Install a MoRI build that ships `mori.ops.dispatch_combine_v2`. On NICs that limit a registered memory region to 2 GiB, the window registration fails once the dispatch arena outgrows that limit (DeepSeek-V4-Pro at EP8 already does at the default 16k batched tokens).
Expand Down
155 changes: 149 additions & 6 deletions tests/kernels/moe/test_mori_epv2_prepare_finalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,18 @@ def test_padding_tokens_are_routed_to_local_experts(monkeypatch):
assert op.sent_ids.tolist() == [[0, 7], [2, 3]]


@pytest.mark.parametrize("world_size,lsa_size", [(8, 8), (8, 4)])
def test_manager_requires_one_lsa_team(monkeypatch, world_size, lsa_size):
@pytest.mark.parametrize(
"world_size,lsa_size,opt_in,gpu_per_node",
[
(8, 8, False, None),
# EP over two hosts: internode, with equal LSA teams, when opted in.
(16, 8, True, 8),
# Not opted in: fails as on main, whatever the guess before startup.
(16, 8, False, "opt-in"),
(8, 3, True, "error"),
],
)
def test_manager_lsa_teams(monkeypatch, world_size, lsa_size, opt_in, gpu_per_node):
from vllm.distributed.device_communicators import all2all
from vllm.distributed.device_communicators.base_device_communicator import (
All2AllManagerBase,
Expand All @@ -161,6 +171,7 @@ def test_manager_requires_one_lsa_team(monkeypatch, world_size, lsa_size):
monkeypatch.setattr(
torch.distributed, "broadcast_object_list", lambda *a, **k: None
)
monkeypatch.setattr(torch.distributed, "barrier", lambda *a, **k: None)
monkeypatch.setattr(all2all, "CudaRTLibrary", Mock)
device_comm = Mock(lsa_size=lsa_size)
calls = Mock() # records the order of op.close and comm.destroy
Expand All @@ -177,8 +188,14 @@ def test_manager_requires_one_lsa_team(monkeypatch, world_size, lsa_size):
GDA_CONNECTION_NONE=0,
),
)
if lsa_size != world_size:
with pytest.raises(ValueError, match="one LSA team"):
monkeypatch.delenv("VLLM_MORI_EP_V2", raising=False)
if opt_in:
monkeypatch.setenv("VLLM_MORI_EP_V2", "1")
if gpu_per_node == "opt-in":
with pytest.raises(ValueError, match="opt-in: set VLLM_MORI_EP_V2=1"):
all2all.MoriEPv2All2AllManager(object())
elif gpu_per_node == "error":
with pytest.raises(ValueError, match="equal LSA teams"):
all2all.MoriEPv2All2AllManager(object())
else:
cfg = dict(
Expand All @@ -192,16 +209,34 @@ def test_manager_requires_one_lsa_team(monkeypatch, world_size, lsa_size):
combine_dtype=None,
schedule=None,
)
configs = []

def make_config(**kw):
config = SimpleNamespace(**(cfg | kw))
config.is_internode = config.__dict__.get("gpu_per_node") is not None
configs.append(config)
return config

monkeypatch.setitem(
sys.modules,
"mori.ops.dispatch_combine_v2",
SimpleNamespace(
EpDispatchCombineConfig=lambda **kw: SimpleNamespace(**(cfg | kw)),
EpDispatchCombineConfig=make_config,
EpDispatchCombineOp=lambda config, comm: calls.op,
),
)
monkeypatch.setitem(
sys.modules,
"mori.ops.dispatch_combine_v2.internode_regions",
SimpleNamespace(internode_regions=lambda config: [("a", 1), ("b", 300)]),
)
manager = all2all.MoriEPv2All2AllManager(object())
assert manager.get_handle({}) is calls.op
(config,) = configs
assert config.__dict__.get("gpu_per_node") == gpu_per_node
assert config.__dict__.get("kernel_backend") == (
"hip" if gpu_per_node else None
)
# The cache holds ops weakly; the manager keeps them alive itself.
manager.handle_cache._cache.clear()
manager.destroy()
Expand All @@ -212,6 +247,112 @@ def test_manager_requires_one_lsa_team(monkeypatch, world_size, lsa_size):
comm.destroy.assert_called_once()


def _internode_manager(monkeypatch, order, regions):
"""A 16-rank MoriEPv2All2AllManager over two LSA teams of 8, MORI faked,
with the internode path opted in."""
from vllm.distributed.device_communicators import all2all
from vllm.distributed.device_communicators.base_device_communicator import (
All2AllManagerBase,
)

monkeypatch.setattr(
All2AllManagerBase,
"__init__",
lambda self, group: self.__dict__.update(
rank=0, world_size=16, cpu_group=group
),
)
monkeypatch.setattr(
torch.distributed,
"broadcast_object_list",
lambda *a, **k: order.append("broadcast"),
)
monkeypatch.setattr(
torch.distributed, "barrier", lambda *a, **k: order.append("barrier")
)
monkeypatch.setattr(all2all, "CudaRTLibrary", Mock)
communicator = Mock()

def init(world_size, rank, uid, per_rank_vmm):
order.append(("init", per_rank_vmm))
return Mock(create_dev_comm=Mock(return_value=Mock(lsa_size=8)))

communicator.init.side_effect = init
monkeypatch.setitem(
sys.modules,
"mori.cco",
SimpleNamespace(
Communicator=communicator,
CCODevCommRequirements=SimpleNamespace,
GDA_CONNECTION_NONE=0,
),
)
configs = []

def make_config(**kw):
config = SimpleNamespace(
dispatch_dtype=None,
combine_dtype=None,
effective_max_recv=0,
schedule=None,
**kw,
)
config.is_internode = True
configs.append(config)
return config

monkeypatch.setitem(
sys.modules,
"mori.ops.dispatch_combine_v2",
SimpleNamespace(
EpDispatchCombineConfig=make_config,
EpDispatchCombineOp=lambda config, comm: Mock(),
),
)
monkeypatch.setitem(
sys.modules, "mori.ops.dispatch_combine_v2.internode_regions", regions
)
monkeypatch.setenv("VLLM_MORI_EP_V2", "1")
return all2all.MoriEPv2All2AllManager(object()), configs


def test_ranks_line_up_before_each_cco_window(monkeypatch):
order: list = []
regions = SimpleNamespace(internode_regions=lambda cfg: [("a", 5 << 30)])
manager, _ = _internode_manager(monkeypatch, order, regions)
# The 5 GiB arena does not fit the first 4 GiB window: a second one opens,
# as it does after weight loading.
manager.get_handle({})
inits = [i for i, step in enumerate(order) if isinstance(step, tuple)]
assert len(inits) == 2
for i in inits:
assert order[i - 2 : i] == ["broadcast", "barrier"]


def test_internode_op_may_set_its_own_kernel_backend(monkeypatch):
regions = SimpleNamespace(internode_regions=lambda cfg: [("a", 1)])
manager, configs = _internode_manager(monkeypatch, [], regions)
manager.get_handle({"kernel_backend": "hip"})
(config,) = configs
assert config.kernel_backend == "hip" and config.gpu_per_node == 8


def test_internode_asyncll_arena_uses_the_asyncll_regions(monkeypatch):
order: list = []
regions = SimpleNamespace(
internode_regions=lambda cfg: [("a", 1)],
asyncll_regions=lambda cfg: [("a", 5 << 30)],
)
manager, _ = _internode_manager(monkeypatch, order, regions)
manager.get_handle({"async_ll": True})
# AsyncLL's per-rank send buffers outgrow the first window; the per-node
# internode table (1 byte here) would have fitted it and overrun.
assert [step for step in order if isinstance(step, tuple)][-1] == (
"init",
8 << 30,
)


def _unique_id_worker(rank, init_method):
from vllm.distributed.device_communicators import all2all
from vllm.distributed.device_communicators.base_device_communicator import (
Expand Down Expand Up @@ -345,8 +486,10 @@ def _quark(qscheme):
"parallel,model,moe_backend,env,ep_v2",
[
({}, {}, "aiter", {}, True),
# EP across hosts: DP over two nodes, even with ep_size <= 8.
# EP across hosts: DP over two nodes, even with ep_size <= 8. MORI's
# internode path is opt-in.
({"data_parallel_size": 16}, {}, "aiter", {}, False),
({"data_parallel_size": 16}, {}, "aiter", {"VLLM_MORI_EP_V2": "1"}, True),
({"data_parallel_size_local": 4}, {}, "aiter", {}, False),
# A DP replica (TP/PP) spanning nodes.
({"nnodes_within_dp": 2}, {}, "aiter", {}, False),
Expand Down
16 changes: 12 additions & 4 deletions vllm/config/parallel.py
Original file line number Diff line number Diff line change
Expand Up @@ -1400,10 +1400,18 @@ def _mori_epv2_unsupported_reason(
* parallel_config.data_parallel_size
* parallel_config.prefill_context_parallel_size
)
# A guess before startup; MoriEPv2All2AllManager checks the actual
# CCO-LSA team.
if not lsa_spans_hosts and _ep_spans_hosts(parallel_config, ep_size):
return f"ep_size={ep_size} across hosts (EPv2 needs one host)"
# The internode path is new: keep EPv1 across hosts unless asked for. A
# guess before startup; MoriEPv2All2AllManager checks the actual CCO-LSA
# team and enforces the opt-in there too.
if (
not envs.VLLM_MORI_EP_V2
and not lsa_spans_hosts
and _ep_spans_hosts(parallel_config, ep_size)
):
return (
f"ep_size={ep_size} across hosts by default (its internode path "
"is opt-in: set VLLM_MORI_EP_V2=1)"
)
sdma = os.environ.get("MORI_ENABLE_SDMA", "0").lower() in ("1", "true")
if sdma and sdma_picks_epv1:
return "MORI_ENABLE_SDMA=1 (SDMA is EPv1-only)"
Expand Down
Loading
Loading