Skip to content
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
71c0fe1
feat(moe): add MORI EPv2 dispatcher and DeepSeek integration
ChengYM5685 Sep 21, 2026
afeaa72
perf(moe): bound EPv2 receive views and reuse combine staging
ChengYM5685 Sep 21, 2026
fb4d14f
Unify entry parameters and select EPv1 or EPv2 via an environment var…
ChengYM5685 Sep 22, 2026
7f90c84
Simplify MoRI receive bounds
ChengYM5685 Sep 22, 2026
cc8c399
Unify MoRI EP dispatchers
ChengYM5685 Sep 23, 2026
81c3115
[AMD] MoRI EP: refine receive bounds and default to EPv2
ChengYM5685 Sep 30, 2026
1ab5903
[AMD] MoRI EP: select EPv2 with boolean SGLANG_MORI_EP_V2
ChengYM5685 Oct 8, 2026
ee5bba0
[AMD] MoRI EP: resolve dispatcher dtypes at load for unquantized and …
ChengYM5685 Oct 8, 2026
a65823d
[AMD] MoRI EP: register EPv2 env vars in Envs and warm up recv caps o…
ChengYM5685 Oct 8, 2026
fc05625
[AMD] MoRI EP: restore the upstream dispatch-token assertion
ChengYM5685 Oct 8, 2026
478df8c
[AMD] MoRI EP: give each EPv2 op its own CCO window
ChengYM5685 Oct 8, 2026
a30e1a3
[AMD] MoRI EP: limit the fp8_direct_cast NaN fallback to EPv2
ChengYM5685 Oct 8, 2026
09f956c
[AMD] MoRI EP: feed EPv2 dispatch counts to the expert distribution r…
ChengYM5685 Oct 8, 2026
695d886
[AMD] MoRI EP: trim diffs against upstream
ChengYM5685 Oct 8, 2026
ef2df01
[AMD] MoRI EP: trim EPv2 env knobs and keep scatter ops on the full r…
ChengYM5685 Oct 8, 2026
184d09e
[AMD] DeepSeek-V4: merge TBO micro-batch outputs without a ResidualSt…
ChengYM5685 Oct 8, 2026
207d929
[AMD] MoRI EP: drop the fused shared-expert slot from recorder counts
ChengYM5685 Oct 8, 2026
f6bf914
[AMD] MoRI EP: honor an explicit SGLANG_MORI_EP_V2=true and size EPv2…
ChengYM5685 Oct 8, 2026
0a79759
Merge branch 'main' into mori_epv2_optimize
HaiShaw Oct 9, 2026
fb2b7a0
Merge branch 'main' into mori_epv2_optimize
ChengYM5685 Oct 9, 2026
4db16a5
Merge branch 'main' into mori_epv2_optimize
HaiShaw Oct 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/docs/references/environment_variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -517,9 +517,19 @@ SGLang supports various environment variables that can be used to configure its
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_MORI_MOE_MAX_INPUT_TOKENS</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Truncate the dispatch buffer to this many rows before MoE computation, reducing kernel overhead on padding tokens. The value must be &gt;= the actual number of received tokens (<code>totalRecvTokenNum</code>); setting it too small causes incorrect results. <code>0</code> disables truncation (use full buffer).</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Manual MoE input row limit for MORI EPv1 and EPv2. A positive value overrides <code>SGLANG_MORI_RECV_BOUND</code> and is used unrounded, capped at the receive capacity; it only narrows the MoE input view, not the communication buffer. It must be &gt;= the actual received tokens, or results are incorrect. <code>0</code> leaves it to <code>SGLANG_MORI_RECV_BOUND</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>0</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_MORI_EP_V2</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Select MORI EPv2 for <code>--moe-a2a-backend mori</code>. When unset, EPv2 is used only for DeepSeek-V4 models on AITER with <code>ep_size</code> &lt;= 8 (on gfx1250, any <code>ep_size</code> that fits one UALink vPOD), a bfloat16 model dtype and <code>MORI_ENABLE_SDMA</code> off, and MORI EPv1 otherwise. Set to <code>true</code> to use EPv2 even when these checks fail (a warning names the failed check), or <code>false</code> to always use EPv1.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>true</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_MORI_RECV_BOUND</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hand the MoE only the receive rows the per-rank DP token counts can fill (rounded up) instead of the full MORI receive buffer. Unverified layouts, missing metadata and two-batch overlap keep the full view.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>false</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum number of dispatched tokens per rank for the PPLX (<code>--moe-a2a-backend pplx</code>) NVSHMEM buffer allocation. Must be &ge; the worst-case per-rank forward tokens (the per-rank prefill chunk <code>chunked_prefill_size / dp_size</code>, or the decode CUDA-graph batch size); the server refuses to start otherwise.</td>
Expand Down
75 changes: 74 additions & 1 deletion python/sglang/srt/arg_groups/moe_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@
from sglang.srt.environ import envs
from sglang.srt.model_executor.cuda_graph_config import Backend, Phase, with_phase
from sglang.srt.runtime_context import attn_dp_enabled_of, get_platform
from sglang.srt.utils.common import is_sm100_supported, parse_connector_type
from sglang.srt.utils.common import (
get_bool_env_var,
is_gfx1250_supported,
is_sm100_supported,
parse_connector_type,
)

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -460,13 +465,24 @@ def handle_a2a_moe(server_args: Any):
)

if a2a_backend == "mori":
is_epv2 = _resolve_mori_ep_v2(server_args)
if cfg.deepep_mode == "auto":
declare_resolution(
server_args,
"_handle_a2a_moe",
deepep_mode="normal",
)
logger.warning("auto set deepep_mode=`normal` for MORI EP")
elif is_epv2 and cfg.deepep_mode == "low_latency":
declare_resolution(
server_args,
"_handle_a2a_moe",
deepep_mode="normal",
)
logger.warning(
"MORI EPv2 has no separate low-latency kernel; "
"using deepep_mode=`normal` for deepep_mode=`low_latency`"
)

# Check chunked prefill for mori
# Skip validation if chunked prefill is disabled (i.e., size <= 0).
Expand Down Expand Up @@ -682,6 +698,63 @@ def validate_flashinfer_a2a_token_budget(server_args: Any) -> None:
)


def _resolve_mori_ep_v2(server_args: Any) -> bool:
"""Whether to use MORI EPv2.

Unset prefers EPv2 and falls back to EPv1 where it is unsupported; an explicit
true keeps EPv2 regardless, and false selects EPv1.
"""
if not envs.SGLANG_MORI_EP_V2.get():
return False

reason = _mori_epv2_unsupported_reason(server_args)
if reason is None:
return True
if envs.SGLANG_MORI_EP_V2.is_set():
logger.warning(
"SGLANG_MORI_EP_V2=true is set explicitly, so MORI EPv2 is used even "
"though it does not support %s.",
reason,
)
return True

logger.warning(
"MORI EPv2 does not support %s; falling back to MORI EPv1. "
"Set SGLANG_MORI_EP_V2=true to force EPv2, or false to select EPv1.",
reason,
)
# Worker processes inherit the env, so every reader agrees on EPv1.
envs.SGLANG_MORI_EP_V2.set(False)
return False


def _mori_epv2_unsupported_reason(server_args: Any) -> str | None:
# The dispatcher falls back from unsupported dispatch/combine dtypes once it
# knows the weight dtype.
import torch

from sglang.srt.configs.model_config import is_deepseek_v4

cfg = resolved_view(server_args)
# gfx1250 forms its LSA team from the UALink vPOD, which can span hosts and is
# only known once cco is up, so init_mori_epv2_op checks that EP fits in it.
if cfg.tp_size > 8 and not is_gfx1250_supported():
return f"ep_size={cfg.tp_size} across hosts (EPv2 is validated within one host)"
if not envs.SGLANG_USE_AITER.get():
return "SGLANG_USE_AITER=0 (EPv2 experts run on the aiter MoE runner)"
if get_bool_env_var("MORI_ENABLE_SDMA", "false"):
return "MORI_ENABLE_SDMA=1 (SDMA is EPv1-only)"
if parse_connector_type(cfg.model_path) == ConnectorType.INSTANCE:
return "a model loaded through an instance connector (cannot validate it)"
model_config = model_config_of(server_args)
if model_config.dtype != torch.bfloat16:
return f"dtype={model_config.dtype} (EPv2 requires bfloat16)"
if not is_deepseek_v4(model_config.hf_config):
architecture = (model_config.hf_config.architectures or [None])[0]
return f"{architecture} (EPv2 is validated on DeepSeek-V4 only)"
return None


def required_mori_dispatch_tokens_per_rank(server_args: Any) -> int:
"""Max tokens a single rank dispatches through MoRI in one forward."""
cfg = resolving_view(server_args)
Expand Down
11 changes: 11 additions & 0 deletions python/sglang/srt/environ.py
Original file line number Diff line number Diff line change
Expand Up @@ -876,6 +876,17 @@ class Envs:
# and relies on the RDMA retry-exceeded timeout only.
SGLANG_MORI_TRANSFER_TIMEOUT_MS = EnvInt(0)
SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK = EnvInt(4096)
# Unset: EPv2 where moe_hook finds it supported (DeepSeek-V4 only), else EPv1.
# true forces EPv2; false forces EPv1.
SGLANG_MORI_EP_V2 = EnvBool(True)
SGLANG_MORI_DISPATCH_DTYPE = EnvStr("auto")
SGLANG_MORI_COMBINE_DTYPE = EnvStr("auto")
SGLANG_MORI_PREALLOC_MAX_RECV_TOKENS = EnvInt(0)
# Hand AITER only ceil32(sum of per-rank DP tokens) rows of the MORI EP receive
# buffer; unproved layouts or metadata keep the full view. Forces DP-synced buckets.
SGLANG_MORI_RECV_BOUND = EnvBool(False)
# Manual MoE input row limit (EPv1 and EPv2); 0 leaves it to SGLANG_MORI_RECV_BOUND.
SGLANG_MORI_MOE_MAX_INPUT_TOKENS = EnvInt(0)

# ===================================================================
# AMD, ROCm, and AITER
Expand Down
11 changes: 8 additions & 3 deletions python/sglang/srt/eplb/expert_distribution.py
Original file line number Diff line number Diff line change
Expand Up @@ -1092,9 +1092,14 @@ def append(self, value: torch.Tensor):
device = self._buffer.device

if self._size == curr_buffer_size:
new_buffer = torch.zeros(
(2 * curr_buffer_size, *self._item_shape), dtype=dtype, device=device
)
# Growth can happen inside a caller's inference_mode (e.g. DSpark draft);
# an inference tensor buffer would reject later appends made outside it.
with torch.inference_mode(False):
new_buffer = torch.zeros(
(2 * curr_buffer_size, *self._item_shape),
dtype=dtype,
device=device,
)
new_buffer[:curr_buffer_size] = self._buffer
self._buffer = new_buffer

Expand Down
2 changes: 2 additions & 0 deletions python/sglang/srt/layers/moe/ep_moe/layer.py
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,8 @@ def forward_cutlass_w4afp8_masked(


def get_moe_impl_class(quant_config: Optional[QuantizationConfig]):
if get_moe_a2a_backend().is_mori_epv2():
return FusedMoE
# [TODO] kk, temporary solution
if (
get_moe_a2a_backend().is_mori()
Expand Down
130 changes: 28 additions & 102 deletions python/sglang/srt/layers/moe/moe_runner/aiter.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,7 @@
register_pre_permute,
)
from sglang.srt.layers.moe.utils import MoeRunnerBackend
from sglang.srt.runtime_context import get_parallel
from sglang.srt.utils import get_bool_env_var, get_int_env_var, is_gfx95_supported
from sglang.srt.utils import get_bool_env_var, is_gfx95_supported

if TYPE_CHECKING:
from sglang.srt.layers.moe.token_dispatcher.base import CombineInput
Expand Down Expand Up @@ -87,6 +86,8 @@ class AiterRunnerInput(RunnerInput):
# Mori-only fused_moe kwargs.
num_local_tokens: Optional[torch.Tensor] = None
output_dtype: Optional[torch.dtype] = None
# Optional MORI EPv2 combine staging view for fused_moe output.
output: Optional[torch.Tensor] = None

@property
def runner_backend(self) -> MoeRunnerBackend:
Expand Down Expand Up @@ -156,95 +157,11 @@ def _aiter_fused_moe_supports_no_combine() -> bool:
return "no_combine" in inspect.signature(fused_moe).parameters


_RECV_BOUND_LOGGED: set[int] = set()
_RECV_BOUND_WARNED = False


def _warn_recv_bound_unavailable() -> None:
global _RECV_BOUND_WARNED
if not _RECV_BOUND_WARNED:
_RECV_BOUND_WARNED = True
logger.warning(
"SGLANG_MORI_RECV_BOUND is set but the per-rank DP token counts do "
"not cover every mori sender, so the receive fan-in is unknown; "
"leaving the receive buffer unbounded."
)


def _mori_decode_recv_bound(recv_rows: int, topk: int) -> int:
"""Live rows mori's receive buffer can hold in decode, or 0 for "do not bound".

Worst case fan-in is every rank routing all of its tokens to this one, so
`sum(per-rank tokens) * topk`, where topk already includes the fused shared
expert. The per-rank counts come from the DP sync, so this is the fan-in for
the batch actually being run rather than an upper bound over all batches.

That is only sound because enabling this gate also makes
`require_mlp_tp_gather()` true for mori, which gives every rank the same
cuda-graph bucket. The value is baked into a captured graph and has to hold
for every later replay; with per-rank buckets a rank on a narrow tier could
be handed rows by a peer on a wider one, and the only bound valid under that
is the widest tier's -- 4-16x looser than the batch being run, which costs
more in expert-GEMM tiles (M 32/64 -> 128) than the trim saves.

Two cases stay unbounded, because a bound below the real fan-in silently
drops rows from the all-to-all -- wrong output rather than an error:

* Prefill, whose per-rank counts are uneven and not knowable here.
* Anything that leaves the per-rank counts unpopulated, or where the EP world
is wider than the DP world so the counts do not cover every sender.
"""
if not get_bool_env_var("SGLANG_MORI_RECV_BOUND", "false"):
return 0
@functools.cache
def _aiter_fused_moe_supports_output() -> bool:
from aiter.fused_moe import fused_moe

from sglang.srt.layers.dp_attention import (
get_dp_global_num_tokens,
get_is_extend_in_batch,
)

if get_is_extend_in_batch():
return 0

per_rank_tokens = get_dp_global_num_tokens()
ep_size = get_parallel().moe_ep_size
if not per_rank_tokens or len(per_rank_tokens) < ep_size:
# Either the DP sync did not publish counts, or they do not cover every
# mori sender. Both mean the fan-in is unknown here.
_warn_recv_bound_unavailable()
return 0

max_tokens = sum(per_rank_tokens)
bound = max_tokens * topk
# Never grow the tensor, and nothing to do when there is nothing to trim.
if not 0 < bound < recv_rows:
return 0

# One INFO line the first time it engages, so an inert bound is not mistaken
# for an active one in the results. Per-tier values go to DEBUG: capture
# visits every tier, and at INFO on every rank that is dozens of lines.
if get_parallel().tp_rank == 0 and bound not in _RECV_BOUND_LOGGED:
first = not _RECV_BOUND_LOGGED
_RECV_BOUND_LOGGED.add(bound)
if first:
logger.info(
"mori recv bound active: %d rows -> %d for this tier "
"(dp_tokens=%d ep=%d topk=%d); per-tier values at DEBUG",
recv_rows,
bound,
max_tokens,
get_parallel().moe_ep_size,
topk,
)
else:
logger.debug(
"mori recv bound: %d rows -> %d (dp_tokens=%d ep=%d topk=%d)",
recv_rows,
bound,
max_tokens,
get_parallel().moe_ep_size,
topk,
)
return bound
return "output" in inspect.signature(fused_moe).parameters


class AiterRunnerCore(MoeRunnerCore):
Expand Down Expand Up @@ -283,8 +200,6 @@ def run(

from aiter.fused_moe import fused_moe

from sglang.srt.environ import envs

a1_scale = (
runner_input.a1_scale
if runner_input.a1_scale is not None
Expand Down Expand Up @@ -330,6 +245,15 @@ def run(
extra["swiglu_limit"] = quant_info.swiglu_limit
if self.config.no_combine:
extra["no_combine"] = True
elif (
runner_input.output is not None
and "output" not in extra
and extra.get("stage2_scatter") is None
and _aiter_fused_moe_supports_output()
):
# MORI EPv2 skips staging when this exact arena view is returned. Older
# AITER versions, no_combine and stage2_scatter keep their old path.
extra["output"] = runner_input.output
extra["moe_sorting_dispatch_policy"] = _MOE_SORTING_DISPATCH_POLICY

# gfx950 small-M MXFP4 kernel (on by default, SGLANG_ROCM_SMALLM_MOE=0 disables): same layouts as aiter, bf16 activations.
Expand Down Expand Up @@ -432,8 +356,8 @@ def pre_permute_standard_to_aiter(


def _is_mori_dispatch_output(dispatch_output: Any) -> bool:
# MoriEP{Normal,LL}DispatchOutput carry the post-mori-permute origin_topk_*
# tensors that the standard DeepEP outputs lack.
# MORI EP dispatch outputs carry the origin_topk_* tensors for combine,
# which the standard DeepEP outputs lack.
return hasattr(dispatch_output, "origin_topk_ids")


Expand Down Expand Up @@ -486,6 +410,7 @@ def _pre_permute_deepep_to_aiter(
a1_scale: Optional[torch.Tensor] = None
num_local_tokens: Optional[torch.Tensor] = None
output_dtype: Optional[torch.dtype] = None
output: Optional[torch.Tensor] = None
quant_type = quant_info.quant_type

if is_mori:
Expand All @@ -494,21 +419,21 @@ def _pre_permute_deepep_to_aiter(
a1_scale = dispatch_output.hidden_states_scale
num_local_tokens = dispatch_output.num_recv_tokens_per_expert
output_dtype = dispatch_output.out_dtype
output = getattr(dispatch_output, "expert_output", None)
mori_max = dispatch_output.recv_cap

# Truncate dispatch tensors to the configured cap; mori combine only
# Truncate dispatch tensors to the configured cap; MORI EP combine only
# reads [0, totalRecvTokenNum), so the truncated result needs no
# padding back.
mori_max = get_int_env_var("SGLANG_MORI_MOE_MAX_INPUT_TOKENS", 0)
if mori_max <= 0:
mori_max = _mori_decode_recv_bound(
hidden_states.shape[0], topk_ids.shape[-1]
)
if mori_max > 0:
# padding back. Slice the optional EPv2 output to the same row count,
# retaining the physical arena and routing stride.
if 0 < mori_max < hidden_states.shape[0]:
hidden_states = hidden_states[:mori_max]
if a1_scale is not None:
a1_scale = a1_scale[:mori_max]
topk_ids = topk_ids[:mori_max]
topk_weights = topk_weights[:mori_max]
if output is not None:
output = output[:mori_max]

# Upscale dispatched activations when there is no AITER kernel for the
# weight/activation dtype pair.
Expand Down Expand Up @@ -595,6 +520,7 @@ def _pre_permute_deepep_to_aiter(
a1_scale=a1_scale,
num_local_tokens=num_local_tokens,
output_dtype=output_dtype,
output=output,
)


Expand Down
Loading
Loading