Skip to content

[ROCm][MORI][Perf] Read the EPv2 AsyncLL segmented receive in place - #66697

Draft
ChengYM5685 wants to merge 14 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-segmented-recv
Draft

ChengYM5685 wants to merge 14 commits into
vllm-project:mainfrom
ChengYM5685:mori-epv2-segmented-recv

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Overview

On MoRI EPv2 AsyncLL, the AITER experts now read MoRI's segmented receive layout (ROCm/mori#918) in place, so MoRI no longer copies the received tokens into a compact buffer first.

Draft, stacked on #65128. This branch contains #65128's commits; only the last commit, [MORI] Read the AsyncLL segmented receive in place, belongs to this PR. It also needs ROCm/mori#918, which is still in review, so the MoRI-side API (async_ll_recv_layout, segment_stride) may change. I will rebase onto main once #65128 merges. The same change for SGLang is sgl-project/sglang#49332.

Claims

  • DeepSeek-V4-Pro, DP8 + EP8 on 8x MI355X, MoRI EPv2 AsyncLL (SDMA), prefill-only load: +1.48% total token throughput and -1.7% mean TTFT without DBO; +0.37% throughput and -1.6% TTFT with DBO.
  • GSM8K accuracy is unchanged within noise.
  • No new flag, env var or config field. MoRI builds without What is the supported input length for qwen? #918, native EPv2, CUDA graph capture and non-AITER experts keep the current compact receive.

Validation

Unit tests: pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.py gives 74 passed, on CPU and in the GPU image below. The 7 new tests cover the in-place receive, the five fallbacks to the compact receive, and the error when a rank's tokens exceed the stride.

End to end: one MI355X node, image vllm/vllm-openai-rocm:nightly-083fbac0 with MoRI #918 (fd718da) installed. Both arms use that image; they differ only in the vLLM Python files (#65128 head vs. this commit). Arms run interleaved on the same node (A B B A), 2 benchmark rounds each.

MORI_ENABLE_SDMA=1 MORI_ASYNCLL_EARLY_NOTIFY=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \
vllm serve DeepSeek-V4-Pro --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \
  --all2all-backend mori --moe-backend aiter --kv-cache-dtype fp8 --max-num-batched-tokens 8192 \
  --max-model-len 16384 --gpu-memory-utilization 0.85 --no-enable-prefix-caching \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' [--enable-dbo]
vllm bench serve --dataset-name random --random-input-len 8192 --random-output-len 1 \
  --max-concurrency 256 --num-prompts 512 --ignore-eos --seed 42
Config Compact receive (#65128) Segmented receive (this PR) Change
No DBO, total tok/s (4 runs) 53,490 / 53,485 / 53,498 / 53,471 54,487 / 53,933 / 54,343 / 54,341 +1.48%
No DBO, mean TTFT 29.76 s 29.25 s -1.7%
DBO, total tok/s (4 runs) 65,615 / 65,636 / 65,665 / 65,656 65,886 / 65,902 / 65,868 / 65,887 +0.37%
DBO, mean TTFT 24.17 s 23.79 s -1.6%

Run-to-run spread within an arm is about 1% without DBO and under 0.1% with DBO. In both configurations, every segmented run is faster than every compact run.

GSM8K (5-shot, 400 questions, lm_eval local-completions): compact 0.9625, segmented 0.9575, segmented + DBO 0.965, all within ±0.01 stderr.

CI coverage: the unit tests run in the AMD kernels/moe step with a fake MoRI op. CI images do not have MoRI #918, so the end-to-end path is validated only by the runs above.

Details

  • Opt-in by capability. all2all_utils sets async_ll_recv_layout="auto" on AsyncLL ops when the installed MoRI has that field (mori_epv2_has_segmented_recv). With "auto", MoRI builds the layout only when the op config supports it (single-node SDMA).
  • Arena sizing. vLLM sizes the cco window before the op exists, and the op resolves "auto" only when it is built. _mori_epv2_arena_bytes therefore sizes for "segmented", which is the larger layout. Otherwise the op could fail inside a window that is too small.
  • One stride on every rank. MoRI requires every rank to use the same stride. _segment_stride derives it only from what all ranks share: the op, graph capture, and dp_metadata.num_tokens_across_dp_cpu (per-microbatch under DBO). The stride is the largest DP batch rounded up to 32 rows. If a rank's own tokens exceed it, the code raises instead of falling back on that rank alone.
  • Fallbacks to the compact receive:
    • non-AITER experts, because the holes between segments carry another rank's expert with weight 0 and only experts honoring expert_mask drop them;
    • CUDA graph capture;
    • a stride larger than MoRI's per-source capacity;
    • DP batches uneven enough that world_size * stride would exceed 1.25x the rows actually sent. This is the same bound as SGLang's MoRI path.
  • MoE input. A segmented dispatch hands the experts all world_size * stride rows, without the receive-row bound or row clearing of the compact path. The experts get no received-row count: it would only repeat the row count, and AITER's MXFP4 W4A4 path faults on small batches given one ( [MORI] Run MORI EPv2 on Quark MXFP4 MoE models; stop passing the received-row count #66733). Combine already writes the expert output into combine_in_view(), which is what MoRI needs.
  • No host sync. The new code reads only the CPU tensor num_tokens_across_dp_cpu.
  • Why the gain is smaller with DBO. I have not profiled this. My guess is that the other microbatch already overlaps part of the receive copy, so removing it saves less.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

Not a duplicate: no open PR uses MoRI's segmented receive. I searched open PRs for "mori segmented", "segment_stride", "async_ll_recv_layout" and "AsyncLL receive"; only #65128, on which this is stacked, matched.

ChengYM5685 and others added 13 commits November 4, 2026 03:38
The memory profile does not microbatch, so it sizes only microbatch 0's
workspace slots. A later microbatch can need as much scratch as the whole
batch (the expert all-to-all receives every rank's tokens), so its slot
grew mid-forward: an empty_cache device sync inside a microbatch thread,
and once the workspace is locked after graph capture, an AssertionError
that kills the thread and leaves its peer parked, hanging the step.

After the profile run, grow each lane's microbatch slots to the largest
slot of that lane, before capture locks them.

The V1 runner's profile run did microbatch, which sized microbatch 0 for
half the tokens; an unsplit step then outgrew the locked workspace. Profile
unsplit there too, then run split once so the later microbatches also
create their persistent resources (e.g. the DeepSeek-V4 ROCm indexer's
pooled K buffer) before the lock.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…efore capture

A microbatch thread's hipBLASLt state is per thread and stream, and creating
it inside graph capture fails with HIP error 900 (operation not permitted when
stream is capturing). The microbatch threads already create it on the compute
and comm streams before capture, but models also fork GEMMs onto side streams
in capture: DeepSeek-V4 on ROCm runs its compressors on aux streams
(execute_in_parallel) and small-batch shared experts on the global aux stream.
Capturing a FULL microbatch graph for decode then fails on the first such GEMM.

Also run the warm-up GEMM on every CUDA stream the model's modules hold. The
streams are found once, by scanning the modules' attributes.

The V1 runner's UBatchWrapper only called current_blas_handle(), which does
not create the hipBLASLt handle at all; do the same warm-up there.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…empty

Under DP, every rank pads to the largest rank's token count before splitting.
A rank whose tokens all fit in the earlier microbatches still splits and runs
its all-padding last microbatch, so the expert all-to-all stays collective.
Such a microbatch cannot replay a FULL graph captured with real tokens in
every microbatch, so the step falls back to eager. The check,
`2 * min_tokens < graph_tokens`, missed the rank whose tokens exactly fill
the earlier microbatches and only covered two microbatches; use
is_last_ubatch_empty instead.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO, the V1 runner's capture_model captures uniform-decode FULL
graphs at or above the decode threshold microbatched only, but
profile_cudagraph_memory sampled every graph unsplit. The microbatched
graphs cost more: on DeepSeek-V4 (DP8/EP8, MI355X, default thresholds)
the graphs took 9.1 GiB against a 4.53 GiB estimate, the KV cache took
the difference, and decode at --gpu-memory-utilization 0.92 ran out of
device memory.

Profile those graphs microbatched, as a group of their own; the
condition is shared with _capture_cudagraphs. Each also allocates
buffers for its microbatch sizes that outlive the graph pool, so their
cost is not shared with the other groups' pool, and it varies too much
by size to extrapolate from two samples (from 2 GiB for the largest to
12 MiB for the small ones here; extrapolating overestimated by 22 GiB).
Capture every one of them and count their cost in full: 9.0 GiB here,
for about 50 s more startup. Clear the UBatchWrapper's graphs afterwards,
as the other wrappers' are, so capture_model does not replay a profiling
graph.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
A DBO microbatch that lies in the DP padding holds the last request with
no query tokens, which the metadata counts as a decode with zero decode
tokens. Two ops broke on it:

- The sparse decode reshaped an empty index tensor and raised, killing the
  microbatch thread and hanging the step. Run the decode only when it has
  rows.
- The MXFP4 indexer compressor launched its pooling kernel over the state
  slot mapping, which spans the DP-padded batch, while the K cache slot
  mapping and token_to_req_indices it reads per token cover only the real
  tokens. On this microbatch both are empty and the read faults the GPU
  (memory access fault at address nil); on other steps the padding tokens
  read stale entries. Launch over the real tokens only.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Give each DBO microbatch its own MORI EPv2 op: both microbatches have a
dispatch or combine in flight at once, and an op's receive and staging
buffers are only valid until its next call. prepare() and finalize() run
dispatch and combine on the comm stream after yielding to the peer
microbatch, so supports_dbo() is true while supports_async() stays false.
Graph capture keeps them on the capture stream. The naive DP/EP path does
the same; move its _comm_region to ubatching.dbo_comm_region and use it
for both.

The receive-row bound needs no change: each microbatch's forward context
carries its own DP token counts, equal on every rank.

Allow --enable-dbo with the mori all2all backend on ROCm when it resolves
to EPv2.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO each microbatch dispatches through its own EPv2 op, and every op was
sized for a full batch (max_num_batched_tokens tokens per rank), so the second
op's arena doubled MORI's footprint and came out of the KV cache.

Microbatch 0's op also serves the steps DBO does not split and keeps the full
size. A split step pads every DP rank to the same N <= max_num_batched_tokens
tokens and cuts it at N // num_ubatches (or on a request boundary for a
uniform spec-decode batch), so a later microbatch holds fewer than
N / num_ubatches + (num_ubatches - 1) * q tokens, q being the query length
(1 + speculative tokens). Size the later ops to that bound; MORI rejects a
batch above an op's size, so an underestimate would fail loudly.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
On GPUs where MORI_ENABLE_SDMA=1 used to select EPv1 (all but gfx1250),
a MoRI with EPv2 AsyncLL (ROCm/mori#768) now keeps EPv2 and builds its
ops with the AsyncLL kernels over SDMA; an older MoRI still falls back to
EPv1. The resolved choice is ParallelConfig.mori_epv2_async_ll.

AsyncLL splits dispatch and combine into a send and a receive, so the
prepare/finalize supports async: the receive is the hook, which DBO runs
on the peer microbatch after its compute is queued (as DeepEP low-latency
does), and the shared experts overlap the combine. Each DBO microbatch
keeps its own op. The arena is sized from MoRI's AsyncLL region table,
and the manager barriers before closing its probe device comm, since
ranks trade SDMA queue handles while creating it. With a MoRI that has
async_ll_own_sdma_queue (ROCm/mori#892), each op asks for its own SDMA
queue: DBO issues both microbatches on the compute stream, where on the
shared queue each receive also waits for the other op's send.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Resolve the modify/delete conflicts with vllm-project#63373 (Remove Model Runner V1) by
taking main's deletion of vllm/v1/worker/gpu_model_runner.py and
vllm/v1/worker/gpu_ubatch_wrapper.py. Their V1-only DBO changes
(microbatched CUDA graphs in the V1 memory profile, side-stream hipBLASLt
workspaces in the V1 ubatch wrapper, V1 workspace sizing) go away with V1;
the V2 counterparts in gpu/model_runner.py, gpu/ubatch_utils.py,
ubatch_utils.py and workspace.py are kept.
With MORI's AsyncLL segmented receive layout (ROCm/mori#918), each source's
rows land in their own fixed-stride segment of the SDMA landing buffer, so
the AITER experts can read them in place instead of MORI first copying the
received rows into a compact buffer.

- AsyncLL ops ask for async_ll_recv_layout="auto" where the installed MORI
  has it; MORI builds the layout only where the op config supports it, and
  the window is sized for it.
- Each dispatch derives one stride from the DP token counts, which every
  rank shares; graph capture, non-AITER experts and DP batches uneven enough
  to pad the experts' input past 1.25x keep the compact receive.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@mergify

mergify Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--66697.org.readthedocs.build/en/66697/

@mergify mergify Bot added documentation Improvements or additions to documentation deepseek Related to DeepSeek models rocm Related to AMD ROCm mrv2 Model Runner V2 specific labels Nov 10, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Nov 10, 2026
In a segmented round the received-row count is world_size * stride, the
row count the experts already get, so it carries no information; and AITER's
MXFP4 W4A4 fused MoE faults on small batches given a count (vllm-project#66733). Hand the
experts no count, as vllm-project#66733 does for the compact receive.

Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
@mergify

mergify Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ChengYM5685.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek Related to DeepSeek models documentation Improvements or additions to documentation mrv2 Model Runner V2 specific needs-rebase rocm Related to AMD ROCm

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

2 participants