Repository navigation
[ROCm][MORI][Perf] Read the EPv2 AsyncLL segmented receive in place - #66697
Draft
ChengYM5685 wants to merge 14 commits into
Draft
ChengYM5685 wants to merge 14 commits into
ChengYM5685 wants to merge 14 commits into
Conversation
The memory profile does not microbatch, so it sizes only microbatch 0's workspace slots. A later microbatch can need as much scratch as the whole batch (the expert all-to-all receives every rank's tokens), so its slot grew mid-forward: an empty_cache device sync inside a microbatch thread, and once the workspace is locked after graph capture, an AssertionError that kills the thread and leaves its peer parked, hanging the step. After the profile run, grow each lane's microbatch slots to the largest slot of that lane, before capture locks them. The V1 runner's profile run did microbatch, which sized microbatch 0 for half the tokens; an unsplit step then outgrew the locked workspace. Profile unsplit there too, then run split once so the later microbatches also create their persistent resources (e.g. the DeepSeek-V4 ROCm indexer's pooled K buffer) before the lock. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…efore capture A microbatch thread's hipBLASLt state is per thread and stream, and creating it inside graph capture fails with HIP error 900 (operation not permitted when stream is capturing). The microbatch threads already create it on the compute and comm streams before capture, but models also fork GEMMs onto side streams in capture: DeepSeek-V4 on ROCm runs its compressors on aux streams (execute_in_parallel) and small-batch shared experts on the global aux stream. Capturing a FULL microbatch graph for decode then fails on the first such GEMM. Also run the warm-up GEMM on every CUDA stream the model's modules hold. The streams are found once, by scanning the modules' attributes. The V1 runner's UBatchWrapper only called current_blas_handle(), which does not create the hipBLASLt handle at all; do the same warm-up there. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
…empty Under DP, every rank pads to the largest rank's token count before splitting. A rank whose tokens all fit in the earlier microbatches still splits and runs its all-padding last microbatch, so the expert all-to-all stays collective. Such a microbatch cannot replay a FULL graph captured with real tokens in every microbatch, so the step falls back to eager. The check, `2 * min_tokens < graph_tokens`, missed the rank whose tokens exactly fill the earlier microbatches and only covered two microbatches; use is_last_ubatch_empty instead. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO, the V1 runner's capture_model captures uniform-decode FULL graphs at or above the decode threshold microbatched only, but profile_cudagraph_memory sampled every graph unsplit. The microbatched graphs cost more: on DeepSeek-V4 (DP8/EP8, MI355X, default thresholds) the graphs took 9.1 GiB against a 4.53 GiB estimate, the KV cache took the difference, and decode at --gpu-memory-utilization 0.92 ran out of device memory. Profile those graphs microbatched, as a group of their own; the condition is shared with _capture_cudagraphs. Each also allocates buffers for its microbatch sizes that outlive the graph pool, so their cost is not shared with the other groups' pool, and it varies too much by size to extrapolate from two samples (from 2 GiB for the largest to 12 MiB for the small ones here; extrapolating overestimated by 22 GiB). Capture every one of them and count their cost in full: 9.0 GiB here, for about 50 s more startup. Clear the UBatchWrapper's graphs afterwards, as the other wrappers' are, so capture_model does not replay a profiling graph. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
A DBO microbatch that lies in the DP padding holds the last request with no query tokens, which the metadata counts as a decode with zero decode tokens. Two ops broke on it: - The sparse decode reshaped an empty index tensor and raised, killing the microbatch thread and hanging the step. Run the decode only when it has rows. - The MXFP4 indexer compressor launched its pooling kernel over the state slot mapping, which spans the DP-padded batch, while the K cache slot mapping and token_to_req_indices it reads per token cover only the real tokens. On this microbatch both are empty and the read faults the GPU (memory access fault at address nil); on other steps the padding tokens read stale entries. Launch over the real tokens only. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Give each DBO microbatch its own MORI EPv2 op: both microbatches have a dispatch or combine in flight at once, and an op's receive and staging buffers are only valid until its next call. prepare() and finalize() run dispatch and combine on the comm stream after yielding to the peer microbatch, so supports_dbo() is true while supports_async() stays false. Graph capture keeps them on the capture stream. The naive DP/EP path does the same; move its _comm_region to ubatching.dbo_comm_region and use it for both. The receive-row bound needs no change: each microbatch's forward context carries its own DP token counts, equal on every rank. Allow --enable-dbo with the mori all2all backend on ROCm when it resolves to EPv2. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Under DBO each microbatch dispatches through its own EPv2 op, and every op was sized for a full batch (max_num_batched_tokens tokens per rank), so the second op's arena doubled MORI's footprint and came out of the KV cache. Microbatch 0's op also serves the steps DBO does not split and keeps the full size. A split step pads every DP rank to the same N <= max_num_batched_tokens tokens and cuts it at N // num_ubatches (or on a request boundary for a uniform spec-decode batch), so a later microbatch holds fewer than N / num_ubatches + (num_ubatches - 1) * q tokens, q being the query length (1 + speculative tokens). Size the later ops to that bound; MORI rejects a batch above an op's size, so an underestimate would fail loudly. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
On GPUs where MORI_ENABLE_SDMA=1 used to select EPv1 (all but gfx1250), a MoRI with EPv2 AsyncLL (ROCm/mori#768) now keeps EPv2 and builds its ops with the AsyncLL kernels over SDMA; an older MoRI still falls back to EPv1. The resolved choice is ParallelConfig.mori_epv2_async_ll. AsyncLL splits dispatch and combine into a send and a receive, so the prepare/finalize supports async: the receive is the hook, which DBO runs on the peer microbatch after its compute is queued (as DeepEP low-latency does), and the shared experts overlap the combine. Each DBO microbatch keeps its own op. The arena is sized from MoRI's AsyncLL region table, and the manager barriers before closing its probe device comm, since ranks trade SDMA queue handles while creating it. With a MoRI that has async_ll_own_sdma_queue (ROCm/mori#892), each op asks for its own SDMA queue: DBO issues both microbatches on the compute stream, where on the shared queue each receive also waits for the other op's send. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Resolve the modify/delete conflicts with vllm-project#63373 (Remove Model Runner V1) by taking main's deletion of vllm/v1/worker/gpu_model_runner.py and vllm/v1/worker/gpu_ubatch_wrapper.py. Their V1-only DBO changes (microbatched CUDA graphs in the V1 memory profile, side-stream hipBLASLt workspaces in the V1 ubatch wrapper, V1 workspace sizing) go away with V1; the V2 counterparts in gpu/model_runner.py, gpu/ubatch_utils.py, ubatch_utils.py and workspace.py are kept.
With MORI's AsyncLL segmented receive layout (ROCm/mori#918), each source's rows land in their own fixed-stride segment of the SDMA landing buffer, so the AITER experts can read them in place instead of MORI first copying the received rows into a compact buffer. - AsyncLL ops ask for async_ll_recv_layout="auto" where the installed MORI has it; MORI builds the layout only where the op config supports it, and the window is sized for it. - Each dispatch derives one stride from the DP token counts, which every rank shares; graph capture, non-AITER experts and DP batches uneven enough to pad the experts' input past 1.25x keep the compact receive. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Contributor
|
Documentation preview: https://vllm--66697.org.readthedocs.build/en/66697/ |
In a segmented round the received-row count is world_size * stride, the row count the experts already get, so it carries no information; and AITER's MXFP4 W4A4 fused MoE faults on small batches given a count (vllm-project#66733). Hand the experts no count, as vllm-project#66733 does for the compact receive. Signed-off-by: ChengYM5685 <Yueming.Cheng@amd.com>
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
On MoRI EPv2 AsyncLL, the AITER experts now read MoRI's segmented receive layout (ROCm/mori#918) in place, so MoRI no longer copies the received tokens into a compact buffer first.
Claims
Validation
Unit tests:
pytest tests/kernels/moe/test_mori_epv2_prepare_finalize.pygives 74 passed, on CPU and in the GPU image below. The 7 new tests cover the in-place receive, the five fallbacks to the compact receive, and the error when a rank's tokens exceed the stride.End to end: one MI355X node, image
vllm/vllm-openai-rocm:nightly-083fbac0with MoRI #918 (fd718da) installed. Both arms use that image; they differ only in the vLLM Python files (#65128 head vs. this commit). Arms run interleaved on the same node (A B B A), 2 benchmark rounds each.MORI_ENABLE_SDMA=1 MORI_ASYNCLL_EARLY_NOTIFY=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1 \ vllm serve DeepSeek-V4-Pro --tensor-parallel-size 1 --data-parallel-size 8 --enable-expert-parallel \ --all2all-backend mori --moe-backend aiter --kv-cache-dtype fp8 --max-num-batched-tokens 8192 \ --max-model-len 16384 --gpu-memory-utilization 0.85 --no-enable-prefix-caching \ --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' [--enable-dbo] vllm bench serve --dataset-name random --random-input-len 8192 --random-output-len 1 \ --max-concurrency 256 --num-prompts 512 --ignore-eos --seed 42Run-to-run spread within an arm is about 1% without DBO and under 0.1% with DBO. In both configurations, every segmented run is faster than every compact run.
GSM8K (5-shot, 400 questions,
lm_evallocal-completions): compact 0.9625, segmented 0.9575, segmented + DBO 0.965, all within ±0.01 stderr.CI coverage: the unit tests run in the AMD
kernels/moestep with a fake MoRI op. CI images do not have MoRI #918, so the end-to-end path is validated only by the runs above.Details
all2all_utilssetsasync_ll_recv_layout="auto"on AsyncLL ops when the installed MoRI has that field (mori_epv2_has_segmented_recv). With"auto", MoRI builds the layout only when the op config supports it (single-node SDMA)."auto"only when it is built._mori_epv2_arena_bytestherefore sizes for"segmented", which is the larger layout. Otherwise the op could fail inside a window that is too small._segment_stridederives it only from what all ranks share: the op, graph capture, anddp_metadata.num_tokens_across_dp_cpu(per-microbatch under DBO). The stride is the largest DP batch rounded up to 32 rows. If a rank's own tokens exceed it, the code raises instead of falling back on that rank alone.expert_maskdrop them;world_size * stridewould exceed 1.25x the rows actually sent. This is the same bound as SGLang's MoRI path.world_size * striderows, without the receive-row bound or row clearing of the compact path. The experts get no received-row count: it would only repeat the row count, and AITER's MXFP4 W4A4 path faults on small batches given one ( [MORI] Run MORI EPv2 on Quark MXFP4 MoE models; stop passing the received-row count #66733). Combine already writes the expert output intocombine_in_view(), which is what MoRI needs.num_tokens_across_dp_cpu.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
Not a duplicate: no open PR uses MoRI's segmented receive. I searched open PRs for "mori segmented", "segment_stride", "async_ll_recv_layout" and "AsyncLL receive"; only #65128, on which this is stacked, matched.