Repository navigation
fix(ep/v2): give each AsyncLL op its own SDMA queue per peer - #892
Conversation
AsyncLL's RecvTransfer drains the SDMA queue it sent on (quietQueue) before waiting for the peers' counts, so that its staging and count/epoch source words are free for the next round. The queue was picked as destPe % sdmaNumQueue, and SDMA queues are process-wide per GPU pair (anvil), so every op in a process posted to the same queue for a given peer. Under two-batch overlap (one op per micro-batch, one stream) every recv is enqueued right behind the other micro-batch's send, so the drain also waited for that unrelated transfer to leave the queue: ~0.4 ms per dispatch recv and ~1 ms per combine recv, ~100 ms per DeepSeek-V4 prefill forward on the critical rank, while the op's own data had arrived milliseconds earlier. Add a per-launch sdmaQueueOffset to EpInterNodeArgs (it takes the alignment gap before the pointers, so no field moves) and pick the queue as (destPe + offset) % sdmaNumQueue. The hip backend gives each AsyncLL op on a device the next offset, so two ops land on different queues whenever the comm has at least two (the default). All of one op's traffic to a peer still shares one queue, so the in-order guarantees the protocol relies on are unchanged; with more ops than queues they fold, which is only slower. test_asyncll.py gains a two-op scenario: two ops on their own communicators, called in TBO's single-stream order with no host sync, checked afterwards, and required to have different queue offsets.
d08b184 to
13f59a7
Compare
yangyuhuiling
left a comment
There was a problem hiding this comment.
Thanks for this -- the diagnosis is right and the change is correct, but on MI355X it trades a single-stream win for a dual-stream regression of similar size, so I don't think it should be on unconditionally yet. I validated 13f59a73 on an 8x MI355X node (ROCm 10 image).
Correctness: passes
test_asyncll.py on the PR head, 0 failed checks in every configuration, including the new two-op scenario:
| TRANSPORT | protocol | two-op offsets |
|---|---|---|
| sdma | legacy (gfx950 default) | 8 queues, [0, 1] |
| sdma | MORI_ASYNCLL_EARLY_NOTIFY=1 |
8 queues, [0, 1] |
| sdma | early + MORI_ASYNCLL_FUSED_RECV=1 |
8 queues, [0, 1] |
| lsa | -- | 2 queues, [0, 1] |
I also checked the premise: anvil's queues are indeed process-global per KFD node pair (anvil.cpp, connect()), so two ops on different communicators did share them. The args field fills the existing 4-byte gap before the pointers, and every queue pick goes through QueueForPe, so a missed site would not compile.
Performance: faster on one stream, slower on two
A TBO-shaped reproducer: two ops, A on the current stream and B either on the same stream (single) or a side stream (dual), issued as A.dispatch_send, B.dispatch_send, A.dispatch_recv, [compute], A.combine_send, B.dispatch_recv, [compute], B.combine_send, A.combine_recv, B.combine_recv. EP8, hidden 7168, top-6, bf16, all ranks loaded. gemm adds two [M,7168]x[7168,4096] GEMMs on every rank before each send and between each recv and combine_send. Base is this PR's merge-base (38dbdd8f). Both builds were installed side by side, with separate JIT caches, and run interleaved in one container. Median of 3 rounds, ms/iter:
| M | protocol | streams | load | base | PR | change |
|---|---|---|---|---|---|---|
| 4096 | early | single | idle | 3.353 | 3.192 | -4.8% |
| 4096 | early | single | gemm | 4.337 | 3.894 | -10.2% |
| 512 | early | single | idle | 0.791 | 0.762 | -3.7% |
| 512 | early | single | gemm | 1.147 | 0.976 | -14.9% |
| 4096 | early | dual | idle | 3.031 | 3.449 | +13.8% |
| 4096 | early | dual | gemm | 4.074 | 4.658 | +14.3% |
| 512 | early | dual | idle | 0.665 | 0.754 | +13.4% |
| 512 | early | dual | gemm | 0.970 | 0.999 | +3.0% |
| 4096 | legacy | single / dual | idle / gemm | within +-2% | ||
| 512 | legacy | single / dual | idle / gemm | within +-3% |
- Single stream: the effect the commit message describes is real. Early notification gains 10-15% when there is compute between the phases.
- Two streams: early notification gets 13-14% slower, reproduced in all three rounds. Legacy is unaffected either way.
Why two streams get slower. Engine contention can be ruled out with a direct test. anvil spreads a pair's queues across its engines by channel index, so offsets 0/1 land on different engines and 0/2 on the same engine. A third op created and closed between A and B gives 0/2, and that pair is exactly as slow as 0/1: 3.420 vs 3.460 ms at M=4096 idle, 4.662 vs 4.671 with gemm, against 3.027 / 4.042 at base. So the cost is splitting the two ops onto separate queues at all.
My reading: on one shared queue, the micro-batch that posted first drains first. On two queues, both micro-batches share the link and both finish later. On one stream that matters less than the drain no longer waiting on the other op. On two streams it is a net loss.
Both shapes exist in practice. The sglang two-batch-overlap path we have been debugging runs the micro-batches on two streams. With #875, which makes early notification the gfx950 default, that is exactly the +14% row.
Suggestions
- Make the per-op offset opt-in rather than the default. For example, a config field the caller sets per op, so a single-stream TBO caller can ask for separate queues and a dual-stream caller keeps the shared one. Alternatively, add numbers from your DeepSeek-V4 setup, and a dual-stream run, so we can judge the trade-off.
- Offsets are never returned (inline). With a third op created and closed between A and B, B gets offset 2. On MI355X's 8 queues that takes seven closed ops to collide. On MI308X, capped at 3 queues, it takes two.
- LSA ops consume offsets too (inline). The lsa run above shows
[0, 1]though LSA never touches an SDMA queue. - Nits:
_make_dev_commstill says the queue is picked asdestPe % sdmaNumQueue.- The PR description is the empty template. The commit message would serve well there, together with your before/after numbers.
A queue of its own helps an op whose calls interleave with another op's on one stream, and hurts when the two ops run on separate streams: there the shared queue drains the first-posted op first, while two queues split the link between both. So the queue offset is no longer given to every AsyncLL op; the caller asks for it with EpDispatchCombineConfig.async_ll_own_sdma_queue, and a two-stream caller keeps the shared queue. Offsets now go to the least-used offset among live ops on the device and are given back in close(), so ops created and closed earlier no longer push later ones round the ring (B landed on offset 2 after a temporary op, and on MI308X's three queues two such ops folded A and B together). LSA-transport ops take no offset: they post nothing to an SDMA queue. test_asyncll's two-op scenario sets the option on both ops and creates and closes an op, and builds one that keeps the shared queue, before B; it checks that B still lands on the offset A leaves free, that the shared-queue op stays on 0, and that LSA ops stay on 0.
ROCm/mori#892 makes the per-op SDMA queue opt-in (EpDispatchCombineConfig.async_ll_own_sdma_queue). TBO issues both micro-batch ops on the compute stream, where on the shared queue each recv also waits for the other op's send (DeepSeek-V4-Pro EP8 prefill: 70.6k vs 79.6k tok/s), so request it for the SDMA transport whenever MORI has the field; builds without it keep their own behavior, and the LSA transport has no queue to ask for. The init log reports own_sdma_queue.
ROCm/mori#892 makes the per-op SDMA queue opt-in (EpDispatchCombineConfig.async_ll_own_sdma_queue). TBO issues both micro-batch ops on the compute stream, where on the shared queue each recv also waits for the other op's send (DeepSeek-V4-Pro EP8 prefill: 70.6k vs 79.6k tok/s), so request it for the SDMA transport whenever MORI has the field; builds without it keep their own behavior, and the LSA transport has no queue to ask for. The init log reports own_sdma_queue.
The comment quoted per-iteration figures for one and two streams. Which setup the option helps and which it slows down is what a caller needs; the measurements live in the PR description.
Thanks for the review. All four points are fixed:
|
Trim the comments, docstrings and docs added for async_ll_own_sdma_queue to what a reader needs: what the option does, when to set it, and that it is slower with separate streams. Comment-only; no code change.
|
Thanks for the quick turnaround. I re-tested the updated head ( Correction: in sglang, two streams got faster with the per-op queueMy review said two streams get ~14% slower with the per-op queue. That number came from a microbenchmark, and the reproducer below still shows it. But a real DeepSeek-V4-Pro prefill run says the opposite: sglang TBO, attn-DP8 / EP8, mxfp8 dispatch, early notification, 8× MI355X, using this PR as of
Throughput is one bench run each. The forward time is from a profiled run. So the per-op queue helps on two streams in practice as well. The microbenchmark's two-stream slowdown doesn't carry over to the real workload: my guess is that rank skew there makes "not waiting for the other op's transfer" worth more than the FIFO ordering it gives up. Keeping it opt-in is still reasonable until there's more end-to-end data, but my earlier "leave it off on two streams" was too strong. Sorry for the noise. Correctness of the updated head: passesThe new head was merged onto main together with #906 (an unrelated AsyncLL kernel change). On 8× MI355X:
Stacked with #906Two-op TBO reproducer at sglang's DeepSeek-V4 dispatch shape: EP8, hidden 7168, top-6/384, fp8 dispatch with scales, early notification, ROCm 10 image.
On one stream the two changes stack almost additively. On two streams the reproducer still shows the slowdown described above, with or without #906. FYI, unrelated to this PR: two-stream hang on the ROCm 7.2 imageOn the |
The option's comment and the AsyncLL docs said it is slower when the two ops run on separate streams. That holds for the two-op microbenchmark, but DeepSeek-V4-Pro prefill with two-stream TBO got faster with it (78.4k -> 80.4k tok/s, and 78.7k -> 80.2k in an earlier run). Say that the two-stream case is not a clear win and should be measured; the default stays off.
Thanks for re-testing end to end. Your two-stream numbers match ours (78,736 → 80,207 tok/s on 11-02); my last reply wrongly went by the microbenchmark alone. I've updated the comment, docs and PR description to drop "leave it off on two streams" and say to measure that case instead. The default stays off; sglang TBO and vLLM DBO run single-stream and will turn it on. On the ROCm 7.2 hang: our runs also use a ROCm 7.2 image, and two-stream TBO without fused receive never hung across many runs (the only hang was fused receive + two streams, which #875 addresses). |
Motivation
SDMA queues are process-wide per GPU pair, and every AsyncLL op picks a peer's queue as
destPe % sdmaNumQueue, so all ops in a process share the same queues. A recv drains the whole queue it sent on (quietQueuewaits for the queue's committed write pointer) before it checks the peer's ready word, only to make sure its own source buffers can be reused.With two-batch overlap issuing both micro-batches on one stream, the order is
A.send, B.send, A.recv, ...: every recv sits right behind the other op's send on the same queue and waits for that copy to finish too. On DeepSeek-V4-Pro EP8 prefill this costs about 100 ms of recv spinning per forward on rank 0.Changes
ep_internode_args.hpp,ep_asyncll_kernel.hpp):EpInterNodeArgsgetsint32_t sdmaQueueOffset(it fills the 4-byte gap before the pointers, so no field moves), andQueueForPereturns(destPe + offset) % n. All 9 call sites (dispatch/combine send, recv drain, legacy count/epoch put, fused recv) pass it. Packets from one op to one peer still go through one queue, so the in-queue ordering the protocol relies on is unchanged.EpDispatchCombineConfig.async_ll_own_sdma_queue: bool = False): by default every op stays on offset 0, which is the behavior before this PR. A caller that issues two ops' calls interleaved on one stream sets it on both ops.hip_backend.py): an op that opts in takes the least-used offset among the live ops on its device (lowest on a tie), and gives it back inclose(). An op created and closed earlier therefore no longer pushes later ops round the ring (with three queues on MI308X, two such ops would fold A and B back together). Only the"sdma"transport takes an offset; LSA posts nothing to an SDMA queue.test_asyncll.py): a two-op scenario that interleaves two ops for 4 layers with no host sync, the way single-stream TBO does, and checks every layer's dispatch and combine. Both ops opt in; before B is built, an op is created and closed, and one is built without the option. The test checks that B lands on the offset A leaves free, that the non-opt-in op stays on 0, and that LSA ops stay on 0.Why opt-in instead of always on: with the two ops on separate streams the result depends on the workload. The two-op microbenchmark gets slower (the shared queue drains the first-posted op first, while two queues split the link between both ops), but DeepSeek-V4-Pro prefill with two-stream TBO got faster (see End to end). Until there is more end-to-end data, the default keeps today's behavior and callers turn it on where they have measured it.
Test results
All on 8x MI355X.
Correctness
tests/python/ops/dispatch_combine_v2/test_asyncll.py, 8 ranks:# PASS: 0 failed checksunder the sdma (legacy), sdma + early notify, and lsa transports. Two-op line: sdmaoffsets [0, 1], closed op 1, shared-queue op 0; lsaoffsets [0, 0]. The reviewer re-ran these onc73f426bmerged onto main with #906 (sdma and lsa, plus a byte-exact fp8 dispatch check at the DeepSeek-V4 shape): all pass.Micro-benchmark
Two ops in the TBO order (EP8, hidden 7168, top-6, bf16, early notify), ms/iter of the slowest rank. Each case was run twice, interleaved, and the runs agreed within 1%.
Legacy protocol (no early notify), M=4096, gemm: one stream −0.3%, two streams +0.9%.
End to end
DeepSeek-V4-Pro, TP8/DP8/EP8, AsyncLL + SDMA + early notify, EPLB, single-stream TBO (sglang#44492), 1024 × 8k-token prefill requests at concurrency 256. Two runs per arm, interleaved (O1, O0, O1, O0), 3 rounds each.
The shared-queue arm matches the build before this PR (70.6k).
Two-stream TBO (micro-batch B on a side stream), same model and load. The own-queue arms ran with the first revision of this PR, where every op got its own queue (
13f59a73, and the same change on the older base for the 11-02 run), which is the same as the option on:So on this workload the microbenchmark's two-stream slowdown does not carry over: the per-op queue helps on two streams as well.
Callers
async_ll_own_sdma_queue=Truefor the sdma transport when the field exists. On MORI builds without the field it sets nothing.