Repository navigation
fix(ep/v2): AsyncLL fused-receive deadlock under two streams; early notification by default on gfx950 - #875
Conversation
A fused receive copy (MORI_ASYNCLL_FUSED_RECV=1) waits for its peers
inside every CTA, so its whole grid stays resident for the length of the
wait. With two micro-batches on two streams (sglang two-batch overlap),
two fused copies spanning the device can leave no room for the send pass
the other stream must run before any peer's wait can end, and the ranks
deadlock. Early notification without fusion waits in eight waves and is
unaffected; so is a single stream, where sends precede receives.
Resolve a grid ceiling in MakeEpInterNodeCfg -- mpCount / 2 unless
MORI_ASYNCLL_FUSED_RECV_CTAS overrides it -- and apply it in the AsyncLL
geometry to the two receive copies of mode 2 only. RecvCopy strides over
its tokens, so a narrower grid does the same work in fewer CTAs. The
ceiling travels in the Cfg, so Geometry() stays a pure function of it.
Measured on MI355X (8 GPUs, dual-stream TBO repro, CUs restricted with
HSA_CU_MASK): uncapped, the repro deadlocks or stalls to ~124 ms/iter;
with the default ceiling 6/6 runs complete at 64 and 128 CUs.
test_asyncll passes with fusion and the default ceiling. Full device,
median of 3, ms/iter:
early fused uncapped fused (cap 128)
M=4096 1 stream 3.386 3.260 3.418
M=4096 2 streams 3.028 3.042 3.056
M=512 1 stream 0.797 0.751 0.765
M=512 2 streams 0.670 0.721 0.691
Fusion gains nothing over early notification on two streams; the docs
now say so and recommend early notification alone for that case.
Co-Authored-By: Claude <noreply@anthropic.com>
Early notification (asyncLlMode 1) posts each payload's ready word on the
same SDMA queue behind the payload, so the receive phase no longer waits
for a separate transfer pass. It was on by default only for gfx1250.
On MI355X it beats the legacy sequence at every token count measured.
EP8, hidden 7168, top-6, bf16, all ranks loaded, full device, median of
3 interleaved runs, ms/iter:
legacy early
M=4096 1 str 3.399 3.337 -1.8%
M=4096 2 str 3.369 3.014 -10.5%
M=512 1 str 0.813 0.790 -2.8%
M=512 2 str 0.722 0.681 -5.7%
M=128 1 str 0.570 0.542 -4.9%
M=128 2 str 0.501 0.485 -3.2%
The largest gain is with two overlapping streams (two-batch overlap),
the case AsyncLL exists for. MORI_ASYNCLL_EARLY_NOTIFY=0 restores the
legacy sequence. gfx942 is unmeasured and keeps legacy.
Since early is now what CI's gfx950 runner gets by default, the sdma leg
of the AsyncLL CI step runs three times: default (early), legacy, and
fused receive, so neither of the other two loses coverage.
Co-Authored-By: Claude <noreply@anthropic.com>
jhchouuu
left a comment
There was a problem hiding this comment.
Tested 874fa999 on MI355X at EP4 and EP8 (hidden 7168, top-6, bf16, sdma transport), with this PR and main each built from source. I used a two-stream reproducer: two ops on one communicator, one micro-batch per stream, dispatch_send → dispatch_recv → combine_send → combine_recv. I restricted CUs with HSA_CU_MASK where noted. In the "skewed" order, odd ranks enqueue micro-batch 1 first.
Early notification by default looks right. It passes test_asyncll.py (early default, legacy and fused) at both 4 and 8 ranks, and test_asyncll_options.py (7 passed). Two streams, full device, median ms/iter:
| M | EP4 main (legacy) | EP4 PR default (early) | EP4 PR fused | EP8 main (legacy) | EP8 PR default (early) | EP8 PR fused |
|---|---|---|---|---|---|---|
| 2048 | 2.04 | 1.82 | 1.86 | 1.68 | 1.32 | 1.36 |
| 512 | 5.62 | 0.51 | 0.51 | 0.48 | 0.40 | 0.41 |
Early is faster everywhere. At EP4, M=512, legacy also stalls with two streams (median 5.6 ms, max 60 ms), and early avoids that, which supports the default switch.
The fused-receive ceiling helps, but in my reproducer it does not remove the deadlock:
| fused receive, M=2048 | main, EP4 | main, EP8 | this PR, EP4 | this PR, EP8 |
|---|---|---|---|---|
| 128 CUs, skewed order | completes, median 37 ms, one 12.4 s stall | deadlock | 2.2 ms | 1.8 ms |
| 64 CUs, skewed order | deadlock | deadlock | deadlock | deadlock |
| 64 CUs, same order on every rank | — | — | 2.5 ms | — |
64 CUs, skewed, MORI_ASYNCLL_FUSED_RECV_CTAS=64 |
— | — | completes, but median 40.8 ms and an 18 s stall | — |
| 64 CUs, skewed, early (no fusion) | — | — | 2.4 ms | 2.3 ms |
| 64 CUs, skewed, legacy | 2.5 ms | 2.3 ms | — | — |
"Deadlock" means no progress before a 240 s timeout, with no error.
I think the reason is that mpCount is the device's full count, while HSA_CU_MASK, or anything else holding CUs, only shrinks what is actually available. With 64 CUs the two capped grids still fill every CTA slot.
The description says the ceiling "cannot rule it out", and the default is early notification without fusion, so this is not blocking. It would help if the docs said fused receive should not be combined with two streams when the device may be shared. Better still would be to size the ceiling from the CUs the process can actually use rather than from multi_processor_count.
…eaves The fused receive grid ceiling defaulted to mpCount / 2, and mpCount is the device's full CU count even when HSA_CU_MASK gives the process a fraction of it. Reported in review: at 64 of 256 CUs, two 128-CTA fused grids still fill every CTA slot, and with micro-batches on two streams enqueued in opposite orders on neighbouring ranks the ranks deadlock just as without a ceiling. When no MORI_ASYNCLL_FUSED_RECV_CTAS override is given and HSA_CU_MASK restricts this device, the host now passes half of the CUs the mask leaves as the ceiling. A mask it cannot read leaves the whole-device default in place rather than failing the op. CUs held by another stream, kernel or process at run time cannot be seen this way, so the docs now say plainly not to combine receive fusion with two streams, and that early notification -- the gfx950 default -- is the dual-stream choice. Co-Authored-By: Claude <noreply@anthropic.com>
|
@jhchouuu Thanks for the thorough test. You're right about the cause, and both suggestions are in bbe293c. Ceiling sized from the CUs the process can use. When
Docs. The Your case, re-run on 8× MI355X (ROCm 10 image): EP8, M=2048, bf16, fused receive, two streams, odd ranks enqueue micro-batch 1 first, 60 iterations per run.
At 128 CUs the smaller grid costs about 4%, the price of the extra headroom. Without a mask nothing changes.
One unrelated note, since you are testing two-stream setups: on the |
jhchouuu
left a comment
There was a problem hiding this comment.
Re-tested bbe293c4 on 8x MI355X (ROCm 7.14, FlyDSL 0.4.0) with the same two-stream reproducer. The ceiling now follows the mask, and the deadlock is gone.
EP8, M=2048, bf16, two streams, odd ranks enqueue micro-batch 1 first, median ms:
874fa999 |
bbe293c4 |
|
|---|---|---|
| fused, 64 CUs | deadlock | 2.45 (3/3) |
| fused, 128 CUs | 1.83 | 1.86 |
| fused, full device | 1.36 | 1.37 |
| early only, 64 CUs | — | 2.27 |
At EP4 with 64 CUs and the same skewed order, it now completes as well (2.67 ms, 2/2); it deadlocked there before. test_asyncll.py with fused receive (8 ranks) and test_asyncll_options.py (8 tests, 21 subtests) pass. The docs paragraph reads well. LGTM.
Before merging: the branch now conflicts with main in docs/api/communication.rst.
…0-early-default # Conflicts: # docs/api/communication.rst
Summary
Two changes to the v2 AsyncLL family (follow-up to #768):
MORI_ASYNCLL_FUSED_RECV=1and two streams. The fused receive grid is now capped at half the device by default.1. Fused receive deadlock (reported from sglang two-batch overlap)
A fused receive copy (asyncLlMode 2) waits for its peers inside every CTA, so its whole grid stays resident for the length of the wait. Two micro-batches on two streams put two fused copies on the device. When those span the device, there is no room left for the send pass the other stream must run before any peer's wait can end, and the ranks deadlock.
Two configurations are unaffected:
Measured occupancy confirms this. A fused recvcopy runs 4 CTAs/CU on gfx950, so two uncapped grids of 256 occupy half the device's CTA slots for the duration of the wait.
Fix:
MakeEpInterNodeCfgresolvesasyncLlFusedRecvCtas = mpCount / 2.MORI_ASYNCLL_FUSED_RECV_CTASoverrides it, as a tuning knob only.Geometry()stays a pure function of the Cfg.Validation on MI355X, 8 GPUs, with a dual-stream TBO reproducer and CUs restricted via
HSA_CU_MASK:test_asyncll(sdma) passes with fused receive and the default ceiling.Cost on the full device, median of 3, ms/iter:
Fusion gains nothing over early notification with two streams. The docs now say so and recommend early notification alone for that case. The ceiling makes the deadlock unlikely but cannot rule it out if another stream fills most of the device.
2. Early notification by default on gfx950
EP8, hidden 7168, top-6, bf16, all ranks loaded, full device, median of 3 interleaved runs, ms/iter:
MORI_ASYNCLL_EARLY_NOTIFY=0restores legacy.Tests
test_jit_core: newAsyncLLMode.FusedReceiveGridIsCappedAtHalfTheDeviceByDefault. It checks the default and explicit ceilings, alignment to worldSize, that the ceiling never widens the grid, and that invalid or misplaced ceilings are rejected.test_asyncll_options: the ceiling env var reaches only mode 2 and rejects bad values; gfx950 defaults to early; gfx942 still requires opt-in.test_asyncll(sdma, early and fused) passed on MI355X. That run used the fix on top of the feat(ep/v2): AsyncLL dispatch/combine on cco, with cross-node and gfx125x support #768 branch; this PR rebases it onto current main.🤖 Generated with Claude Code