Skip to content

fix(ep/v2): give each AsyncLL op its own SDMA queue per peer - #892

Merged
yangyuhuiling merged 6 commits into
ROCm:mainfrom
ChengYM5685:fix/asyncll-per-op-sdma-queue
Nov 5, 2026
Merged

yangyuhuiling merged 6 commits into
ROCm:mainfrom
ChengYM5685:fix/asyncll-per-op-sdma-queue

Conversation

@ChengYM5685

@ChengYM5685 ChengYM5685 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

SDMA queues are process-wide per GPU pair, and every AsyncLL op picks a peer's queue as destPe % sdmaNumQueue, so all ops in a process share the same queues. A recv drains the whole queue it sent on (quietQueue waits for the queue's committed write pointer) before it checks the peer's ready word, only to make sure its own source buffers can be reused.

With two-batch overlap issuing both micro-batches on one stream, the order is A.send, B.send, A.recv, ...: every recv sits right behind the other op's send on the same queue and waits for that copy to finish too. On DeepSeek-V4-Pro EP8 prefill this costs about 100 ms of recv spinning per forward on rank 0.

Changes

  1. Per-op queue offset (ep_internode_args.hpp, ep_asyncll_kernel.hpp): EpInterNodeArgs gets int32_t sdmaQueueOffset (it fills the 4-byte gap before the pointers, so no field moves), and QueueForPe returns (destPe + offset) % n. All 9 call sites (dispatch/combine send, recv drain, legacy count/epoch put, fused recv) pass it. Packets from one op to one peer still go through one queue, so the in-queue ordering the protocol relies on is unchanged.
  2. Opt-in (EpDispatchCombineConfig.async_ll_own_sdma_queue: bool = False): by default every op stays on offset 0, which is the behavior before this PR. A caller that issues two ops' calls interleaved on one stream sets it on both ops.
  3. Offset allocation (hip_backend.py): an op that opts in takes the least-used offset among the live ops on its device (lowest on a tie), and gives it back in close(). An op created and closed earlier therefore no longer pushes later ops round the ring (with three queues on MI308X, two such ops would fold A and B back together). Only the "sdma" transport takes an offset; LSA posts nothing to an SDMA queue.
  4. Test (test_asyncll.py): a two-op scenario that interleaves two ops for 4 layers with no host sync, the way single-stream TBO does, and checks every layer's dispatch and combine. Both ops opt in; before B is built, an op is created and closed, and one is built without the option. The test checks that B lands on the offset A leaves free, that the non-opt-in op stays on 0, and that LSA ops stay on 0.

Why opt-in instead of always on: with the two ops on separate streams the result depends on the workload. The two-op microbenchmark gets slower (the shared queue drains the first-posted op first, while two queues split the link between both ops), but DeepSeek-V4-Pro prefill with two-stream TBO got faster (see End to end). Until there is more end-to-end data, the default keeps today's behavior and callers turn it on where they have measured it.

Test results

All on 8x MI355X.

Correctness

tests/python/ops/dispatch_combine_v2/test_asyncll.py, 8 ranks: # PASS: 0 failed checks under the sdma (legacy), sdma + early notify, and lsa transports. Two-op line: sdma offsets [0, 1], closed op 1, shared-queue op 0; lsa offsets [0, 0]. The reviewer re-ran these on c73f426b merged onto main with #906 (sdma and lsa, plus a byte-exact fp8 dispatch check at the DeepSeek-V4 shape): all pass.

Micro-benchmark

Two ops in the TBO order (EP8, hidden 7168, top-6, bf16, early notify), ms/iter of the slowest rank. Each case was run twice, interleaved, and the runs agreed within 1%.

M streams load shared queue own queue change
4096 one idle 3.381 3.000 −11.3%
4096 one gemm 4.365 3.757 −13.9%
4096 two idle 2.326 3.411 +46.6%
4096 two gemm 3.841 4.831 +25.8%
512 one idle 0.494 0.446 −9.8%
512 one gemm 0.812 0.653 −19.6%
512 two idle 0.342 0.474 +38.8%
512 two gemm 0.569 0.736 +29.5%

Legacy protocol (no early notify), M=4096, gemm: one stream −0.3%, two streams +0.9%.

End to end

DeepSeek-V4-Pro, TP8/DP8/EP8, AsyncLL + SDMA + early notify, EPLB, single-stream TBO (sglang#44492), 1024 × 8k-token prefill requests at concurrency 256. Two runs per arm, interleaved (O1, O0, O1, O0), 3 rounds each.

arm tok/s gsm8k (200)
shared queue (option off) 71,235 0.960–0.980
own queue (option on) 80,542 (+13.1%) 0.965–0.985

The shared-queue arm matches the build before this PR (70.6k).

Two-stream TBO (micro-batch B on a side stream), same model and load. The own-queue arms ran with the first revision of this PR, where every op got its own queue (13f59a73, and the same change on the older base for the 11-02 run), which is the same as the option on:

run shared queue own queue
11-02, 3 rounds each 78,736 tok/s 80,207 tok/s (+1.9%)
reviewer's re-test, one bench run each 78,416 tok/s 80,404 tok/s (+2.5%)

So on this workload the microbenchmark's two-stream slowdown does not carry over: the per-op queue helps on two streams as well.

Callers

  • sglang ([Feat] [AMD] mori: Run MORI EPv2 dispatch/combine on AsyncLL kernels for TBO sgl-project/sglang#44492): its TBO issues both micro-batches on the compute stream and sets async_ll_own_sdma_queue=True for the sdma transport when the field exists. On MORI builds without the field it sets nothing.
  • vLLM DBO: also single-stream; it will set the option the same way.
  • Callers that run micro-batches on separate streams: the option also helped DeepSeek-V4 two-stream TBO (above), but a two-op microbenchmark is slower, so measure before turning it on. Leaving it off keeps today's behavior.

AsyncLL's RecvTransfer drains the SDMA queue it sent on (quietQueue) before
waiting for the peers' counts, so that its staging and count/epoch source
words are free for the next round. The queue was picked as
destPe % sdmaNumQueue, and SDMA queues are process-wide per GPU pair (anvil),
so every op in a process posted to the same queue for a given peer.

Under two-batch overlap (one op per micro-batch, one stream) every recv is
enqueued right behind the other micro-batch's send, so the drain also waited
for that unrelated transfer to leave the queue: ~0.4 ms per dispatch recv and
~1 ms per combine recv, ~100 ms per DeepSeek-V4 prefill forward on the
critical rank, while the op's own data had arrived milliseconds earlier.

Add a per-launch sdmaQueueOffset to EpInterNodeArgs (it takes the alignment
gap before the pointers, so no field moves) and pick the queue as
(destPe + offset) % sdmaNumQueue. The hip backend gives each AsyncLL op on a
device the next offset, so two ops land on different queues whenever the
comm has at least two (the default). All of one op's traffic to a peer still
shares one queue, so the in-order guarantees the protocol relies on are
unchanged; with more ops than queues they fold, which is only slower.

test_asyncll.py gains a two-op scenario: two ops on their own communicators,
called in TBO's single-stream order with no host sync, checked afterwards,
and required to have different queue offsets.
@ChengYM5685
ChengYM5685 force-pushed the fix/asyncll-per-op-sdma-queue branch from d08b184 to 13f59a7 Compare November 2, 2026 13:54
@ChengYM5685 ChengYM5685 changed the title Fix/asyncll per op sdma queue fix(ep/v2): give each AsyncLL op its own SDMA queue per peer Nov 2, 2026

@yangyuhuiling yangyuhuiling left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this -- the diagnosis is right and the change is correct, but on MI355X it trades a single-stream win for a dual-stream regression of similar size, so I don't think it should be on unconditionally yet. I validated 13f59a73 on an 8x MI355X node (ROCm 10 image).

Correctness: passes

test_asyncll.py on the PR head, 0 failed checks in every configuration, including the new two-op scenario:

TRANSPORT protocol two-op offsets
sdma legacy (gfx950 default) 8 queues, [0, 1]
sdma MORI_ASYNCLL_EARLY_NOTIFY=1 8 queues, [0, 1]
sdma early + MORI_ASYNCLL_FUSED_RECV=1 8 queues, [0, 1]
lsa -- 2 queues, [0, 1]

I also checked the premise: anvil's queues are indeed process-global per KFD node pair (anvil.cpp, connect()), so two ops on different communicators did share them. The args field fills the existing 4-byte gap before the pointers, and every queue pick goes through QueueForPe, so a missed site would not compile.

Performance: faster on one stream, slower on two

A TBO-shaped reproducer: two ops, A on the current stream and B either on the same stream (single) or a side stream (dual), issued as A.dispatch_send, B.dispatch_send, A.dispatch_recv, [compute], A.combine_send, B.dispatch_recv, [compute], B.combine_send, A.combine_recv, B.combine_recv. EP8, hidden 7168, top-6, bf16, all ranks loaded. gemm adds two [M,7168]x[7168,4096] GEMMs on every rank before each send and between each recv and combine_send. Base is this PR's merge-base (38dbdd8f). Both builds were installed side by side, with separate JIT caches, and run interleaved in one container. Median of 3 rounds, ms/iter:

M protocol streams load base PR change
4096 early single idle 3.353 3.192 -4.8%
4096 early single gemm 4.337 3.894 -10.2%
512 early single idle 0.791 0.762 -3.7%
512 early single gemm 1.147 0.976 -14.9%
4096 early dual idle 3.031 3.449 +13.8%
4096 early dual gemm 4.074 4.658 +14.3%
512 early dual idle 0.665 0.754 +13.4%
512 early dual gemm 0.970 0.999 +3.0%
4096 legacy single / dual idle / gemm within +-2%
512 legacy single / dual idle / gemm within +-3%
  • Single stream: the effect the commit message describes is real. Early notification gains 10-15% when there is compute between the phases.
  • Two streams: early notification gets 13-14% slower, reproduced in all three rounds. Legacy is unaffected either way.

Why two streams get slower. Engine contention can be ruled out with a direct test. anvil spreads a pair's queues across its engines by channel index, so offsets 0/1 land on different engines and 0/2 on the same engine. A third op created and closed between A and B gives 0/2, and that pair is exactly as slow as 0/1: 3.420 vs 3.460 ms at M=4096 idle, 4.662 vs 4.671 with gemm, against 3.027 / 4.042 at base. So the cost is splitting the two ops onto separate queues at all.

My reading: on one shared queue, the micro-batch that posted first drains first. On two queues, both micro-batches share the link and both finish later. On one stream that matters less than the drain no longer waiting on the other op. On two streams it is a net loss.

Both shapes exist in practice. The sglang two-batch-overlap path we have been debugging runs the micro-batches on two streams. With #875, which makes early notification the gfx950 default, that is exactly the +14% row.

Suggestions

  1. Make the per-op offset opt-in rather than the default. For example, a config field the caller sets per op, so a single-stream TBO caller can ask for separate queues and a dual-stream caller keeps the shared one. Alternatively, add numbers from your DeepSeek-V4 setup, and a dual-stream run, so we can judge the trade-off.
  2. Offsets are never returned (inline). With a third op created and closed between A and B, B gets offset 2. On MI355X's 8 queues that takes seven closed ops to collide. On MI308X, capped at 3 queues, it takes two.
  3. LSA ops consume offsets too (inline). The lsa run above shows [0, 1] though LSA never touches an SDMA queue.
  4. Nits:
    • _make_dev_comm still says the queue is picked as destPe % sdmaNumQueue.
    • The PR description is the empty template. The commit message would serve well there, together with your before/after numbers.

A queue of its own helps an op whose calls interleave with another op's on one
stream, and hurts when the two ops run on separate streams: there the shared
queue drains the first-posted op first, while two queues split the link
between both. So the queue offset is no longer given to every AsyncLL op;
the caller asks for it with EpDispatchCombineConfig.async_ll_own_sdma_queue,
and a two-stream caller keeps the shared queue.

Offsets now go to the least-used offset among live ops on the device and are
given back in close(), so ops created and closed earlier no longer push later
ones round the ring (B landed on offset 2 after a temporary op, and on
MI308X's three queues two such ops folded A and B together). LSA-transport
ops take no offset: they post nothing to an SDMA queue.

test_asyncll's two-op scenario sets the option on both ops and creates and
closes an op, and builds one that keeps the shared queue, before B; it checks
that B still lands on the offset A leaves free, that the shared-queue op stays
on 0, and that LSA ops stay on 0.
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Nov 3, 2026
ROCm/mori#892 makes the per-op SDMA queue opt-in
(EpDispatchCombineConfig.async_ll_own_sdma_queue). TBO issues both
micro-batch ops on the compute stream, where on the shared queue each recv
also waits for the other op's send (DeepSeek-V4-Pro EP8 prefill: 70.6k vs
79.6k tok/s), so request it for the SDMA transport whenever MORI has the
field; builds without it keep their own behavior, and the LSA transport has
no queue to ask for. The init log reports own_sdma_queue.
ChengYM5685 added a commit to ChengYM5685/sglang that referenced this pull request Nov 3, 2026
ROCm/mori#892 makes the per-op SDMA queue opt-in
(EpDispatchCombineConfig.async_ll_own_sdma_queue). TBO issues both
micro-batch ops on the compute stream, where on the shared queue each recv
also waits for the other op's send (DeepSeek-V4-Pro EP8 prefill: 70.6k vs
79.6k tok/s), so request it for the SDMA transport whenever MORI has the
field; builds without it keep their own behavior, and the LSA transport has
no queue to ask for. The init log reports own_sdma_queue.
@ChengYM5685
ChengYM5685 marked this pull request as ready for review November 3, 2026 08:32
The comment quoted per-iteration figures for one and two streams. Which
setup the option helps and which it slows down is what a caller needs;
the measurements live in the PR description.
@ChengYM5685

Copy link
Copy Markdown
Contributor Author

Thanks for this -- the diagnosis is right and the change is correct, but on MI355X it trades a single-stream win for a dual-stream regression of similar size, so I don't think it should be on unconditionally yet. I validated 13f59a73 on an 8x MI355X node (ROCm 10 image).

Correctness: passes

test_asyncll.py on the PR head, 0 failed checks in every configuration, including the new two-op scenario:

TRANSPORT protocol two-op offsets
sdma legacy (gfx950 default) 8 queues, [0, 1]
sdma MORI_ASYNCLL_EARLY_NOTIFY=1 8 queues, [0, 1]
sdma early + MORI_ASYNCLL_FUSED_RECV=1 8 queues, [0, 1]
lsa -- 2 queues, [0, 1]
I also checked the premise: anvil's queues are indeed process-global per KFD node pair (anvil.cpp, connect()), so two ops on different communicators did share them. The args field fills the existing 4-byte gap before the pointers, and every queue pick goes through QueueForPe, so a missed site would not compile.

Performance: faster on one stream, slower on two

A TBO-shaped reproducer: two ops, A on the current stream and B either on the same stream (single) or a side stream (dual), issued as A.dispatch_send, B.dispatch_send, A.dispatch_recv, [compute], A.combine_send, B.dispatch_recv, [compute], B.combine_send, A.combine_recv, B.combine_recv. EP8, hidden 7168, top-6, bf16, all ranks loaded. gemm adds two [M,7168]x[7168,4096] GEMMs on every rank before each send and between each recv and combine_send. Base is this PR's merge-base (38dbdd8f). Both builds were installed side by side, with separate JIT caches, and run interleaved in one container. Median of 3 rounds, ms/iter:

M protocol streams load base PR change
4096 early single idle 3.353 3.192 -4.8%
4096 early single gemm 4.337 3.894 -10.2%
512 early single idle 0.791 0.762 -3.7%
512 early single gemm 1.147 0.976 -14.9%
4096 early dual idle 3.031 3.449 +13.8%
4096 early dual gemm 4.074 4.658 +14.3%
512 early dual idle 0.665 0.754 +13.4%
512 early dual gemm 0.970 0.999 +3.0%
4096 legacy single / dual idle / gemm within +-2%
512 legacy single / dual idle / gemm within +-3%

  • Single stream: the effect the commit message describes is real. Early notification gains 10-15% when there is compute between the phases.
  • Two streams: early notification gets 13-14% slower, reproduced in all three rounds. Legacy is unaffected either way.

Why two streams get slower. Engine contention can be ruled out with a direct test. anvil spreads a pair's queues across its engines by channel index, so offsets 0/1 land on different engines and 0/2 on the same engine. A third op created and closed between A and B gives 0/2, and that pair is exactly as slow as 0/1: 3.420 vs 3.460 ms at M=4096 idle, 4.662 vs 4.671 with gemm, against 3.027 / 4.042 at base. So the cost is splitting the two ops onto separate queues at all.

My reading: on one shared queue, the micro-batch that posted first drains first. On two queues, both micro-batches share the link and both finish later. On one stream that matters less than the drain no longer waiting on the other op. On two streams it is a net loss.

Both shapes exist in practice. The sglang two-batch-overlap path we have been debugging runs the micro-batches on two streams. With #875, which makes early notification the gfx950 default, that is exactly the +14% row.

Suggestions

  1. Make the per-op offset opt-in rather than the default. For example, a config field the caller sets per op, so a single-stream TBO caller can ask for separate queues and a dual-stream caller keeps the shared one. Alternatively, add numbers from your DeepSeek-V4 setup, and a dual-stream run, so we can judge the trade-off.

  2. Offsets are never returned (inline). With a third op created and closed between A and B, B gets offset 2. On MI355X's 8 queues that takes seven closed ops to collide. On MI308X, capped at 3 queues, it takes two.

  3. LSA ops consume offsets too (inline). The lsa run above shows [0, 1] though LSA never touches an SDMA queue.

  4. Nits:

    • _make_dev_comm still says the queue is picked as destPe % sdmaNumQueue.
    • The PR description is the empty template. The commit message would serve well there, together with your before/after numbers.

Thanks for the review. All four points are fixed:

  1. The per-op queue is now opt-in through async_ll_own_sdma_queue and off by default, so existing callers keep the shared queue. I also see the dual-stream slowdown, so only single-stream callers (sglang TBO, vLLM DBO) will turn it on.
  2. Offsets are returned in close().
  3. LSA ops no longer take an offset.
  4. The _make_dev_comm comment and the PR description are updated, and the docs now describe the option.

Trim the comments, docstrings and docs added for async_ll_own_sdma_queue
to what a reader needs: what the option does, when to set it, and that
it is slower with separate streams. Comment-only; no code change.
@yangyuhuiling

Copy link
Copy Markdown
Contributor

Thanks for the quick turnaround. I re-tested the updated head (c73f426b) and need to correct part of my earlier review.

Correction: in sglang, two streams got faster with the per-op queue

My review said two streams get ~14% slower with the per-op queue. That number came from a microbenchmark, and the reproducer below still shows it. But a real DeepSeek-V4-Pro prefill run says the opposite: sglang TBO, attn-DP8 / EP8, mxfp8 dispatch, early notification, 8× MI355X, using this PR as of 13f59a73.

sglang TBO shared queue per-op queue
micro-batches on two streams: throughput 78,416 tok/s 80,404 tok/s
two streams: rank-0 forward 802 ms 784 ms
one stream: throughput 70,462 tok/s 79,642 tok/s

Throughput is one bench run each. The forward time is from a profiled run.

So the per-op queue helps on two streams in practice as well. The microbenchmark's two-stream slowdown doesn't carry over to the real workload: my guess is that rank skew there makes "not waiting for the other op's transfer" worth more than the FIFO ordering it gives up. Keeping it opt-in is still reasonable until there's more end-to-end data, but my earlier "leave it off on two streams" was too strong. Sorry for the noise.

Correctness of the updated head: passes

The new head was merged onto main together with #906 (an unrelated AsyncLL kernel change). On 8× MI355X:

  • test_asyncll.py passes for sdma and lsa, including the new two-op checks: offsets [0, 1], a closed op's offset is reused, and LSA ops stay on 0.
  • A byte-exact fp8 dispatch check (DeepSeek-V4 shape, 224 B e8m0 scales) passes.

Stacked with #906

Two-op TBO reproducer at sglang's DeepSeek-V4 dispatch shape: EP8, hidden 7168, top-6/384, fp8 dispatch with scales, early notification, ROCm 10 image. async_ll_own_sdma_queue=True in the two right-hand columns. Median of 3, ms/iter.

M streams load main this PR #906 both
4096 one GEMM 3.870 3.108 (−19.7%) 3.699 (−4.4%) 2.997 (−22.6%)
4096 one idle 2.653 2.518 (−5.1%) 2.575 (−2.9%) 2.469 (−6.9%)
512 one GEMM 1.116 1.008 (−9.7%) 1.093 (−2.1%) 0.986 (−11.6%)
4096 two GEMM 3.765 3.913 (+3.9%) 3.785 (+0.5%) 3.837 (+1.9%)
4096 two idle 2.326 2.680 (+15.2%) 2.236 (−3.9%) 2.534 (+8.9%)
512 two GEMM 0.971 0.953 (−1.9%) 0.938 (−3.4%) 0.925 (−4.7%)

On one stream the two changes stack almost additively. On two streams the reproducer still shows the slowdown described above, with or without #906.

FYI, unrelated to this PR: two-stream hang on the ROCm 7.2 image

On the rocm/sgl-dev:v0.5.23-rocm720 image, the two-stream reproducer (M=512, GEMM on every rank, early notification, no fused receive) hangs in about 29% of 200-iteration runs: 9 of 31. It hangs equally on main, this PR, #906 and both, and with bf16 as well as fp8. On the ROCm 10 image the same runs never hung: 0 of 32. I'm investigating separately and will report there; it doesn't block this PR.

The option's comment and the AsyncLL docs said it is slower when the two ops
run on separate streams. That holds for the two-op microbenchmark, but
DeepSeek-V4-Pro prefill with two-stream TBO got faster with it (78.4k ->
80.4k tok/s, and 78.7k -> 80.2k in an earlier run). Say that the
two-stream case is not a clear win and should be measured; the default stays
off.
@ChengYM5685

Copy link
Copy Markdown
Contributor Author

Thanks for the quick turnaround. I re-tested the updated head (c73f426b) and need to correct part of my earlier review.

Correction: in sglang, two streams got faster with the per-op queue

My review said two streams get ~14% slower with the per-op queue. That number came from a microbenchmark, and the reproducer below still shows it. But a real DeepSeek-V4-Pro prefill run says the opposite: sglang TBO, attn-DP8 / EP8, mxfp8 dispatch, early notification, 8× MI355X, using this PR as of 13f59a73.

sglang TBO shared queue per-op queue
micro-batches on two streams: throughput 78,416 tok/s 80,404 tok/s
two streams: rank-0 forward 802 ms 784 ms
one stream: throughput 70,462 tok/s 79,642 tok/s
Throughput is one bench run each. The forward time is from a profiled run.

So the per-op queue helps on two streams in practice as well. The microbenchmark's two-stream slowdown doesn't carry over to the real workload: my guess is that rank skew there makes "not waiting for the other op's transfer" worth more than the FIFO ordering it gives up. Keeping it opt-in is still reasonable until there's more end-to-end data, but my earlier "leave it off on two streams" was too strong. Sorry for the noise.

Correctness of the updated head: passes

The new head was merged onto main together with #906 (an unrelated AsyncLL kernel change). On 8× MI355X:

  • test_asyncll.py passes for sdma and lsa, including the new two-op checks: offsets [0, 1], a closed op's offset is reused, and LSA ops stay on 0.
  • A byte-exact fp8 dispatch check (DeepSeek-V4 shape, 224 B e8m0 scales) passes.

Stacked with #906

Two-op TBO reproducer at sglang's DeepSeek-V4 dispatch shape: EP8, hidden 7168, top-6/384, fp8 dispatch with scales, early notification, ROCm 10 image. async_ll_own_sdma_queue=True in the two right-hand columns. Median of 3, ms/iter.

M streams load main this PR #906 both
4096 one GEMM 3.870 3.108 (−19.7%) 3.699 (−4.4%) 2.997 (−22.6%)
4096 one idle 2.653 2.518 (−5.1%) 2.575 (−2.9%) 2.469 (−6.9%)
512 one GEMM 1.116 1.008 (−9.7%) 1.093 (−2.1%) 0.986 (−11.6%)
4096 two GEMM 3.765 3.913 (+3.9%) 3.785 (+0.5%) 3.837 (+1.9%)
4096 two idle 2.326 2.680 (+15.2%) 2.236 (−3.9%) 2.534 (+8.9%)
512 two GEMM 0.971 0.953 (−1.9%) 0.938 (−3.4%) 0.925 (−4.7%)
On one stream the two changes stack almost additively. On two streams the reproducer still shows the slowdown described above, with or without #906.

FYI, unrelated to this PR: two-stream hang on the ROCm 7.2 image

On the rocm/sgl-dev:v0.5.23-rocm720 image, the two-stream reproducer (M=512, GEMM on every rank, early notification, no fused receive) hangs in about 29% of 200-iteration runs: 9 of 31. It hangs equally on main, this PR, #906 and both, and with bf16 as well as fp8. On the ROCm 10 image the same runs never hung: 0 of 32. I'm investigating separately and will report there; it doesn't block this PR.

Thanks for re-testing end to end. Your two-stream numbers match ours (78,736 → 80,207 tok/s on 11-02); my last reply wrongly went by the microbenchmark alone. I've updated the comment, docs and PR description to drop "leave it off on two streams" and say to measure that case instead. The default stays off; sglang TBO and vLLM DBO run single-stream and will turn it on.

On the ROCm 7.2 hang: our runs also use a ROCm 7.2 image, and two-stream TBO without fused receive never hung across many runs (the only hang was fused receive + two streams, which #875 addresses).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants