Repository navigation
Description
Part of #626. A host-launched SDMA (copy engine) prototype for the flydsl_v2 combine on MI455X: what it does, what it gains, what it needs, the GPU hang it hit and the open questions. Paused since 2026-11-06. The code is on an exploration branch, not in a PR.
In short
SDMA does not raise the bandwidth ceiling; its value is that the copies take no CUs. With all copies on SDMA, one engine per peer and the reduce deferred until after the GEMM, the combine beats the fastest CU version at a 32-CU budget, alone and next to a GEMM, at EP2 and EP4. From 64 CUs the CU version is better. Getting there needs two process-wide HIP environment variables, direct ROCr calls and an API change (the caller launches the reduce). One EP4 setting made MES stop responding three times out of four, twice needing a node reboot; the cause is unknown. Whether any training setup runs the combine on about 32 CUs next to a GEMM is not settled, so the work is paused.
Why SDMA
- The copy engine reaches 1.54 TB/s to one peer; the TDM copy kernel 1.45-1.55 TB/s. So SDMA is not faster when DeepEP has many CUs.
- With few CUs it is: one CU pushes at most about 34 GB/s to a peer, so
flydsl_v2on 32 CUs tops out at about 1.09 TB/s, and training gives DeepEP 32-64 CUs. - Next to a GEMM, CUs used for communication slow the GEMM; SDMA copies do not. A sustained 1.3 TB/s peer SDMA copy did not slow an 8192 x 7168 x 7168 bf16 GEMM next to it (EP2 and EP4).
- Both only become an end-to-end gain if the combine overlaps compute. In the Primus DeepSeek-V3 setup, communication and compute take turns today.
What the prototype does
PRIMUS_TURBO_DEEPEP_V2_SDMA=1selects a separate host runtime. Only the combine changes; the dispatch is the same.- After a barrier, one copy stream per peer. Per segment of rows: one 1-D
hipMemcpyAsyncfor the rows, one for the weights, a 4-byte copy for the segment flag. The combine slot is split (rows contiguous, weights apart) because a 2-D peer copy is a blit kernel, not SDMA. - A reduce-only kernel waits on the segment flags and reduces.
- Deferred reduce: the combine only queues the copies; the caller launches the reduce later (after the GEMM) on all CUs, and the next dispatch or combine waits for it.
- EP4: copies through ROCr
hsa_amd_memory_async_copy_on_engine, one SDMA engine per peer. - The output is bit-identical to the CU path in every configuration measured.
Results
One MI455X node (4 GPUs), DeepSeek-V3 proxy shape: 8192 tokens a rank, hidden 7168, bf16, top-8 of 128 experts. "Alone" is the combine by itself. "Exposed" is the time 40 combines add to 40 interleaved 8192 x 7168 x 7168 bf16 GEMMs, per combine, median over several processes. The same configuration differs 10-15 us between batches, so compare rows of one batch. The CU reference is the roofline exploration branch (the fastest CU version, faster than dev/mi455x_v4).
Copy bandwidth, per rank, copies only:
| Case | TB/s |
|---|---|
| EP2, one peer, HIP (second node) | 1.34-1.54 |
| EP4, three peers, engines picked by HIP | 0.97-1.02 |
| EP4, three peers on one engine | 0.74-0.86 |
| EP4, one engine per peer (ROCr) | 1.39-1.45 |
Combine, us a call:
| EP | Method | CUs | Alone | Exposed |
|---|---|---|---|---|
| 2 | CU (roofline) | 32 / 64 / 160 | 238 / 151 / 115 | 102 / 97 / 68 |
| 2 | SDMA, 2048-row segments | 32 | 197 | 70 |
| 2 | SDMA, deferred reduce, one segment a peer | 32 | 168 | 51 |
| 4 | CU (roofline) | 32 / 64 / 160 | 540 / 319 / 250 | 389 / 164 / 142 |
| 4 | SDMA, 2048-row segments, HIP engines (other batch) | 32 / 64 | 736 / 738 | 625 / 634 |
| 4 | SDMA, deferred reduce, HIP engines | 32 | 575 | 585 |
| 4 | SDMA, deferred reduce, one engine a peer (ROCr) | 32 | 314 | 250 |
| 4 | RCCL all-to-all, peers only / with the own rank, no reduce (other batch) | - | 282 / 466 | 208 / 348 |
The 250 us still exposed at EP4 is not broken down: the reduce runs on the GEMM stream, the next copy waits for the last reduce and for a barrier kernel that has to get CUs from the GEMM, and the host takes about 190 us a combine to launch the copies.
What it needs
All three, or there is no gain:
- Every copy on SDMA. By default HIP turns peer copies under 1 MB, and any copy under 16 KB, into blit kernels that run on CUs;
ROC_P2P_SDMA_SIZE=0 GPU_FORCE_BLIT_COPY_SIZE=0prevents that, but they are process-wide and change every small HIP copy in the process. A blit kernel followed by an SDMA copy on one stream also costs about 50 us at each switch. - One SDMA engine per peer at EP4. HIP puts the copies to three peers on the same one or two engines (0.97-1.02 TB/s); picking the engine needs a direct ROCr call, which bypasses HIP.
- The reduce deferred until after the GEMM, so that no CU waits on copies during the overlap. This is an API change: the combine returns before the reduce, the caller launches it, the next dispatch or combine waits for it. Waiting for all copies and then reducing, without deferring, is 23-28% slower alone at 16-32 CUs.
GPU hang
- Setting: EP4, deferred reduce (a reduce-only kernel of 192-256 blocks), several segments a peer (2048 rows).
- What happened: 3 of 4 timed processes made MES stop responding to REMOVE_QUEUE / SUSPEND, each 9-10 s after the process started. Twice the Mode2 reset failed and the node had to be rebooted; once the driver reset the process's queues and recovered. HIP copies and ROCr copies both hit it.
- Not seen: deferred reduce with one segment a peer (9 processes), 2048-row segments without deferring (7 or more).
- Without a hang, the HIP path with the deferred reduce also stalled 94-1614 ms in the overlap measurement; the ROCr path with one segment a peer was steady.
- Suspects, none confirmed:
- a path nobody else uses: ROCr's own blit SDMA queues, with the two variables above putting the 4-byte flag copies on SDMA as well;
- tens of thousands of lanes polling a system-scope flag with no sleep while copies are slow; MES is known to fail when it removes a queue while a kernel spins;
- 9-10 s matches NUMA balancing on these nodes, which evicts each process about every 10 s: MES may have to suspend the compute queues and busy SDMA queues together;
- the container's ROCm user space is much newer than the host driver (amdgpu 7.1.0).
- One segment a peer is as fast, so the hanging setting is not run again. It hangs often enough to make a repro for the driver / MES team.
MORI on gfx1250, for comparison
- MORI's
ccoSdmawrites SDMA packets from the kernel on its own SDMA queues, signals completion with an SDMA ATOMIC on the same queue, and waits withs_sleep. FlyDSL kernels can call it (mori.cco.device.flydsl, linked as bitcode). That would remove the process-wide variables, the host launch and the busy polling. - MORI's own numbers (
dispatch_combine_v2/README.md, "Single-node, EP4 (gfx1250, wave32)": hidden 7168, top-8, bf16, CUDA graph, dispatch + combine, 64-2048 tokens a rank): its AsyncLL path over SDMA is 1.38-1.64x slower than its native path, which writes straight into the peer (2048 tokens: 355.4 vs 246.1 us). The README gives the reason: AsyncLL stages the payload three times (the shape an RDMA leg needs), and the benchmark puts nothing between send and receive, so it measures the cost of the protocol and none of the overlap. It does not test SDMA on a small CU budget or next to a GEMM, so it neither confirms nor contradicts the results above. - Related: feat(cco): enable SDMA broadcast + poll_regmem packets on gfx1250 ROCm/mori#880 (gfx1250 SDMA packets need scope bits), fix(ep/v2): give each AsyncLL op its own SDMA queue per peer ROCm/mori#892 (a shared SDMA queue makes a receive wait for another op's sends), fix(ep/v2): AsyncLL fused-receive deadlock under two streams; early notification by default on gfx950 ROCm/mori#875 (a fused receive that keeps the grid resident deadlocks with two streams).
Not done
- The dispatch on SDMA (only the combine was tried).
- End to end in Primus: it needs the deferred-reduce API and a model step where the combine overlaps a GEMM.
- Other shapes: only 8192 tokens a rank, the DeepSeek-V3 shape and one large GEMM as the overlap; no EP8.
- The code: on an exploration branch based on an old refactor(deep_ep): separate the FlyDSL DeepEP V1 and V2, with correctness fixes and a faster combine on gfx1250 #602 head, not rebased onto
dev/mi455x_v4, not in a PR. Results come from one node; the two MI455X nodes behave differently ([DeepEP Tracking] FlyDSL DeepEP on MI455X: what is in, results, open work and known issues #626, known issue 4), so both need a rerun.
Open questions
- Which case is it for? SDMA only wins when the combine has about 32 CUs and overlaps a GEMM. If no Primus MI455X setup runs that way, it stays paused.
- If it goes on: move to MORI's device-side
ccoSdma, then measure the hang rate at EP2 on an idle node before EP4. - Report the MES hang to the driver / MES team with a repro.
Activity