Skip to content

perf(ep/v2): AsyncLL slot assignment by block, receive copy over the whole grid - #906

Closed
yangyuhuiling wants to merge 1 commit into
mainfrom
xiangch/asyncll-slotassign-recvcopy
Closed

yangyuhuiling wants to merge 1 commit into
mainfrom
xiangch/asyncll-slotassign-recvcopy

Conversation

@yangyuhuiling

Copy link
Copy Markdown
Contributor

Motivation

In a DeepSeek-V4-Pro prefill trace with TBO on a single stream (8× MI355X, EP8), AsyncLL's copy kernels add about 54 ms per forward to the compute stream. Two of them were well short of what their work needs:

  • dispatch_slotassign moves only a few hundred KB of routing, yet took ~60 µs.
  • dispatch_recvcopy ran at about 4 TB/s, while the other copy passes reach 4.5–5.

Technical Details

Both changes are in ep_asyncll_kernel.hpp.

SlotAssign: slots are claimed per block.

  • Before: every surviving entry did its own atomicAdd on one of worldSize counters. At EP8, top-6, 4096 tokens that is about 18k atomics on 8 addresses.
  • Now: each warp counts its claims per destination PE with a ballot and reserves a run inside the block through LDS. Then one thread per PE reserves the block's whole run from the global counter, about 200 global atomics in all.
  • No behaviour change downstream: slot order was already atomic order, and RecvCopy records where each token came from.

RecvCopy: the whole grid walks the output.

  • Before: the grid was split evenly across source PEs, so the PE that sent the most set the kernel's length while the other groups sat idle.
  • Now: the whole grid walks the concatenated output, and each row's source PE is read off the existing prefix sum with one ballot.
  • The payload copy keeps four 16 B loads in flight per lane. The source is what SDMA just wrote, so it comes from HBM.
  • The short metadata fields (indices, weights, scales) are copied one 4 B word per lane instead of through WarpCopy's byte-at-a-time tail. The new LaneCopyWords helper checks length and alignment at runtime and falls back to bytes.

The grid stays aligned to worldSize. The LSA transfer passes still need it; the two comments that cited RecvCopy as the reason are updated.

Not changed. The other copy passes already run at 4.5–5 TB/s. Two SendCopy variants were tried and dropped as slower:

  • reading each token once for all of its destinations: +19%;
  • a 4-deep WarpCopy: +25%.

Test Plan

On 8× MI355X:

  • test_asyncll.py with TRANSPORT=sdma (legacy, MORI_ASYNCLL_EARLY_NOTIFY=1, early + MORI_ASYNCLL_FUSED_RECV=1) and TRANSPORT=lsa.
  • A per-kernel benchmark at sglang's DeepSeek-V4 dispatch shape: EP8, hidden 7168, top-6 of 384 experts, 4096 tokens/rank, fp8 OCP dispatch with 224 B of e8m0 scales per token, bf16 gather combine, sdma, early notification. It checks every received row's hidden bytes, indices, weights and scales byte for byte against its source, and the combined output against the expected sum.
  • A two-op TBO-shaped reproducer at the same shape, alone and stacked with fix(ep/v2): give each AsyncLL op its own SDMA queue per peer #892.

Test Result

test_asyncll: PASS, 0 failed checks, in all four configurations. The byte-exact fp8 check passes.

Per-kernel time, mean of 8 ranks, 3 interleaved runs, µs:

kernel main this PR
dispatch slotassign 58.6 6.6
dispatch recvcopy 65.4 53.9
dispatch sendcopy 58.4 58.1 (unchanged)
combine sendcopy 103.2 103.2 (unchanged)
combine recvcopy 67.5 67.7 (unchanged)

That is about 64 µs per MoE call, roughly 7.8 ms per forward of the trace above (122 calls).

Two-op TBO reproducer at the same shape (fp8, early notification, ROCm 10 image), median of 3, ms/iter. "GEMM" adds two [M,7168]×[7168,4096] GEMMs on every rank before each send and between each receive and combine send.

M streams load main this PR + #892 (per-op queue on)
4096 one GEMM 3.870 3.699 (−4.4%) 2.997 (−22.6%)
4096 one idle 2.653 2.575 (−2.9%) 2.469 (−6.9%)
4096 two GEMM 3.765 3.785 (+0.5%) 3.837 (+1.9%)
4096 two idle 2.326 2.236 (−3.9%) 2.534 (+8.9%)
512 one GEMM 1.116 1.093 (−2.1%) 0.986 (−11.6%)
512 two GEMM 0.971 0.938 (−3.4%) 0.925 (−4.7%)

The two changes are independent. With #892, the gains largely add up on a single stream.

Submission Checklist

🤖 Generated with Claude Code

…whole grid

Two dispatch passes of the AsyncLL family were well short of what their
work needs. Measured at sglang's DeepSeek-V4 shape (EP8, hidden 7168,
top-6 of 384 experts, 4096 tokens/rank, fp8 dispatch with 224 B of e8m0
scales, sdma, early notification), mean of 8 ranks, 3 interleaved runs:

                       before    after
  dispatch slotassign  58.6 us   6.6 us
  dispatch recvcopy    65.4 us  53.9 us

SlotAssign claimed each entry's slot with its own atomicAdd on one of
worldSize counters: ~18k atomics on 8 addresses, for a pass that moves a
few hundred KB of routing. Warps now count their claims per destination
PE with a ballot and take a run inside the block through LDS; one thread
per PE then claims the block's run from the global counter, ~200 global
atomics in all. Which slot an entry gets was already atomic order, and
RecvCopy records each token's origin, so nothing downstream depends on it.

RecvCopy split its grid evenly across source PEs, so the PE that sent the
most set the kernel's length while the other groups idled. The whole grid
now walks the concatenated output and finds each row's source PE from
the prefix sum with one ballot. The payload copy keeps four 16 B loads in
flight per lane (the source is what SDMA just wrote, so it comes from
HBM), and the short metadata fields go a 4 B word per lane, with a byte
fallback for unaligned fields, instead of through WarpCopy's byte tail.

The other copy passes already run at 4.5-5 TB/s and are unchanged. Two
SendCopy variants were tried and dropped: reading each token once for all
its destinations (+19%) and a 4-deep copy (+25%).

test_asyncll passes on 8x MI355X for sdma (legacy, early, early + fused
receive) and lsa. The fp8 benchmark above also checks every received
row, index, weight and scale byte for byte against its source, and the
combined output against the expected sum.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant