Repository navigation
perf(ep/v2): AsyncLL slot assignment by block, receive copy over the whole grid - #906
Closed
yangyuhuiling wants to merge 1 commit into
Closed
yangyuhuiling wants to merge 1 commit into
yangyuhuiling wants to merge 1 commit into
Conversation
…whole grid
Two dispatch passes of the AsyncLL family were well short of what their
work needs. Measured at sglang's DeepSeek-V4 shape (EP8, hidden 7168,
top-6 of 384 experts, 4096 tokens/rank, fp8 dispatch with 224 B of e8m0
scales, sdma, early notification), mean of 8 ranks, 3 interleaved runs:
before after
dispatch slotassign 58.6 us 6.6 us
dispatch recvcopy 65.4 us 53.9 us
SlotAssign claimed each entry's slot with its own atomicAdd on one of
worldSize counters: ~18k atomics on 8 addresses, for a pass that moves a
few hundred KB of routing. Warps now count their claims per destination
PE with a ballot and take a run inside the block through LDS; one thread
per PE then claims the block's run from the global counter, ~200 global
atomics in all. Which slot an entry gets was already atomic order, and
RecvCopy records each token's origin, so nothing downstream depends on it.
RecvCopy split its grid evenly across source PEs, so the PE that sent the
most set the kernel's length while the other groups idled. The whole grid
now walks the concatenated output and finds each row's source PE from
the prefix sum with one ballot. The payload copy keeps four 16 B loads in
flight per lane (the source is what SDMA just wrote, so it comes from
HBM), and the short metadata fields go a 4 B word per lane, with a byte
fallback for unaligned fields, instead of through WarpCopy's byte tail.
The other copy passes already run at 4.5-5 TB/s and are unchanged. Two
SendCopy variants were tried and dropped: reading each token once for all
its destinations (+19%) and a 4-deep copy (+25%).
test_asyncll passes on 8x MI355X for sdma (legacy, early, early + fused
receive) and lsa. The fp8 benchmark above also checks every received
row, index, weight and scale byte for byte against its source, and the
combined output against the expected sum.
Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
In a DeepSeek-V4-Pro prefill trace with TBO on a single stream (8× MI355X, EP8), AsyncLL's copy kernels add about 54 ms per forward to the compute stream. Two of them were well short of what their work needs:
dispatch_slotassignmoves only a few hundred KB of routing, yet took ~60 µs.dispatch_recvcopyran at about 4 TB/s, while the other copy passes reach 4.5–5.Technical Details
Both changes are in
ep_asyncll_kernel.hpp.SlotAssign: slots are claimed per block.
atomicAddon one ofworldSizecounters. At EP8, top-6, 4096 tokens that is about 18k atomics on 8 addresses.RecvCopyrecords where each token came from.RecvCopy: the whole grid walks the output.
WarpCopy's byte-at-a-time tail. The newLaneCopyWordshelper checks length and alignment at runtime and falls back to bytes.The grid stays aligned to
worldSize. The LSA transfer passes still need it; the two comments that citedRecvCopyas the reason are updated.Not changed. The other copy passes already run at 4.5–5 TB/s. Two
SendCopyvariants were tried and dropped as slower:WarpCopy: +25%.Test Plan
On 8× MI355X:
test_asyncll.pywithTRANSPORT=sdma(legacy,MORI_ASYNCLL_EARLY_NOTIFY=1, early +MORI_ASYNCLL_FUSED_RECV=1) andTRANSPORT=lsa.Test Result
test_asyncll: PASS, 0 failed checks, in all four configurations. The byte-exact fp8 check passes.Per-kernel time, mean of 8 ranks, 3 interleaved runs, µs:
That is about 64 µs per MoE call, roughly 7.8 ms per forward of the trace above (122 calls).
Two-op TBO reproducer at the same shape (fp8, early notification, ROCm 10 image), median of 3, ms/iter. "GEMM" adds two [M,7168]×[7168,4096] GEMMs on every rank before each send and between each receive and combine send.
The two changes are independent. With #892, the gains largely add up on a single stream.
Submission Checklist
🤖 Generated with Claude Code