Repository navigation
fix(EPv2): give routing handles their own buffers on request (snapshot=True) - #733
Conversation
…t=True) A handle from dispatch(..., return_routing=True) aliases the op's routing_dest_map and total_recv, so the next return_routing dispatch on the same op overwrites every earlier handle: combine with an older handle returns wrong tokens, and replaying one on flydsl faults in ep_dispatch_0. The HIP intranode combine also zeroed the total_recv it was passed, so even a handle with private buffers could combine only once. Fixes ROCm#715. dispatch() takes snapshot=True (with return_routing=True). The handle then gets a fresh torch.empty_like dest map that the dispatch kernel writes directly (no fill and no copy: dispatch writes every (tok, k) entry below num_tokens), plus a 4-byte copy of total_recv. The count is copied rather than handed to the kernel because local_expert_count and the StdMoE convert read it through the op. snapshot=False stays the default and allocation-free; the EpDispatchRoutingHandle docstring now states that such a handle is valid until the next return_routing dispatch on the same op. snapshot=True is rejected on internode and scatter configs, whose combines also read routing state kept in the arena. The total_recv reset moves to the producer. The HIP intranode dispatch zeroes it in its preamble (ordered before the Phase-2 adds by the __syncthreads() on the same warp), and the combine kernels no longer write it: the generic HIP combine, the arch-specific intranode combine body (its dispatch already assigns the count instead of accumulating), and the flydsl combines (flydsl dispatch already self-resets). This drops the host-side total_recv.zero_() before each HIP dispatch and with it the KernelSet.self_resets_counters flag, which no backend sets to False anymore. The internode kernels are unchanged: copystaging already zeroes the count at dispatch, and EpCombineAll's reset sits with its blockFlagCounter reset, which cannot be tested here. Test Plan New tests/python/ops/dispatch_combine_v2/test_routing_snapshot.py (both backends, wired into ci_cco.yml at EP8): two snapshot handles from back-to-back dispatches; combine with the older one, the newer one, then the older one again; counts survive combine; flydsl replay of the older handle with a new payload; snapshot needs return_routing; the default handle still aliases the op's buffers. ``` torchrun --standalone --nproc_per_node=2 test_routing_snapshot.py # also 4 MORI_V2_KERNEL_BACKEND={flydsl,hip} REPLAY=<1 on flydsl> SWEEP=8,128,512 torchrun --standalone --nproc_per_node={2,4} test_op.py MORI_V2_KERNEL_BACKEND=flydsl {STDMOE=1 | COMBINE=scatter | QUANT=fp8_direct_cast} torchrun --standalone --nproc_per_node=2 test_op.py MORI_V2_KERNEL_BACKEND=hip RESET=1 torchrun --standalone --nproc_per_node=2 test_op.py MORI_V2_KERNEL_BACKEND={flydsl,hip} torchrun --standalone --nproc_per_node=4 test_graph_capture.py MORI_V2_KERNEL_BACKEND={flydsl,hip} torchrun --standalone --nproc_per_node=2 test_asym_dtype.py torchrun --standalone --nproc_per_node={2,4} test_ep_backend_parity.py torchrun --standalone --nproc_per_node=2 test_op_lifecycle.py ``` On gfx950: the new test passes 18/18 checks at EP2 and EP4. The existing tests above give the same 87 PASS / 0 FAIL lines on main and on this branch. On main, emulating snapshot with clones (the issue's workaround) fails the HIP second combine and the count-survives check on both backends. Latency is unchanged at EP2, hidden 7168 bf16, top-k 8, 4096 tokens/rank (A/B/A/B medians); at 128 tokens HIP dispatch gets about 4 us faster without the host memset, and snapshot=True adds about 2-4 us. Authored with Claude.
Whitespace and line breaks only, from clang-format 20.1.8 with the repo's .clang-format. ROCm#723 left this file out of format, so the pre-commit check fails on any PR that touches it (the check runs on every changed file); this change touches it. Stripping whitespace, the file is identical before and after. Authored with Claude.
|
@jeffdaily Hi jeff, you need to merge or rebase your branch to main for latest ci test. |
Two conflicts, both from main re-touching code this branch removes. hip_backend.py: main set self_resets_counters=self._is1250 on the intranode KernelSet. This branch deletes that field (the dispatch kernels reset total_recv themselves on every arch), so the line is dropped again. ep_intranode_1250x.hpp: main rewrote the dispatch body that this branch had only clang-formatted. Took main's file as is and re-applied the branch's one change to it, removing the two total_recv resets from EpCombine1250xBody. Authored with Claude.
|
@kawhil-amd rebase done, please review. |
Question: what is the intended contract for
|
combine(..., weights=...) reads the forwarded weights from the arena, not from the handle, so snapshot=True does not make a weighted combine with an older handle correct. State that the caller is responsible: a weighted combine is valid with the handle of the latest dispatch, or after a replay dispatch with the handle, which rewrites the weights at its slots. StdMoE's convert also reads the op's slot map and forwarded weights, so snapshot=True now rejects enable_std_moe, as it already rejects the internode and scatter modes. test_routing_snapshot.py dispatches distinct weights per call and checks the fold for the newest handle and, on backends with replay, for the older handle after replaying it. Authored with Claude.
|
My agent came up with this analysis, but I've read and approved it.
|
The routing handle copy works around ROCm/mori#715. Once a MoRI release includes ROCm/mori#733, dispatch(..., snapshot=True) makes the copy unnecessary. Authored with Claude.
jhchouuu
left a comment
There was a problem hiding this comment.
LGTM, and I have tested on gfx950, it passed.
Since the PR also changes the gfx1250 intranode body (ep_intranode_1250x.hpp), we will test it on gfx1250 before approving and merging.
kawhil-amd
left a comment
There was a problem hiding this comment.
LGTM, verified in gfx1250.
Resolve hip_backend.py: main (#733) removed KernelSet.self_resets_counters, so drop it from the refactored internode KernelSet as well. Internode copystaging still zeroes total_recv at dispatch. Co-Authored-By: Claude <noreply@anthropic.com>
Two textual conflicts and two that git merged cleanly but broke.
Textual:
.github/workflows/ci_cco.yml -- both sides added a step after backend
parity (this PR's AsyncLL step, main's routing snapshot step); keep both.
dispatch_combine_op.py -- both edited the dispatch dest-map choice: this PR
takes the kernel table as a parameter (for the split API), #733 adds the
snapshot branch. Combined.
Merged without a conflict marker, but broken:
* dispatch() gained `snapshot` from #733 and _dispatch_impl's body uses it,
but _dispatch_impl is this PR's function and its signature had no such
parameter, and dispatch() did not pass it on -- every dispatch would have
raised NameError. Added the parameter and pass it through.
* #733 removed KernelSet.self_resets_counters (with the host total_recv
reset it gated), but the AsyncLL KernelSet still passed it, so every
async_ll op would have failed construction with a TypeError. Dropped. That
reset was redundant for AsyncLL in every mode -- its RecvCopy assigns
totalRecvTokenNum rather than accumulating -- so MORI_ASYNCLL_SKIP_RESET
no longer selects anything. It is still parsed, so existing settings
validate, and docs/api/communication.rst now says it has no effect.
Also refuse dispatch(..., snapshot=True) on async_ll. A snapshot handle exists
to be combined after later dispatches; AsyncLL's combine reads
dispReceiverIdxMap, which every dispatch's RecvCopy rewrites, and its call-order
check refuses exactly the dispatch-A, dispatch-B, combine-A pattern anyway.
Co-Authored-By: Claude <noreply@anthropic.com>
…dles copy the recv maps #733 moved the total_recv reset into the dispatch and lets a routing handle keep a copy of the count and combine more than once (dispatch(..., snapshot=True)). The LL128 combines still cleared totalRecvTokenNum at their end, and the bf16 LL128 combine read each slot's source and expert ids from the arena, which the next dispatch overwrites. - An LL128 op's count is the low half of a 64-bit word whose high half is the call number the count is for (two ints instead of one, allocated with the op). The dispatch's block 0 starts the word over with one store when the call starts, and each slot is one 64-bit add: an add that returns the last call's number landed before that store, so the wave takes its slot again once the store is in. Nothing else waits. Both LL128 combines only read the count, which stays until the op's next call. Callers and routing handles still see total_recv as one int, and reset() zeroes the whole word. - EpArgs gains recvToSrc / recvIdx, where the bf16 LL128 combine reads each slot's source and ids: the op's own arena regions, or for a snapshot routing handle (EpDispatchRoutingHandle.recv_maps) copies of them taken right after its dispatch. The LL128 dispatch writes both regions from this rank, so the copies need no barrier. The dispatch itself is unchanged. Snapshot handles are gather only, so the push (fp4 / fp8) LL128 combine keeps reading the arena. EP4 on C17, hip, bf16 LL128 64x16: test_routing_snapshot.py 9/9 (5/9 before). CHECK fp4 1/7/64/128, fp8 1/16/128, bf16 1/16/64 pass. A count test (two ops of one config alternating their calls, reset() between calls, some or all ranks sending no tokens, each count checked against the routing and each combine against the input) passes 34/34; on the merge it fails at its first combine, which clears the count. With block 0's store held back about 25 us, so that the adds for every row past the first token's land before it (dispatch 35 us), the count test and every CHECK above still pass. Three alternating rounds against the merge (ab_1020_095313_1520748 and ab_1020_095615_1527455 fp4, ab_1020_095513_1525146 fp8, ab_1020_095413_1522928 bf16): fp4 dispatch +0.1-0.8 us at 64 / 128 tokens in one run, mixed-signed in the other and at 1 / 7; fp8 dispatch 0.3-1.5 faster at 1 token, mixed at 16, +0.1-0.3 at 128; bf16 dispatch mixed-signed at every size. Against writing the snapshot copies from inside the dispatch behind a null check, in the same rounds: never slower in all three rounds, fp8 dispatch 0.6-0.9 faster at 1 token and combine 0.15-0.3 faster at 16. Zeroing the count when the call starts and having each wave wait for a mark cost +0.2-0.7 us of dispatch at fp8 128 / bf16 64, and two alternating slot counters with every slot also added to the count +0.3-1.7.
Motivation
In
mori.ops.dispatch_combine_v2, the handle fromdispatch(..., return_routing=True)aliases the op-ownedrouting_dest_mapandtotal_recv. The nextreturn_routing=Truedispatch on the same op therefore overwrites every earlier handle. Combine with an older handle returns wrong tokens, and replaying an older handle on flydsl faults inep_dispatch_0. Training hits this with more than one MoE layer per op, because the backward runs the layers in reverse. The HIP intranode combine also zeroed thetotal_recvit was passed, so even a handle with private buffers could combine only once.Fixes #715.
Technical Details
This follows the shape agreed on the issue (option 1).
dispatch()takessnapshot=True, which is valid only withreturn_routing=True. The handle then gets its own dest map fromtorch.empty_like(self.routing_dest_map), and the dispatch kernel writes into it directly. It needs no fill and no copy, because dispatch writes every (tok, k) entry belownum_tokensand nothing reads past it. The handle also gets a 4-byte copy oftotal_recv. The count is copied rather than handed to the kernel becauselocal_expert_count()and the StdMoE convert read it through the op, so the op's own buffer has to stay current.snapshot=Falsestays the default and does not allocate, so the serving path is unchanged. TheEpDispatchRoutingHandledocstring now gives the lifetime: a non-snapshot handle is valid until the nextreturn_routingdispatch on the same op. It also notes that only the routing is snapshotted. The arena outputs, including the reverse map until its first access clones it and the forwarded weights that combine's weight fold reads, still belong to the latest dispatch.snapshot=Trueraises on internode and scatter configs. Their combines also read routing state the op keeps in the arena (the internode send and dest maps, and the reverse map), so a private dest map would not make an older handle correct there.The
total_recvreset now lives with the producer:EpDispatchBody) zeroes it in its preamble with one 4-byte store. The__syncthreads()before Phase 2 orders that store before the Phase-2atomicAdds, which run on the same warp.EpCombineBodyno longer writes it, and neither does the arch-specific intranode combine body (two stores removed). That body's dispatch already assigns the count instead of accumulating into it.reset_total_recvis removed frommake_combineandmake_combine_scatter). flydsl dispatch already self-resets the count.total_recv.zero_()before each HIP dispatch goes away. So doesKernelSet.self_resets_counters, which no backend sets to False anymore.The internode kernels are unchanged.
copystagingalready zeroes the count at dispatch, andEpCombineAll's reset sits next to itsblockFlagCounterreset, which I cannot test without RDMA NICs.New test:
tests/python/ops/dispatch_combine_v2/test_routing_snapshot.py. It runs both backends in one process and is added toci_cco.ymlat EP8. Two stale comments that described the old combine-side reset are updated (bench_ep.py,test_ep_backend_parity.py).Test Plan
gfx950, torch 2.15.0a0 (ROCm), flydsl 0.3.4.1, HIP_VISIBLE_DEVICES limited to 2 or 4 GPUs:
Also tested: the reproducer from #715 with
snapshot=True(flydsl with and withoutREPLAY=1, and hip), plus a latency A/B/A/B on main and this branch. The latency setup is EP2, hidden 7168 bf16, top-k 8, 4096 and 128 tokens/rank, using median CUDA-event times over 100 alternating dispatch/combine pairs.Test Result
test_routing_snapshot.py, EP2 and EP4snapshot=); with the issue's clone workaround: HIP second combine FAIL, count after combine FAIL on both backendssnapshot=Truesnapshot=Truesnapshot=TrueThe default path is unchanged at 4096 tokens. At 128 tokens, HIP dispatch is about 4 us faster without the host memset.
snapshot=Truecosts about 2-4 us at 128 tokens and is within noise at 4096. On hardware that uses the arch-specific intranode body (a single GPU, with the ranks sharing that device), test_routing_snapshot.py passed at 1, 2 and 4 ranks on both backends, including the second-combine and count-survives-combine checks, which run that body's main combine path. Its fp4 overlapped combine path (QUANT=fp4_blockwise at 4 or more ranks with the combine warp count pinned to 24) was not run there; @kawhil-amd, could you check that path? The first of the two removed stores is in it, after every read of the count. black, ruff and clang-format are clean on the changed lines.This PR was authored with Claude (AI assistant) and reviewed by me.