Skip to content

fix(EPv2): give routing handles their own buffers on request (snapshot=True) - #733

Merged
kawhil-amd merged 4 commits into
ROCm:mainfrom
jeffdaily:fix/epv2-routing-snapshot
Oct 19, 2026
Merged

kawhil-amd merged 4 commits into
ROCm:mainfrom
jeffdaily:fix/epv2-routing-snapshot

Conversation

@jeffdaily

Copy link
Copy Markdown
Collaborator

Motivation

In mori.ops.dispatch_combine_v2, the handle from dispatch(..., return_routing=True) aliases the op-owned routing_dest_map and total_recv. The next return_routing=True dispatch on the same op therefore overwrites every earlier handle. Combine with an older handle returns wrong tokens, and replaying an older handle on flydsl faults in ep_dispatch_0. Training hits this with more than one MoE layer per op, because the backward runs the layers in reverse. The HIP intranode combine also zeroed the total_recv it was passed, so even a handle with private buffers could combine only once.

Fixes #715.

Technical Details

This follows the shape agreed on the issue (option 1).

dispatch() takes snapshot=True, which is valid only with return_routing=True. The handle then gets its own dest map from torch.empty_like(self.routing_dest_map), and the dispatch kernel writes into it directly. It needs no fill and no copy, because dispatch writes every (tok, k) entry below num_tokens and nothing reads past it. The handle also gets a 4-byte copy of total_recv. The count is copied rather than handed to the kernel because local_expert_count() and the StdMoE convert read it through the op, so the op's own buffer has to stay current.

snapshot=False stays the default and does not allocate, so the serving path is unchanged. The EpDispatchRoutingHandle docstring now gives the lifetime: a non-snapshot handle is valid until the next return_routing dispatch on the same op. It also notes that only the routing is snapshotted. The arena outputs, including the reverse map until its first access clones it and the forwarded weights that combine's weight fold reads, still belong to the latest dispatch.

snapshot=True raises on internode and scatter configs. Their combines also read routing state the op keeps in the arena (the internode send and dest maps, and the reverse map), so a private dest map would not make an older handle correct there.

The total_recv reset now lives with the producer:

  • HIP intranode dispatch (EpDispatchBody) zeroes it in its preamble with one 4-byte store. The __syncthreads() before Phase 2 orders that store before the Phase-2 atomicAdds, which run on the same warp.
  • EpCombineBody no longer writes it, and neither does the arch-specific intranode combine body (two stores removed). That body's dispatch already assigns the count instead of accumulating into it.
  • The flydsl combines no longer write it either (reset_total_recv is removed from make_combine and make_combine_scatter). flydsl dispatch already self-resets the count.
  • With every intranode dispatch resetting its own count, the host-side total_recv.zero_() before each HIP dispatch goes away. So does KernelSet.self_resets_counters, which no backend sets to False anymore.

The internode kernels are unchanged. copystaging already zeroes the count at dispatch, and EpCombineAll's reset sits next to its blockFlagCounter reset, which I cannot test without RDMA NICs.

New test: tests/python/ops/dispatch_combine_v2/test_routing_snapshot.py. It runs both backends in one process and is added to ci_cco.yml at EP8. Two stale comments that described the old combine-side reset are updated (bench_ep.py, test_ep_backend_parity.py).

Test Plan

gfx950, torch 2.15.0a0 (ROCm), flydsl 0.3.4.1, HIP_VISIBLE_DEVICES limited to 2 or 4 GPUs:

torchrun --standalone --nproc_per_node=2 test_routing_snapshot.py
torchrun --standalone --nproc_per_node=4 test_routing_snapshot.py
MORI_V2_KERNEL_BACKEND=flydsl REPLAY=1 SWEEP=8,128,512 torchrun --standalone --nproc_per_node=2 test_op.py   # and 4
MORI_V2_KERNEL_BACKEND=hip SWEEP=8,128,512 torchrun --standalone --nproc_per_node=2 test_op.py              # and 4
MORI_V2_KERNEL_BACKEND=flydsl STDMOE=1 torchrun --standalone --nproc_per_node=2 test_op.py
MORI_V2_KERNEL_BACKEND=flydsl COMBINE=scatter torchrun --standalone --nproc_per_node=2 test_op.py
MORI_V2_KERNEL_BACKEND=flydsl QUANT=fp8_direct_cast torchrun --standalone --nproc_per_node=2 test_op.py
MORI_V2_KERNEL_BACKEND=hip RESET=1 torchrun --standalone --nproc_per_node=2 test_op.py
MORI_V2_KERNEL_BACKEND=flydsl torchrun --standalone --nproc_per_node=4 test_graph_capture.py                 # and hip
MORI_V2_KERNEL_BACKEND=flydsl torchrun --standalone --nproc_per_node=2 test_asym_dtype.py                    # and hip
torchrun --standalone --nproc_per_node=2 test_ep_backend_parity.py                                          # and 4
torchrun --standalone --nproc_per_node=2 test_op_lifecycle.py

Also tested: the reproducer from #715 with snapshot=True (flydsl with and without REPLAY=1, and hip), plus a latency A/B/A/B on main and this branch. The latency setup is EP2, hidden 7168 bf16, top-k 8, 4096 and 128 tokens/rank, using median CUDA-event times over 100 alternating dispatch/combine pairs.

Test Result

main (d999f1e) this PR
test_routing_snapshot.py, EP2 and EP4 n/a (no snapshot=); with the issue's clone workaround: HIP second combine FAIL, count after combine FAIL on both backends 18/18 PASS at each world size
#715 reproducer, snapshot=True n/a combine with the older handle PASS (max abs err 0.000) on flydsl and hip; flydsl replay of the older handle returns, no fault
#715 reproducer, default handles FAIL, flydsl replay faults unchanged (FAIL, faults), as documented
existing tests above (15 runs) 87 PASS / 0 FAIL lines 87 PASS / 0 FAIL lines
median us, EP2 flydsl dispatch flydsl combine hip dispatch hip combine
4096 tok, main 1111 / 1111 1149 / 1151 1132 / 1122 1111 / 1105
4096 tok, this PR 1109 / 1108 1149 / 1149 1126 / 1129 1108 / 1113
4096 tok, this PR, snapshot=True 1111 / 1110 1149 / 1150 1122 / 1129 1107 / 1112
128 tok, main 74.0 / 73.6 85.8 / 85.9 48.9 / 48.6 48.3 / 48.2
128 tok, this PR 74.2 / 74.4 86.2 / 86.4 44.9 / 44.9 48.1 / 48.3
128 tok, this PR, snapshot=True 77.5 / 78.1 90.4 / 92.0 46.5 / 46.6 48.2 / 48.4

The default path is unchanged at 4096 tokens. At 128 tokens, HIP dispatch is about 4 us faster without the host memset. snapshot=True costs about 2-4 us at 128 tokens and is within noise at 4096. On hardware that uses the arch-specific intranode body (a single GPU, with the ranks sharing that device), test_routing_snapshot.py passed at 1, 2 and 4 ranks on both backends, including the second-combine and count-survives-combine checks, which run that body's main combine path. Its fp4 overlapped combine path (QUANT=fp4_blockwise at 4 or more ranks with the combine warp count pinned to 24) was not run there; @kawhil-amd, could you check that path? The first of the two removed stores is in it, after every read of the count. black, ruff and clang-format are clean on the changed lines.

This PR was authored with Claude (AI assistant) and reviewed by me.

…t=True)

A handle from dispatch(..., return_routing=True) aliases the op's routing_dest_map and total_recv, so the next return_routing dispatch on the same op overwrites every earlier handle: combine with an older handle returns wrong tokens, and replaying one on flydsl faults in ep_dispatch_0. The HIP intranode combine also zeroed the total_recv it was passed, so even a handle with private buffers could combine only once. Fixes ROCm#715.

dispatch() takes snapshot=True (with return_routing=True). The handle then gets a fresh torch.empty_like dest map that the dispatch kernel writes directly (no fill and no copy: dispatch writes every (tok, k) entry below num_tokens), plus a 4-byte copy of total_recv. The count is copied rather than handed to the kernel because local_expert_count and the StdMoE convert read it through the op. snapshot=False stays the default and allocation-free; the EpDispatchRoutingHandle docstring now states that such a handle is valid until the next return_routing dispatch on the same op. snapshot=True is rejected on internode and scatter configs, whose combines also read routing state kept in the arena.

The total_recv reset moves to the producer. The HIP intranode dispatch zeroes it in its preamble (ordered before the Phase-2 adds by the __syncthreads() on the same warp), and the combine kernels no longer write it: the generic HIP combine, the arch-specific intranode combine body (its dispatch already assigns the count instead of accumulating), and the flydsl combines (flydsl dispatch already self-resets). This drops the host-side total_recv.zero_() before each HIP dispatch and with it the KernelSet.self_resets_counters flag, which no backend sets to False anymore. The internode kernels are unchanged: copystaging already zeroes the count at dispatch, and EpCombineAll's reset sits with its blockFlagCounter reset, which cannot be tested here.

Test Plan

New tests/python/ops/dispatch_combine_v2/test_routing_snapshot.py (both backends, wired into ci_cco.yml at EP8): two snapshot handles from back-to-back dispatches; combine with the older one, the newer one, then the older one again; counts survive combine; flydsl replay of the older handle with a new payload; snapshot needs return_routing; the default handle still aliases the op's buffers.

```
torchrun --standalone --nproc_per_node=2 test_routing_snapshot.py   # also 4
MORI_V2_KERNEL_BACKEND={flydsl,hip} REPLAY=<1 on flydsl> SWEEP=8,128,512 torchrun --standalone --nproc_per_node={2,4} test_op.py
MORI_V2_KERNEL_BACKEND=flydsl {STDMOE=1 | COMBINE=scatter | QUANT=fp8_direct_cast} torchrun --standalone --nproc_per_node=2 test_op.py
MORI_V2_KERNEL_BACKEND=hip RESET=1 torchrun --standalone --nproc_per_node=2 test_op.py
MORI_V2_KERNEL_BACKEND={flydsl,hip} torchrun --standalone --nproc_per_node=4 test_graph_capture.py
MORI_V2_KERNEL_BACKEND={flydsl,hip} torchrun --standalone --nproc_per_node=2 test_asym_dtype.py
torchrun --standalone --nproc_per_node={2,4} test_ep_backend_parity.py
torchrun --standalone --nproc_per_node=2 test_op_lifecycle.py
```

On gfx950: the new test passes 18/18 checks at EP2 and EP4. The existing tests above give the same 87 PASS / 0 FAIL lines on main and on this branch. On main, emulating snapshot with clones (the issue's workaround) fails the HIP second combine and the count-survives check on both backends. Latency is unchanged at EP2, hidden 7168 bf16, top-k 8, 4096 tokens/rank (A/B/A/B medians); at 128 tokens HIP dispatch gets about 4 us faster without the host memset, and snapshot=True adds about 2-4 us.

Authored with Claude.
Whitespace and line breaks only, from clang-format 20.1.8 with the repo's .clang-format. ROCm#723 left this file out of format, so the pre-commit check fails on any PR that touches it (the check runs on every changed file); this change touches it. Stripping whitespace, the file is identical before and after.

Authored with Claude.
@kawhil-amd

Copy link
Copy Markdown
Contributor

@jeffdaily Hi jeff, you need to merge or rebase your branch to main for latest ci test.

Two conflicts, both from main re-touching code this branch removes.

hip_backend.py: main set self_resets_counters=self._is1250 on the intranode KernelSet. This branch deletes that field (the dispatch kernels reset total_recv themselves on every arch), so the line is dropped again.

ep_intranode_1250x.hpp: main rewrote the dispatch body that this branch had only clang-formatted. Took main's file as is and re-applied the branch's one change to it, removing the two total_recv resets from EpCombine1250xBody.

Authored with Claude.
@jeffdaily

Copy link
Copy Markdown
Collaborator Author

@kawhil-amd rebase done, please review.

@kawhil-amd

Copy link
Copy Markdown
Contributor

Question: what is the intended contract for snapshot=True + the weight fold?

snapshot=True gives the handle its own dest map and total_recv. It does not
cover the arena's out_wts, and the handle docstring says so explicitly
(dispatch_combine_op.py:643). What I can't tell from the PR is which side of the
line that is meant to be.

The asymmetry that prompts the question: combine re-stages out_tok from the
caller's input on every call, which is why an older handle's token output is
correct. out_wts is not re-staged — the fold reads whatever is in the arena
(ep_intranode_kernel.hpp:430):

srcWeightPtrs[j] = EpPeer<float>(win, destPe, args.offOutWts) + destLocalTokId * kTopk;

So is the caller responsible for ensuring out_wts already matches the handle's
layout before calling combine(..., weights=...) — the same way it is responsible
for passing that handle's expert output as input — or is this a gap that
snapshot=True was meant to close?

Two things that make me think the answer may be backend-dependent rather than
uniform:

  • flydsl's replay dispatch rewrites out_wts at the handle's slots (the weight
    scatter in intranode_kernels.py is outside the const_expr(replay) branch), so
    a backward that replays with a handle before combining with it would refresh the
    region. The HIP backend has dispatch_replay=None and has no equivalent path.
  • enable_std_moe isn't in the is_internode / is_scatter guard list, and
    convert_combine_input() uses out_wts as multipliers
    (intranode_kernels.py:1508:
    out_tok[recv_t][e] = sum_k wts[k]*packed_x[slot_k][e]), so there the same
    question lands on the main token output rather than on a side tensor.

Test gap

Whatever the intended answer is, test_routing_snapshot.py doesn't pin it down:
combine_ok() passes weights=None on every call, so the fold never runs under an
older handle. The new test covers the dest map and the count thoroughly, but the
out_wts boundary — the one part of the lifetime that is documented in prose only
— is untested on both backends.

combine(..., weights=...) reads the forwarded weights from the arena, not from the handle, so snapshot=True does not make a weighted combine with an older handle correct. State that the caller is responsible: a weighted combine is valid with the handle of the latest dispatch, or after a replay dispatch with the handle, which rewrites the weights at its slots. StdMoE's convert also reads the op's slot map and forwarded weights, so snapshot=True now rejects enable_std_moe, as it already rejects the internode and scatter modes.

test_routing_snapshot.py dispatches distinct weights per call and checks the fold for the newest handle and, on backends with replay, for the older handle after replaying it.

Authored with Claude.
@jeffdaily

Copy link
Copy Markdown
Collaborator Author

My agent came up with this analysis, but I've read and approved it.

The intent is that out_wts is the caller's responsibility, like input. snapshot=True covers the routing only (dest map and count). The weight fold reads the forwarded weights from the arena, so a weighted combine is valid with the handle of the latest dispatch, or after a replay dispatch(..., routing=handle), which rewrites out_wts at that handle's slots (flydsl; hip has no replay). Pushed 2706c6a, which:

  • states this in the EpDispatchRoutingHandle and combine docstrings;
  • makes snapshot=True reject enable_std_moe, since StdMoE's convert reads the op's slot map and out_wts, so it only works with the latest dispatch (same reasoning as the existing internode/scatter guard);
  • gives test_routing_snapshot.py distinct weights per dispatch and checks the fold (expected U[t] * w[t]) for the newest handle on both backends, and for the older handle after a replay on flydsl.

Checked on MI300X at EP2 and EP4: hip 9/9 checks, flydsl 12/12. A scratch negative check (older handle, no replay) gives a wrong fold on every token on both backends while the tokens stay correct, so the boundary the docstring describes is real.

jeffdaily added a commit to ROCm/pytorch that referenced this pull request Oct 16, 2026
The routing handle copy works around ROCm/mori#715. Once a MoRI release includes ROCm/mori#733, dispatch(..., snapshot=True) makes the copy unnecessary.

Authored with Claude.

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, and I have tested on gfx950, it passed.

Since the PR also changes the gfx1250 intranode body (ep_intranode_1250x.hpp), we will test it on gfx1250 before approving and merging.

@kawhil-amd kawhil-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, verified in gfx1250.

@kawhil-amd
kawhil-amd merged commit eb211db into ROCm:main Oct 19, 2026
15 checks passed
QizhouZhang97 added a commit that referenced this pull request Oct 19, 2026
Resolve hip_backend.py: main (#733) removed KernelSet.self_resets_counters,
so drop it from the refactored internode KernelSet as well. Internode
copystaging still zeroes total_recv at dispatch.

Co-Authored-By: Claude <noreply@anthropic.com>
yangyuhuiling pushed a commit that referenced this pull request Oct 20, 2026
Two textual conflicts and two that git merged cleanly but broke.

Textual:
  .github/workflows/ci_cco.yml -- both sides added a step after backend
    parity (this PR's AsyncLL step, main's routing snapshot step); keep both.
  dispatch_combine_op.py -- both edited the dispatch dest-map choice: this PR
    takes the kernel table as a parameter (for the split API), #733 adds the
    snapshot branch. Combined.

Merged without a conflict marker, but broken:
  * dispatch() gained `snapshot` from #733 and _dispatch_impl's body uses it,
    but _dispatch_impl is this PR's function and its signature had no such
    parameter, and dispatch() did not pass it on -- every dispatch would have
    raised NameError. Added the parameter and pass it through.
  * #733 removed KernelSet.self_resets_counters (with the host total_recv
    reset it gated), but the AsyncLL KernelSet still passed it, so every
    async_ll op would have failed construction with a TypeError. Dropped. That
    reset was redundant for AsyncLL in every mode -- its RecvCopy assigns
    totalRecvTokenNum rather than accumulating -- so MORI_ASYNCLL_SKIP_RESET
    no longer selects anything. It is still parsed, so existing settings
    validate, and docs/api/communication.rst now says it has no effect.

Also refuse dispatch(..., snapshot=True) on async_ll. A snapshot handle exists
to be combined after later dispatches; AsyncLL's combine reads
dispReceiverIdxMap, which every dispatch's RecvCopy rewrites, and its call-order
check refuses exactly the dispatch-A, dispatch-B, combine-A pattern anyway.

Co-Authored-By: Claude <noreply@anthropic.com>
zhangfei829 added a commit that referenced this pull request Oct 20, 2026
…dles copy the recv maps

#733 moved the total_recv reset into the dispatch and lets a routing handle keep a copy of the count and combine more than once (dispatch(..., snapshot=True)). The LL128 combines still cleared totalRecvTokenNum at their end, and the bf16 LL128 combine read each slot's source and expert ids from the arena, which the next dispatch overwrites.

- An LL128 op's count is the low half of a 64-bit word whose high half is the call number the count is for (two ints instead of one, allocated with the op). The dispatch's block 0 starts the word over with one store when the call starts, and each slot is one 64-bit add: an add that returns the last call's number landed before that store, so the wave takes its slot again once the store is in. Nothing else waits. Both LL128 combines only read the count, which stays until the op's next call. Callers and routing handles still see total_recv as one int, and reset() zeroes the whole word.

- EpArgs gains recvToSrc / recvIdx, where the bf16 LL128 combine reads each slot's source and ids: the op's own arena regions, or for a snapshot routing handle (EpDispatchRoutingHandle.recv_maps) copies of them taken right after its dispatch. The LL128 dispatch writes both regions from this rank, so the copies need no barrier. The dispatch itself is unchanged. Snapshot handles are gather only, so the push (fp4 / fp8) LL128 combine keeps reading the arena.

EP4 on C17, hip, bf16 LL128 64x16: test_routing_snapshot.py 9/9 (5/9 before). CHECK fp4 1/7/64/128, fp8 1/16/128, bf16 1/16/64 pass. A count test (two ops of one config alternating their calls, reset() between calls, some or all ranks sending no tokens, each count checked against the routing and each combine against the input) passes 34/34; on the merge it fails at its first combine, which clears the count. With block 0's store held back about 25 us, so that the adds for every row past the first token's land before it (dispatch 35 us), the count test and every CHECK above still pass.

Three alternating rounds against the merge (ab_1020_095313_1520748 and ab_1020_095615_1527455 fp4, ab_1020_095513_1525146 fp8, ab_1020_095413_1522928 bf16): fp4 dispatch +0.1-0.8 us at 64 / 128 tokens in one run, mixed-signed in the other and at 1 / 7; fp8 dispatch 0.3-1.5 faster at 1 token, mixed at 16, +0.1-0.3 at 128; bf16 dispatch mixed-signed at every size. Against writing the snapshot copies from inside the dispatch behind a null check, in the same rounds: never slower in all three rounds, fp8 dispatch 0.6-0.9 faster at 1 token and combine 0.15-0.3 faster at 16. Zeroing the count when the call starts and having each wave wait for a mark cost +0.2-0.7 us of dispatch at fp8 128 / bf16 64, and two alternating slot counters with every slot also added to the count +0.3-1.7.
@zhangfei829 zhangfei829 mentioned this pull request Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EPv2: routing handles alias op-owned buffers; replaying an older handle faults

3 participants