Repository navigation
[HIP] Add all_gather_lastdim_add to the custom all-reduce - #6960
TennyWang1223 merged 9 commits into
Conversation
out = bf16(bf16(all_gather(inp, dim=-1) + add_b) [+ add_c]) in one launch: each rank reads its peers' [rows, n] shards straight from the registered IPC buffers and writes the [rows, n * world] result with the residual adds fused, instead of a dim-0 all_gather, a movedim copy and one or two adds. The rounding matches the unfused sequence (bf16 after each add), so the result is bit-identical. Under graph capture the input is read in place (capture registers it); eagerly it is staged into the registered input pool first. Kimi-K3 on MI355X (TP8) uses it for the vocab-parallel logits gather and the routed-expert up-projection gather (SGLang rocm/k3-mi355). Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit f7cdef4)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
One backend per PR: PR title tags & labels: |
all_gather_lastdim_add to the custom all-reduceall_gather_lastdim_add to the custom all-reduce
TennyWang1223
left a comment
There was a problem hiding this comment.
● Review Summary — PR #6960: all_gather_lastdim_add
Verdict: Request changes. The kernel itself is correct. What blocks merge is a lint failure in CI and two input-validation gaps in the wrapper that produce wrong results without raising.
Validation (local, PR head merged onto current main, 8× MI300X gfx942, TP8)
- The PR's own test (test_custom_allreduce.py, 5 shapes) passes bit-exact (atol=rtol=0) in both eager and graph mode.
- Extra cases I ran also come out bit-exact:
- the add_c=None path, eager and graph;
- changing the input in place, then replaying the captured graph.
- The bit-exact claim also holds on gfx942: module_custom_all_reduce builds with OPUS_FP32_to_BF16_DEFAULT=5 (round-to-nearest-even), so its rounding matches torch's.
CI
- ❌ Check Code Style with Ruff fails because of this PR, in aiter/ops/custom_all_reduce.py: No changes this session
- I001: the import block is unsorted (from typing import Optional must come before import torch).
- UP045: write torch.Tensor | None instead of Optional[torch.Tensor].
- That failure makes check-signal fail, so every GPU job is skipped, the ci:multi-gpu tests included, and the new test has not run in CI yet. After fixing the lint, re-run CI and confirm the Multi-GPU Tests (8 GPU) job passes.
Issues
⚠️ [verified] A non-contiguous inp gives a silently wrong result.- custom_all_reduce.py:1909-1916 checks add_b, add_c and out for contiguity, but not inp. The C++ side then copies numel * element_size bytes from inp.data_ptr() as if it were
contiguous. - Repro: inp = wide[:, :n] raises no error, and the max abs error is 7.16.
- Fix: add assert is_weak_contiguous(inp), the same check should_custom_ag applies.
- custom_all_reduce.py:1909-1916 checks add_b, add_c and out for contiguity, but not inp. The C++ side then copies numel * element_size bytes from inp.data_ptr() as if it were
⚠️ [verified] add_b, add_c and out have no dtype check.- Only inp is checked to be bf16. An fp32 add_b is read as bf16 with no error; the max abs error is 7.39.
- Fix: assert that add_b, add_c and out have the same dtype and device as inp.
- 📝 Test coverage gaps.
- The test always passes add_c, so the kHasC=false kernel instantiation never runs in CI.
- The graph test never changes the input between replays, although reading inp in place under graph capture is exactly what this PR adds.
- Fix: add a case without add_c, and a case that modifies the input before replay.
- 📝 [inferred] The registered capture path skips _capture_registered(inp).
- Every other collective calls it, and it raises a clear error when inp cannot be IPC-exported (the expandable_segments case, issue #4174). Without it, that failure would show
up later, at flush or replay, with an unclear message. - Fix: reuse _capture_registered(inp) here. I have not reproduced this one.
- Every other collective calls it, and it raises a clear error when inp cannot be IPC-exported (the expandable_segments case, issue #4174). Without it, that failure would show
…t add_c=None and input updates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Thanks for the thorough review. Fixed in fc6dbbd:
|
|
…lias/shape checks in Python and C++, fp16 support Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
WAITING ON AUTHORPR Self-ReviewRevision Waiting for the PR author. The readiness report will appear after the self-review answers are complete for this revision. Questions to answer in the PR description
Edit the Missing or invalid answers
Shadow mode reports what would block but keeps the status successful. Set |
… test predicate and rejected inputs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Thanks, all fair. Changes in a95a988 and b33ea4b:
bf16 is shown; fp16 is within noise of it. In Kimi-K3 this is the attention-residual / logits gather. Turning it off costs 1-6% end-to-end throughput (sgl-project/sglang#46656).
|
|
@TennyWang1223 The requested changes have been addressed. There are no CI failures so far, with a few checks still running. Could you please take another look and confirm whether this is ready to merge once CI is green? |
|
Thanks for the follow-ups. All of songpiao's items look addressed, and I re-ran fp16/bf16 eager+graph at TP2 on MI300X (merged with latest main), all bit-exact. CI's only failure (test_custom_allreduce_capture_registration.py, HIPCachingAllocator assert in teardown) also happens on main and is unrelated. One remaining issue: with an empty shard (rows == 0, e.g. an idle DP-attention rank, same case as #7302), should_custom_ag_lastdim_add returns True but all_gather_lastdim_add then raises AssertionError: ... out aliasing add_b or add_c, because all empty tensors have data_ptr() == 0. Please early-return out when inp.numel() == 0 (Python), and skip the alias check, the memcpy and the launch in C++ for that case. Optional: consider honoring AITER_CUSTOM_AG_MAX_SIZE (_cag_max_size) in the predicate the way should_custom_ag now does on main. |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Thanks. a526188: the Python wrapper returns |
* Add all_gather_lastdim_add to the custom all-reduce out = bf16(bf16(all_gather(inp, dim=-1) + add_b) [+ add_c]) in one launch: each rank reads its peers' [rows, n] shards straight from the registered IPC buffers and writes the [rows, n * world] result with the residual adds fused, instead of a dim-0 all_gather, a movedim copy and one or two adds. The rounding matches the unfused sequence (bf16 after each add), so the result is bit-identical. Under graph capture the input is read in place (capture registers it); eagerly it is staged into the registered input pool first. Kimi-K3 on MI355X (TP8) uses it for the vocab-parallel logits gather and the routed-expert up-projection gather (SGLang rocm/k3-mi355). Co-Authored-By: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit f7cdef4) * fixup! Add all_gather_lastdim_add to the custom all-reduce Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * all_gather_lastdim_add: validate inputs, use _capture_registered, test add_c=None and input updates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * all_gather_lastdim_add: add should_custom_ag_lastdim_add, alignment/alias/shape checks in Python and C++, fp16 support Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * all_gather_lastdim_add: check contiguity/device/inp alignment in C++; test predicate and rejected inputs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * all_gather_lastdim_add: return early on an empty shard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 74a3010)
Motivation
K3's vocab-parallel logits gather and its routed-expert up-projection gather each run a dim-0 all_gather, a movedim copy, and one or two adds.
Modifications
out = bf16(bf16(all_gather(inp, dim=-1) + add_b) [+ add_c])runs in one launch.Tests
op_tests/multigpu_tests/test_custom_allreduce.pychecks bit-exactness against the unfused reference in eager and graph mode on MI355X TP8. Used by the Kimi-K3 attention-residual PR in SGLang.PR Self-Review
Reviewed revision
b33ea4b
Behavior and intent
Adds
all_gather_lastdim_add:out = all_gather(inp, dim=-1) + add_b [+ add_c]in one launch, reading peer shards from the registered IPC buffers. Today the same result needs a dim-0 all-gather, a movedim copy and one or two adds (3-4 launches and an extra round trip through HBM). Kimi-K3 runs it for the vocab-parallel logits gather and the routed-expert up-projection gather. Existing collectives are unchanged.Validation and unverified scope
torchrun --nproc_per_node=8 op_tests/multigpu_tests/test_custom_allreduce.pyon 8x MI355X (gfx950), bf16 and fp16, eager and CUDA graph: all cases bit-exact (atol=rtol=0) againsttorch.cat(shards, -1) + add_b [+ add_c].custom_all_gather(dim=0)+ reshape + adds), TP8 MI355X, graph mode, 200 replays, max over ranks: 1.6x at 8-256 rows, 1.1-1.4x at 1 row (table in the review thread).PR-specific question
Selected only through
should_custom_ag_lastdim_add(inp)and the wrapper, which require: communicator enabled and op built (not gfx1250), world size 2/4/8 and fully connected above 2, bf16 or fp16, contiguous 2Dinpwith last dim % 8 == 0, 16-byte aligned, andinpno larger than the registered pool.out/add_b/add_cmust be contiguous[rows, n * world_size], same dtype and device, 16-byte aligned, andoutmust not aliasadd_b/add_c. The C++ entry repeats these checks and raises, so direct callers can't reach the kernel with unsupported inputs.Changed op tests
op_tests/multigpu_tests/test_custom_allreduce.py: adds the only test ofall_gather_lastdim_add, at Kimi-K3 shapes, bf16 and fp16, eager and graph, with and withoutadd_c, plus an input rewrite before graph replay and rejection of unsupported inputs (predicate False cases, aliased, misaligned and wrong-shape tensors). No existing test covers this op.