Repository navigation
[Perf] [NVIDIA] muse-spark: Fuse the o_proj all-reduce into the attention boundary - #45220
Open
vincentzed wants to merge 1 commit into
Open
vincentzed wants to merge 1 commit into
vincentzed wants to merge 1 commit into
Conversation
vincentzed
force-pushed
the
muse-attn-ar-fusion
branch
from
October 19, 2026 19:39
5dab46b to
0329c43
Compare
Collaborator
|
@vincentzed Please rebase it on main branch. The helion kernel PRs are not blockers |
vincentzed
force-pushed
the
muse-attn-ar-fusion
branch
from
October 20, 2026 00:36
0329c43 to
04834d4
Compare
vincentzed
marked this pull request as ready for review
October 20, 2026 00:36
vincentzed
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
October 20, 2026 00:36
…tion boundary - o_proj skips its all-reduce for <= 16 rows on B300 TP4; the attention boundary kernel reduces the partials itself (unicast push, fp32 sum in rank order, bit-exact with the 1shot push all-reduce). - New op communication.muse_spark_all_reduce_norm_residual reuses the all-gather boundary kernel's cluster path. - 4-GPU kernel test against custom_all_reduce + fused_norm_residual.
vincentzed
force-pushed
the
muse-attn-ar-fusion
branch
from
October 20, 2026 01:45
04834d4 to
91c9b1c
Compare
Contributor
Author
|
Done, rebased on main (91c9b1c). |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on main after #45112.
cc @Fridge003 @BBuf
Motivation
Modifications
o_projskips its all-reduce when the fusion applies.muse_spark_all_gather_norm_residualand adds an optional fp32 router output. New opcommunication.muse_spark_all_reduce_norm_residual, with a KernelSpec. The model calls it with the boundary'sjit_kernel_kwargs(), like the FFN all-gather.Accuracy Tests
custom_all_reduce+fused_norm_residualfor 1, 3, 8 and 16 rows, on both halves of the push workspace.fused_norm_residualon the full branch for 3 and 128 rows. This covers both the cluster kernel and the row kernel, since the push code is now shared.Speed Tests and Profiling
Kernel time per layer on B300 TP4, CUDA graph of 62 calls:
End-to-end output tok/s on B300 TP4/EP4 with DFlash (8 draft tokens), greedy 512-token outputs, median of 5 runs, main at 28448a8 (before #45112, same kernel and gating):
Server:
Client
CI States
Latest PR Test (Base): ❌ Run #39355212879
Latest PR Test (Extra): ❌ Run #39355212812
Latest PR Test (AMD ROCm 10): ❌ Run #39355212867