Repository navigation
[Perf] muse-spark: remove o-proj and moe tail from Helion verify megakernels - #44798
Open
ethche wants to merge 2 commits into
Conversation
ethche
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
October 16, 2026 21:47
Contributor
Author
|
@Fridge003 Hi, could you cherry pick / merge this into your PR: #44668? this should improve performance for the helion megakernels. thanks! |
Fridge003
force-pushed
the
codex/muse-spark-stack-05-helion-verify
branch
from
October 18, 2026 04:52
10151d3 to
2352216
Compare
Fridge003
requested review from
Edwardf0t1,
FlamingoPg,
Fridge003,
Qiaolin-Yu,
Ying1123,
ch-wan,
hebiao064,
ishandhanani,
ispobock,
merrymercy,
sogalin and
yizhang2077
as code owners
October 18, 2026 04:52
Fridge003
force-pushed
the
codex/muse-spark-stack-05-helion-verify
branch
3 times, most recently
from
October 18, 2026 08:02
90c02c0 to
ecf68d3
Compare
Fridge003
requested review from
JustinTong0323,
hnyls2002,
mickqian and
yhyang201
as code owners
October 18, 2026 08:02
Fridge003
requested review from
Alisehen,
AniZpZ,
CatherineSue,
OrangeRedeng,
ShangmingCai,
alphabetc1,
b8zhong,
hanming-lu,
huangtingwei9988,
hzh0425,
liusy58,
mmangkad,
slin1237 and
xiezhq-hermann
as code owners
October 18, 2026 08:02
…ion verify megakernels On GB300 the stock CuTe GEMMs on either side of the expert path stream their weights in under PDL while the previous kernel still runs; a megakernel cannot load anything before its dependency wait. The attention-boundary megakernel's O-projection / all-reduce / norm roots lose to the stock TgvGemm + one-shot all-reduce + norm, and the tail megakernel loses to the stock unpooler GEMM + all-gather-norm + QKV GEMM. Verify now runs: attention megakernel (sliding-window layers) -> stock O-projection, all-reduce and attention-boundary norms -> boundary megakernel (pooler, router top-8, NVFP4 quant) -> MoE megakernel -> stock unpooler and FFN boundary. - attention_boundary_verify_megakernel.py: drop roots A/B/C; the input is the stock pre-FFN-normed hidden state. Maxes go through hl.reduce, since Helion 1.5.0 lowers amax to a helper the cross-loop dry pass treats as impure. Pooler K block 64 and 8 warps (tuned on GB300). - moe_tp4_verify_megakernel.py: trigger dependents at entry so the stock unpooler GEMM prefetches during R3; remove the unused tail megakernel, its config and tests. - integration.py / megakernel_verify.py: compose the stock modules around the megakernels. - muse_spark.py: drop the next-layer QKV hand-off the tail megakernel used (bind_next_layer / _verify_qkv). 4 x GB300, E07 OpenHands replay, c=1, DFlash2 T=8: TPOT mean vs the default EP4 path 1.081x (Helion 1.5.0) / 1.089x (Helion 1.5.1), was 0.99x. Applies on codex/muse-spark-stack-05-helion-verify (e297cab); meant to be folded into that commit.
ethche
force-pushed
the
muse-spark-helion-verify-stock-gemms-stack05
branch
from
October 20, 2026 04:36
612af78 to
a04d168
Compare
…KV head The verify attention megakernel's MERGE root read att_ws/lse_ws through their 5-D layout while ATTEND wrote them through the row views, so Helion could not match the two and put a barrier across all of ATTEND before the first merge. MERGE now tiles and reads the row views, which gives one readiness counter per KV head: a merge task waits only for its own head's splits. The output is bitwise identical (Helion 1.5.1 and main). B200, T=8, sliding window: cold-L2 kernel time 10.8-11.0 -> 10.3-10.6 us, in-graph per call 8.4-8.8 -> 8.1-8.3 us.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With optimizations to the o-proj and tail kernels, the TP4 verify megakernels' lead over today's default (EP4) on the E07 OpenHands replay shrank from 1.10x to 1.03x TPOT.
We simply remove the helion overrides for tail and o-proj and restore:
devo-projection, all-reduce, and attention-boundary norms. This brings us back to 1.08x output tok/s vs dev EP4 on the OpenHands eval.This keeps the performant moe and attention megakernels:
Modifications
attention_boundary_verify_megakernel.py: drop roots A/B/C; the input is the stock pre-FFN-normed hidden state. Pooler K block 64 and 8 warps (tuned on GB300).moe_tp4_verify_megakernel.py: trigger dependents at entry so the stock unpooler GEMM prefetches during R3. The tail megakernel stays in-tree, unused (can go in a follow-up).integration.py/megakernel_verify.py: compose the stock modules around the megakernels.Accuracy Tests
Megakernel TP4 on this branch, real outputs (no forced acceptance):
GB300): 58/60 pass@1 (96.7%), pass@2 30/30, 0 truncated, 0 errors (reference 94.6%).
Speed Tests and Profiling
For 2x4xGB300, using PD disagg we have:
PD disaggregation (2 x 4 GB300: EP4 prefill on node 0, decode arms in turn on node 1, mooncake over RDMA; same
workload, c=1, 3 repeats, 0 failed). Decode speed is ITL p50 / tokens per step (evalscope's TPOT reads low under PD,
since the first streamed chunk lands after a few decode steps):
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): 🚫 Run #39373026264
Latest PR Test (Extra): ❌ Run #39373026157
Latest PR Test (AMD ROCm 10): ❌ Run #39373026161