Skip to content

[Perf] muse-spark: remove o-proj and moe tail from Helion verify megakernels - #44798

Open
ethche wants to merge 2 commits into
sgl-project:codex/muse-spark-stack-05-helion-verifyfrom
ethche:muse-spark-helion-verify-stock-gemms-stack05
Open

ethche wants to merge 2 commits into
sgl-project:codex/muse-spark-stack-05-helion-verifyfrom
ethche:muse-spark-helion-verify-stock-gemms-stack05

Conversation

@ethche

@ethche ethche commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

With optimizations to the o-proj and tail kernels, the TP4 verify megakernels' lead over today's default (EP4) on the E07 OpenHands replay shrank from 1.10x to 1.03x TPOT.

We simply remove the helion overrides for tail and o-proj and restore: dev o-projection, all-reduce, and attention-boundary norms. This brings us back to 1.08x output tok/s vs dev EP4 on the OpenHands eval.

This keeps the performant moe and attention megakernels:

Megakernel Default EP4 kernels Default Helion Speedup
Attention QK norm + RoPE (Triton), attention (FlashInfer), norm + gate (CUDA) 12.3 µs 10.1 µs 1.22×
Router pooler (FlashInfer), router GEMM (cuBLASLt), quant + top-8 (CUDA) 13.4 µs 12.7 µs 1.06×
MoE MoE (FlashInfer), finalize + combine (CuTe DSL) 61.9 µs 55.6 µs 1.11×
LM head GEMM + split-K reduce (cuBLASLt) 137.4 µs 135.9 µs 1.01×

Modifications

  • Restore stock O-projection, all-reduce and attention-boundary norms. Got rid of tail megakernel.
  • attention_boundary_verify_megakernel.py: drop roots A/B/C; the input is the stock pre-FFN-normed hidden state. Pooler K block 64 and 8 warps (tuned on GB300).
  • moe_tp4_verify_megakernel.py: trigger dependents at entry so the stock unpooler GEMM prefetches during R3. The tail megakernel stays in-tree, unused (can go in a follow-up).
  • integration.py / megakernel_verify.py: compose the stock modules around the megakernels.

Accuracy Tests

Megakernel TP4 on this branch, real outputs (no forced acceptance):

  • AIME26 (sgl-eval, thinking, reasoning effort xhigh, temperature 1, top-p 0.95, 400K max tokens, 2 repeats, 4 x
    GB300): 58/60 pass@1 (96.7%), pass@2 30/30, 0 truncated, 0 errors (reference 94.6%).

Speed Tests and Profiling

  • 4 x GB300
  • E07 OpenHands replay (13-turn conversations, ~79K-token prompts, 220 output tokens per turn), c=1, DFlash2
  • T=8 with forced acceptance 4.5, 3 repeats on each of 2 nodes.
arm TTFT p50 TTFT p90 TPOT p50 TPOT p90 TPOT mean ITL p50 interactivity p50 verify step (sglang timer)
dev EP4 (default) 255 ms 384 ms 2.12 ms 2.19 ms 2.108 ms 9.52 ms 472.8 tok/s 9.38 ms
megakernel TP4 (this PR) 255 ms 431 ms 1.94 ms 2.00 ms 1.945 ms 8.74 ms 515.5 tok/s 8.65 ms
ratio 1.09x 1.09x 1.084x 1.089x 1.090x

For 2x4xGB300, using PD disagg we have:

PD disaggregation (2 x 4 GB300: EP4 prefill on node 0, decode arms in turn on node 1, mooncake over RDMA; same
workload, c=1, 3 repeats, 0 failed). Decode speed is ITL p50 / tokens per step (evalscope's TPOT reads low under PD,
since the first streamed chunk lands after a few decode steps):

setup TTFT p50 TTFT p90 TTFT p99 decode ms/token vs EP4 decode verify step
PD -> megakernel TP4 decode (this PR) 302 ms 490 ms 2.06 s 1.940 1.092x 8.62 ms
PD -> dev EP4 decode 301 ms 475 ms 2.06 s 2.118 1.000x 9.37 ms
PD -> dev TP4 decode 298 ms 338 ms 2.06 s 2.198 0.964x 9.75 ms

Checklist

  • Format your code according to the Format code with pre-commit.
  • Add unit tests according to the Run and add unit tests. Updated: test_helion_attention_boundary_verify_megakernel (8 passed), test_muse_spark_helion_megakernel_integration (11 passed); unchanged and passing: test_helion_moe_tp4_verify_megakernel (7), test_helion_attention_verify_megakernel (32), test_helion_lm_head_verify_megakernel (2).
  • Update documentation according to Write documentations. (No user-facing change.)
  • Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
  • Follow the SGLang code style guidance.

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #39373026264
Latest PR Test (Extra): ❌ Run #39373026157
Latest PR Test (AMD ROCm 10): ❌ Run #39373026161

@ethche

ethche commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

@Fridge003 Hi, could you cherry pick / merge this into your PR: #44668? this should improve performance for the helion megakernels. thanks!

@ethche ethche changed the title [Perf] muse-spark: Run stock O-projection and FFN tail around the Helion verify megakernels [Perf] muse-spark: remove o-proj and moe tail from Helion verify megakernels Oct 16, 2026
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-05-helion-verify branch from 10151d3 to 2352216 Compare October 18, 2026 04:52
@Fridge003
Fridge003 force-pushed the codex/muse-spark-stack-05-helion-verify branch 3 times, most recently from 90c02c0 to ecf68d3 Compare October 18, 2026 08:02
…ion verify megakernels

On GB300 the stock CuTe GEMMs on either side of the expert path stream their
weights in under PDL while the previous kernel still runs; a megakernel cannot
load anything before its dependency wait. The attention-boundary megakernel's
O-projection / all-reduce / norm roots lose to the stock TgvGemm + one-shot
all-reduce + norm, and the tail megakernel loses to the stock unpooler GEMM +
all-gather-norm + QKV GEMM.

Verify now runs: attention megakernel (sliding-window layers) -> stock
O-projection, all-reduce and attention-boundary norms -> boundary megakernel
(pooler, router top-8, NVFP4 quant) -> MoE megakernel -> stock unpooler and
FFN boundary.

- attention_boundary_verify_megakernel.py: drop roots A/B/C; the input is the
  stock pre-FFN-normed hidden state. Maxes go through hl.reduce, since Helion
  1.5.0 lowers amax to a helper the cross-loop dry pass treats as impure.
  Pooler K block 64 and 8 warps (tuned on GB300).
- moe_tp4_verify_megakernel.py: trigger dependents at entry so the stock
  unpooler GEMM prefetches during R3; remove the unused tail megakernel, its
  config and tests.
- integration.py / megakernel_verify.py: compose the stock modules around the
  megakernels.
- muse_spark.py: drop the next-layer QKV hand-off the tail megakernel used
  (bind_next_layer / _verify_qkv).

4 x GB300, E07 OpenHands replay, c=1, DFlash2 T=8: TPOT mean vs the default
EP4 path 1.081x (Helion 1.5.0) / 1.089x (Helion 1.5.1), was 0.99x.

Applies on codex/muse-spark-stack-05-helion-verify (e297cab); meant to be
folded into that commit.
@ethche
ethche force-pushed the muse-spark-helion-verify-stock-gemms-stack05 branch from 612af78 to a04d168 Compare October 20, 2026 04:36
…KV head

The verify attention megakernel's MERGE root read att_ws/lse_ws through their
5-D layout while ATTEND wrote them through the row views, so Helion could not
match the two and put a barrier across all of ATTEND before the first merge.
MERGE now tiles and reads the row views, which gives one readiness counter per
KV head: a merge task waits only for its own head's splits.

The output is bitwise identical (Helion 1.5.1 and main). B200, T=8, sliding
window: cold-L2 kernel time 10.8-11.0 -> 10.3-10.6 us, in-graph per call
8.4-8.8 -> 8.1-8.3 us.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant