Skip to content

[Perf] [NVIDIA] muse-spark: Fuse the o_proj all-reduce into the attention boundary - #45220

Open
vincentzed wants to merge 1 commit into
sgl-project:mainfrom
vincentzed:muse-attn-ar-fusion
Open

vincentzed wants to merge 1 commit into
sgl-project:mainfrom
vincentzed:muse-attn-ar-fusion

Conversation

@vincentzed

@vincentzed vincentzed commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main after #45112.

cc @Fridge003 @BBuf

Motivation

  • At decode and DFlash verify, each Muse Spark layer runs the o_proj all-reduce and the attention-boundary kernel (post-norm, gated residual, pre-norm) as two kernels.
  • For at most 16 rows on B300 TP4, this PR does the all-reduce inside the boundary kernel. It saves about 1 us per layer, over 62 layers per step.

Modifications

  1. o_proj skips its all-reduce when the fusion applies.
  2. The boundary kernel stores each rank's bf16 partial into every rank's push workspace with unicast stores. It then sums the four partials in fp32 in rank order, the same order as the 1shot push all-reduce, so the result is bit-exact.
  3. The reduce reuses the cluster path of muse_spark_all_gather_norm_residual and adds an optional fp32 router output. New op communication.muse_spark_all_reduce_norm_residual, with a KernelSpec. The model calls it with the boundary's jit_kernel_kwargs(), like the FFN all-gather.
  4. The fusion runs for bf16, TP4 with attention TP equal to TP, SM103, at most 16 rows, the JIT boundary backend, and room in the push workspace. 16 rows is where the fused kernel stops winning on B300 (table below).

Accuracy Tests

  • New 4-GPU kernel test, 10 passed on B300:
    • The reduce is bit-exact against custom_all_reduce + fused_norm_residual for 1, 3, 8 and 16 rows, on both halves of the push workspace.
    • The existing FFN all-gather is bit-exact against fused_norm_residual on the full branch for 3 and 128 rows. This covers both the cluster kernel and the row kernel, since the push code is now shared.
    python test/registered/kernels/ops/communication/test_muse_spark_all_reduce_norm_residual.py
  • Greedy outputs are identical to main for 8 of 8 prompts: 4 at bs=1 and 2 pairs at bs=2, 384 tokens each.
  • AIME26 with sgl-eval 0.1.3, B300 TP4/EP4 with DFlash. Concurrency 2 keeps every verify step at 16 rows, so the fused kernel runs throughout.
main This PR
AIME26 pass@1 (30 x 4) 95.83% (SEM 2.50%) 95.83% (SEM 0.83%)
Truncated at 65536 tokens 3 / 120 4 / 120
Avg output tokens 10952 10882
  • The scores are equal; the truncation gap is one sample, within noise.
sgl-eval run aime26 --base-url http://127.0.0.1:30000/v1 --num-threads 2 --n-repeats 4 \
  --max-tokens 65536 --temperature 1.0 --top-p 1.0 --reasoning-effort xhigh

Speed Tests and Profiling

Kernel time per layer on B300 TP4, CUDA graph of 62 calls:

Rows All-reduce + boundary (us) Fused (us)
1 4.85 3.90
8 5.52 4.46
16 6.28 5.57
24 6.91 7.32
64 8.93 12.20

End-to-end output tok/s on B300 TP4/EP4 with DFlash (8 draft tokens), greedy 512-token outputs, median of 5 runs, main at 28448a8 (before #45112, same kernel and gating):

Batch main This PR
1 817.3 (788.2 to 826.3) 824.5 (808.9 to 831.3)
2 1440.9 (1320.5 to 1441.2) 1452.7 (1348.7 to 1453.3)
  • bs=1 is +0.9% (the ranges overlap) and bs=2 is +0.8%. Accept length is 7.88 in both.
  • At bs=3 and above a verify step has more than 16 rows, so the fusion is off. On main 8e7601a, bs=4 and bs=16 were within noise (1511 vs 1514 and 4566 vs 4579 tok/s).

Server:

python -m sglang.launch_server --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
  --tp-size 4 --ep-size 4 --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm \
  --mem-fraction-static 0.8 --port 30000 \
  --speculative-algorithm DFLASH --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
  --speculative-draft-attention-backend fa4 --speculative-num-draft-tokens 8 \
  --reasoning-parser muse --tool-call-parser muse
Client
import json, statistics, threading, time, urllib.request

TOPICS = ["numerical linear algebra", "the history of compilers", "GPU memory hierarchies",
          "speculative decoding", "the Roman road network", "protein folding",
          "sorting algorithms", "TCP congestion control", "the French revolution",
          "photosynthesis", "black holes", "the history of chess", "jazz harmony",
          "volcanoes", "cryptographic hash functions", "the printing press"]


def gen(i, results):
    body = {"text": f"Write a long, detailed essay about {TOPICS[i]}. Essay {i}:",
            "sampling_params": {"max_new_tokens": 512, "min_new_tokens": 512, "temperature": 0}}
    req = urllib.request.Request("http://127.0.0.1:30000/generate", data=json.dumps(body).encode(),
                                 headers={"Content-Type": "application/json"})
    results[i] = json.loads(urllib.request.urlopen(req).read())["meta_info"]["completion_tokens"]


for bs, reps in ((1, 5), (2, 5)):
    vals = []
    for _ in range(reps):
        results = [0] * bs
        threads = [threading.Thread(target=gen, args=(i, results)) for i in range(bs)]
        start = time.time()
        [t.start() for t in threads]
        [t.join() for t in threads]
        vals.append(sum(results) / (time.time() - start))
    print(f"bs={bs} out_tok/s median={statistics.median(vals):.1f}")

CI States

Latest PR Test (Base): ❌ Run #39355212879
Latest PR Test (Extra): ❌ Run #39355212812
Latest PR Test (AMD ROCm 10): ❌ Run #39355212867

@github-actions github-actions Bot added documentation Improvements or additions to documentation quant LLM Quantization amd deepseek blackwell SM100/SM120 diffusion SGLang Diffusion jit-kernel unified-radix-cache memory-pool labels Oct 19, 2026
@Fridge003

Copy link
Copy Markdown
Collaborator

@vincentzed Please rebase it on main branch. The helion kernel PRs are not blockers

@vincentzed
vincentzed marked this pull request as ready for review October 20, 2026 00:36
…tion boundary

- o_proj skips its all-reduce for <= 16 rows on B300 TP4; the attention
  boundary kernel reduces the partials itself (unicast push, fp32 sum in
  rank order, bit-exact with the 1shot push all-reduce).
- New op communication.muse_spark_all_reduce_norm_residual reuses the
  all-gather boundary kernel's cluster path.
- 4-GPU kernel test against custom_all_reduce + fused_norm_residual.
@vincentzed

Copy link
Copy Markdown
Contributor Author

Done, rebased on main (91c9b1c).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd blackwell SM100/SM120 deepseek diffusion SGLang Diffusion documentation Improvements or additions to documentation jit-kernel memory-pool quant LLM Quantization unified-radix-cache

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants