Skip to content

fuse shared experts with routed experts in MegaMoE's DeepGEMM - #39313

Merged
BBuf merged 16 commits into
sgl-project:mainfrom
xikronz:fix/fuse-shared-experts-MegaMoE
Sep 29, 2026
Merged

BBuf merged 16 commits into
sgl-project:mainfrom
xikronz:fix/fuse-shared-experts-MegaMoE

Conversation

@xikronz

@xikronz xikronz commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

for #38700 (* - *)

Motivation

Context: Deepseek-v4-pro uses DeepseekMoE (served as MegaMoE) as its mixure of experts with a shared expert that is activated across all tokens and a fine-grained routed experts.

  • flash model: 1 shared + 256 routed
  • pro model: 1 shared + 384 routed

Note that the authors also chose to use FP4 with QAT such that routed expert weights are in MXFP4 while all other parameters (including the shared expert in the official checkpoints) stay in blockwise fp8 format.

For each token $t_i$ , a V4 MoE layer does two different GEMM operations:

  1. shared - dense MLP on every token (in FP8)
  2. routed - grouped GEMM over the top 6 selected experts (MXFP4)

Launching each kernel independently wastes launches and stream syncs on every layer (this is true even if they shared a dtype). DeepGEMM already caught this and added native MegaMoE support via (num_shared_experts, separate shared_l1/shared_l2 weights, mixed FP8 and MXFP4).

The existing Sglang fusion logic in deepseek-v2.py

        # num_fused_shared_experts drives weight remapping in deepseek_weight_loader:
        # mlp.shared_experts → mlp.experts.256 when > 0.
        self.num_fused_shared_experts = 0 if _fusion_disabled else n_shared_experts

only works if shared and routed share a dtype/layout (from deepseel-v4.py):

        # Need to disable if quant precision mismatch, even if
        # --enforce-shared-experts-fusion is specified
        if quant_blocks_shared_experts_fusion(quant_config):
            return (
                "Quantization keeps shared experts at a higher precision than the "
                "routed experts, so they cannot be fused into the quantized "
                "routed-expert path."
            )

but the official deepseek-ai checkpoints do not, so that path does not apply on MegaMoE (it can still run on the trtllm-gen runner if you --enforce-shared-experts-fusion and requantize shared FP8→MXFP4, which is what #35505 did)

Modifications

Changes mainly in mega_moe.py:
This PR went through a lot of iterations, starting with a naive index implementation that was supposed to be easier to maintain and more local w.r.t. number of files changed. The memory and throughput overhead of maintaining a python cache of expert indices was pretty bad (as seen below) which motivated for a kernel level change in mega_moe_predispatch.cuh.

Placing the FP8 shared expert inside routed MegaMoE (removing six standalone kernels per layer) does this but it regressed throughput because it bypassed the existing alternate-stream fork in mega_moe.py. In the unfused profiles, the shared expert’s second GEMM overlapped with the routed MegaMoE in 184/184 observed calls per rank, covering a median 94.1–95.6% of its duration. Naively fusing the path without modifying forward_mega_moe() function returned before the existing alternate-stream fork, leaving the whole router/top-k/pre-dispatch/MegaMoE region on the current stream...

  if moe.mega_shared_l1_weights is not None:
          return _run_mega_routed(
              moe, hidden_states, forward_batch, input_ids_global, num_tokens
          )
  
      sbo_overlap_flag = (
          moe.alt_stream is not None
          and moe.num_fused_shared_experts == 0
          and num_tokens > 0
          and get_is_capture_mode()
      )

The current iteration of this PR keeps the same kernel changes but forks the whole fused region in forward_mega_moe() which lets other graph work overlap it; at BS1, main, kernel-only, and kernel-plus-fork achieved 756.90, 721.97, and 759.08 tok/s.

Major files changes:

  1. _get_mega_moe_symm_buffer includes num_shared_experts in its cache key and allocates the shared-expert buffer for the fused configuration
  2. _run_mega_routed forwards the shared-expert count and transformed L1/L2 weights through gate and top-k to run_mega_routed_experts
  3. run_mega_routed_experts selects the consuming MegaMoE BLOCK_M, passes the shared-scale destination to pre-dispatch, and launches fp8_fp4_mega_moe with the shared weights
  4. forward_mega_moe forks the entire fused router/top-k/pre-dispatch/MegaMoE region onto the existing alternate stream during CUDA-graph capture and joins before returning

Accuracy Tests

Accuracy evals are run on deepseek-ai/DeepSeek-V4-Flash-0731 with the bundled DSpark draft via 4x B300 (SM103) with sgl-deep-gemm 0.1.7. On GPTQA-diamond with sgl-eval and thinking enabled, reasoning effort max, max_tokens=200000, and
top_p=0.95:

Workload n main fusion index (no kernel) fusion kernel fusion kernel + fork (PR)
Greedy accuracy 198 0.899 0.894 0.883 0.897
Greedy empty at cap 198 8 4 8 8
Sampled pass@1 792 0.900 0.922 0.910 0.910
Parsing failure 792 0 0 0 0
Request failures - 0 0 0 0

paired per-request comparison:

Workload Pair Answer flips main right, arm wrong arm right, main wrong One-sided empty answers
Greedy main vs fusion index 14 2 8 6
Greedy main vs fusion kernel 17 5 7 8
Greedy main vs fusion kernel + fork 22 5 11 10
Sampled main vs fusion index 50 11 28 0
Sampled main vs fusion kernel 55 19 27 0
Sampled main vs fusion kernel + fork (PR) 58 19 27 0

All request errors are empty answers truncated at 200k tokens. No request failed

Speed Tests and Profiling

  • Model: deepseek-ai/DeepSeek-V4-Flash-0731 with sgl-deep-gemm 0.1.7 on 4xB300
  • Workload: 1,024 input tokens, 256 requested output tokens, batch sizes 1, 8, 32, and 64
  • DSpark performance uses SGLANG_SIMULATE_ACC_LEN=6 while regular inference omits --speculative-algorithm

DSpark

Decode throughput (output tok/s)

Batch main fusion index (no kernel) fusion kernel fusion kernel + fork
1 756.90 701.62 721.97 759.08
8 4,582.62 4,292.23 4,431.16 4,598.75
32 14,357.79 13,606.42 14,049.60 14,493.22
64 25,977.82 24,734.04 25,413.47 26,165.51

Prefill throughput (input tok/s)

Batch main fusion index (no kernel) fusion kernel fusion kernel + fork
1 12,717.26 12,737.81 14,355.81 14,392.83
8 96,307.46 70,973.86 102,971.41 107,059.48
32 165,250.51 159,007.99 142,998.49 169,604.66
64 147,197.12 173,145.35 164,764.98 174,694.23

End-to-end throughput (overall tok/s)

Batch main fusion index (no kernel) fusion kernel fusion kernel + fork
1 3,052.04 2,870.52 3,001.36 3,125.89
8 19,255.64 17,281.11 18,901.10 19,624.04
32 53,236.74 50,710.59 50,370.68 54,022.42
64 76,347.80 78,686.38 78,806.15 81,739.39

Regular inference, no DSpark

Decode throughput (output tok/s)

Batch main fusion index (no kernel) fusion kernel fusion kernel + fork
1 154.08 144.31 149.06 155.84
8 1,087.39 1,024.38 1,051.05 1,093.97
32 3,532.20 3,376.20 3,452.26 3,552.20
64 6,331.32 6,012.39 6,238.97 6,409.05

Prefill throughput (input tok/s)

Batch main fusion index (no kernel) fusion kernel fusion kernel + fork
1 13,658.51 14,851.47 15,349.32 15,019.27
8 77,148.75 89,425.99 108,876.45 106,129.73
32 150,395.92 123,628.46 139,848.67 150,440.94
64 170,862.41 157,327.51 165,910.17 155,020.38

End-to-end throughput (overall tok/s)

Batch main fusion index (no kernel) fusion kernel fusion kernel + fork
1 737.25 694.47 717.37 748.13
8 5,147.25 4,938.40 5,059.74 5,252.39
32 16,123.12 15,393.83 15,723.31 16,245.61
64 27,562.54 26,171.25 27,118.78 27,506.76

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #36500558133
Latest PR Test (Extra): ❌ Run #36500558119
Latest PR Test (AMD ROCm 10): ❌ Run #36500558338

@xikronz
xikronz marked this pull request as ready for review September 14, 2026 01:33
Comment thread python/sglang/srt/models/deepseek_v4.py Resolved
Comment thread python/sglang/srt/environ.py Resolved
@b8zhong

b8zhong commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci again

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/kernels/ops/moe/test_mega_moe_pre_dispatch.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

🚀 ubuntu-latest (2 tests): ❌ View workflow run

cd test/ && python3 registered/unit/layers/moe/test_mega_moe_deepgemm_api.py
cd test/ && python3 registered/spec/dspark/test_dspark_draft_path_default.py

Comment thread python/sglang/srt/environ.py Resolved
@b8zhong

b8zhong commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/kernels/ops/moe/test_mega_moe_pre_dispatch.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

🚀 ubuntu-latest (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/layers/moe/test_mega_moe_deepgemm_api.py
cd test/ && python3 registered/spec/dspark/test_dspark_draft_path_default.py

@b8zhong b8zhong added the run-ci CI: run the baseline test suite on this PR label Sep 26, 2026
@b8zhong

b8zhong commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

/rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py again

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/kernels/ops/moe/test_mega_moe_pre_dispatch.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

🚀 ubuntu-latest (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/layers/moe/test_mega_moe_deepgemm_api.py
cd test/ && python3 registered/spec/dspark/test_dspark_draft_path_default.py

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py again:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/kernels/ops/moe/test_mega_moe_pre_dispatch.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

🚀 ubuntu-latest (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/layers/moe/test_mega_moe_deepgemm_api.py
cd test/ && python3 registered/spec/dspark/test_dspark_draft_path_default.py

⛔ again: No test file found matching again under test/registered/ or python/sglang/multimodal_gen/test/.

@b8zhong b8zhong added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) deepseek jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants