Repository navigation
fuse shared experts with routed experts in MegaMoE's DeepGEMM - #39313
Conversation
|
/rerun-failed-ci again |
|
Results for 🚀 🚀 |
|
/rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py |
|
Results for 🚀 🚀 |
|
/rerun-test test_deepseek_v4_flash_fp4_megamoe_b200.py test_mega_moe_pre_dispatch.py test_deepseek_v4_flash_fp4_b200_cp.py test_mega_moe_deepgemm_api.py test_dspark_draft_path_default.py again |
|
Results for 🚀 🚀 |
|
Results for 🚀 🚀 ⛔ |
for #38700 (* - *)
Motivation
Context: Deepseek-v4-pro uses DeepseekMoE (served as MegaMoE) as its mixure of experts with a shared expert that is activated across all tokens and a fine-grained routed experts.
For each token$t_i$ , a V4 MoE layer does two different GEMM operations:
Launching each kernel independently wastes launches and stream syncs on every layer (this is true even if they shared a dtype). DeepGEMM already caught this and added native MegaMoE support via (num_shared_experts, separate shared_l1/shared_l2 weights, mixed FP8 and MXFP4).
The existing Sglang fusion logic in
deepseek-v2.pyonly works if shared and routed share a dtype/layout (from
deepseel-v4.py):but the official deepseek-ai checkpoints do not, so that path does not apply on MegaMoE (it can still run on the trtllm-gen runner if you
--enforce-shared-experts-fusionand requantize shared FP8→MXFP4, which is what #35505 did)Modifications
Changes mainly in mega_moe.py:
This PR went through a lot of iterations, starting with a naive index implementation that was supposed to be easier to maintain and more local w.r.t. number of files changed. The memory and throughput overhead of maintaining a python cache of expert indices was pretty bad (as seen below) which motivated for a kernel level change in
mega_moe_predispatch.cuh.Placing the FP8 shared expert inside routed MegaMoE (removing six standalone kernels per layer) does this but it regressed throughput because it bypassed the existing alternate-stream fork in
mega_moe.py. In the unfused profiles, the shared expert’s second GEMM overlapped with the routed MegaMoE in 184/184 observed calls per rank, covering a median 94.1–95.6% of its duration. Naively fusing the path without modifyingforward_mega_moe()function returned before the existing alternate-stream fork, leaving the whole router/top-k/pre-dispatch/MegaMoE region on the current stream...The current iteration of this PR keeps the same kernel changes but forks the whole fused region in
forward_mega_moe()which lets other graph work overlap it; at BS1, main, kernel-only, and kernel-plus-fork achieved 756.90, 721.97, and 759.08 tok/s.Major files changes:
_get_mega_moe_symm_bufferincludesnum_shared_expertsin its cache key and allocates the shared-expert buffer for the fused configuration_run_mega_routedforwards the shared-expert count and transformed L1/L2 weights through gate and top-k torun_mega_routed_expertsrun_mega_routed_expertsselects the consuming MegaMoEBLOCK_M, passes the shared-scale destination to pre-dispatch, and launchesfp8_fp4_mega_moewith the shared weightsforward_mega_moeforks the entire fused router/top-k/pre-dispatch/MegaMoE region onto the existing alternate stream during CUDA-graph capture and joins before returningAccuracy Tests
Accuracy evals are run on
deepseek-ai/DeepSeek-V4-Flash-0731with the bundled DSpark draft via 4x B300 (SM103) with sgl-deep-gemm 0.1.7. On GPTQA-diamond with sgl-eval and thinking enabled, reasoning effortmax,max_tokens=200000, andtop_p=0.95:paired per-request comparison:
Speed Tests and Profiling
deepseek-ai/DeepSeek-V4-Flash-0731with sgl-deep-gemm 0.1.7 on 4xB300SGLANG_SIMULATE_ACC_LEN=6while regular inference omits--speculative-algorithmDSpark
Decode throughput (output tok/s)
Prefill throughput (input tok/s)
End-to-end throughput (overall tok/s)
Regular inference, no DSpark
Decode throughput (output tok/s)
Prefill throughput (input tok/s)
End-to-end throughput (overall tok/s)
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #36500558133
Latest PR Test (Extra): ❌ Run #36500558119
Latest PR Test (AMD ROCm 10): ❌ Run #36500558338