Repository navigation
Conversation
`MegaMoEScheduler::create_task` rebuilt the expert -> pool block prefix over all local experts for every task: one warp inclusive sum plus one shuffle per 32 experts. The scheduler warp runs it between claiming a task and publishing it, so with many experts and short tasks the consumers wait for the scheduler. Compute the offsets once, after the receive counts arrive (`store_pool_block_offsets`, scheduler warp only), and stop at the owning expert. Every task is built exactly as before, so outputs are bitwise identical. Signed-off-by: Gin-Sin <ginxian@foxmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
MegaMoEScheduler::create_taskmaps a task's pool block to its expert by rebuilding the expert → pool block prefix over all local experts on every call: onemath::warp_inclusive_sumplus oneptx::exchangeper 32 experts. With 512 local experts that is 16 rounds, about 100 warp shuffles per task. The scheduler warp does this between claiming a task and publishing it, so with many short tasks (small and medium token counts, small L2 tiles) the TMA and MMA warps wait for the scheduler.This PR computes the offsets once per call, right after the receive counts arrive (
store_pool_block_offsets(), scheduler warp only), keeps them in registers next tostored_num_tokens_per_expert, and letscreate_taskstop at the owning expert. Every task is built exactly as before, so outputs are bitwise identical.It is one header,
deep_gemm/include/deep_gemm/scheduler/mega_moe.cuh(+19 / −5), and applies to every MegaMoE kernel that uses this scheduler (FP8 × FP4, FP8 × FP8, BF16, FP4 × FP4). The same per-task scan is in deepseek-ai/DeepGEMMmain. No open PR in either repository touches it.Claims
dev:yand the receive stats match in 1520 cases covering 4 MMA types, EP1 and EP4, 19 token counts from 1 to 16384 per rank, uniform and hot-expert routing, and 2 seeds.dev:MXFP8 × MXFP4 gains more: 20–26% less time with uniform routing and up to 30% less with hot experts.
Inside the engine, with real routing and weights, MegaMoE calls are 9.8% faster at decode batch 64, 22% faster on 1k-token prefills and 4% faster on 8k–16k-token prefill chunks.
devmeasure 0.0%. It comes from code generation, not from the added work; see Limitations. It does not show in the engine: 1-token decode calls are 1.0% faster there, and c1 TPOT is unchanged.DecodeBenchConnector, 1024 out): TPOT 4.8–5.6% lower (better) and output throughput 5.0–5.8% higher at c16–c128. Every placement is 4.5–5.8% lower, against ≤ 0.2% between same-arm servers. Unchanged at c1.devservers. Greedy outputs are 200/200 identical across all pairs of the four servers.Validation
Microbenchmarks
Each MegaMoE call runs inside a CUDA graph, timed with CUPTI (kernel time only):
devand this PR in A-B-B-A order in fresh processes, alternating which arm goes first. Each arm's value is the mean of its two runs.The routed experts use Qwen3.8-Flash-Next's shape (hidden 2560, intermediate 640, top-10 of 512 experts) with
num_max_tokens_per_rank= 16384. The two routings are:EP1, MXFP8 (512 local experts)
dev(µs)dev→ PRdev(µs)EP4, MXFP8 (128 local experts per rank, 4 × B300 over NVLink)
dev(µs)dev(µs)Δ time is the median over placements of PR /
dev− 1 of the kernel time, so negative means the PR is faster. SOL is max(FLOPs / 4.5 PFLOP/s, ideal bytes / 8 TB/s, NVLink bytes / 0.9 TB/s) for the busiest rank:xandy.All token counts, MXFP8 × MXFP4, spread, ablation, register usage and method
All rows. Spread = (max − min) / median over the 8 placements of each arm's time. At EP1 small token counts and in every hot EP4 row it mostly reflects routing: each placement has its own seed, so the touched experts differ. The per-placement PR /
devratio is much tighter; for example, EP4 hot at 128 tokens ranges from −5.4% to −3.4%.dev(µs)dev/ PRdev% SOLAblation (EP1, MXFP8; 8 GPUs, one run per arm and GPU in rotated order; kernel time, median over GPUs of arm /
dev− 1 with the range; negative means faster thandev):dev(µs)dev+ earlybreakonlybreak)fetch_expert_recv_countBoth halves help, and together they are the fastest at every size from 8 tokens up. Computing the offsets in
fetch_expert_recv_count(which the dispatch warps also call) instead of inmainloopmeasures the same; this PR keeps them in the scheduler warp only. Control: two byte-identical copies ofdev, anddevwith a comment changed (a different JIT cache key), measure +0.0% at 1, 2, 4 and 8 tokens.Register usage (
cuobjdump -res-usageon the JIT cubins of every config compiled in these runs):Method. B300 SXM6 (148 SMs), driver 595.112.07, CUDA 13.1, torch 2.14.0+cu132.
dev= 9918052, this PR = f2def50; same extension build flags. Each arm has its own JIT cache. Timing runs the kernel alone: inputs are copied into the symmetric buffer before graph capture, and the timed output is checked against the eager output's hash. Raw CUPTI kernel durations, median of ≥ 400 launches per process. The harness is a ~200-line script outside this repository; I can add it undertests/if useful.Correctness
dev: sha256 ofyand ofcumulative_local_expert_recv_stats, all identical in 1520 cases:pytest tests/test_mega_moe_block_config.py: 24 passed.tests/test_fp4_fp4_mega_moe.py, default and--masked-ratio 0.3 --num-max-removed-tokens 512: passed.tests/test_nvfp4_mega_moe.py: passed with the default NVFP4 config and with--mma-type fp8xfp8/fp8xfp4at the Qwen3.8 shape (--hidden 2560 --intermediate-hidden 640 --experts 512 --topk 10 --shared-experts 0), 4 ranks and 1 rank.tests/test_mega_moe.py, default and--mma-type fp8xfp8at the Qwen3.8 shape: runs and benchmarks. Its correctness check compares against DeepEP + TileLang, which are not installed here; the bitwise A/B above covers it.tests/test_mega_moe.py --post-expert-norm --mma-type bf16xbf16 --num-shared-experts 0: fails identically ondev(2 of 2 runs per arm) at the CUDA-graphassert_close, after its eager checks pass. This is pre-existing and not touched here.E2E perf
Setup.
main(5d55ef1590) + [Model][Qwen4Exp] Run routed experts on DeepGEMM MegaMoE (MXFP8 or MXFP4) vllm#64315, the open PR that runs Qwen4Exp's routed experts on DeepGEMM MegaMoE (MXFP8 by default for the block-FP8 checkpoint).DEEPGEMM_SRC_DIR=<arm checkout>vendors it (tools/build_deepgemm_C.pyplus the cmake install list); each arm has its own JIT cache. The arms differ only inscheduler/mega_moe.cuh.--kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}'. The connector fills the KV cache with dummy values instead of running prefill, so every step is a decode step over an 8k context.devand the PR swap GPUs.dev/devand PR/PR pairs, run the same way, measure the noise. Δ is PR /dev− 1: the mean over placements, with the per-placement range in brackets. Values are medians (TTFT, TPOT) and totals (output tok/s) fromvllm bench serve, averaged over the 8 runs per arm.Prefill + decode, 8192 in / 1024 out
dev→ PRdev→ PRdev→ PRSame-arm pairs (B / A − 1): TTFT −4.5% … +0.2%, TPOT −1.0% … +1.0%, output tok/s −0.5% … +1.1%.
Decode only, 8192 cached in / 1024 out (
DecodeBenchConnector)dev→ PRdev→ PRSame-arm pairs: TPOT −0.2% … +1.0% (c1), −0.2% … −0.1% (c16–c128); output tok/s −1.1% … +0.3%.
Reading the tables.
In-engine kernel times, other loads (short prompts, 1024 / 512, multi-GPU decode), setup
In-engine MegaMoE time per call (TP1, torch profiler, one window per shape, real routing and weights; median over all calls in the window):
dev(µs)Real routing is more skewed than the uniform microbenchmark routing, so the in-engine 1k-token calls look like the hot-expert microbenchmark rows (23% less time at 1024 tokens).
Engine flags.
--kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' --enable-expert-parallel(an EP group of one, so 512 local experts),--max-num-seqs 256 --max-num-batched-tokens 16384 --max-model-len 32768 --no-enable-prefix-caching, no MTP.--data-parallel-size 4 --enable-expert-parallel --max-num-seqs 128 --max-num-batched-tokens 8192 --max-model-len 16384 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'.vllm bench serve --dataset-name random --ignore-eos --temperature 0 --seed 0. Prompts per run: 8 / 32 / 96 / 256 for 8192 / 1024 at c1 / c4 / c16 / c64; 8 / 96 / 256 / 384 for decode-only at c1 / c16 / c64 / c128; 64 / 128 for 1024 / 16 at c1 / c4; 16 / 256 for 1024 / 512 at c1 / c64; 24 / 64 / 192 / 384 for 8192 / 16 at c1 / c4 / c16 / c64; 64 / 256 / 928 for 2048 / 1024 at c16 / c64 / c464. Every server first runs a warm-up load.Short prompts (1024 / 16, TP1). Qwen4Exp defaults to breakable CUDA graphs, which leaves torch.compile off, so a prefill step above the 512-token capture size runs eagerly. At 1–4 short prompts such a step is CPU-bound: in a torch-profiler trace, a 1024-token step keeps the GPU busy for about 34 ms against about 90 ms TTFT without the profiler. The PR cuts MegaMoE time by 3.8 ms per forward (17.6 → 13.8 ms), and throughput does not move:
devOther loads (an earlier round, same method; n = placements):
dev--max-num-batched-tokens 16384). MegaMoE calls gain 2.5–4% there and are about 20% of GPU time, so ≈ 0.5% is expected, below the noise.E2E accuracy
Same TP1 MegaMoE serving config, two servers per arm:
dev(1)dev(2)Greedy identity: 200 GSM8K questions with 8-shot prompts (1330–1440 tokens), 256 output tokens each, sent one at a time. All six server pairs are 200/200 identical. The kernel is bitwise identical, so this only confirms that nothing else differs between the arms.
Reproduce
Details
Design
stored_pool_block_offsets[kNumExpertsPerLane]uses the same lane layout asstored_num_tokens_per_expert: entryiof lanelis the first pool block of experti * 32 + l.store_pool_block_offsets()fills it with the samewarp_inclusive_sum+exchangescan thatcreate_taskused to run per task.mainloop()right afterfetch_expert_recv_count(), so only the scheduler warp pays for it. The dispatch warps of the MegaMoE kernels also callfetch_expert_recv_count(), and they do not need the offsets.create_taskkeeps its ballot over each group of 32 experts. Experts own disjoint pool block ranges, so at most one lane matches, and the loop stops after the first group that has an owner. The values written totask_info_tare unchanged.Limitations
devwith only the earlybreakadded (strictly less work) is 1–1.5% slower there, while identical copies ofdevmeasure 0.0%. So the cost comes from how the scheduler code compiles, not from the added scan; none of the variants above avoids it. At EP4, 1 token per rank is 0.4% faster.devand 6% higher with this PR. This PR's end-to-end gain, single-GPU decode from c16 up, therefore narrows a gap to the default backend rather than making MegaMoE the faster choice there. For single-GPU prefill, where MegaMoE is the faster backend, the throughput gain is within noise (≤ 0.5% expected).Possible follow-ups (not in this PR)
kNumScheduleStages = 2slots. A deeper task ring would let the scheduler run further ahead of the consumers; in earlier experiments it made ≤ 4k-token calls 11–14% faster. With the deeper ring, the scheduler is closer to the critical path, so this PR's saving grows at prefill sizes (8k–16k tokens per call); a follow-up PR will measure the combination ondev.redinstead of the returning system-scopeatomic_add. Together with the deeper ring, it took a decode-shaped EP4 call from 110 to 102 µs in earlier experiments.AI assistance was used to implement, benchmark and test this change. 🤖 Generated with Claude Code