Skip to content

[SM100] MegaMoE: compute the expert pool block offsets once per call - #50

Draft
Gin-Sin wants to merge 1 commit into
vllm-project:devfrom
Gin-Sin:megamoe-sched-prefix
Draft

Gin-Sin wants to merge 1 commit into
vllm-project:devfrom
Gin-Sin:megamoe-sched-prefix

Conversation

@Gin-Sin

@Gin-Sin Gin-Sin commented Nov 2, 2026 •

Copy link
Copy Markdown

Overview

MegaMoEScheduler::create_task maps a task's pool block to its expert by rebuilding the expert → pool block prefix over all local experts on every call: one math::warp_inclusive_sum plus one ptx::exchange per 32 experts. With 512 local experts that is 16 rounds, about 100 warp shuffles per task. The scheduler warp does this between claiming a task and publishing it, so with many short tasks (small and medium token counts, small L2 tiles) the TMA and MMA warps wait for the scheduler.

This PR computes the offsets once per call, right after the receive counts arrive (store_pool_block_offsets(), scheduler warp only), keeps them in registers next to stored_num_tokens_per_expert, and lets create_task stop at the owning expert. Every task is built exactly as before, so outputs are bitwise identical.

It is one header, deep_gemm/include/deep_gemm/scheduler/mega_moe.cuh (+19 / −5), and applies to every MegaMoE kernel that uses this scheduler (FP8 × FP4, FP8 × FP8, BF16, FP4 × FP4). The same per-task scan is in deepseek-ai/DeepGEMM main. No open PR in either repository touches it.

Claims

  1. Bitwise identical to dev: y and the receive stats match in 1520 cases covering 4 MMA types, EP1 and EP4, 19 token counts from 1 to 16384 per rank, uniform and hot-expert routing, and 2 seeds.
  2. Faster at 512 local experts (EP1). On B300, Qwen3.8-Flash-Next's routed experts (hidden 2560, intermediate 640, top-10 of 512), MXFP8, kernel time vs dev:
    • uniform routing: 7–9% less time from 8 to 2048 tokens, 2.5% less at 8192–16384;
    • hot experts: 10–26% less time from 64 to 2048 tokens.
      MXFP8 × MXFP4 gains more: 20–26% less time with uniform routing and up to 30% less with hot experts.
      Inside the engine, with real routing and weights, MegaMoE calls are 9.8% faster at decode batch 64, 22% faster on 1k-token prefills and 4% faster on 8k–16k-token prefill chunks.
  3. Smaller at 128 local experts (EP4): MXFP8 kernel time is 1–1.5% lower (uniform) and 3–5% lower (hot) from 32 to 256 tokens per rank, and unchanged from 512 up.
  4. Cost: in the microbenchmark, EP1 calls with 1–2 tokens are 2–3% (≈ 1.5 µs) slower. This is real: two identical copies of dev measure 0.0%. It comes from code generation, not from the added work; see Limitations. It does not show in the engine: 1-token decode calls are 1.0% faster there, and c1 TPOT is unchanged.
  5. E2E, Qwen3.8-Flash-Next-FP8 on one B300 (TP1, MegaMoE as an EP group of one):
    • Decode only (8192 cached input tokens via DecodeBenchConnector, 1024 out): TPOT 4.8–5.6% lower (better) and output throughput 5.0–5.8% higher at c16–c128. Every placement is 4.5–5.8% lower, against ≤ 0.2% between same-arm servers. Unchanged at c1.
    • Prefill + decode (8192 in / 1024 out): TPOT 2.7–2.8% lower and output throughput 2.5–2.7% higher at c16 / c64. Every placement is 1.7–4.4% lower, against ≤ 1.0% between same-arm servers. TTFT is unchanged beyond noise, and c1 / c4 are unchanged.
    • Multi-GPU decode (DP4 + EP, MTP3, c16–c464): no change beyond noise, as expected from the EP4 kernel times at those sizes.
  6. Accuracy: GSM8K is 0.965 / 0.969 on two PR servers vs 0.968 / 0.968 on two dev servers. Greedy outputs are 200/200 identical across all pairs of the four servers.

Validation

Microbenchmarks

Each MegaMoE call runs inside a CUDA graph, timed with CUPTI (kernel time only):

  • Cold L2: each graph holds 8 launches on 8 copies of the expert weights, so every launch reads its weights from HBM.
  • Placement: each value is the median over 8 placements (EP1: 8 GPUs with seed = GPU; EP4: 4 GPU groups × 2 seeds).
  • A-B-B-A: each placement runs dev and this PR in A-B-B-A order in fresh processes, alternating which arm goes first. Each arm's value is the mean of its two runs.
  • Within-arm noise: the same arm, run twice in one placement, differs by 0.05% (median) and 0.4% (p90).
  • EP4: the time is the slowest rank's.

The routed experts use Qwen3.8-Flash-Next's shape (hidden 2560, intermediate 640, top-10 of 512 experts) with num_max_tokens_per_rank = 16384. The two routings are:

  • Uniform: top-10 of Gaussian scores.
  • Hot: a Zipf-like bias of log(1 / popularity rank) is added to the scores. The most popular expert then receives about 97% of the tokens; at 4096 tokens, 255 of the 512 experts are touched.

EP1, MXFP8 (512 local experts)

Tokens Uniform: dev (µs) PR (µs) Δ time % SOL dev → PR Hot: dev (µs) PR (µs) Δ time
1 66.5 66.8 +2.3% 10% → 10% 68.1 68.8 +2.2%
8 93.8 87.6 −7.0% 50% → 53% 72.5 73.6 +0.5%
64 325.8 296.3 −9.1% 72% → 79% 103.6 93.1 −9.7%
256 444.8 407.4 −8.6% 73% → 80% 196.1 154.7 −20.7%
1024 470.6 434.6 −7.6% 72% → 78% 337.4 261.3 −22.7%
2048 510.5 468.3 −8.3% 68% → 75% 399.3 328.5 −17.9%
4096 581.7 557.1 −4.2% 64% → 67% 513.6 471.8 −8.5%
16384 1461.2 1428.0 −2.5% 36% → 37% 1439.7 1394.4 −3.0%

EP4, MXFP8 (128 local experts per rank, 4 × B300 over NVLink)

Tokens per rank Uniform: dev (µs) PR (µs) Δ time Hot: dev (µs) PR (µs) Δ time
1 45.8 46.0 −0.4% 44.1 43.9 −0.4%
8 81.1 80.7 −0.5% 49.1 48.8 −0.6%
64 130.3 128.4 −1.3% 76.3 74.4 −3.3%
128 133.1 131.0 −1.5% 102.8 97.4 −5.2%
256 136.1 134.3 −1.2% 112.5 107.6 −4.4%
512 146.0 146.0 −0.0% 142.7 141.1 −0.6%
2048 287.2 286.7 −0.2% 406.6 405.9 −0.1%
16384 1941.1 1941.6 +0.0% 2874.2 2875.7 +0.1%

Δ time is the median over placements of PR / dev − 1 of the kernel time, so negative means the PR is faster. SOL is max(FLOPs / 4.5 PFLOP/s, ideal bytes / 8 TB/s, NVLink bytes / 0.9 TB/s) for the busiest rank:

  • FLOPs: 2 · received tokens · 3 · hidden · intermediate.
  • Ideal bytes: every byte is read or written once: the touched experts' weights and scale factors, the FP8 activations into and out of each GEMM, the BF16 L2 output, x and y.
  • NVLink bytes: received tokens · hidden · 3 (FP8 dispatch plus BF16 combine), EP4 only.
All token counts, MXFP8 × MXFP4, spread, ablation, register usage and method

All rows. Spread = (max − min) / median over the 8 placements of each arm's time. At EP1 small token counts and in every hot EP4 row it mostly reflects routing: each placement has its own seed, so the touched experts differ. The per-placement PR / dev ratio is much tighter; for example, EP4 hot at 128 tokens ranges from −5.4% to −3.4%.

EP MMA Routing Tokens / rank dev (µs) PR (µs) Δ time Spread dev / PR dev % SOL PR % SOL
1 MXFP8 uniform 1 66.5 66.8 +2.3% 12.2% / 12.3% 10% 10%
1 MXFP8 uniform 8 93.8 87.6 −7.0% 6.5% / 4.1% 50% 53%
1 MXFP8 uniform 32 220.9 201.8 −8.6% 6.6% / 6.4% 68% 75%
1 MXFP8 uniform 64 325.8 296.3 −9.1% 5.5% / 4.5% 72% 79%
1 MXFP8 uniform 128 413.3 376.3 −8.9% 3.8% / 2.6% 73% 80%
1 MXFP8 uniform 256 444.8 407.4 −8.6% 0.8% / 0.6% 73% 80%
1 MXFP8 uniform 512 459.1 418.0 −8.9% 0.7% / 0.5% 72% 79%
1 MXFP8 uniform 1024 470.6 434.6 −7.6% 0.5% / 0.2% 72% 78%
1 MXFP8 uniform 2048 510.5 468.3 −8.3% 0.4% / 0.3% 68% 75%
1 MXFP8 uniform 4096 581.7 557.1 −4.2% 1.0% / 1.0% 64% 67%
1 MXFP8 uniform 8192 812.1 795.1 −2.5% 1.6% / 2.1% 52% 53%
1 MXFP8 uniform 16384 1461.2 1428.0 −2.5% 2.5% / 2.5% 36% 37%
1 MXFP8 hot 1 68.1 68.8 +2.2% 7.5% / 5.8% 9% 9%
1 MXFP8 hot 8 72.5 73.6 +0.5% 7.1% / 3.3% 27% 26%
1 MXFP8 hot 32 87.4 83.2 −5.2% 9.4% / 6.0% 42% 44%
1 MXFP8 hot 64 103.6 93.1 −9.7% 7.6% / 6.3% 46% 52%
1 MXFP8 hot 128 135.5 114.4 −15.2% 10.2% / 8.5% 46% 55%
1 MXFP8 hot 256 196.1 154.7 −20.7% 7.4% / 8.9% 40% 51%
1 MXFP8 hot 512 300.0 223.5 −25.6% 5.2% / 6.5% 32% 43%
1 MXFP8 hot 1024 337.4 261.3 −22.7% 3.9% / 4.7% 37% 47%
1 MXFP8 hot 2048 399.3 328.5 −17.9% 3.6% / 2.7% 40% 49%
1 MXFP8 hot 4096 513.6 471.8 −8.5% 1.5% / 1.5% 41% 45%
1 MXFP8 hot 8192 808.4 780.7 −3.4% 2.8% / 2.9% 36% 37%
1 MXFP8 hot 16384 1439.7 1394.4 −3.0% 2.3% / 3.0% 29% 30%
1 MXFP8 × MXFP4 uniform 1 65.3 66.2 +1.0% 11.8% / 12.4% 5% 5%
1 MXFP8 × MXFP4 uniform 8 87.2 78.3 −10.7% 4.6% / 4.7% 27% 30%
1 MXFP8 × MXFP4 uniform 32 180.2 136.9 −24.0% 7.3% / 6.3% 42% 55%
1 MXFP8 × MXFP4 uniform 64 260.8 193.9 −25.7% 5.0% / 4.7% 45% 60%
1 MXFP8 × MXFP4 uniform 128 328.0 243.2 −25.7% 3.2% / 3.1% 46% 62%
1 MXFP8 × MXFP4 uniform 256 352.9 263.8 −25.3% 1.0% / 0.7% 47% 62%
1 MXFP8 × MXFP4 uniform 512 364.3 272.7 −25.0% 1.1% / 1.3% 46% 62%
1 MXFP8 × MXFP4 uniform 1024 376.5 286.4 −23.9% 0.7% / 0.5% 46% 61%
1 MXFP8 × MXFP4 uniform 2048 406.6 326.9 −19.6% 0.7% / 1.0% 46% 57%
1 MXFP8 × MXFP4 uniform 4096 484.5 446.4 −8.0% 2.0% / 1.5% 44% 47%
1 MXFP8 × MXFP4 uniform 8192 723.6 699.5 −3.5% 1.2% / 1.2% 36% 37%
1 MXFP8 × MXFP4 uniform 16384 1357.6 1305.4 −3.7% 1.1% / 1.7% 27% 28%
1 MXFP8 × MXFP4 hot 1 66.7 67.6 +0.8% 7.6% / 7.2% 5% 5%
1 MXFP8 × MXFP4 hot 8 71.2 70.8 −0.3% 7.6% / 6.7% 14% 14%
1 MXFP8 × MXFP4 hot 32 85.6 77.4 −9.6% 6.4% / 5.4% 22% 24%
1 MXFP8 × MXFP4 hot 64 96.3 85.2 −11.4% 5.0% / 5.1% 25% 29%
1 MXFP8 × MXFP4 hot 128 124.0 98.7 −19.9% 8.2% / 6.6% 26% 32%
1 MXFP8 × MXFP4 hot 256 182.0 132.1 −27.3% 6.9% / 7.6% 22% 31%
1 MXFP8 × MXFP4 hot 512 285.3 198.6 −30.5% 3.6% / 4.3% 18% 26%
1 MXFP8 × MXFP4 hot 1024 315.6 225.0 −28.8% 3.1% / 3.1% 22% 30%
1 MXFP8 × MXFP4 hot 2048 367.2 275.4 −25.1% 2.2% / 2.4% 25% 34%
1 MXFP8 × MXFP4 hot 4096 464.4 414.1 −10.7% 1.2% / 1.1% 28% 32%
1 MXFP8 × MXFP4 hot 8192 747.5 711.4 −4.8% 1.8% / 1.6% 26% 27%
1 MXFP8 × MXFP4 hot 16384 1358.7 1303.9 −4.0% 1.9% / 1.6% 26% 27%
4 MXFP8 uniform 1 45.8 46.0 −0.4% 8.3% / 7.8% 18% 18%
4 MXFP8 uniform 8 81.1 80.7 −0.5% 12.3% / 11.3% 52% 53%
4 MXFP8 uniform 32 123.6 122.1 −0.9% 3.7% / 2.9% 62% 63%
4 MXFP8 uniform 64 130.3 128.4 −1.3% 1.2% / 1.2% 63% 64%
4 MXFP8 uniform 128 133.1 131.0 −1.5% 1.1% / 0.8% 62% 63%
4 MXFP8 uniform 256 136.1 134.3 −1.2% 0.7% / 1.0% 62% 63%
4 MXFP8 uniform 512 146.0 146.0 −0.0% 0.8% / 1.6% 60% 60%
4 MXFP8 uniform 1024 181.5 179.9 −0.9% 0.8% / 0.7% 52% 52%
4 MXFP8 uniform 2048 287.2 286.7 −0.2% 1.1% / 1.1% 61% 61%
4 MXFP8 uniform 4096 520.2 519.7 −0.2% 0.5% / 0.5% 67% 67%
4 MXFP8 uniform 8192 986.5 987.3 +0.1% 0.2% / 0.2% 71% 71%
4 MXFP8 uniform 16384 1941.1 1941.6 +0.0% 0.3% / 0.3% 72% 72%
4 MXFP8 hot 1 44.1 43.9 −0.4% 8.5% / 9.2% 13% 13%
4 MXFP8 hot 8 49.1 48.8 −0.6% 7.1% / 7.6% 24% 24%
4 MXFP8 hot 32 61.1 59.5 −2.5% 10.9% / 12.7% 31% 32%
4 MXFP8 hot 64 76.3 74.4 −3.3% 11.4% / 10.3% 33% 34%
4 MXFP8 hot 128 102.8 97.4 −5.2% 13.2% / 12.9% 29% 31%
4 MXFP8 hot 256 112.5 107.6 −4.4% 9.4% / 8.9% 34% 35%
4 MXFP8 hot 512 142.7 141.1 −0.6% 16.8% / 17.5% 48% 49%
4 MXFP8 hot 1024 227.5 226.1 −0.4% 20.5% / 20.4% 61% 61%
4 MXFP8 hot 2048 406.6 405.9 −0.1% 21.5% / 21.5% 68% 68%
4 MXFP8 hot 4096 765.4 765.1 −0.1% 21.4% / 21.4% 72% 72%
4 MXFP8 hot 8192 1482.6 1483.6 +0.1% 21.3% / 21.3% 74% 74%
4 MXFP8 hot 16384 2874.2 2875.7 +0.1% 22.6% / 22.6% 76% 76%

Ablation (EP1, MXFP8; 8 GPUs, one run per arm and GPU in rotated order; kernel time, median over GPUs of arm / dev − 1 with the range; negative means faster than dev):

Routing Tokens dev (µs) dev + early break only Offsets only (no break) This PR Offsets in fetch_expert_recv_count
uniform 1 66.6 +1.1% [−4.7, +2.0] +2.0% [+1.5, +3.6] +1.8% [+0.0, +3.1] +2.2% [−0.4, +3.5]
uniform 2 68.8 +1.5% [−0.5, +2.5] +3.2% [+0.6, +3.8] +3.1% [+0.2, +3.8] +3.2% [+0.1, +4.0]
uniform 8 93.9 −2.6% [−4.7, −0.2] −6.0% [−7.3, −4.2] −6.4% [−7.9, −4.4] −6.5% [−7.8, −4.2]
uniform 64 325.9 −5.3% [−5.6, −4.1] −8.2% [−8.6, −7.7] −9.1% [−9.5, −8.3] −9.0% [−9.4, −8.1]
uniform 512 459.1 −5.0% [−5.1, −4.5] −6.7% [−7.0, −6.5] −8.9% [−9.1, −8.8] −8.9% [−9.1, −8.8]
uniform 4096 580.4 −1.8% [−2.1, −1.6] +0.0% [−0.2, +0.3] −4.2% [−4.7, −3.8] −4.3% [−4.7, −4.0]
hot 1 68.3 +1.4% [−1.1, +2.3] +1.8% [−4.5, +3.2] +2.0% [+0.6, +3.7] +1.8% [−3.4, +3.0]
hot 64 103.6 −3.9% [−4.8, −2.6] −6.9% [−7.8, −6.3] −9.4% [−10.0, −7.5] −9.4% [−10.2, −8.5]
hot 512 299.9 −11.7% [−13.0, −8.5] −19.1% [−19.6, −18.6] −25.6% [−25.8, −24.6] −25.1% [−26.4, −24.5]
hot 4096 513.7 −4.7% [−5.3, −4.3] −2.5% [−3.1, −2.2] −8.5% [−9.2, −7.5] −8.4% [−9.1, −7.7]

Both halves help, and together they are the fastest at every size from 8 tokens up. Computing the offsets in fetch_expert_recv_count (which the dispatch warps also call) instead of in mainloop measures the same; this PR keeps them in the scheduler warp only. Control: two byte-identical copies of dev, and dev with a comment changed (a different JIT cache key), measure +0.0% at 1, 2, 4 and 8 tokens.

Register usage (cuobjdump -res-usage on the JIT cubins of every config compiled in these runs):

  • Registers: 128 per thread in both arms, for every kernel.
  • Stack frame: unchanged at EP4 and for the BF16 and FP4 × FP4 kernels. Grows by 8–16 bytes for the FP8 × FP4 / FP8 × FP8 kernels at EP1 (512 experts per rank, 16 cached experts per lane).

Method. B300 SXM6 (148 SMs), driver 595.112.07, CUDA 13.1, torch 2.14.0+cu132. dev = 9918052, this PR = f2def50; same extension build flags. Each arm has its own JIT cache. Timing runs the kernel alone: inputs are copied into the symmetric buffer before graph capture, and the timed output is checked against the eager output's hash. Raw CUPTI kernel durations, median of ≥ 400 launches per process. The harness is a ~200-line script outside this repository; I can add it under tests/ if useful.

Correctness

  • Bitwise A/B against dev: sha256 of y and of cumulative_local_expert_recv_stats, all identical in 1520 cases:
    • MMA types: FP8 × FP8, FP8 × FP4, BF16, all at hidden 2560 and intermediate 640. FP4 × FP4 runs at intermediate 768, because it requires a multiple of 256.
    • EP1 and EP4 (every rank compared).
    • Tokens per rank: 1, 2, 3, 8, 16, 32, 64, 100, 128, 256, 333, 512, 1000, 1024, 2048, 4096, 8192, 12288, 16384.
    • Uniform and hot routing, 2 seeds.
    • Every case runs twice per arm and both runs agree, so the kernel is deterministic here. The 4608 timed CUDA-graph launches also reproduce their eager hashes.
  • Repository tests (both arms, B300, 4 GPUs unless noted):
    • pytest tests/test_mega_moe_block_config.py: 24 passed.
    • tests/test_fp4_fp4_mega_moe.py, default and --masked-ratio 0.3 --num-max-removed-tokens 512: passed.
    • tests/test_nvfp4_mega_moe.py: passed with the default NVFP4 config and with --mma-type fp8xfp8 / fp8xfp4 at the Qwen3.8 shape (--hidden 2560 --intermediate-hidden 640 --experts 512 --topk 10 --shared-experts 0), 4 ranks and 1 rank.
    • tests/test_mega_moe.py, default and --mma-type fp8xfp8 at the Qwen3.8 shape: runs and benchmarks. Its correctness check compares against DeepEP + TileLang, which are not installed here; the bitwise A/B above covers it.
    • tests/test_mega_moe.py --post-expert-norm --mma-type bf16xbf16 --num-shared-experts 0: fails identically on dev (2 of 2 runs per arm) at the CUDA-graph assert_close, after its eager checks pass. This is pre-existing and not touched here.

E2E perf

Setup.

  • vLLM: main (5d55ef1590) + [Model][Qwen4Exp] Run routed experts on DeepGEMM MegaMoE (MXFP8 or MXFP4) vllm#64315, the open PR that runs Qwen4Exp's routed experts on DeepGEMM MegaMoE (MXFP8 by default for the block-FP8 checkpoint).
  • DeepGEMM per arm: vendored the way DEEPGEMM_SRC_DIR=<arm checkout> vendors it (tools/build_deepgemm_C.py plus the cmake install list); each arm has its own JIT cache. The arms differ only in scheduler/mega_moe.cuh.
  • Model: Qwen3.8-Flash-Next-FP8 on B300.
  • Loads: the two standard ones, both on one GPU (TP1):
    • Prefill + decode: 8192 in / 1024 out at c1 / c4 / c16 / c64.
    • Decode only: 8192 cached input tokens / 1024 out at c1 / c16 / c64 / c128, with --kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}'. The connector fills the KV cache with dummy values instead of running prefill, so every step is a decode step over an 8k context.
  • Method: each load runs in A-B-B-A order on two servers at a time, over 4 placements in which dev and the PR swap GPUs. dev/dev and PR/PR pairs, run the same way, measure the noise. Δ is PR / dev − 1: the mean over placements, with the per-placement range in brackets. Values are medians (TTFT, TPOT) and totals (output tok/s) from vllm bench serve, averaged over the 8 runs per arm.

Prefill + decode, 8192 in / 1024 out

Concurrency TTFT (ms, lower is better) dev → PR Δ TTFT TPOT (ms, lower is better) dev → PR Δ TPOT Output tok/s (higher is better) dev → PR Δ output tok/s
1 204 → 204 +0.1% [−2.6, +2.1] 7.02 → 7.02 +0.1% [−0.6, +1.0] 139 → 139 −0.1% [−1.0, +0.5]
4 619 → 619 +0.0% [−1.8, +0.8] 8.15 → 8.17 +0.2% [−0.5, +0.8] 457 → 456 −0.3% [−0.9, +0.4]
16 1621 → 1577 −2.7% [−8.4, +1.8] 11.72 → 11.39 −2.8% [−3.9, −1.7] 1207 → 1238 +2.5% [+1.8, +3.1]
64 1578 → 1574 −0.2% [−1.8, +0.8] 26.51 → 25.78 −2.7% [−4.4, −2.0] 2271 → 2332 +2.7% [+1.8, +3.6]

Same-arm pairs (B / A − 1): TTFT −4.5% … +0.2%, TPOT −1.0% … +1.0%, output tok/s −0.5% … +1.1%.

Decode only, 8192 cached in / 1024 out (DecodeBenchConnector)

Concurrency TPOT (ms, lower is better) dev → PR Δ TPOT Output tok/s (higher is better) dev → PR Δ output tok/s
1 7.11 → 7.09 −0.3% [−2.2, +0.7] 139.5 → 140.0 +0.4% [−0.6, +2.2]
16 12.41 → 11.81 −4.8% [−5.2, −4.5] 1276 → 1341 +5.0% [+4.7, +5.4]
64 23.81 → 22.48 −5.6% [−5.8, −5.5] 2656 → 2810 +5.8% [+5.8, +5.9]
128 31.80 → 30.24 −4.9% [−5.2, −4.6] 3942 → 4146 +5.2% [+4.9, +5.5]

Same-arm pairs: TPOT −0.2% … +1.0% (c1), −0.2% … −0.1% (c16–c128); output tok/s −1.1% … +0.3%.

Reading the tables.

  • Decode is where the gain is. From c16 up, each decode step runs 48 MegaMoE calls of 16–128 tokens, the range where the kernel is 7–9% faster, and MegaMoE is a large share of the step. The in-engine profile at decode batch 64 predicts 4.8% less step time (MegaMoE 201.7 → 182.0 µs per call, below); the decode-only load measures 5.6%. At c1 the kernel is about 1% faster, too small to show.
  • Prefill + decode gains about half as much: decode steps share the GPU with 8k-token prefill chunks, where MegaMoE calls gain only 2.5–4% (about 0.5% expected end to end). TTFT at c16 is 2.7% lower on average, but the placements range from −8.4% to +1.8% and same-arm pairs differ by up to 4.5%, so TTFT does not change beyond noise.
  • Decode-only caveats. The KV contents are dummy values (the int8 parts of the cache are filled with zeros), so routing follows the dummy context rather than real text; the work per step has the same shape. TTFT in this load measures the dummy KV fill, not prefill, and is not reported.
In-engine kernel times, other loads (short prompts, 1024 / 512, multi-GPU decode), setup

In-engine MegaMoE time per call (TP1, torch profiler, one window per shape, real routing and weights; median over all calls in the window):

Window Calls dev (µs) PR (µs) Δ time
8192-token prompts, c4: prefill chunks 144 788.9 756.7 −4.1%
same window: 4-token decode steps 144 77.0 77.1 +0.1%
1024-token prompts, c1: prefill 384 352.3 274.0 −22.2%
same window: 1-token decode steps 1152 70.5 69.8 −1.0%
decode, batch 64 3264 201.7 182.0 −9.8%

Real routing is more skewed than the uniform microbenchmark routing, so the in-engine 1k-token calls look like the hot-expert microbenchmark rows (23% less time at 1024 tokens).

Engine flags.

  • TP1: --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' --enable-expert-parallel (an EP group of one, so 512 local experts), --max-num-seqs 256 --max-num-batched-tokens 16384 --max-model-len 32768 --no-enable-prefix-caching, no MTP.
  • DP4 + EP: the same kernel config, plus --data-parallel-size 4 --enable-expert-parallel --max-num-seqs 128 --max-num-batched-tokens 8192 --max-model-len 16384 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'.
  • Loads: vllm bench serve --dataset-name random --ignore-eos --temperature 0 --seed 0. Prompts per run: 8 / 32 / 96 / 256 for 8192 / 1024 at c1 / c4 / c16 / c64; 8 / 96 / 256 / 384 for decode-only at c1 / c16 / c64 / c128; 64 / 128 for 1024 / 16 at c1 / c4; 16 / 256 for 1024 / 512 at c1 / c64; 24 / 64 / 192 / 384 for 8192 / 16 at c1 / c4 / c16 / c64; 64 / 256 / 928 for 2048 / 1024 at c16 / c64 / c464. Every server first runs a warm-up load.

Short prompts (1024 / 16, TP1). Qwen4Exp defaults to breakable CUDA graphs, which leaves torch.compile off, so a prefill step above the 512-token capture size runs eagerly. At 1–4 short prompts such a step is CPU-bound: in a torch-profiler trace, a 1024-token step keeps the GPU busy for about 34 ms against about 90 ms TTFT without the profiler. The PR cuts MegaMoE time by 3.8 ms per forward (17.6 → 13.8 ms), and throughput does not move:

Load Metric dev PR Δ mean [range] (n) Same-arm pairs
1024 / 16, c1 input tok/s (higher is better) 5272 5282 +0.2% [−1.6, +2.2] (8) −1.2% … +1.7%
1024 / 16, c4 input tok/s (higher is better) 13960 13923 −0.3% [−1.5, +2.8] (8) −0.4% … +1.3%

Other loads (an earlier round, same method; n = placements):

Config Load (in / out, concurrency) Metric dev PR Δ mean [range] (n) Same-arm pairs
TP1 1024 / 512, c64 TPOT (ms, lower is better) 19.87 18.93 −4.7% [−5.4, −4.2] (8) −0.9% … +0.2%
TP1 1024 / 512, c1 TPOT (ms, lower is better) 6.98 6.98 +0.1% [−2.7, +1.3] (10) −2.1% … +0.8%
TP1 8192 / 16, c16 input tok/s (higher is better) 50000 50124 +0.3% [−0.5, +1.6] (10) −1.5% … +1.3%
TP1 8192 / 16, c64 input tok/s (higher is better) 52249 52296 +0.1% [−0.8, +1.4] (10) −1.5% … +1.1%
DP4 + EP, MTP3 2048 / 1024, c64 TPOT (ms, lower is better) 8.32 8.21 −1.3% [−1.8, −0.8] (2) +1.6%
DP4 + EP, MTP3 2048 / 1024, c464 output tok/s (higher is better) 17351 17267 −0.5% [−1.9, +0.9] (2) +0.7%
DP4 + EP, MTP3 2048 / 1024, c464 time per verify step (ms, lower is better) 48.82 49.16 +0.7% [+0.4, +1.0] (2) +1.4%
  • 8192 / 16 fills each prefill step to about 16k tokens (--max-num-batched-tokens 16384). MegaMoE calls gain 2.5–4% there and are about 20% of GPU time, so ≈ 0.5% is expected, below the noise.
  • DP4 + EP with MTP3: at c464 each rank verifies about 460 tokens per step, where the EP4 kernel is unchanged. Time per verify step = median TPOT × MTP acceptance length (2.39–2.44), which removes the run-to-run acceptance noise.

E2E accuracy

Same TP1 MegaMoE serving config, two servers per arm:

dev (1) dev (2) PR (1) PR (2)
GSM8K, 1319 questions, 5-shot, chat, thinking off, temperature 0 0.968 0.968 0.965 0.969

Greedy identity: 200 GSM8K questions with 8-shot prompts (1330–1440 tokens), 256 output tokens each, sent one at a time. All six server pairs are 200/200 identical. The kernel is bitwise identical, so this only confirms that nothing else differs between the arms.

Reproduce

# Repository tests
python -m pytest -q tests/test_mega_moe_block_config.py
python tests/test_fp4_fp4_mega_moe.py
python tests/test_nvfp4_mega_moe.py --num-processes 4 --mma-type fp8xfp8 \
  --hidden 2560 --intermediate-hidden 640 --experts 512 --topk 10 --shared-experts 0
# Quick speed check with the repository's CUDA-graph benchmark (MXFP8 x MXFP4, 512 experts on one GPU).
# On one B300 (warm L2, CUDA events), dev -> this PR in us:
#   8: 86.3 -> 77.9, 64: 262.7 -> 196.3, 512: 369.0 -> 277.2, 2048: 407.8 -> 321.3, 8192: 692.5 -> 661.5
python tests/bench_mega_moe_cudagraph.py --num-processes 1 --hidden 2560 --intermediate-hidden 640 \
  --num-experts 512 --num-topk 10 --mma-types fp8xfp4 --token-counts 8,64,512,2048,8192
# E2E: vLLM built with DEEPGEMM_SRC_DIR=<this branch> (vs <dev>), plus vllm-project/vllm#64315
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}' --enable-expert-parallel \
  --max-num-seqs 256 --max-num-batched-tokens 16384 --max-model-len 32768 --no-enable-prefix-caching
vllm bench serve --dataset-name random --random-input-len 8192 --random-output-len 1024 --ignore-eos \
  --temperature 0 --seed 0 --max-concurrency 64 --num-prompts 256
# Decode only: same serve command plus
#   --kv-transfer-config '{"kv_connector":"DecodeBenchConnector","kv_role":"kv_both"}'
# and the same bench command at --max-concurrency 1 / 16 / 64 / 128
python tests/evals/gsm8k/gsm8k_eval.py --use-chat-completions --temperature 0 --max-tokens 1024 \
  --num-questions 1319 --chat-template-kwargs '{"enable_thinking": false}'

Details

Design

  • stored_pool_block_offsets[kNumExpertsPerLane] uses the same lane layout as stored_num_tokens_per_expert: entry i of lane l is the first pool block of expert i * 32 + l. store_pool_block_offsets() fills it with the same warp_inclusive_sum + exchange scan that create_task used to run per task.
  • It runs in mainloop() right after fetch_expert_recv_count(), so only the scheduler warp pays for it. The dispatch warps of the MegaMoE kernels also call fetch_expert_recv_count(), and they do not need the offsets.
  • create_task keeps its ballot over each group of 32 experts. Experts own disjoint pool block ranges, so at most one lane matches, and the loop stops after the first group that has an owner. The values written to task_info_t are unchanged.
  • No host, API or layout change.

Limitations

  • 1–2 tokens per call at EP1 (512 local experts) is 2–3% (≈ 1.5 µs) slower. Even dev with only the early break added (strictly less work) is 1–1.5% slower there, while identical copies of dev measure 0.0%. So the cost comes from how the scheduler code compiles, not from the added scan; none of the variants above avoids it. At EP4, 1 token per rank is 0.4% faster.
  • At 512+ tokens per rank with 128 local experts (EP4), the kernel time is unchanged. The scan is short there (4 rounds per task instead of 16), and the scheduler is no longer the bottleneck.
  • Where the EP1 gains matter in vLLM. For Qwen3.8-Flash-Next on one GPU, MegaMoE (as an EP group of one) beats vLLM's default FlashInfer TRTLLM FP8 MoE on prefill, with 14–16% higher input throughput with 8k-token prompts at c16/c64. It also beats DeepGEMM's grouped FP8 MoE, with 12–19% higher input throughput; from 4k tokens per forward up, its MoE time per layer is 35–45% lower than either. It loses on decode: its TPOT is 63% higher (worse) at c1, and at c64 it is 12% higher with dev and 6% higher with this PR. This PR's end-to-end gain, single-GPU decode from c16 up, therefore narrows a gap to the default backend rather than making MegaMoE the faster choice there. For single-GPU prefill, where MegaMoE is the faster backend, the throughput gain is within noise (≤ 0.5% expected).
  • Measured on B300 only. The change is architecture-neutral within the SM100 MegaMoE kernels.

Possible follow-ups (not in this PR)

  • The scheduler publishes tasks through kNumScheduleStages = 2 slots. A deeper task ring would let the scheduler run further ahead of the consumers; in earlier experiments it made ≤ 4k-token calls 11–14% faster. With the deeper ring, the scheduler is closer to the critical path, so this PR's saving grows at prefill sizes (8k–16k tokens per call); a follow-up PR will measure the combination on dev.
  • Publishing the per-expert receive counts with a non-returning red instead of the returning system-scope atomic_add. Together with the deeper ring, it took a decode-shaped EP4 call from 110 to 102 µs in earlier experiments.

AI assistance was used to implement, benchmark and test this change. 🤖 Generated with Claude Code

`MegaMoEScheduler::create_task` rebuilt the expert -> pool block prefix
over all local experts for every task: one warp inclusive sum plus one
shuffle per 32 experts. The scheduler warp runs it between claiming a
task and publishing it, so with many experts and short tasks the
consumers wait for the scheduler.

Compute the offsets once, after the receive counts arrive
(`store_pool_block_offsets`, scheduler warp only), and stop at the
owning expert. Every task is built exactly as before, so outputs are
bitwise identical.

Signed-off-by: Gin-Sin <ginxian@foxmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant