Skip to content

[Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks - #43363

Merged
Fridge003 merged 2 commits into
sgl-project:mainfrom
JackChuang:horenc/pr-moe-cutlass-autotune-prefill
Oct 20, 2026
Merged

Fridge003 merged 2 commits into
sgl-project:mainfrom
JackChuang:horenc/pr-moe-cutlass-autotune-prefill

Conversation

@JackChuang

@JackChuang JackChuang commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

TL;DR: This PR enables sglang's startup autotune to tune FlashInfer's CUTLASS MoE kernel for prefill sizes too, not only for decode sizes. sglang already autotunes this kernel (cutlass_fused_moe, FlashInfer's MoE kernel, not its attention backend) at startup, but each call tells FlashInfer to tune only up to its own token count. The startup run is one decode-sized batch, so only decode sizes get a tuned kernel configuration (tactic), and prefill chunks above that size run FlashInfer's untuned fallback tactic (its default kernel configuration). This PR gives FlashInfer one tuning range per server instead: the largest prefill batch the MoE layer sees (chunk size × attention data-parallel size, capped at 8192, FlashInfer's own default range for this call), so the same startup autotune also covers prefill. No new flag, kernel or extra pass. It applies on every GPU this kernel runs on (SM90, SM100, SM110, SM120), where it is the default MoE kernel of DeepSeek-V4.1/V4, Ling-3.0, Nemotron-H, gpt-oss and ModelOpt NVFP4 MoE models (which GPU for which model: Design notes), and to any other MoE model that runs on it. Other MoE kernels and the decode servers of prefill-decode (PD) disaggregation are unchanged. Measured on H200 (SM90).

  • Performance (H200): faster prefill on all seven MoE models measured (DeepSeek-V4.1-Flash, Nemotron-3.5, Ling-3.0, gpt-oss-20b, GLM-5.3-Flash, MiniMax-M3, Gemma-4): TTFT (time to first token) for 8k-token prompts 30.3–57.8% lower (DeepSeek-V4.1-Flash: 573.9 → 400.0 ms, 30.3%), for 32k-token prompts 23.0–52.8% lower, prefill throughput 43.4–143.4% higher; decode ITL (inter-token latency) unchanged on the three models measured for it. It complements open PR [Perf] flashinfer: Autotune kernels at prefill sizes by default #42736 (an extra startup tuning pass): with both, 8k-token TTFT on DeepSeek-V4.1-Flash is 24.7% lower than with [Perf] flashinfer: Autotune kernels at prefill sizes by default #42736 alone.
  • Accuracy: unchanged by design (only the tile configuration, or tactic, of the same GEMM changes); GSM8K on all seven models and the DeepSeek-V4-Flash CI GSM8K test stays within the 95% noise range (−0.83 to +0.15 points on the seven models).
  • Cost: a longer startup autotune when no saved results exist: 41–42 → 176–177 s on DeepSeek-V4.1-Flash 8×H200 (server ready 153–154 → 288–289 s), 116 → 604 s on Ling-3.0-flash, and 4–26 s more on the other five models. Saved results are reused on one GPU, but multi-GPU servers currently re-tune at every start; --disable-flashinfer-autotune turns the startup autotune off. Each GPU has 0.4–1.2 GiB (0.3–0.9% of an H200) less free memory while the server runs, mostly the buffer FlashInfer keeps for timing tactics at startup, now sized for the prefill chunk instead of the decode batch; the KV cache size is unchanged. PD decode servers pay nothing.

Motivation

sglang runs FlashInfer's autotune at startup (#12306, on by default since #14357). For cutlass_fused_moe, FlashInfer keeps one tactic per token-count bucket (powers of two up to 256, then larger steps up to 8192: 21 buckets) and tunes every bucket up to the caller's tune_max_num_tokens; the same value also decides which bucket a call uses. Both sglang call sites (the flashinfer_cutlass runner, used for BF16, FP8 and NVFP4 experts, and the CUTLASS path of flashinfer_mxfp4, used for MXFP4 experts) pass next_power_of_2(x.shape[0]), which dates from #7327, before the startup autotune existed. This has two effects:

  1. Prefill runs an untuned tactic. The startup batch is decode-sized, one token per request up to the request limit (256 tokens on DeepSeek-V4.1-Flash, 138 on Nemotron-3.5-Lightning, 4096 on gpt-oss-20b), so nothing above that size is tuned: an 8k-token chunk maps to bucket 8192 and gets FlashInfer's fallback tactic. The server log shows nothing, because sglang sets FlashInfer's logger to ERROR.
  2. The tactic depends on earlier requests. Every cutlass_fused_moe call that runs Python code (the startup autotune, CUDA graph capture, and every serving forward that is not a CUDA graph replay) first calls MoERunner.refine_tuning_config(tune_max_num_tokens). That method is wrapped in functools.lru_cache and rebuilds FlashInfer's class-wide bucket list only the first time it sees a value, so a call with a value seen before keeps the list built for the latest new value. On DeepSeek-V4.1-Flash with --cuda-graph-max-bs-decode 64, the first decode batch of 65–128 requests while serving runs without a CUDA graph and rebuilds the list for 128 tokens (a short prompt of a new size likewise rebuilds it, for its own size); later prefill chunks of an already-seen size, such as every 8k-token chunk after the first, then run the 128-token bucket's tactic, because a list built for 128 tokens ends at bucket 128 and every larger call falls into that last bucket.

flashinfer-ai/flashinfer#5625 was filed by an sglang user who hit both effects serving DeepSeek-V4-Flash on 4×H100 with flashinfer_mxfp4. One value for the whole server, computed from its configuration and used by every call, as large as the largest prefill batch (up to 8192 tokens), removes both.

Modifications

  • python/sglang/srt/layers/moe/moe_runner/flashinfer_cutlass.py -- the new _cutlass_moe_tune_max_num_tokens(num_tokens) gives both cutlass_fused_moe calls their tune_max_num_tokens: min(prefill chunk size × attention DP size, 8192), with the chunk size (--chunked-prefill-size, or --max-prefill-tokens when chunked prefill is off) read the way flashinfer_autotune.py already does (max_prefill_buffer_tokens() or get_schedule().max_prefill_tokens). PD decode servers keep next_power_of_2(num_tokens).
  • test/registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py (CPU suite) -- new TestCutlassMoeTuneMaxNumTokens. With the kernel stubbed, it checks at both calls that token counts 1, 257, 777, 2048, 4097, 8192 and 16384 map to a bucket the startup autotune tunes, in five server configurations, and that PD decode servers keep the per-call value. 60 of its 80 sub-cases fail on main.
  • test/registered/unit/layers/quantization/test_mxfp4_sm{90,120}_cutlass.py, test/registered/unit/layers/moe/test_moe_runner_config_placement.py -- their fused-MoE calls now run inside get_context().override_server_args(), because the helper reads the server configuration.

Design notes:

  • Where this kernel is the default MoE path: on SM90 and SM120, DeepSeek-V4.1/V4 and Ling-3.0 (MXFP4 experts, through flashinfer_mxfp4) and BF16/FP8 Nemotron-H (flashinfer_cutlass); on SM120 also gpt-oss and ModelOpt NVFP4 MoE models, the latter on SM110 too since fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends #41765. On SM100 no model uses it by default (flashinfer_mxfp4 runs a TRT-LLM kernel there). Any other MoE model reaches it with --moe-runner-backend flashinfer_cutlass (BF16, ModelOpt FP8/NVFP4) or flashinfer_mxfp4 (MXFP4).
  • DP attention: sglang splits --chunked-prefill-size across the DP ranks, and the MoE layer runs on the tokens gathered from all of them, so the largest batch the MoE layer sees is each rank's chunk × the DP size, which is the --chunked-prefill-size the user set.
  • The 8192 cap is a choice, not a limit on chunk size: it is FlashInfer's own default range for this call, and chunks larger than 8192 tokens still run, using bucket 8192's tactic, which in a one-layer test with DeepSeek-V4.1-Flash's MoE shape is as fast as a tactic tuned for each larger size (One MoE call, under Experimental Results); tuning beyond 8192 would only lengthen startup. Omitting the argument would also give one value, but PD decode servers and servers with smaller chunks would then tune sizes they never use.
  • Servers whose startup decode batch is above 8192 tokens (for example attention DP 8 with the default request limit) tune fewer buckets than main and start faster, at no MoE run-time cost: main cannot use those larger buckets at run time either (item 2), and in a one-layer test bucket 8192's tactic is about as fast as each size's own tactic at 12288–32768 tokens (One MoE call, under Experimental Results).

Experimental Results

Performance

Setup: in every prefill row below, every server with this PR beats every server without it (main, or #42736 alone). Every number in this section was measured with this PR's code at head a4c96d4080f on main c7f3ae1dd59 (H200, CUDA 13.0, torch 2.14.1, FlashInfer 0.7.0.post1). Every measurement starts a new server for each version, alternating which goes first; each value is the median over server starts (for TTFT, of each server's median over the 8 prompts). TTFT rows send 8 random prompts one at a time (2k, 8k and 32k mean 2,000, 8,000 and 32,000 tokens; percentages are computed from unrounded medians). Prompts are prefilled in chunks of 8192 tokens (--chunked-prefill-size 8192, set on DeepSeek-V4.1-Flash and the default on the other models here), so a 32k prompt runs as four chunks and every MoE call has at most 8192 tokens. With this chunk size this PR's tuning range is also 8192 (it follows the chunk size; the 8192 cap is FlashInfer's default range for this call, see Design notes); throughput rows send 64 prompts at 8 concurrent requests with 1 output token, so they measure prefill throughput (bench_serving total token throughput). Launch commands are under Commands below; settings not listed there are the defaults.

This table measures the main claim: on a newly started main server, prefill chunks larger than the startup decode batch run FlashInfer's untuned fallback tactic (Motivation, item 1), and with this PR a tuned one. Result: on all seven models, TTFT is 30.3–57.8% lower (8k-token prompts) and 23.0–52.8% lower (32k), and prefill throughput 43.4–143.4% higher.

Prefill speed, all models (each server newly started; main → this PR, improvement; +% = this PR better)

Model (expert weights), GPUs, MoE runner TTFT, 8k-token prompts (ms) TTFT, 32k-token prompts (ms) Prefill throughput, 8k-token prompts (tok/s)
DeepSeek-V4.1-Flash (MXFP4 experts), 8×H200, flashinfer_mxfp4 (default) 573.9 → 400.0 (+30.3%) 2,685.0 → 1,994.3 (+25.7%) 14,755 → 21,154 (+43.4%)
Nemotron-3.5-Lightning-30B-A3B (BF16), 1×H200, flashinfer_cutlass (default) 309.1 → 136.6 (+55.8%) 1,227.5 → 599.9 (+51.1%) 27,320 → 63,131 (+131.1%)
Ling-3.0-flash (MXFP4 experts), 1×H200, flashinfer_mxfp4 (default) 430.4 → 254.7 (+40.8%) 1,761.0 → 1,073.3 (+39.1%) 18,961 → 31,557 (+66.4%)
gpt-oss-20b (MXFP4 experts), 1×H200, flashinfer_mxfp4 ¹ 422.0 → 177.9 (+57.8%) 1,790.9 → 846.2 (+52.8%) 20,301 → 49,408 (+143.4%)
GLM-5.3-Flash (BF16), 8×H200, flashinfer_cutlass 495.1 → 250.3 (+49.5%) 1,896.4 → 1,019.3 (+46.3%) 17,160 → 34,122 (+98.8%)
MiniMax-M3 (BF16), 8×H200, flashinfer_cutlass ² 631.2 → 294.4 (+53.4%) 2,547.0 → 1,224.8 (+51.9%) 13,657 → 30,302 (+121.9%)
Gemma-4-26B-A4B-it (BF16), 1×H200, flashinfer_cutlass 408.3 → 244.3 (+40.2%) 2,798.8 → 2,156.1 (+23.0%) 19,978 → 34,302 (+71.7%)

Runners not marked "(default)" are set with --moe-runner-backend. ¹ With gpt-oss's default request limit of 4096, main already tunes buckets up to 4096; this PR adds bucket 8192. ² Both versions also run --disable-prefill-cuda-graph: capturing MiniMax-M3's prefill CUDA graph fails at startup on main too.

DeepSeek-V4.1-Flash on a server that has already served traffic (+% = this PR better)

On main, the tactic a prefill chunk gets depends on what the server has served before (Motivation, item 2). On a newly started server (the table above), an 8k chunk gets FlashInfer's untuned fallback tactic. After one decode batch larger than the CUDA graph limit (here 65–128 requests, above --cuda-graph-max-bs-decode 64), later 8k chunks get the tactic tuned for 128 tokens instead: faster than the fallback, but not tuned for 8k. This PR uses the tactic tuned for its 8192-token chunks in both cases. A long-running main server stays in whichever state its last new batch size left it: a busy server whose decode batches often exceed the CUDA graph limit tends to end in the second state, a lightly loaded one in the first. This test measures the second case. Both servers first get the same requests (listed under Commands), and every main server's log shows such a batch.

Result: this PR is faster in both cases. main's 8k-token TTFT is 573.9 ms on a new server and 496.5 ms after this traffic; this PR's is 400.0 ms and 398.2 ms:

Benchmark main this PR improvement
TTFT, 8k-token prompts (ms) 496.5 398.2 +19.8%
TTFT, 32k-token prompts (ms) 2,383.5 1,991.9 +16.4%
Prefill throughput, 8k-token prompts (tok/s) 17,152 21,152 +23.3%

This table measures the two costs of tuning more buckets, a longer startup autotune and a larger buffer for timing tactics, and checks the KV cache size. Result: the KV cache size is unchanged, and each GPU gives up 0.4–1.2 GiB (0.3–0.9% of an H200); with no saved results, the startup autotune takes 4–26 s more on five models, about 135 s more on DeepSeek-V4.1-Flash (8×H200, paid at every start) and 488 s more on Ling-3.0-flash (1×H200, paid once).

Cost of this PR: startup time and GPU memory (main → this PR, from the server logs of the servers above)

Model Tuned buckets per MoE GEMM FlashInfer autotune at startup, no saved results (s) ³ Free memory per GPU after startup, lowest GPU (GiB; in parentheses, the extra memory this PR uses as a share of the H200's 140.4 GiB, so here + is a cost) KV cache size (tokens), unchanged
DeepSeek-V4.1-Flash, 8×H200 9 → 21 41–42 → 176–177 20.72 → 19.64 (+0.77%) 21,542,656
Nemotron-3.5-Lightning, 1×H200 9 → 21 6–7 → 21–22 12.46 → 11.88 (+0.41%) 6,137,490
Ling-3.0-flash, 1×H200 8 → 21 116 → 604 11.80 → 11.37 (+0.31%) 4,239,637
gpt-oss-20b, 1×H200 20 → 21 132 → 157–158 5.78 → 5.28 (+0.36%) 2,641,507
GLM-5.3-Flash, 8×H200 9 → 21 7–8 → 17 16.28–16.48 → 15.27 (+0.72–0.86%) 2,118,400
MiniMax-M3, 8×H200 16 → 21 16–17 → 21 9.88 → 9.15 (+0.52%) 407,296
Gemma-4-26B-A4B-it, 1×H200 16 → 21 11 → 18–19 20.88 → 20.38 (+0.36%) 382,802

³ This column leaves out each model's first start after download, whose autotune window also JIT-compiled other kernels (main: Ling-3.0 170 s, GLM-5.3-Flash 61 s, Gemma-4 33 s); that start's other values are included.

The extra memory is mostly the buffer FlashInfer uses at startup to time each candidate tactic. It is sized for the largest token count being tuned: the prefill chunk size with this PR (capped at 8192, so 8192 tokens in these measurements) instead of the decode batch size on main, and FlashInfer keeps it while the server runs. The KV cache is sized before the startup autotune runs, so its size is unchanged and the buffer comes out of the memory left outside it.

Together with open PR #42736 (DeepSeek-V4.1-Flash, same setup; baseline: main with #42736 applied). This shows what this PR adds when both land: #42736 adds a startup pass that tunes the same 21 prefill-sized buckets, and this PR makes the prefill calls use them, so the two work well together, and the improvements are what this PR adds on top of #42736's own gain (#42736's own gain over main, from a separate set of server starts: 8k-token TTFT 574.8 → 522.7 ms, 9.1% lower): with both, 8k-token TTFT is 24.7% lower than with #42736 alone (527.1 → 396.8 ms), and 19.8% lower on a server that has already served the traffic above (497.0 → 398.7 ms). For this kernel's prefill, both together match this PR alone (396.8 vs 400.0 ms; 398.7 vs 398.2 ms after the traffic), as this PR already tunes the same buckets; #42736 also tunes other FlashInfer operations. This combination was measured on DeepSeek-V4.1-Flash; since both PRs act on the same buckets in the same way, the other models above should also gain on top of #42736 (expected from the mechanism; not measured here).

Together with #42736: table and mechanism (baseline: main + #42736; +% = with both better)

#42736 at its head ef17b191caf0 (2026-10-06), on the same main. Startup is the same for #42736 alone, this PR alone and both (autotune 176–179 s, 21 buckets).

Benchmark #42736 alone #42736 + this PR improvement
TTFT, 8k-token prompts (ms) 527.1 396.8 +24.7%
TTFT, 32k-token prompts (ms) 2,511.8 1,996.8 +20.5%
Prefill throughput, 8k-token prompts (tok/s) 16,006 21,152 +32.2%
After the traffic above: TTFT, 8k-token prompts (ms) 497.0 398.7 +19.8%
Same: TTFT, 32k-token prompts (ms) 2,376.5 2,003.6 +15.7%
Same: prefill throughput, 8k-token prompts (tok/s) 17,157 21,190 +23.5%

With #42736 alone, CUDA graph capture runs after its extra pass and captures from the largest batch down to 1 (decode_cuda_graph_runner.py), so FlashInfer's cached bucket list ends as the 1-token list (item 2) and the prefill calls look up small buckets even though the large ones are tuned; this PR's fixed range makes them find the tuned large bucket. A profile agrees (TP rank 0, three 8k-token prefills): every MoE GEMM runs a 64×16 or 128×16 tile with #42736 alone and a 128×64 tile with both; per prefill both MoE GEMMs together take 123 → 58 ms, the all-reduce that waits for them 119 → 75 ms, and the GPU time 480 → 370 ms (22.9% less). FlashInfer does not log the bucket, so the list state is inferred from the code and the tile shapes. #42736 also tunes other FlashInfer operations; this PR does not overlap with that part.

Decode (+% = this PR better)

This PR targets prefill, but it also changes the tuning range that decode calls pass, so this checks that decode does not get slower. Result: decode ITL is unchanged on all three models, and TTFT and throughput improve, mostly at 8 and 32 concurrent requests, where several 2k-token prompts are prefilled together in batches of up to 8192 tokens, sizes main leaves untuned.

DeepSeek-V4.1-Flash, newly started servers running 1, then 8, then 32 concurrent requests; 2k-token prompts, 256 output tokens.

Concurrent requests median ITL (ms) throughput (tok/s) TTFT (ms)
1 7.48 → 7.24 (+3.2%) 1,085 → 1,121 (+3.3%) 168.0 → 162.0 (+3.6%)
8 9.68 → 9.67 (+0.1%) 5,186 → 5,615 (+8.3%) 978.9 → 725.4 (+25.9%)
32 13.96 → 13.95 (+0.1%) 9,779 → 11,558 (+18.2%) 2,520.3 → 1,785.4 (+29.2%)

GLM-5.3-Flash and MiniMax-M3 (8×H200, flashinfer_cutlass), the same requests:

Model Concurrent requests median ITL (ms) throughput (tok/s) TTFT (ms)
GLM-5.3-Flash 1 6.62 → 6.63 (−0.2%) 1,189 → 1,217 (+2.4%) 198.4 → 153.2 (+22.8%)
GLM-5.3-Flash 8 10.24 → 10.25 (0.0%) 5,068 → 5,707 (+12.6%) 781.0 → 480.7 (+38.5%)
GLM-5.3-Flash 32 15.83 → 15.82 (0.0%) 10,035 → 12,870 (+28.3%) 1,909.6 → 1,007.6 (+47.2%)
MiniMax-M3 1 8.61 → 8.61 (0.0%) 964.5 → 964.8 (0.0%) 138.4 → 137.3 (+0.9%)
MiniMax-M3 8 15.16 → 15.17 (−0.1%) 3,616 → 4,069 (+12.5%) 878.2 → 501.7 (+42.9%)
MiniMax-M3 32 23.77 → 23.76 (0.0%) 6,922 → 8,958 (+29.4%) 2,589.1 → 1,212.2 (+53.2%)

On all three models, decode batches use buckets 1–256, which main already tunes, so ITL is unchanged (DeepSeek-V4.1-Flash's 1-request difference is smaller than main's own spread between restarts, 3.4%); the throughput and TTFT gains at 8 and 32 requests come from these batched prefills (MiniMax-M3's main already tunes up to 2048 tokens, so a single 2k-token prompt gains only 0.9% there).

One MoE call

The tables above time whole requests; this test times one MoE call alone to answer two questions. (1) Does the gain come from the tactic, that is, from replacing FlashInfer's fallback tactic with a tuned one? (2) This PR tunes up to 8192 tokens, so a chunk larger than 8192 tokens reuses bucket 8192's tactic: is that slower than a tactic tuned for the chunk's own size? Result: (1) yes: at 8192 tokens the tuned tactic is 2.6× faster than the fallback; (2) no: above 8192 tokens, bucket 8192's tactic is never slower than each size's own tactic. It even measured 1.5–3.4% faster, but that is within the measurement spread: at 8192 tokens, where the last two columns run the very same tactic, the last column also measured 2.3% slower (1.549 vs 1.514 ms). cutlass_fused_moe on one H200, DeepSeek-V4.1-Flash's per-GPU shape under EP8 (48 of 384 experts, hidden size 5120, intermediate size 2304, top-6, uniform random routing; median time per call), tuned once up to 8192 tokens (this PR's range with 8192-token chunks) and once up to 32768 tokens. Time per call (ms):

Tokens fallback tactic bucket 8192's tactic (this PR) tactic tuned for the call's own bucket
8192 3.990 1.514 1.549 (the same tactic)
12288 5.900 2.138 2.194
16384 7.744 2.733 2.825
24576 11.482 3.976 4.035
32768 14.998 5.137 5.222

Tuning this layer's two GEMMs up to 8192 tokens takes 185 s, about as long as the server's whole startup autotune (176–177 s), because FlashInfer tunes each input shape once and every MoE layer has this shape; up to 32768 tokens it takes 232 s. The three tactics give bit-identical output at every size in the table.

Commands
python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4.1-Flash \
  --trust-remote-code --tp-size 8 --ep-size 8 --mem-fraction-static 0.8 --chunked-prefill-size 8192 \
  --attention-backend dsv4 --moe-runner-backend flashinfer_mxfp4 \
  --max-running-requests 256 --cuda-graph-max-bs-decode 64 --watchdog-timeout 7200
python3 -m sglang.launch_server --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --trust-remote-code \
  --moe-runner-backend auto --watchdog-timeout 7200   # request limit 138 (set by the Mamba state cache)
python3 -m sglang.launch_server --model-path inclusionAI/Ling-3.0-flash-fp4 --trust-remote-code --moe-runner-backend auto --watchdog-timeout 7200
python3 -m sglang.launch_server --model-path openai/gpt-oss-20b --moe-runner-backend flashinfer_mxfp4 --watchdog-timeout 7200
python3 -m sglang.launch_server --model-path zai-org/GLM-5.3-Flash-BF16 --trust-remote-code --tp-size 8 --ep-size 8 \
  --moe-runner-backend flashinfer_cutlass --watchdog-timeout 7200
python3 -m sglang.launch_server --model-path MiniMaxAI/MiniMax-M3 --trust-remote-code --tp-size 8 --ep-size 8 \
  --moe-runner-backend flashinfer_cutlass --disable-prefill-cuda-graph --watchdog-timeout 7200
python3 -m sglang.launch_server --model-path google/gemma-4-26B-A4B-it --trust-remote-code \
  --moe-runner-backend flashinfer_cutlass --watchdog-timeout 7200
# --watchdog-timeout is not needed: the scheduler's watchdog starts after the startup autotune
B="python3 -m sglang.bench_serving --backend sglang --dataset-name random --random-range-ratio 1.0 --seed 5 --flush-cache"
$B --random-input-len 8000  --random-output-len 1 --num-prompts 8  --max-concurrency 1
$B --random-input-len 32000 --random-output-len 1 --num-prompts 8  --max-concurrency 1
$B --random-input-len 8000  --random-output-len 1 --num-prompts 64 --max-concurrency 8
for nc in "8 1" "64 8" "128 32"; do set -- $nc   # decode rows
  $B --random-input-len 2000 --random-output-len 256 --num-prompts $1 --max-concurrency $2; done
# requests before the "already served traffic" rows (--seed 7): --random-input-len 400/800/1600/3000/6000 --random-output-len 1
#   --num-prompts 2 --max-concurrency 1, then --random-input-len 2000 --random-output-len 64 --num-prompts 384 --max-concurrency 96

Accuracy

By design this PR does not affect accuracy: it only changes which tile configuration (tactic) of the same MoE GEMM runs, so the GEMM computes the same values, at most with a different summation order (with DeepSeek-V4.1-Flash's MoE shape, one call at 8192–32768 tokens is bit-identical to the fallback tactic). Measured: GSM8K stays within sampling noise on all seven models (−0.83 to +0.15 points) and in the DeepSeek-V4-Flash CI test (+1.5).

Test main this PR difference, percentage points (95% noise range)
GSM8K, all 1,319 test questions, 20-shot, temperature 0: DeepSeek-V4.1-Flash 91.43% 90.60% −0.83 (±2.2)
Same: Nemotron-3.5-Lightning 87.64% 87.79% +0.15 (±2.5)
Same: Ling-3.0-flash 86.28% 86.05% −0.23 (±2.6)
Same: gpt-oss-20b 80.14% 79.38% −0.76 (±3.1)
Same, through the chat API (up to 4096 output tokens): GLM-5.3-Flash 97.35% 96.82% −0.53 (±1.3)
Same, chat API: MiniMax-M3 97.19% 97.27% +0.08 (±1.3)
Same, chat API: Gemma-4-26B-A4B-it 96.36% 96.06% −0.30 (±1.5)
CI TestDSV4FlashFP4H200FlashInferCutlass.test_gsm8k (DeepSeek-V4-Flash, TP4 + EAGLE, 200 questions; threshold 93%) 96.0% 97.5% +1.5 (±3.5)

The noise range is 1.96 standard errors of the difference between two scores over that many questions (an independent-sample approximation, conservative here because both versions answer the same questions). GPQA and long-context evals were not run.

GSM8K command

The 20 shots are the first 20 training examples, so no test question appears in the prompt.

D=https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data
{ curl -sL $D/train.jsonl | head -n 20; curl -sL $D/test.jsonl; } > gsm8k_k20.jsonl
python3 -m sglang.test.run_eval --eval-name gsm8k --num-shots 20 --num-examples 1319 \
  --num-threads 32 --api completion --gsm8k-data-path gsm8k_k20.jsonl
# GLM-5.3-Flash, MiniMax-M3, Gemma-4: --api chat --max-tokens 4096 instead of --api completion

CI

No file fails only with this PR, and the new CPU test passes all 80 sub-cases (60 fail on main). Local CI on 8×H200 with the commands CI uses: every CPU and GPU suite this machine can run (1,427 files) with this PR, and on its parent main every CPU suite in full plus every file that failed with this PR. SM100, SM110 and SM120 cannot run here; the CI tests that reach the changed call there (test_mxfp4_sm120_cutlass.py on the RTX 5090 runner, test_nvfp4_moe_backends.py::test_flashinfer_cutlass on B200) are in upstream CI; this command runs them with the other changed test files and the DeepSeek-V4-Flash H200 test: /rerun-test -c registered/unit/layers/quantization/test_nvfp4_moe_backends.py registered/e2e/models/test_deepseek_v4_flash_fp4_h200.py::TestDSV4FlashFP4H200FlashInferCutlass.

Results per test and per suite, and why the other files fail
Test main this PR
New CPU test TestCutlassMoeTuneMaxNumTokens 60 of 80 sub-cases fail all 80 pass
test_mxfp4_sm90_cutlass.py / test_mxfp4_sm120_cutlass.py 19 passed / 2 passed, 2 skipped (those 2 need an SM120 GPU) the same
TestDSV4FlashFP4H200FlashInferCutlass (TP4 + EAGLE) 8 of 9 pass (test_bs_1_speed: accept length 2.790, threshold 2.8; borderline and unrelated to MoE tactics) 9 of 9 pass

Other files failing with this PR (failing / run), every one also failing on main: base-a-test-cpu 4/793, base-a-test-1-gpu-small 2/5, base-b-test-1-gpu-small 13/127, base-b-test-1-gpu-large 10/129, base-b-kernel-unit-test-1-gpu-large 1/209, base-b-test-2-gpu-large 12/38, base-c-test-4-gpu-h100 3/16, base-c-test-8-gpu-h200 2/6, and the Intel/ARM CPU suites 44/104. Why:

  • This machine lacks what they need: network access for uncached models and datasets (including the CI model lmsys/sglang-ci-dsv3-test, which needs CI credentials), RDMA for PD/Mooncake KV transfer, SM120, or the CPU kernel build and packages of the Intel/ARM suites (16 of those 44 also hit our 300 s per-file limit).
  • They fail on main at this PR's base for their own reasons: kernels/ops/attention/test_dsa_metadata_replay.py (fixed on main by [DSA] Fix pooled page table refresh for target verify CUDA graph replay #42750 after this PR's base); pp/test_pp_gemma4.py (accuracy 0.498, threshold 0.65); core/test_basic_sanity_eagle3.py (fwd_occupancy 97.7, threshold 98); core/test_basic_sanity_dspark.py; prefill_only/test_embedding_models.py; prefill_only/test_pooled_hidden_states.py; unit/mem_cache/test_unified_radix_cache_unittest.py; unit/mem_cache/test_rust_unified_radix_cache_unittest.py; and in base-a-test-cpu, test_disagg_idle_step_counters.py, test_page_interleave_shard.py, test_vit_npu_graph_runner.py, test_numa_utils.py.

Related work

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #39387222701
Latest PR Test (Extra): ❌ Run #39387222528
Latest PR Test (AMD ROCm 10): ❌ Run #39387222971

Both cutlass_fused_moe call sites pass tune_max_num_tokens =
next_power_of_2(M), where M is the call's token count. The startup
FlashInfer autotune runs one decode-sized dummy batch, so it tunes only
the token buckets up to that batch. A prefill chunk larger than that
batch then maps to a bucket that was never tuned and runs FlashInfer's
untuned fallback tactic.

Pass one value per server at both call sites instead: the largest
prefill chunk (max_prefill_buffer_tokens(), else max_prefill_tokens)
times the attention data-parallel size, since the MoE layer runs on the
tokens gathered from all DP ranks. The value is capped at FlashInfer's
own default tune_max_num_tokens for cutlass_fused_moe, which bounds the
extra startup autotune time and the tuning workspace FlashInfer
keeps after it. The same startup autotune then also tunes the
prefill-sized buckets, and larger calls map to its top bucket, which is
tuned too. Kernels, flags and the KV cache are unchanged.

PD decode servers (--disaggregation-mode decode) run no prefill, so
they keep the per-call value, as before, and pay no extra startup time
or memory.

FlashInfer also caches MoERunner.refine_tuning_config per value with
lru_cache (flashinfer-ai/flashinfer#5625), so with per-call values the
bucket a call uses depends on earlier calls. One value per server
removes that dependence too. flashinfer-ai/flashinfer#5631 fixes only
the dependence and keeps the per-call range, so this change is still
needed after it.

A new CPU test checks, with the kernel stubbed, that prefill-sized
token counts map to buckets the startup autotune tunes, and that PD
decode servers keep the per-call value. The SM90/SM120 CUTLASS MXFP4
GPU tests and the CPU test of the MoE runner placement now run the
fused-MoE call inside override_server_args(), because the new helper
reads the server configuration.
detain added a commit to detain/sglang that referenced this pull request Oct 20, 2026
…/deferred/avoid map

- 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619,
  SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420,
  mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821.
- 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202,
  PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565,
  draft embed/lm_head VRAM sgl-project#37155.
- sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39).
- stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
@Fridge003

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py test/registered/unit/layers/moe/test_moe_runner_config_placement.py test/registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit 6a39ddf1f2e0.

Re-run was not dispatched. What to do:

  • Rebase your branch onto the latest main and push again
  • Follow issue #21065 for context
  • CI-fix PRs may request the bypass-maintenance label to skip this check

@Fridge003

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py test/registered/unit/layers/moe/test_moe_runner_config_placement.py test/registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py

⛔ test/registered/unit/layers/moe/test_moe_runner_config_placement.py: test/registered/unit/layers/moe/test_moe_runner_config_placement.py is registered for Intel Xeon (suite stage-b-test-cpu-intel), not the ubuntu-latest CPU pool; rerun-test.yml has no job for it. Rerun it with /rerun-failed-ci, or dispatch the pr-test-xeon.yml workflow manually.

@github-actions

This comment has been minimized.

@Fridge003 Fridge003 changed the title [MoE] Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks [Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks Oct 20, 2026
@Fridge003
Fridge003 merged commit f8fd41c into sgl-project:main Oct 20, 2026
107 of 124 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants