Repository navigation
Conversation
Both cutlass_fused_moe call sites pass tune_max_num_tokens = next_power_of_2(M), where M is the call's token count. The startup FlashInfer autotune runs one decode-sized dummy batch, so it tunes only the token buckets up to that batch. A prefill chunk larger than that batch then maps to a bucket that was never tuned and runs FlashInfer's untuned fallback tactic. Pass one value per server at both call sites instead: the largest prefill chunk (max_prefill_buffer_tokens(), else max_prefill_tokens) times the attention data-parallel size, since the MoE layer runs on the tokens gathered from all DP ranks. The value is capped at FlashInfer's own default tune_max_num_tokens for cutlass_fused_moe, which bounds the extra startup autotune time and the tuning workspace FlashInfer keeps after it. The same startup autotune then also tunes the prefill-sized buckets, and larger calls map to its top bucket, which is tuned too. Kernels, flags and the KV cache are unchanged. PD decode servers (--disaggregation-mode decode) run no prefill, so they keep the per-call value, as before, and pay no extra startup time or memory. FlashInfer also caches MoERunner.refine_tuning_config per value with lru_cache (flashinfer-ai/flashinfer#5625), so with per-call values the bucket a call uses depends on earlier calls. One value per server removes that dependence too. flashinfer-ai/flashinfer#5631 fixes only the dependence and keeps the per-call range, so this change is still needed after it. A new CPU test checks, with the kernel stubbed, that prefill-sized token counts map to buckets the startup autotune tunes, and that PD decode servers keep the per-call value. The SM90/SM120 CUTLASS MXFP4 GPU tests and the CPU test of the MoE runner placement now run the fused-MoE call inside override_server_args(), because the new helper reads the server configuration.
…/deferred/avoid map - 3A [24-35]: NVFP4 MoE boot sgl-project#39807/sgl-project#40173, block-FP8 TP refine sgl-project#42566, k-pool indexer split sgl-project#43619, SM120 skinny GEMM sgl-project#42865/sgl-project#42867, DeepGEMM warmup guards sgl-project#32686/sgl-project#43712, autotune sgl-project#43363/sgl-project#40420, mHC overlap sgl-project#42985, FP8-einsum precompile sgl-project#42821. - 3B [36-43]: PCIe CustomAllreduce VMM sgl-project#44448, EP local expert ids sgl-project#44480, shm TP broadcast sgl-project#44202, PCIe-IPC all-gather sgl-project#42863 + multicast guard sgl-project#41888, TP-sampling consistency sgl-project#41916/sgl-project#38565, draft embed/lm_head VRAM sgl-project#37155. - sgl-project#39807 toolchain commit verified safe against our CUDA_HOME=12.9 pin (cu13 fallback needs CUDA_HOME unset AND glibc>=2.41; we are 2.39). - stack_want.txt: execution-status banner. Branch tip 5dd6ec7.
|
/rerun-test test/registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py test/registered/unit/layers/moe/test_moe_runner_config_placement.py test/registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py |
|
|
/rerun-test test/registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py test/registered/unit/layers/moe/test_moe_runner_config_placement.py test/registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py test/registered/unit/layers/quantization/test_mxfp4_sm120_cutlass.py |
|
🚀 🚀 🚀 ⛔ |
Summary
TL;DR: This PR enables sglang's startup autotune to tune FlashInfer's CUTLASS MoE kernel for prefill sizes too, not only for decode sizes. sglang already autotunes this kernel (
cutlass_fused_moe, FlashInfer's MoE kernel, not its attention backend) at startup, but each call tells FlashInfer to tune only up to its own token count. The startup run is one decode-sized batch, so only decode sizes get a tuned kernel configuration (tactic), and prefill chunks above that size run FlashInfer's untuned fallback tactic (its default kernel configuration). This PR gives FlashInfer one tuning range per server instead: the largest prefill batch the MoE layer sees (chunk size × attention data-parallel size, capped at 8192, FlashInfer's own default range for this call), so the same startup autotune also covers prefill. No new flag, kernel or extra pass. It applies on every GPU this kernel runs on (SM90, SM100, SM110, SM120), where it is the default MoE kernel of DeepSeek-V4.1/V4, Ling-3.0, Nemotron-H, gpt-oss and ModelOpt NVFP4 MoE models (which GPU for which model: Design notes), and to any other MoE model that runs on it. Other MoE kernels and the decode servers of prefill-decode (PD) disaggregation are unchanged. Measured on H200 (SM90).--disable-flashinfer-autotuneturns the startup autotune off. Each GPU has 0.4–1.2 GiB (0.3–0.9% of an H200) less free memory while the server runs, mostly the buffer FlashInfer keeps for timing tactics at startup, now sized for the prefill chunk instead of the decode batch; the KV cache size is unchanged. PD decode servers pay nothing.Motivation
sglang runs FlashInfer's autotune at startup (#12306, on by default since #14357). For
cutlass_fused_moe, FlashInfer keeps one tactic per token-count bucket (powers of two up to 256, then larger steps up to 8192: 21 buckets) and tunes every bucket up to the caller'stune_max_num_tokens; the same value also decides which bucket a call uses. Both sglang call sites (theflashinfer_cutlassrunner, used for BF16, FP8 and NVFP4 experts, and the CUTLASS path offlashinfer_mxfp4, used for MXFP4 experts) passnext_power_of_2(x.shape[0]), which dates from #7327, before the startup autotune existed. This has two effects:cutlass_fused_moecall that runs Python code (the startup autotune, CUDA graph capture, and every serving forward that is not a CUDA graph replay) first callsMoERunner.refine_tuning_config(tune_max_num_tokens). That method is wrapped infunctools.lru_cacheand rebuilds FlashInfer's class-wide bucket list only the first time it sees a value, so a call with a value seen before keeps the list built for the latest new value. On DeepSeek-V4.1-Flash with--cuda-graph-max-bs-decode 64, the first decode batch of 65–128 requests while serving runs without a CUDA graph and rebuilds the list for 128 tokens (a short prompt of a new size likewise rebuilds it, for its own size); later prefill chunks of an already-seen size, such as every 8k-token chunk after the first, then run the 128-token bucket's tactic, because a list built for 128 tokens ends at bucket 128 and every larger call falls into that last bucket.flashinfer-ai/flashinfer#5625 was filed by an sglang user who hit both effects serving DeepSeek-V4-Flash on 4×H100 with
flashinfer_mxfp4. One value for the whole server, computed from its configuration and used by every call, as large as the largest prefill batch (up to 8192 tokens), removes both.Modifications
python/sglang/srt/layers/moe/moe_runner/flashinfer_cutlass.py-- the new_cutlass_moe_tune_max_num_tokens(num_tokens)gives bothcutlass_fused_moecalls theirtune_max_num_tokens:min(prefill chunk size × attention DP size, 8192), with the chunk size (--chunked-prefill-size, or--max-prefill-tokenswhen chunked prefill is off) read the wayflashinfer_autotune.pyalready does (max_prefill_buffer_tokens() or get_schedule().max_prefill_tokens). PD decode servers keepnext_power_of_2(num_tokens).test/registered/unit/layers/moe/test_flashinfer_cutlass_swiglu_params.py(CPU suite) -- newTestCutlassMoeTuneMaxNumTokens. With the kernel stubbed, it checks at both calls that token counts 1, 257, 777, 2048, 4097, 8192 and 16384 map to a bucket the startup autotune tunes, in five server configurations, and that PD decode servers keep the per-call value. 60 of its 80 sub-cases fail on main.test/registered/unit/layers/quantization/test_mxfp4_sm{90,120}_cutlass.py,test/registered/unit/layers/moe/test_moe_runner_config_placement.py-- their fused-MoE calls now run insideget_context().override_server_args(), because the helper reads the server configuration.Design notes:
flashinfer_mxfp4) and BF16/FP8 Nemotron-H (flashinfer_cutlass); on SM120 also gpt-oss and ModelOpt NVFP4 MoE models, the latter on SM110 too since fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends #41765. On SM100 no model uses it by default (flashinfer_mxfp4runs a TRT-LLM kernel there). Any other MoE model reaches it with--moe-runner-backend flashinfer_cutlass(BF16, ModelOpt FP8/NVFP4) orflashinfer_mxfp4(MXFP4).--chunked-prefill-sizeacross the DP ranks, and the MoE layer runs on the tokens gathered from all of them, so the largest batch the MoE layer sees is each rank's chunk × the DP size, which is the--chunked-prefill-sizethe user set.Experimental Results
Performance
Setup: in every prefill row below, every server with this PR beats every server without it (main, or #42736 alone). Every number in this section was measured with this PR's code at head
a4c96d4080fon mainc7f3ae1dd59(H200, CUDA 13.0, torch 2.14.1, FlashInfer 0.7.0.post1). Every measurement starts a new server for each version, alternating which goes first; each value is the median over server starts (for TTFT, of each server's median over the 8 prompts). TTFT rows send 8 random prompts one at a time (2k, 8k and 32k mean 2,000, 8,000 and 32,000 tokens; percentages are computed from unrounded medians). Prompts are prefilled in chunks of 8192 tokens (--chunked-prefill-size 8192, set on DeepSeek-V4.1-Flash and the default on the other models here), so a 32k prompt runs as four chunks and every MoE call has at most 8192 tokens. With this chunk size this PR's tuning range is also 8192 (it follows the chunk size; the 8192 cap is FlashInfer's default range for this call, see Design notes); throughput rows send 64 prompts at 8 concurrent requests with 1 output token, so they measure prefill throughput (bench_servingtotal token throughput). Launch commands are under Commands below; settings not listed there are the defaults.This table measures the main claim: on a newly started main server, prefill chunks larger than the startup decode batch run FlashInfer's untuned fallback tactic (Motivation, item 1), and with this PR a tuned one. Result: on all seven models, TTFT is 30.3–57.8% lower (8k-token prompts) and 23.0–52.8% lower (32k), and prefill throughput 43.4–143.4% higher.
Prefill speed, all models (each server newly started; main → this PR, improvement; +% = this PR better)
flashinfer_mxfp4(default)flashinfer_cutlass(default)flashinfer_mxfp4(default)flashinfer_mxfp4¹flashinfer_cutlassflashinfer_cutlass²flashinfer_cutlassRunners not marked "(default)" are set with
--moe-runner-backend. ¹ With gpt-oss's default request limit of 4096, main already tunes buckets up to 4096; this PR adds bucket 8192. ² Both versions also run--disable-prefill-cuda-graph: capturing MiniMax-M3's prefill CUDA graph fails at startup on main too.DeepSeek-V4.1-Flash on a server that has already served traffic (+% = this PR better)
On main, the tactic a prefill chunk gets depends on what the server has served before (Motivation, item 2). On a newly started server (the table above), an 8k chunk gets FlashInfer's untuned fallback tactic. After one decode batch larger than the CUDA graph limit (here 65–128 requests, above
--cuda-graph-max-bs-decode 64), later 8k chunks get the tactic tuned for 128 tokens instead: faster than the fallback, but not tuned for 8k. This PR uses the tactic tuned for its 8192-token chunks in both cases. A long-running main server stays in whichever state its last new batch size left it: a busy server whose decode batches often exceed the CUDA graph limit tends to end in the second state, a lightly loaded one in the first. This test measures the second case. Both servers first get the same requests (listed under Commands), and every main server's log shows such a batch.Result: this PR is faster in both cases. main's 8k-token TTFT is 573.9 ms on a new server and 496.5 ms after this traffic; this PR's is 400.0 ms and 398.2 ms:
This table measures the two costs of tuning more buckets, a longer startup autotune and a larger buffer for timing tactics, and checks the KV cache size. Result: the KV cache size is unchanged, and each GPU gives up 0.4–1.2 GiB (0.3–0.9% of an H200); with no saved results, the startup autotune takes 4–26 s more on five models, about 135 s more on DeepSeek-V4.1-Flash (8×H200, paid at every start) and 488 s more on Ling-3.0-flash (1×H200, paid once).
Cost of this PR: startup time and GPU memory (main → this PR, from the server logs of the servers above)
³ This column leaves out each model's first start after download, whose autotune window also JIT-compiled other kernels (main: Ling-3.0 170 s, GLM-5.3-Flash 61 s, Gemma-4 33 s); that start's other values are included.
The extra memory is mostly the buffer FlashInfer uses at startup to time each candidate tactic. It is sized for the largest token count being tuned: the prefill chunk size with this PR (capped at 8192, so 8192 tokens in these measurements) instead of the decode batch size on main, and FlashInfer keeps it while the server runs. The KV cache is sized before the startup autotune runs, so its size is unchanged and the buffer comes out of the memory left outside it.
Together with open PR #42736 (DeepSeek-V4.1-Flash, same setup; baseline: main with #42736 applied). This shows what this PR adds when both land: #42736 adds a startup pass that tunes the same 21 prefill-sized buckets, and this PR makes the prefill calls use them, so the two work well together, and the improvements are what this PR adds on top of #42736's own gain (#42736's own gain over main, from a separate set of server starts: 8k-token TTFT 574.8 → 522.7 ms, 9.1% lower): with both, 8k-token TTFT is 24.7% lower than with #42736 alone (527.1 → 396.8 ms), and 19.8% lower on a server that has already served the traffic above (497.0 → 398.7 ms). For this kernel's prefill, both together match this PR alone (396.8 vs 400.0 ms; 398.7 vs 398.2 ms after the traffic), as this PR already tunes the same buckets; #42736 also tunes other FlashInfer operations. This combination was measured on DeepSeek-V4.1-Flash; since both PRs act on the same buckets in the same way, the other models above should also gain on top of #42736 (expected from the mechanism; not measured here).
Together with #42736: table and mechanism (baseline: main + #42736; +% = with both better)
#42736 at its head
ef17b191caf0(2026-10-06), on the same main. Startup is the same for #42736 alone, this PR alone and both (autotune 176–179 s, 21 buckets).With #42736 alone, CUDA graph capture runs after its extra pass and captures from the largest batch down to 1 (
decode_cuda_graph_runner.py), so FlashInfer's cached bucket list ends as the 1-token list (item 2) and the prefill calls look up small buckets even though the large ones are tuned; this PR's fixed range makes them find the tuned large bucket. A profile agrees (TP rank 0, three 8k-token prefills): every MoE GEMM runs a 64×16 or 128×16 tile with #42736 alone and a 128×64 tile with both; per prefill both MoE GEMMs together take 123 → 58 ms, the all-reduce that waits for them 119 → 75 ms, and the GPU time 480 → 370 ms (22.9% less). FlashInfer does not log the bucket, so the list state is inferred from the code and the tile shapes. #42736 also tunes other FlashInfer operations; this PR does not overlap with that part.Decode (+% = this PR better)
This PR targets prefill, but it also changes the tuning range that decode calls pass, so this checks that decode does not get slower. Result: decode ITL is unchanged on all three models, and TTFT and throughput improve, mostly at 8 and 32 concurrent requests, where several 2k-token prompts are prefilled together in batches of up to 8192 tokens, sizes main leaves untuned.
DeepSeek-V4.1-Flash, newly started servers running 1, then 8, then 32 concurrent requests; 2k-token prompts, 256 output tokens.
GLM-5.3-Flash and MiniMax-M3 (8×H200,
flashinfer_cutlass), the same requests:On all three models, decode batches use buckets 1–256, which main already tunes, so ITL is unchanged (DeepSeek-V4.1-Flash's 1-request difference is smaller than main's own spread between restarts, 3.4%); the throughput and TTFT gains at 8 and 32 requests come from these batched prefills (MiniMax-M3's main already tunes up to 2048 tokens, so a single 2k-token prompt gains only 0.9% there).
One MoE call
The tables above time whole requests; this test times one MoE call alone to answer two questions. (1) Does the gain come from the tactic, that is, from replacing FlashInfer's fallback tactic with a tuned one? (2) This PR tunes up to 8192 tokens, so a chunk larger than 8192 tokens reuses bucket 8192's tactic: is that slower than a tactic tuned for the chunk's own size? Result: (1) yes: at 8192 tokens the tuned tactic is 2.6× faster than the fallback; (2) no: above 8192 tokens, bucket 8192's tactic is never slower than each size's own tactic. It even measured 1.5–3.4% faster, but that is within the measurement spread: at 8192 tokens, where the last two columns run the very same tactic, the last column also measured 2.3% slower (1.549 vs 1.514 ms).
cutlass_fused_moeon one H200, DeepSeek-V4.1-Flash's per-GPU shape under EP8 (48 of 384 experts, hidden size 5120, intermediate size 2304, top-6, uniform random routing; median time per call), tuned once up to 8192 tokens (this PR's range with 8192-token chunks) and once up to 32768 tokens. Time per call (ms):Tuning this layer's two GEMMs up to 8192 tokens takes 185 s, about as long as the server's whole startup autotune (176–177 s), because FlashInfer tunes each input shape once and every MoE layer has this shape; up to 32768 tokens it takes 232 s. The three tactics give bit-identical output at every size in the table.
Commands
Accuracy
By design this PR does not affect accuracy: it only changes which tile configuration (tactic) of the same MoE GEMM runs, so the GEMM computes the same values, at most with a different summation order (with DeepSeek-V4.1-Flash's MoE shape, one call at 8192–32768 tokens is bit-identical to the fallback tactic). Measured: GSM8K stays within sampling noise on all seven models (−0.83 to +0.15 points) and in the DeepSeek-V4-Flash CI test (+1.5).
TestDSV4FlashFP4H200FlashInferCutlass.test_gsm8k(DeepSeek-V4-Flash, TP4 + EAGLE, 200 questions; threshold 93%)The noise range is 1.96 standard errors of the difference between two scores over that many questions (an independent-sample approximation, conservative here because both versions answer the same questions). GPQA and long-context evals were not run.
GSM8K command
The 20 shots are the first 20 training examples, so no test question appears in the prompt.
D=https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data { curl -sL $D/train.jsonl | head -n 20; curl -sL $D/test.jsonl; } > gsm8k_k20.jsonl python3 -m sglang.test.run_eval --eval-name gsm8k --num-shots 20 --num-examples 1319 \ --num-threads 32 --api completion --gsm8k-data-path gsm8k_k20.jsonl # GLM-5.3-Flash, MiniMax-M3, Gemma-4: --api chat --max-tokens 4096 instead of --api completionCI
No file fails only with this PR, and the new CPU test passes all 80 sub-cases (60 fail on main). Local CI on 8×H200 with the commands CI uses: every CPU and GPU suite this machine can run (1,427 files) with this PR, and on its parent main every CPU suite in full plus every file that failed with this PR. SM100, SM110 and SM120 cannot run here; the CI tests that reach the changed call there (
test_mxfp4_sm120_cutlass.pyon the RTX 5090 runner,test_nvfp4_moe_backends.py::test_flashinfer_cutlasson B200) are in upstream CI; this command runs them with the other changed test files and the DeepSeek-V4-Flash H200 test:/rerun-test -c registered/unit/layers/quantization/test_nvfp4_moe_backends.py registered/e2e/models/test_deepseek_v4_flash_fp4_h200.py::TestDSV4FlashFP4H200FlashInferCutlass.Results per test and per suite, and why the other files fail
TestCutlassMoeTuneMaxNumTokenstest_mxfp4_sm90_cutlass.py/test_mxfp4_sm120_cutlass.pyTestDSV4FlashFP4H200FlashInferCutlass(TP4 + EAGLE)test_bs_1_speed: accept length 2.790, threshold 2.8; borderline and unrelated to MoE tactics)Other files failing with this PR (failing / run), every one also failing on main:
base-a-test-cpu4/793,base-a-test-1-gpu-small2/5,base-b-test-1-gpu-small13/127,base-b-test-1-gpu-large10/129,base-b-kernel-unit-test-1-gpu-large1/209,base-b-test-2-gpu-large12/38,base-c-test-4-gpu-h1003/16,base-c-test-8-gpu-h2002/6, and the Intel/ARM CPU suites 44/104. Why:lmsys/sglang-ci-dsv3-test, which needs CI credentials), RDMA for PD/Mooncake KV transfer, SM120, or the CPU kernel build and packages of the Intel/ARM suites (16 of those 44 also hit our 300 s per-file limit).kernels/ops/attention/test_dsa_metadata_replay.py(fixed on main by [DSA] Fix pooled page table refresh for target verify CUDA graph replay #42750 after this PR's base);pp/test_pp_gemma4.py(accuracy 0.498, threshold 0.65);core/test_basic_sanity_eagle3.py(fwd_occupancy97.7, threshold 98);core/test_basic_sanity_dspark.py;prefill_only/test_embedding_models.py;prefill_only/test_pooled_hidden_states.py;unit/mem_cache/test_unified_radix_cache_unittest.py;unit/mem_cache/test_rust_unified_radix_cache_unittest.py; and inbase-a-test-cpu,test_disagg_idle_step_counters.py,test_page_interleave_shard.py,test_vit_npu_graph_runner.py,test_numa_utils.py.Related work
SGLANG_FLASHINFER_AUTOTUNE_EXTEND=1extra pass. It keeps the per-call value, so item 2 still applies to it.Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #39387222701
Latest PR Test (Extra): ❌ Run #39387222528
Latest PR Test (AMD ROCm 10): ❌ Run #39387222971