Repository navigation
[Klaud Cold] minimaxm31flash-fp4-h200-vllm-agentic-dspark: day-zero MiniMax-M3.1-Flash H200 DSpark recipe #4423
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8750a02
7fe8926
b795420
28349f5
d91b266
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| base: | ||
| schema: 2 | ||
| name: minimaxm31flash-fp4-h200-vllm-agentic-dspark | ||
| resources: | ||
| gpu_type: h200 | ||
| gpus_per_node: 8 | ||
| frontend: | ||
| type: vllm | ||
| enable_multiple_frontends: false | ||
| observability: | ||
| enabled: false | ||
| tachometer: | ||
| enabled: false | ||
| engine: | ||
| type: vllm | ||
| connector: null | ||
| # The engine may take the full VLLM_ENGINE_READY_TIMEOUT_S to become ready. | ||
| health_check: | ||
| interval_seconds: 10 | ||
| max_attempts: 720 | ||
| roles: | ||
| agg: | ||
| nodes: 1 | ||
| workers: 1 | ||
| # Serve flags follow the Hopper TP8 recipe at | ||
| # https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash. Hopper has no FP4 | ||
| # tensor cores, so the KV and indexer caches are FP8 and the NVFP4 experts | ||
| # run on Marlin. DSpark follows the GB200 recipe's FLASH_ATTN drafter. | ||
| args: | ||
| served-model-name: MiniMaxAI/MiniMax-M3.1-Flash | ||
| language-model-only: true | ||
| trust-remote-code: true | ||
| reasoning-parser: minimax_m3 | ||
| tool-call-parser: minimax_m3 | ||
| enable-auto-tool-choice: true | ||
| # MSA sparse/index cache needs 128-token blocks. | ||
| block-size: 128 | ||
| kv-cache-dtype: fp8 | ||
| attention-config: '{"indexer_kv_dtype":"fp8"}' | ||
| moe-backend: marlin | ||
| # Three-token DSpark with probabilistic drafting and block rejection, as | ||
| # measured for minimaxm31flash_dspark. Throughput runs switch to synthetic | ||
| # rejection at the golden acceptance length. | ||
| speculative-config: '{"method":"dspark","num_speculative_tokens":3,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","kv_cache_dtype":"fp8","rejection_sample_method":"block"}' | ||
| disable-hybrid-kv-cache-manager: true | ||
| enable-prefix-caching: true | ||
| # Upstream caps Hopper at 256K, but the AgentX corpus for this prefix is | ||
| # uncapped; keep the native 1M context like the Blackwell arms. | ||
| max-model-len: 1048576 | ||
| max-num-batched-tokens: 8192 | ||
| gpu-memory-utilization: 0.90 | ||
| compilation-config: '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' | ||
| disable-uvicorn-access-log: true | ||
| # vLLM lazy-mmaps checkpoints on h200-dgxc's VIRTIOFS HF cache ~14x slower; | ||
| # stream the shards into page cache with parallel readers first. | ||
| safetensors-load-strategy: prefetch | ||
| env: | ||
| PYTHONNOUSERSITE: '1' | ||
| PYTHONUNBUFFERED: '1' | ||
| VLLM_ENGINE_READY_TIMEOUT_S: '7200' | ||
| # Atomic reduction for the Marlin MoE path. | ||
| VLLM_MARLIN_USE_ATOMIC_ADD: '1' | ||
| benchmark: | ||
| env: | ||
| AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:' | ||
|
|
||
| # Native zip expansion creates one variant per concurrency. The scheduler batch | ||
| # is 2x CONC (AgentX subagent fan-out) and graph capture is the smallest power of | ||
| # two covering it at four tokens per sequence (three drafts), up to 2048. | ||
| zip_override_tp8: | ||
| roles: | ||
| agg: | ||
| gpus: 8 | ||
| args: | ||
| tensor-parallel-size: 8 | ||
| max-num-seqs: [2, 4, 8, 16, 24, 32, 40, 48] | ||
| max-cudagraph-capture-size: [8, 16, 32, 64, 128, 128, 256, 256] | ||
| benchmark: | ||
| env: | ||
| CONC: ['1', '2', '4', '8', '12', '16', '20', '24'] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5294,6 +5294,29 @@ minimaxm31flash-fp4-h100-vllm-agentic-dspark: | |
| # KV per 80 GB GPU, about 2x MiniMax-M3 H100 (cliff c5-c6). | ||
| - { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8, 10, 12], srt-recipe: agentic.yaml } | ||
|
|
||
| # MiniMax-M3.1-Flash AgentX on H200: day-zero vLLM image from | ||
| # https://vllm.ai/blog/2026-10-23-minimax-m31-flash, multiarch manifest verified | ||
| # 2026-10-23 (amd64 is minimax-m31-flash-x86_64-cu134). Serve flags follow the | ||
| # Hopper TP8 recipe: FP8 KV and indexer caches, NVFP4 experts on Marlin MoE. | ||
| # Upstream does not enable DSpark on Hopper; this adds the GB200 recipe's | ||
| # three-token DSpark, with synthetic acceptance from minimaxm31flash_dspark.yaml. | ||
|
Comment on lines
+5297
to
+5302
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 (optional) These new comments break the mandatory AGENTS.md comment rules, which forbid header comments and history or provenance comments. The block above the master entry describes the config and records where it came from ("day-zero vLLM image from ", "verified 2026-10-23"). Fix: delete the header and provenance text, keep only short per-line technical reasons, and leave the history to perf-changelog, which covers the 4 sites listed. Same pattern at 4 sites (configs/nvidia-master.yaml:5274-5279, configs/nvidia-master.yaml:5293-5294, h200-fp4-mtp/agentic.yaml:26-29, h200-fp4-mtp/agentic.yaml:65-67). Why this was flaggedAGENTS.md:64 says never add broad comments above a recipe, override or section that say what the config is. AGENTS.md:65 forbids comments about where something was derived from or about past results. nvidia-master.yaml:5274-5279 is a header above the new entry that cites the blog URL, a verification date and upstream provenance ("Upstream does not enable DSpark on Hopper; this adds the GB200 recipe's..."). nvidia-master.yaml:5293-5294 compares against a past result ("about 1.3x MiniMax-M3 H200 (run to c14)"). agentic.yaml:26-29 says "Serve flags follow the Hopper TP8 recipe at https://recipes.vllm.ai/..." and "DSpark follows the GB200 recipe", which is derivation. agentic.yaml:65-67 is a section comment above the zip_override_tp8 block. The base branch has none of these comments. Nothing enforces the rule automatically, so the comments merge as written. Verification: nit. Nothing breaks at runtime. The comments break the mandatory comment rules in AGENTS.md. Line 64 says "No header comments... above a recipe, override, or section." Line 65 says "No history... where it was derived from, what it replaced, or past results." The header at nvidia-master.yaml:5274-5279 sits directly above |
||
| minimaxm31flash-fp4-h200-vllm-agentic-dspark: | ||
| image: vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25ea92802353727164ae4eb03b37b18baf9119b805ffaec2a7ef07c | ||
| model: MiniMaxAI/MiniMax-M3.1-Flash | ||
| model-prefix: minimaxm31flash | ||
| runner: cluster:h200-dgxc | ||
| precision: fp4 | ||
| framework: vllm | ||
| multinode: false | ||
| srt-recipe-dir: minimaxm31flash/vllm/h200-fp4-mtp | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| # TP8 is the upstream Hopper layout. The 250 GB checkpoint leaves ~88 GiB of | ||
| # KV per 141 GB GPU, about 1.3x MiniMax-M3 H200 (run to c14). | ||
| - { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 12, 16, 20, 24], srt-recipe: agentic.yaml } | ||
|
Comment on lines
+5303
to
+5318
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Every throughput run of this new config will fail before benchmarking, because its golden acceptance curve is not in the repo. The vLLM speculative-config is dspark, so the srt driver looks up Why this was flaggedAny non-eval agentic run of Verification: Trigger: any non-eval agentic throughput run of this config key. There is no minimaxm31flash_dspark.yaml in infx/golden_al_distribution/. |
||
|
|
||
| qwen3.5-fp4-b200-sglang-agentic-mtp: | ||
| image: lmsysorg/sglang:v0.5.16-cu130 | ||
| model: nvidia/Qwen3.5-397B-A17B-NVFP4 | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 (optional) Throughput results may pair a thinking_on golden acceptance length with replay traffic served in the template's default thinking mode. That mode may not be thinking-on. Every MiniMax-M3 vLLM AgentX recipe pins
default-chat-template-kwargs: '{"thinking_mode":"enabled"}', but this new recipe's args do not. The AgentX client sends no thinking toggle. Fix: adddefault-chat-template-kwargs: '{"thinking_mode":"enabled"}'toroles.agg.args, or confirm the M3.1-Flash template defaults to thinking-on. Either way the served mode matches the thinking_on curve the driver picks.Why this was flagged
The condition is the MiniMax-M3.1-Flash chat template not defaulting to thinking-on. I could not check this from the checkout, and the M3 template's toggle is the
thinking_modekwarg (speedbench/minimaxm3_fp4_b300_vllm.sh:43-44). The srt planner hard-codes THINKING_MODE = "thinking_on" (infx/srt_slurm/generate.py:32). That value selects the golden curve in acceptance_length (infx/srt_slurm/synthetic_acceptance.py:131). The recipe args at agentic.yaml:30-58 set no default-chat-template-kwargs. A grep of infx/bench finds no chat_template_kwargs or thinking field in the AgentX client. So the server falls back to the template's default mode. All seven MiniMax-M3 vLLM/ATOM agentic recipes (for example minimaxm3/vllm/h200-fp8-mtp/agentic.yaml:40) pin thinking_mode enabled. perf-changelog.yaml:6361 explains the pin as "Pin thinking-on so the run matches the golden ... thinking_on" acceptance target. If the default is off, the new config publishes numbers whose output-length profile does not match the thinking_on AL it simulates. No validator compares the two.Verification: Trigger: the MiniMax-M3.1-Flash chat template does not default to thinking-on. The new recipe's
roles.agg.args(agentic.yaml lines 30-58) set nodefault-chat-template-kwargs. infx/srt_slurm/generate.py:32 hard-codesTHINKING_MODE = "thinking_on". In that case the throughput numbers model thinking-on acceptance on traffic that is not in thinking mode.