Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
base:
schema: 2
name: minimaxm31flash-fp4-h200-vllm-agentic-dspark
resources:
gpu_type: h200
gpus_per_node: 8
frontend:
type: vllm
enable_multiple_frontends: false
observability:
enabled: false
tachometer:
enabled: false
engine:
type: vllm
connector: null
# The engine may take the full VLLM_ENGINE_READY_TIMEOUT_S to become ready.
health_check:
interval_seconds: 10
max_attempts: 720
roles:
agg:
nodes: 1
workers: 1
# Serve flags follow the Hopper TP8 recipe at
# https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3.1-Flash. Hopper has no FP4
# tensor cores, so the KV and indexer caches are FP8 and the NVFP4 experts
# run on Marlin. DSpark follows the GB200 recipe's FLASH_ATTN drafter.
args:
served-model-name: MiniMaxAI/MiniMax-M3.1-Flash
language-model-only: true
trust-remote-code: true
reasoning-parser: minimax_m3
tool-call-parser: minimax_m3
enable-auto-tool-choice: true
Comment on lines +30 to +35

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Throughput results may pair a thinking_on golden acceptance length with replay traffic served in the template's default thinking mode. That mode may not be thinking-on. Every MiniMax-M3 vLLM AgentX recipe pins default-chat-template-kwargs: '{"thinking_mode":"enabled"}', but this new recipe's args do not. The AgentX client sends no thinking toggle. Fix: add default-chat-template-kwargs: '{"thinking_mode":"enabled"}' to roles.agg.args, or confirm the M3.1-Flash template defaults to thinking-on. Either way the served mode matches the thinking_on curve the driver picks.

Why this was flagged

The condition is the MiniMax-M3.1-Flash chat template not defaulting to thinking-on. I could not check this from the checkout, and the M3 template's toggle is the thinking_mode kwarg (speedbench/minimaxm3_fp4_b300_vllm.sh:43-44). The srt planner hard-codes THINKING_MODE = "thinking_on" (infx/srt_slurm/generate.py:32). That value selects the golden curve in acceptance_length (infx/srt_slurm/synthetic_acceptance.py:131). The recipe args at agentic.yaml:30-58 set no default-chat-template-kwargs. A grep of infx/bench finds no chat_template_kwargs or thinking field in the AgentX client. So the server falls back to the template's default mode. All seven MiniMax-M3 vLLM/ATOM agentic recipes (for example minimaxm3/vllm/h200-fp8-mtp/agentic.yaml:40) pin thinking_mode enabled. perf-changelog.yaml:6361 explains the pin as "Pin thinking-on so the run matches the golden ... thinking_on" acceptance target. If the default is off, the new config publishes numbers whose output-length profile does not match the thinking_on AL it simulates. No validator compares the two.

Verification: Trigger: the MiniMax-M3.1-Flash chat template does not default to thinking-on. The new recipe's roles.agg.args (agentic.yaml lines 30-58) set no default-chat-template-kwargs. infx/srt_slurm/generate.py:32 hard-codes THINKING_MODE = "thinking_on". In that case the throughput numbers model thinking-on acceptance on traffic that is not in thinking mode.

# MSA sparse/index cache needs 128-token blocks.
block-size: 128
kv-cache-dtype: fp8
attention-config: '{"indexer_kv_dtype":"fp8"}'
moe-backend: marlin
# Three-token DSpark with probabilistic drafting and block rejection, as
# measured for minimaxm31flash_dspark. Throughput runs switch to synthetic
# rejection at the golden acceptance length.
speculative-config: '{"method":"dspark","num_speculative_tokens":3,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","kv_cache_dtype":"fp8","rejection_sample_method":"block"}'
disable-hybrid-kv-cache-manager: true
enable-prefix-caching: true
# Upstream caps Hopper at 256K, but the AgentX corpus for this prefix is
# uncapped; keep the native 1M context like the Blackwell arms.
max-model-len: 1048576
max-num-batched-tokens: 8192
gpu-memory-utilization: 0.90
compilation-config: '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
disable-uvicorn-access-log: true
# vLLM lazy-mmaps checkpoints on h200-dgxc's VIRTIOFS HF cache ~14x slower;
# stream the shards into page cache with parallel readers first.
safetensors-load-strategy: prefetch
env:
PYTHONNOUSERSITE: '1'
PYTHONUNBUFFERED: '1'
VLLM_ENGINE_READY_TIMEOUT_S: '7200'
# Atomic reduction for the Marlin MoE path.
VLLM_MARLIN_USE_ATOMIC_ADD: '1'
benchmark:
env:
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'

# Native zip expansion creates one variant per concurrency. The scheduler batch
# is 2x CONC (AgentX subagent fan-out) and graph capture is the smallest power of
# two covering it at four tokens per sequence (three drafts), up to 2048.
zip_override_tp8:
roles:
agg:
gpus: 8
args:
tensor-parallel-size: 8
max-num-seqs: [2, 4, 8, 16, 24, 32, 40, 48]
max-cudagraph-capture-size: [8, 16, 32, 64, 128, 128, 256, 256]
benchmark:
env:
CONC: ['1', '2', '4', '8', '12', '16', '20', '24']
23 changes: 23 additions & 0 deletions inferencex-e2e/configs/nvidia-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5294,6 +5294,29 @@ minimaxm31flash-fp4-h100-vllm-agentic-dspark:
# KV per 80 GB GPU, about 2x MiniMax-M3 H100 (cliff c5-c6).
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 6, 8, 10, 12], srt-recipe: agentic.yaml }

# MiniMax-M3.1-Flash AgentX on H200: day-zero vLLM image from
# https://vllm.ai/blog/2026-10-23-minimax-m31-flash, multiarch manifest verified
# 2026-10-23 (amd64 is minimax-m31-flash-x86_64-cu134). Serve flags follow the
# Hopper TP8 recipe: FP8 KV and indexer caches, NVFP4 experts on Marlin MoE.
# Upstream does not enable DSpark on Hopper; this adds the GB200 recipe's
# three-token DSpark, with synthetic acceptance from minimaxm31flash_dspark.yaml.
Comment on lines +5297 to +5302

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) These new comments break the mandatory AGENTS.md comment rules, which forbid header comments and history or provenance comments. The block above the master entry describes the config and records where it came from ("day-zero vLLM image from ", "verified 2026-10-23"). Fix: delete the header and provenance text, keep only short per-line technical reasons, and leave the history to perf-changelog, which covers the 4 sites listed. Same pattern at 4 sites (configs/nvidia-master.yaml:5274-5279, configs/nvidia-master.yaml:5293-5294, h200-fp4-mtp/agentic.yaml:26-29, h200-fp4-mtp/agentic.yaml:65-67).

Why this was flagged

AGENTS.md:64 says never add broad comments above a recipe, override or section that say what the config is. AGENTS.md:65 forbids comments about where something was derived from or about past results. nvidia-master.yaml:5274-5279 is a header above the new entry that cites the blog URL, a verification date and upstream provenance ("Upstream does not enable DSpark on Hopper; this adds the GB200 recipe's..."). nvidia-master.yaml:5293-5294 compares against a past result ("about 1.3x MiniMax-M3 H200 (run to c14)"). agentic.yaml:26-29 says "Serve flags follow the Hopper TP8 recipe at https://recipes.vllm.ai/..." and "DSpark follows the GB200 recipe", which is derivation. agentic.yaml:65-67 is a section comment above the zip_override_tp8 block. The base branch has none of these comments. Nothing enforces the rule automatically, so the comments merge as written.

Verification: nit. Nothing breaks at runtime. The comments break the mandatory comment rules in AGENTS.md. Line 64 says "No header comments... above a recipe, override, or section." Line 65 says "No history... where it was derived from, what it replaced, or past results." The header at nvidia-master.yaml:5274-5279 sits directly above minimaxm31flash-fp4-h200-vllm-agentic-dspark:.

minimaxm31flash-fp4-h200-vllm-agentic-dspark:
image: vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25ea92802353727164ae4eb03b37b18baf9119b805ffaec2a7ef07c
model: MiniMaxAI/MiniMax-M3.1-Flash
model-prefix: minimaxm31flash
runner: cluster:h200-dgxc
precision: fp4
framework: vllm
multinode: false
srt-recipe-dir: minimaxm31flash/vllm/h200-fp4-mtp
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
# TP8 is the upstream Hopper layout. The 250 GB checkpoint leaves ~88 GiB of
# KV per 141 GB GPU, about 1.3x MiniMax-M3 H200 (run to c14).
- { tp: 8, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 12, 16, 20, 24], srt-recipe: agentic.yaml }
Comment on lines +5303 to +5318

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Every throughput run of this new config will fail before benchmarking, because its golden acceptance curve is not in the repo. The vLLM speculative-config is dspark, so the srt driver looks up minimaxm31flash_dspark.yaml in infx/golden_al_distribution/. That lookup raises "No committed golden curve" (curves.py:68), and the PR description confirms the file is not committed. Fix: commit minimaxm31flash_dspark.yaml in this PR, or hold back the master entry and its perf-changelog entry until the curve lands, so no run can be scheduled without it.

Why this was flagged

Any non-eval agentic run of minimaxm31flash-fp4-h200-vllm-agentic-dspark has SPEC_DECODING=mtp (nvidia-master.yaml:5295). It also has a vLLM speculative-config with method dspark (agentic.yaml:44). So acceptance_length in infx/srt_slurm/synthetic_acceptance.py calls golden_length("minimaxm31flash", spec, ...). load_curve raises ValueError "No committed golden curve minimaxm31flash_dspark.yaml in ..." at infx/golden_al_distribution/curves.py:68. The only files under golden_al_distribution/ for this family are minimaxm3_eagle3.yaml and minimaxm3_eagle3_gqa.yaml. Nothing checks this before merge: only the golden_al_distribution unit tests use GOLDEN_DIR, and they do not cross-check master configs. The perf-changelog entry adds this key to the benchmark set, so once merged every scheduled sweep point (8 concurrencies) fails at override construction and produces no results. MODELS.md and the changelog also cite a curve that does not exist. The PR description says the sweep is "on hold", but nothing in the code enforces that hold.

Verification: Trigger: any non-eval agentic throughput run of this config key. There is no minimaxm31flash_dspark.yaml in infx/golden_al_distribution/. load_curve raises at curves.py:68, and main returns 1 before any srtctl apply is submitted. No test checks master entries against the curve files, so CI will not catch this before merge.


qwen3.5-fp4-b200-sglang-agentic-mtp:
image: lmsysorg/sglang:v0.5.16-cu130
model: nvidia/Qwen3.5-397B-A17B-NVFP4
Expand Down
9 changes: 9 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10919,3 +10919,12 @@
- "Update the SGLang ROCm image from v0.5.21-rocm720-mi30x to v0.5.22-rocm720-mi30x."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4408
scenario-type: [agentic-coding]

- config-keys:
- minimaxm31flash-fp4-h200-vllm-agentic-dspark
description:
- "Add day-zero MiniMax-M3.1-Flash (MiniMaxAI/MiniMax-M3.1-Flash, NVFP4 experts + MXFP8 dense) AgentX on H200 with vLLM image minimax-m31-flash (https://vllm.ai/blog/2026-10-23-minimax-m31-flash)."
- "Serve flags follow the recipes.vllm.ai Hopper TP8 recipe: block size 128, FP8 KV and indexer cache, Marlin MoE backend for the NVFP4 experts, FULL_AND_PIECEWISE graphs, hybrid KV cache manager disabled, minimax_m3 reasoning and tool parsers, three-token DSpark (FLASH_ATTN drafter, fp8 draft KV, probabilistic drafting, block rejection, from the GB200 recipe; upstream does not enable DSpark on Hopper). Context stays at 1M (upstream Hopper example uses 256K) because the AgentX corpus for this prefix is uncapped."
- "Search space: TP8 only, conc 1-24, no KV offloading. Throughput runs use synthetic acceptance from the minimaxm31flash_dspark golden curve (AL 2.88 at 3 draft tokens, #4425)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4423
scenario-type: [agentic-coding]
Loading