Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,9 @@ override_tp4_c30_dram:
args:
tensor-parallel-size: 4
max-num-seqs: 60
max-cudagraph-capture-size: 240
max-cudagraph-capture-size: 4096
cudagraph-capture-sizes: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 256, 320, 384, 448, 512, 640, 768, 896, 1024, 1280, 1536, 1792, 2048, 2560, 3072, 3584, 4096]
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1073000000000,"lazy_offload":true}}'
env:
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
Expand Down Expand Up @@ -257,7 +259,9 @@ override_tp4_c38_dram:
args:
tensor-parallel-size: 4
max-num-seqs: 76
max-cudagraph-capture-size: 304
max-cudagraph-capture-size: 4096
cudagraph-capture-sizes: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 384, 448, 512, 640, 768, 896, 1024, 1280, 1536, 1792, 2048, 2560, 3072, 3584, 4096]
attention-config: '{"backend":"FLASHINFER","use_trtllm_attention":true,"indexer_kv_dtype":"fp8","minimax_m3_msa_decode_backend":"cutlass"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1073000000000,"lazy_offload":true}}'
env:
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
Expand All @@ -283,3 +287,133 @@ override_tp4_c40_dram:
CONC: '40'
KV_OFFLOADING: dram
TOTAL_CPU_DRAM_GB: '1073'

# Low-concurrency arms that cap the number of sequences the scheduler keeps active
# (max-num-active-seqs) below CONC, with a three- or five-token draft. Selected by
# name from the master config.
Comment on lines +291 to +293

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): The new section comments break the repo's comment policy, which bans header comments above a recipe, override or section. sweep:^# (Low-concurrency arms|TP4 DRAM concurrency 20|CONC 32 with active) Lines 291-293 say what the next overrides are and how they are selected, so they are a section header, not a technical note. Line 377 is the same kind of comment, and its "cut to" wording also describes history. Fix: delete these blocks. If a value needs a reason, keep only a one-line note on that line, for example why max-num-active-seqs sits below CONC.

Why this was flagged

AGENTS.md "Code comments (mandatory)" applies to recipes and configs. It says "Never add broad, overarching comments ... above a recipe, override, or section" and "Do not describe what the file, topology, or config is". It also bans history wording. The diff adds three such comments in agentic.yaml. Lines 291-293 begin "# Low-concurrency arms that cap ... Selected by name from the master config." Lines 333-334 begin "# TP4 DRAM concurrency 20 with 4 and 3 active sequences on the c10 key's image". Line 377 begins "# CONC 32 with active sequences capped below CONC ... the CPU KV pool is cut to the 1,073 GB node allowance." Each one describes the override group below it and does not explain a specific value. The base branch has none of them.

Verification: AGENTS.md lines 60-67 apply to recipes and say: "Never add broad, overarching comments at the top of a file or above a recipe, override, or section." The diff adds such comments in agentic.yaml at lines 291-293 and 333-334, and line 377's "cut to" is history wording. Nothing breaks at runtime; the only effect is a breach of a rule the repository wrote down.


override_cap1_k5_tp4_c10:
roles:
agg:
gpus: 4
args:
tensor-parallel-size: 4
max-num-active-seqs: 1
gpu-memory-utilization: 0.925
max-cudagraph-capture-size: 4096
cudagraph-capture-sizes: [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 640, 768, 1024, 1536, 2048, 3072, 4096]
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":5,"attention_backend":"FLASHINFER"}'
env:
VLLM_FLASHINFER_ALLREDUCE_BACKEND: mnnvl
benchmark:
env:
CONC: '10'
KV_OFFLOADING: none

override_cap1_k5_tp8_c10:
roles:
agg:
gpus: 8
args:
tensor-parallel-size: 8
max-num-active-seqs: 1
gpu-memory-utilization: 0.9
max-cudagraph-capture-size: 4096
cudagraph-capture-sizes: [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 640, 768, 1024, 1536, 2048, 3072, 4096]
no-enable-flashinfer-autotune: false
enable-flashinfer-autotune: true
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":5,"attention_backend":"FLASHINFER"}'
env:
VLLM_FLASHINFER_ALLREDUCE_BACKEND: mnnvl
benchmark:
env:
CONC: '10'
KV_OFFLOADING: none

# TP4 DRAM concurrency 20 with 4 and 3 active sequences on the c10 key's image; the CPU KV
# pool is the 1,073 GB node allowance.
override_cap4_tp4_c20_dram:
roles:
agg:
gpus: 4
args:
tensor-parallel-size: 4
max-num-active-seqs: 4
gpu-memory-utilization: 0.9
max-cudagraph-capture-size: 8192
cudagraph-capture-sizes: [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 640, 768, 1024, 1536, 2048, 3072, 4096, 6144, 8192]
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1073000000000,"lazy_offload":true}}'
env:
VLLM_FLASHINFER_ALLREDUCE_BACKEND: mnnvl
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
benchmark:
env:
CONC: '20'
KV_OFFLOADING: dram
TOTAL_CPU_DRAM_GB: '1073'

override_cap3_tp4_c20_dram:
roles:
agg:
gpus: 4
args:
tensor-parallel-size: 4
max-num-active-seqs: 3
gpu-memory-utilization: 0.9
max-cudagraph-capture-size: 8192
cudagraph-capture-sizes: [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 640, 768, 1024, 1536, 2048, 3072, 4096, 6144, 8192]
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1073000000000,"lazy_offload":true}}'
env:
VLLM_FLASHINFER_ALLREDUCE_BACKEND: mnnvl
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
benchmark:
env:
CONC: '20'
KV_OFFLOADING: dram
TOTAL_CPU_DRAM_GB: '1073'

# CONC 32 with active sequences capped below CONC on the c10 key's image; the CPU KV pool is cut to the 1,073 GB node allowance.

override_cap6_tp4_c32_dram:
roles:
agg:
gpus: 4
args:
tensor-parallel-size: 4
max-num-active-seqs: 6
gpu-memory-utilization: 0.925
max-cudagraph-capture-size: 4096
cudagraph-capture-sizes: [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 640, 768, 1024, 1536, 2048, 3072, 4096]
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1073000000000,"lazy_offload":true}}'
env:
VLLM_FLASHINFER_ALLREDUCE_BACKEND: mnnvl
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
benchmark:
env:
CONC: '32'
KV_OFFLOADING: dram
TOTAL_CPU_DRAM_GB: '1073'

override_cap16_tp4_c32_dram:
roles:
agg:
gpus: 4
args:
tensor-parallel-size: 4
max-num-active-seqs: 16
gpu-memory-utilization: 0.925
max-cudagraph-capture-size: 4096
cudagraph-capture-sizes: [1, 2, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, 640, 768, 1024, 1536, 2048, 3072, 4096]
speculative-config: '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER"}'
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":1073000000000,"lazy_offload":true}}'
env:
VLLM_FLASHINFER_ALLREDUCE_BACKEND: mnnvl
VLLM_USE_SIMPLE_KV_OFFLOAD: '1'
benchmark:
env:
CONC: '32'
KV_OFFLOADING: dram
TOTAL_CPU_DRAM_GB: '1073'
31 changes: 26 additions & 5 deletions inferencex-e2e/configs/nvidia-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5558,7 +5558,7 @@ minimaxm3-fp4-b300-trtllm-agentic-mtp:
# enabled by framework=dynamo-vllm.

minimaxm3-fp4-b200-vllm-agentic-mtp:
image: vllm/vllm-openai:nightly-af7f9488c2210d67e1033ecdc845b087ee7fe92b
image: vllm/vllm-openai:nightly-af7f9488c2210d67e1033ecdc845b087ee7fe92b@sha256:8e2ffbb9f866de678b22e7446903b1ee5a5766c51a948013277477c462a68a77
model: nvidia/MiniMax-M3-NVFP4
model-prefix: minimaxm3
runner: cluster:b200-nscale
Expand All @@ -5570,11 +5570,33 @@ minimaxm3-fp4-b200-vllm-agentic-mtp:
agentic-coding:
- dram-utilization: 0.683
search-space:
- { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml }
- { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 5, 10, 15, 20], srt-recipe: agentic.yaml }
- { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [15], srt-recipe: agentic.yaml:override_tp4_c15 }
- dram-utilization: 1.0
search-space:
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [30], srt-recipe: agentic.yaml:override_tp4_c30_dram }
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [32], srt-recipe: agentic.yaml:override_tp4_c32_dram }
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [38], srt-recipe: agentic.yaml:override_tp4_c38_dram }
Comment on lines 5570 to +5578

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Maintainers inherit 11 overrides in the B200 MiniMax-M3 agentic.yaml that no config runs anymore. The master config now names a selector for every point. Overrides left without a selector include override_tp8_c1, override_tp4_c1/c5/c10/c20, and the dram c15/c20/c25/c34/c36/c40 blocks. Fix: delete the overrides no config references, or keep them reachable from a search-space entry. AGENTS.md says to delete recipes and assets no active config uses rather than leave them in place.

Why this was flagged

On the base branch, both search-space groups used bare srt-recipe: agentic.yaml. With no selector, expand_variants in infx/srt_slurm/variants.py:102-104 expands every override_* block, so all of them were reachable. After this diff, nvidia-master.yaml:5527-5553 points each entry at one named override. The only ones used are override_tp4_c15, override_tp4_c30_dram, override_tp4_c32_dram, override_tp4_c38_dram and the six new cap overrides. The only two configs that use srt-recipe-dir minimaxm3/vllm/b200-fp4-mtp are at nvidia-master.yaml:5522 and :5541. That leaves 11 dead blocks in agentic.yaml: override_tp8_c1 (line 56), override_tp4_c1 (69), c5 (82), c10 (95), c20 (121), c15_dram (134), c20_dram (151), c25_dram (168), c34_dram (221), c36_dram (238) and c40_dram (274). They still look like supported benchmark points, but nothing runs or checks them, which conflicts with the AGENTS.md rule to remove unused assets.

Verification: On the base, both search-space groups used bare srt-recipe: agentic.yaml. With no selector, expand_variants (inferencex-e2e/infx/srt_slurm/variants.py) expands every override_* key. After this diff, every entry at nvidia-master.yaml:5527-5553 names exactly one override. That leaves 11 overrides in agentic.yaml that nothing selects.

minimaxm3-fp4-b200-vllm-agentic-mtp-c10:
image: vllm/vllm-openai:nightly-ac68c3087215e0a4f3cdfa218508c6aada57235d@sha256:98adb2311118263a453ec0acd5d4b72bda3db26cecc0fe8ffc94b9ab03c951f9
model: nvidia/MiniMax-M3-NVFP4
model-prefix: minimaxm3
runner: cluster:b200-nscale
precision: fp4
framework: vllm
multinode: false
srt-recipe-dir: minimaxm3/vllm/b200-fp4-mtp
scenarios:
agentic-coding:
- dram-utilization: 0.683
search-space:
- { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [10], srt-recipe: agentic.yaml:override_cap1_k5_tp4_c10 }
- { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [10], srt-recipe: agentic.yaml:override_cap1_k5_tp8_c10 }
- dram-utilization: 1.0
search-space:
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [15, 20, 25, 30, 32, 34, 36, 38, 40], srt-recipe: agentic.yaml }
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [20], srt-recipe: agentic.yaml:override_cap4_tp4_c20_dram }
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [20], srt-recipe: agentic.yaml:override_cap3_tp4_c20_dram }
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [32], srt-recipe: agentic.yaml:override_cap6_tp4_c32_dram }
- { tp: 4, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [32], srt-recipe: agentic.yaml:override_cap16_tp4_c32_dram }
Comment on lines +5596 to +5599

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Two pairs of new points can't be told apart by the tooling: cap4/cap3 at c20 and cap6/cap16 at c32. Their results collide, so reuse validation and per-experiment reruns fail. The c10 key reuses TP4 with the same conc and offload settings, and only the srt-recipe override differs. Exp names (generate.py:1062) and agentic_key (results/artifacts.py:123) both ignore the override. Fix: give every point a distinct identity, for example by keeping one capped variant per (tp, conc, offload), or by adding the recipe or override to the exp name and to agentic_key.

Why this was flagged

In minimaxm3-fp4-b200-vllm-agentic-mtp-c10, nvidia-master.yaml:5550-5551 are both tp 4, conc 20, dram, vllm-simple. Lines 5552-5553 are both tp 4, conc 32, dram, vllm-simple, and so is override_tp4_c32_dram at :5531 in the main key, on the same runner cluster:b200-nscale. generate.py:1062-1066 builds exp_name only from model_code, tp, conc, offload suffix and spec, so each pair gets the same exp_name. filter_exp_names (generate.py:1437-1441) then raises "Experiment name(s) matched multiple rows" when a test-config run targets either point. For single-node runs, agentic_key (results/artifacts.py:166-180) keys on hw, model prefix, framework, precision, tp, pp, dcp, pcp, ep, dp_attention, conc and offload, not the recipe. So once a sweep finishes, validate_agentic_artifacts reports "agentic point artifacts contain N duplicate row(s)". That breaks the /use <run_id> reuse path validate_reusable_sweep_artifacts.py:59. The artifact names include RECIPE_FINGERPRINT, so uploads don't collide, but the identity check doesn't use it. On the base branch every (tp, conc, offload) point was unique.

Verification: In configs/nvidia-master.yaml, lines 5550 and 5551 differ only in srt-recipe, and so do lines 5552 and 5553. infx/matrix/generate.py:1062-1066 builds the exp_name without the override, so filter_exp_names raises "Experiment name(s) matched multiple rows". agentic_key (infx/results/artifacts.py:169-183) also ignores it, so reuse validation fails. The base had no such duplicates.

minimaxm31flash-fp4-b200-vllm-agentic-dspark:
image: vllm/vllm-openai:minimax-m31-flash@sha256:86f582dae25ea92802353727164ae4eb03b37b18baf9119b805ffaec2a7ef07c
model: MiniMaxAI/MiniMax-M3.1-Flash
Expand Down Expand Up @@ -5612,7 +5634,6 @@ minimaxm31flash-fp4-gb200-vllm-agentic-dspark:
- search-space:
- { tp: 4, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 16, 32], srt-recipe: agentic.yaml }
- { tp: 4, ep: 4, dp-attn: true, spec-decoding: mtp, kv-offloading: none, conc-list: [64, 128], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml }

minimaxm3-fp4-b200-trtllm-agentic-mtp:
image: nvcr.io#nvidia/tensorrt-llm/release:1.3.0rc23.post1
model: nvidia/MiniMax-M3-NVFP4
Expand Down
10 changes: 10 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11043,3 +11043,13 @@
description:
- 'Refine GB300 Kimi-K3 AgentX points to be good on all four p90 charts: replace the decode-capped 1P2D-2P5D points with uncapped 1P2D c48-64 and 3P4D c240, add specialized-decode 1P3D c16 and 1P2D c24/32, keep 2P3D c240 and aggregated TP16 c1, and drop aggregated TP8 c2/c4. Pin cp-kv-cache-interleave-size to 1 for the specialized-decode disaggregated points.'
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4373

- config-keys:
- minimaxm3-fp4-b200-vllm-agentic-mtp
- minimaxm3-fp4-b200-vllm-agentic-mtp-c10
scenario-type:
- agentic-coding
description:
- "Pin the MiniMax-M3 B200 vLLM images by digest and retune the TP4 concurrency 15-38 entries, with DRAM KV offload sized to the 1,073 GB node allowance."
- "Add minimaxm3-fp4-b200-vllm-agentic-mtp-c10 with TP4 and TP8 concurrency 10 and TP4 DRAM concurrency 20 and 32 entries that cap active sequences."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4382
Loading