Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ base:
engine:
type: vllm
connector: null
# Cold loads of the shared-cache checkpoint run past half an hour.
# The engine may take the full VLLM_ENGINE_READY_TIMEOUT_S to become ready.
health_check:
interval_seconds: 5
max_attempts: 1440
interval_seconds: 10
max_attempts: 720
roles:
agg:
nodes: 1
Expand All @@ -29,19 +29,12 @@ base:
tool-call-parser: deepseek_v41
enable-auto-tool-choice: true
reasoning-parser: deepseek_v41
engram-config: '{"cpu_offload":true}'
kernel-config: '{"enable_flashinfer_autotune":true}'
attention-config: '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
kv-cache-dtype: fp8
# Five-token DSpark with probabilistic drafting. Throughput runs replace
# block rejection with the golden acceptance length.
speculative-config: '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
max-model-len: 1048576
# Piecewise graph capture sizes are multiples of the six-token DSpark
# verification block, denser for small batches.
compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
max-cudagraph-capture-size: 8190
max-num-batched-tokens: 8192
gpu-memory-utilization: 0.93
disable-uvicorn-access-log: true
env:
VLLM_USE_V2_MODEL_RUNNER: '1'
Expand All @@ -54,29 +47,36 @@ base:

# Native zip expansion creates one variant per concurrency.

# TP4 covers low concurrency with THP Engram tables.
zip_override_tp4:
roles:
agg:
gpus: 4
args:
tensor-parallel-size: 4
attention-config: '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
engram-config: '{"cpu_offload":true,"use_thp":true}'
max-num-seqs: [8, 8, 16, 32]
kernel-config: '{"enable_flashinfer_autotune":true}'
# Specialized decode: the SM103 latency-specialized TP4 decode executor.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): The comment above additional-config only repeats what the next line says, which AGENTS.md's comment rules forbid. Fix: delete the comment, or swap it for the reason TP4 needs this executor (for example, a measured decode-latency gain or an SM103-only constraint).

Why this was flagged

agentic.yaml:59 adds # Specialized decode: the SM103 latency-specialized TP4 decode executor. directly above additional-config: '{"decode_backend":"latency_specialized"}' at line 60. Root CLAUDE.md points to AGENTS.md. Under its mandatory comment rules, a comment may only briefly explain a specific line or block, such as why a value was chosen or a hardware quirk, and must never restate the code. This comment only names what the value already says, so it adds a convention violation the base recipe did not have. There is no runtime impact.

Verification: nit. Line 59 of the new agentic.yaml in zip_override_tp4 is # Specialized decode: the SM103 latency-specialized TP4 decode executor., directly above line 60, additional-config: '{"decode_backend":"latency_specialized"}'. AGENTS.md lines 54-58 say to "never restate the code." Only "SM103" adds a detail, and it gives no reason. Nothing at runtime is affected.

additional-config: '{"decode_backend":"latency_specialized"}'
max-num-seqs: [1, 8, 16, 64, 64, 64, 64]
# Keeps decode batches small; max-num-seqs still sizes the runner and graphs.
max-num-active-seqs: [null, 1, 2, 3, 4, 6, 8]
prefill-schedule-interval: [1, 16, 96, 96, 96, 96, 96]
max-num-batched-tokens: [4096, 4096, 8192, 8192, 8192, 8192, 8192]
compilation-config:
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
max-cudagraph-capture-size: [4092, 4092, 8190, 8190]
max-num-batched-tokens: [4096, 4096, 8192, 8192]
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[5,6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
max-cudagraph-capture-size: [4092, 4092, 8190, 8190, 8190, 8190, 8190]
benchmark:
env:
CONC: ['1', '4', '8', '16']
CONC: ['1', '4', '10', '16', '24', '32', '48']

# DEP2: TP1 x DP2 + EP2 with MegaMoE behind a consistent-hash router;
# MegaAttention at CONC 128 and above.
zip_override_dep2:
zip_override_dep4:
frontend:
type: vllm-router
args:
Expand All @@ -88,30 +88,38 @@ zip_override_dep2:
setup_script: vllm-router.sh
roles:
agg:
gpus: 2
gpus: 4
args:
tensor-parallel-size: 1
data-parallel-size: 2
data-parallel-size: 4
enable-expert-parallel: true
kernel-config: '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe"}'
engram-config: '{"cpu_offload":true,"embedding_across_dp":true}'
max-num-seqs: [16, 16, 32, 64, 128, 192]
attention-config:
- '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
- '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
- '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
- '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
- '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
- '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
compilation-config:
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048]}'
max-cudagraph-capture-size: [8190, 8190, 8190, 8190, 8190, 4092]
max-num-batched-tokens: [8192, 8192, 8192, 8192, 16384, 16384]
enable-cumem-allocator: true
enable-ep-weight-filter: true
attention-config: '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true,"sparse_mla_prefill_workspace_factor":4}'
engram-config: '{"cpu_offload":true,"embedding_across_dp":true,"use_thp":false}'
kernel-config: '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe","sparse_indexer_topk_backend":"deep_select"}'
max-num-seqs: [128, 128, 128, 128, 128, 128, 192, 224, 256, 320]
prefill-schedule-interval: 96
max-num-batched-tokens: 65536
# gpu-memory-utilization sizing reserves this prefill budget's activation peak and leaves too little KV.
kv-cache-memory: [25769803776, 30064771072, 34359738368, 42949672960, 51539607552, 60129542144, 77309411328, 103079215104, 120259084288, 137438953472]
# A cold long prompt on one DP rank would otherwise stall the other ranks' decode for a full budget step.
long-prefill-token-threshold: 8192
compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
max-cudagraph-capture-size: 8190
# Binds each DP rank's EngineCore and worker to its GPU's NUMA node.
numa-bind: true
numa-bind-nodes: [[0, 0, 1, 1]]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 GB300 DeepSeek-V4.1-Flash operatorx layer cases that fall back to this recipe would get server args that serve_argv cannot render or was meant to drop. That happens if vLLM types --numa-bind-nodes as an int list. serve_argv (operatorx/recipes.py:129) sends numa-bind-nodes as one JSON token, which argparse would reject. kv-cache-memory also gets past the kv-cache-memory-bytes exclusion. Fix: add kv-cache-memory, numa-bind and numa-bind-nodes to _NOT_APPLIED, or spell the recipe key kv-cache-memory-bytes. Also have serve_argv emit list values as separate tokens.

Why this was flagged

The operatorx CI calls find_recipe (operatorx/ci.py:213) for each GB300 case whose sources name deepseek-ai/DeepSeek-V4.1-Flash. _variants expands this recipe with no selector, and zip keys are sorted (synthetic_acceptance.py:376), so zip_override_dep4[0] comes first. A case whose split matches no variant exactly, such as a single-device or tp2 MoE case, falls back to pool[0] (recipes.py:114-120) and gets the DEP4 c136 args. serve_argv (recipes.py:133-137) turns numa-bind-nodes: [0, 0, 1, 1] into --numa-bind-nodes "[0, 0, 1, 1]". engine.py:59 passes that to vLLM's parse_known_args. If vLLM declares the flag as a list[int] with nargs and type=int, argparse exits on an invalid int value and the shard dies. On the base branch the fallback was zip_override_dep2[0], which set neither key. Neither key has an exclusion in _NOT_APPLIED.

Verification: Severity: normal. Trigger: the recipe image's vLLM declares --numa-bind-nodes as a list[int] flag (nargs="+", type=int). serve_argv (recipes.py:133-138) adds --numa-bind-nodes and then json.dumps(v) as one token, "[0, 0, 1, 1]". Before this change, the same slot held dep2[0], which had no numa or kv-cache-memory args.

env:
VLLM_RPC_TIMEOUT: '600000'
VLLM_LOG_STATS_INTERVAL: '1'
VLLM_SERVER_DEV_MODE: '1'
VLLM_ALLREDUCE_USE_SYMM_MEM: '0'
VLLM_RANDOMIZE_DP_DUMMY_INPUTS: '1'
NCCL_CUMEM_ENABLE: '1'
NCCL_MNNVL_ENABLE: '1'
NCCL_NVLS_ENABLE: '1'
PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
benchmark:
env:
CONC: ['8', '16', '32', '64', '128', '192']
CONC: ['136', '170', '200', '256', '320', '384', '512', '640', '768', '1024']
10 changes: 3 additions & 7 deletions inferencex-e2e/configs/nvidia-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5784,7 +5784,7 @@ dsv41flash-fp4-b200-sglang-agentic-dspark:
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, spec-decoding: mtp, conc-list: [48, 64, 96, 128, 192, 256], router: { name: sglang-router, version: "0.3.2" }, srt-recipe: agentic.yaml }

dsv41flash-fp4-gb300-vllm-agentic-dspark:
image: vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7
image: vllm/vllm-openai:cu134-nightly-a2bcb5657a881a002e79d0adab3c6b5e39436121@sha256:76a533475156ffdb3656ccc389d0c71f5cec16897c0f490ac43e4f379c4b4f0d
model: deepseek-ai/DeepSeek-V4.1-Flash
model-prefix: dsv41flash
runner: cluster:gb300-nv
Expand All @@ -5796,12 +5796,8 @@ dsv41flash-fp4-gb300-vllm-agentic-dspark:
agentic-coding:
- dram-utilization: 0.80
search-space:
# Engram weights use UVA DRAM; the KV cache stays GPU-resident.
# TP4 covers low concurrency with FlashInfer attention and THP Engram tables.
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 8, 16], srt-recipe: agentic.yaml }
# DEP2 is native TP1 x DP2 + EP2 with MegaMoE behind a consistent-hash vLLM
# Router; FlashInfer attention up to conc 64, MegaAttention above.
- { tp: 2, ep: 2, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [8, 16, 32, 64, 128, 192], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml }
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 4, 10, 16, 24, 32, 48], srt-recipe: agentic.yaml }
- { tp: 4, ep: 4, dp-attn: true, kv-offloading: none, spec-decoding: mtp, conc-list: [136, 170, 200, 256, 320, 384, 512, 640, 768, 1024], router: { name: vllm-router, version: "0.1.14" }, srt-recipe: agentic.yaml }

# MiniMax-M3.1-Flash AgentX on GB300: the B200 recipe and search space plus a
# DEP4 c96 point, with the vLLM blog's FlashInfer PrimiTS DSpark drafter and the
Expand Down
9 changes: 9 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12099,3 +12099,12 @@
- "Disaggregated: TP8/DCP8 prefill with a HiCache host tier feeding either one TP16 (DCP8 x attention DP2, EP16 MegaMoE) decode worker (3P1D c300, 2P1D c180, 1P1D c76) or TP8/DCP8 decode workers (1P3D c60 and c36, 1P4D c32)."
- "Aggregated low-latency points: one TP16 EP1 worker with the NVIDIA megakernel (--enable-nvidia-megakernel), max-running-requests equal to the concurrency (c1, c2, c4)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4811

- config-keys:
- dsv41flash-fp4-gb300-vllm-agentic-dspark
description:
- "Update the GB300 DeepSeek-V4.1-Flash vLLM AgentX image from vllm/vllm-openai:nightly-dev-arm64-cu130-ac9126e58aa7 to the vLLM main nightly vllm/vllm-openai:cu134-nightly-a2bcb5657a881a002e79d0adab3c6b5e39436121 (main a2bcb565, pinned to index sha256:76a533475156ffdb3656ccc389d0c71f5cec16897c0f490ac43e4f379c4b4f0d; arm64 sha256:24541db891c590e3a4022c7206c3ebb00e4be35442f54c20f2edf12eddfef4bf) and replace the search space with seventeen single-node points. Every point sets gpu-memory-utilization 0.93 and runs five-token DSpark with adaptive verification off."
- "TP4 with the DeepSeek-V4.1 latency-specialized decode executor (additional-config decode_backend latency_specialized, FlashInfer sparse MLA, THP Engram host tables) behind vLLM's own frontend at concurrency 1 (max-num-seqs 1, prefill-schedule-interval 1), 4 (max-num-seqs 8, max-num-active-seqs 1, prefill-schedule-interval 16), 10 (max-num-seqs 16, max-num-active-seqs 2, prefill-schedule-interval 96) and 16, 24, 32 and 48 (max-num-seqs 64, max-num-active-seqs 3, 4, 6 and 8, prefill-schedule-interval 96)."
- "DEP4 (TP1 x DP4 + EP4, DeepGEMM MegaMoE, FlashMLA mega-attention with sparse-MLA prefill workspace factor 4, deep_select top-k, Engram host tables shared across DP ranks, prefill-schedule-interval 96, NUMA binding) behind the consistent-hash vLLM Router at concurrency 136 (24 GiB KV pool), 170 (28 GiB), 200 (32 GiB), 256 (40 GiB), 320 (48 GiB), 384 (56 GiB), 512 (72 GiB, max-num-seqs 192), 640 (96 GiB, max-num-seqs 224), 768 (112 GiB, max-num-seqs 256) and 1024 (128 GiB, max-num-seqs 320), max-num-seqs 128 otherwise, each with a 65536-token prefill budget and long-prefill-token-threshold 8192. Removes TP4 at concurrency 8 and DEP2 at concurrency 8-192."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
scenario-type: [agentic-coding]
Loading