Skip to content
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,17 @@ base:
VLLM_USE_V2_MODEL_RUNNER: '1'
VLLM_FASTSAFETENSORS_QUEUE_SIZE: '-1'
# K3.1 has no torch.compile submodule, so only breakable graphs capture the prefill
# sizes the throughput points list. The conc-1 override turns this off.
# sizes the throughput points list. The conc-1 and conc-4 overrides turn this off.
VLLM_USE_BREAKABLE_CUDAGRAPH: '1'
VLLM_ROCM_AITER_MLA_ASM_PADDING: asm
# Verify the draft tokens with the AITER assembly DCP kernel.
VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm
VLLM_ROCM_DCP_AITER_ALLGATHER: '1'
# DCP V-up reduce-scatter; needs VLLM_ROCM_DCP_AITER_ALLGATHER.
VLLM_ROCM_DCP_VUP_RS: '1'
VLLM_ROCM_DSA_MULTISTREAM: '1'
# VRAM reserve for unprofiled DCP prefill transients.
VLLM_ROCM_MEM_HEADROOM_GIB: '8'
VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1'
# AITER reads its own variable for the same kernels.
AITER_FLYDSL_SPARSE_MLA: '1'
Expand Down Expand Up @@ -94,8 +99,9 @@ base:
AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'

# max-num-seqs stays below CONC from c4 to c32, and equals it at c1 and c48. The capture list is
# the decode sizes, multiples of one plus the draft token count, then the prefill ladder.
# max-num-seqs stays below CONC from c4 to c32, and equals it at c1 and c48. c1 and c4 capture
# decode sizes only; the others list decode sizes, multiples of one plus the draft token count,
# then the prefill ladder.
# Throughput runs replace block rejection with the golden acceptance length.
override_gpu_kv_c1:
roles:
Expand All @@ -112,44 +118,43 @@ override_gpu_kv_c1:
env:
# This point captures only the decode sizes, so breakable graphs buy nothing.
VLLM_USE_BREAKABLE_CUDAGRAPH: '0'
# Runs each DSA layer's decode prologue as one launch; needs DCP1.
VLLM_KIMI_SD_DSA_PROLOGUE: '1'
benchmark:
env:
CONC: '1'
KV_OFFLOADING: 'none'

override_dram_c4:
override_gpu_kv_c4:
roles:
agg:
args:
max-num-seqs: 2
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[8,16,24,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '4'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

override_dram_c10:
roles:
agg:
args:
max-num-seqs: 7
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,18,24,30,36,42,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":12,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12]}'
additional-config: '{"decode_backend":"latency_specialized","kda_prefill_backend":"fused"}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
kv-transfer-config: null
env:
# Decode-only capture sizes, as at conc 1.
VLLM_USE_BREAKABLE_CUDAGRAPH: '0'
benchmark:
env:
CONC: '10'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'
CONC: '4'
KV_OFFLOADING: 'none'

override_dram_c16:
roles:
agg:
args:
max-num-seqs: 8
# Admit prefill only every 32nd engine step so decode runs in longer uninterrupted stretches.
prefill-schedule-interval: 32
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
env:
# max-num-seqs is below CONC, so requests always wait and vLLM would otherwise
# drop the prefill hold (capacity escape); keep it.
VLLM_PREFILL_CADENCE_NO_ESCAPE: '1'
Comment thread
seungrokj marked this conversation as resolved.
Resolved
benchmark:
env:
CONC: '16'
Expand All @@ -161,8 +166,11 @@ override_dram_c24:
agg:
args:
max-num-seqs: 13
prefill-schedule-interval: 32
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
env:
VLLM_PREFILL_CADENCE_NO_ESCAPE: '1'
benchmark:
env:
CONC: '24'
Expand All @@ -176,12 +184,11 @@ override_dram_c32:
max-num-seqs: 20
max-num-batched-tokens: 10240
gpu-memory-utilization: 0.91
# Admit prefill only every 8th engine step so decode runs in longer uninterrupted stretches.
prefill-schedule-interval: 8
long-prefill-token-threshold: 10240
long-prefill-token-threshold-adaptive: true
prefill-schedule-interval: 32
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
env:
VLLM_PREFILL_CADENCE_NO_ESCAPE: '1'
benchmark:
env:
CONC: '32'
Expand All @@ -195,15 +202,10 @@ override_dram_c48:
max-num-seqs: 48
max-num-batched-tokens: 10240
gpu-memory-utilization: 0.91
# Admit prefill only every 8th engine step so decode runs in longer uninterrupted stretches.
prefill-schedule-interval: 8
long-prefill-token-threshold: 10240
long-prefill-token-threshold-adaptive: true
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224,231,238,245,252,259,266,273,280,287,294,301,308,315,322,329,336,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":6,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
benchmark:
env:
CONC: '48'
KV_OFFLOADING: 'dram'
TOTAL_CPU_DRAM_GB: '1799'

7 changes: 4 additions & 3 deletions inferencex-e2e/configs/amd-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -974,7 +974,7 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark:
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [32, 64], srt-recipe: agentic.yaml }

kimik3.1-fp4-mi355x-vllm-agentic-dspark:
image: vllm/vllm-openai-rocm:kimi-k31-d73dd529@sha256:5b54f223a150a9baad41a5d511e159e7cbf43e308c6cd44ad3f6a2463c245b3c
image: vllm/vllm-openai-rocm:kimi-k31-6d74a15@sha256:c470d0de88e8bb664a07006d44e6603a413c9fffd36cc4add4d4b420d4bde5e0
model: moonshotai/Kimi-K3.1-Pro
model-prefix: kimik3.1
runner: cluster:mi355x-amds
Expand All @@ -987,9 +987,10 @@ kimik3.1-fp4-mi355x-vllm-agentic-dspark:
# 0.6 resolves to 1799 GB of host DRAM, which the recipe's Simple CPU offload splits across the 8 TP ranks.
- dram-utilization: 0.6
search-space:
# The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. c1 keeps the KV cache on the GPUs.
# The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. c1 and c4 keep the KV cache on the GPUs.
- { tp: 8, ep: 1, dcp-size: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml }
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 16, 24, 32, 48], srt-recipe: agentic.yaml }
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [4], srt-recipe: agentic.yaml }
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [16, 24, 32, 48], srt-recipe: agentic.yaml }

# MiniMax-M3.1-Flash AgentX on MI355X with SGLang: day-zero image from the SGLang
# cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1),
Expand Down
16 changes: 16 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12855,3 +12855,19 @@
- "The new ATOM brings DeepSeek-V4.1 DP attention (ROCm/ATOM#2376), the V4.1 MXFP8 KV cache with DSpark on an FP8 cache (ROCm/ATOM#2725), DP-attention prefill balancing and graph replay (ROCm/ATOM#2551, #2696, #2697) and the default HIP Mega-MHC kernel on gfx950 (ROCm/ATOM#2568)."
- "No change to the DSpark draft's weight precision: ATOM loads the checkpoint's mtp.0-2 layers as shipped (FP8 E4M3 attention, shared experts and main_proj with 32x32 UE8M0 scales, MXFP4 routed experts, BF16 router, norms and heads) at both images. The draft shares the target's KV pool, so its KV cache follows the target from BF16 to MXFP8."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3965

- config-keys:
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
scenario-type:
- agentic-coding
description:
- "Update the image from vllm/vllm-openai-rocm:kimi-k31-d73dd529 (sha256:5b54f223) to vllm/vllm-openai-rocm:kimi-k31-6d74a15 (sha256:c470d0de), vLLM mum-tracking 6d74a152f."
- "Split concurrency 4 out of the DRAM-offload row into its own TP8 x DCP8 row with kv-offloading none (recipe override override_dram_c4 renamed override_gpu_kv_c4). Concurrency 1 (TP8 x DCP1, GPU KV) and 16, 24, 32 and 48 (TP8 x DCP8, Simple CPU DRAM offload) keep their layouts. Remove override_dram_c10, unused since concurrency 10 left the sweep."
- "Every point sets VLLM_ROCM_DCP_VUP_RS=1 (DCP V-up reduce-scatter, on top of the existing VLLM_ROCM_DCP_AITER_ALLGATHER=1), VLLM_ROCM_DSA_MULTISTREAM=1 and VLLM_ROCM_MEM_HEADROOM_GIB=8."
- "Concurrency 1 (latency_specialized decode backend) sets VLLM_KIMI_SD_DSA_PROLOGUE=1, which runs each DSA layer's decode prologue in one launch."
- "Concurrency 4 keeps max-num-seqs 2 and switches to the latency_specialized decode backend, 5 draft tokens (golden AL 3.41, from 7), decode-only capture sizes 6 and 12 and breakable CUDA graphs off."
- "Concurrency 16, 24 and 32 set prefill-schedule-interval 32 (from 1, 1 and 8) and VLLM_PREFILL_CADENCE_NO_ESCAPE=1. Their max-num-seqs is below the concurrency, so requests always wait; without the flag vLLM drops the prefill hold (capacity escape). Concurrency 32 drops long-prefill-token-threshold 10240 and long-prefill-token-threshold-adaptive."
- "Concurrency 48 uses 3 draft tokens (golden AL 2.88, from 6) with decode capture sizes 4-192, and drops prefill-schedule-interval 8, long-prefill-token-threshold 10240 and long-prefill-token-threshold-adaptive (back to vLLM defaults)."
- "The new vLLM (mum-tracking d73dd529 to 6d74a152f) zeroes the KV null block after CUDA graph capture in the V2 runner (seungrokj/mum#17) and adds the opt-in VLLM_PREFILL_CADENCE_NO_ESCAPE hold (seungrokj/mum#21), the DCP V-up-first combine and multi-stream DSA decode, VLLM_ROCM_MEM_HEADROOM_GIB and the one-launch DSA decode prologue. It also fuses the MXFP4 indexer decode prep, the q_a/kv_a RMSNorm with the sigmoid gate, the KDA f_b_proj GEMM, the AttnRes register tile and the latent-MoE norm, compiles the FlyDSL sparse MLA at load and sizes the packed DSA KV blocks tightly."
- "No change to the DSpark draft's data types: speculative-config does not pin a revision, so Inferact/Kimi-K3.1-Pro-DSpark-preview resolves to Hugging Face main 6a2ef20a (current since 2026-11-05), whose weights are all BF16 with no quantization_config. vLLM 6d74a152f loads it unquantized with an FP8 KV cache, as d73dd529 did; VLLM_ROCM_DCP_VUP_RS also combines the draft's DCP attention output, in BF16 with an FP32 LSE."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/5043
Loading