Skip to content

Commit e85978a

Browse files
seungrokjclaude
andcommitted
perf(amd): Kimi-K3.1 MI355X vLLM AgentX: image kimi-k31-5c14fe8, SP prefill, c1/c4 decode levers
Bump the image to vllm/vllm-openai-rocm:kimi-k31-5c14fe8 (mum-tracking 5c14fe8de: seungrokj/mum#26 and seungrokj/mum#28 on top of 6d74a152f). Enable VLLM_K3_SP_PREFILL on every point, and set the latency_specialized decode levers explicitly at concurrency 1 and 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 879bb37 commit e85978a

3 files changed

Lines changed: 30 additions & 1 deletion

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,8 @@ base:
7272
VLLM_ROCM_DSA_MULTISTREAM: '1'
7373
# VRAM reserve for unprofiled DCP prefill transients.
7474
VLLM_ROCM_MEM_HEADROOM_GIB: '8'
75+
# Token-shard the residual stream across TP for prefill batches of >= 4096 tokens.
76+
VLLM_K3_SP_PREFILL: '1'
7577
VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1'
7678
# AITER reads its own variable for the same kernels.
7779
AITER_FLYDSL_SPARSE_MLA: '1'
@@ -120,6 +122,15 @@ override_gpu_kv_c1:
120122
VLLM_USE_BREAKABLE_CUDAGRAPH: '0'
121123
# Runs each DSA layer's decode prologue as one launch; needs DCP1.
122124
VLLM_KIMI_SD_DSA_PROLOGUE: '1'
125+
# D1a also runs the MXFP4 indexer prep; needs VLLM_KIMI_SD_DSA_PROLOGUE.
126+
VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER: '1'
127+
# latency_specialized decode levers; read only under that backend.
128+
VLLM_K31_BEAT_MARKOV: '1'
129+
VLLM_K31_BEAT_DRAFTMETA: '1'
130+
VLLM_K31_BEAT_K2SYNC: '1'
131+
VLLM_K31_BEAT_EAGERPREP: '1'
132+
VLLM_K31_BEAT_KDAK1: '1'
133+
VLLM_K3_INDEXER_SAMPLED_TOPK: '1'
123134
benchmark:
124135
env:
125136
CONC: '1'
@@ -137,6 +148,13 @@ override_gpu_kv_c4:
137148
env:
138149
# Decode-only capture sizes, as at conc 1.
139150
VLLM_USE_BREAKABLE_CUDAGRAPH: '0'
151+
# latency_specialized decode levers; read only under that backend.
152+
VLLM_K31_BEAT_MARKOV: '1'
153+
VLLM_K31_BEAT_DRAFTMETA: '1'
154+
VLLM_K31_BEAT_K2SYNC: '1'
155+
VLLM_K31_BEAT_EAGERPREP: '1'
156+
VLLM_K31_BEAT_KDAK1: '1'
157+
VLLM_K3_INDEXER_SAMPLED_TOPK: '1'
140158
benchmark:
141159
env:
142160
CONC: '4'

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -974,7 +974,7 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark:
974974
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [32, 64], srt-recipe: agentic.yaml }
975975

976976
kimik3.1-fp4-mi355x-vllm-agentic-dspark:
977-
image: vllm/vllm-openai-rocm:kimi-k31-6d74a15@sha256:c470d0de88e8bb664a07006d44e6603a413c9fffd36cc4add4d4b420d4bde5e0
977+
image: vllm/vllm-openai-rocm:kimi-k31-5c14fe8@sha256:a61996b4c270ec09111802f62b44f5c20ad4cb73068be022b2804a287b924b9a
978978
model: moonshotai/Kimi-K3.1-Pro
979979
model-prefix: kimik3.1
980980
runner: cluster:mi355x-amds

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12898,3 +12898,14 @@
1289812898
- "Sweep the B200 Wan2.2-T2V-A14B vLLM-Omni baseline on 1, 2, 4 and 8 GPUs at C1 on the same wan22.yaml release-v1 workload, so the dashboard shows a GPU-count curve."
1289912899
- "Deployment per count: 1 GPU runs both CFG branches with Ulysses 1 and no VAE patch parallelism; 2 GPUs CFG parallel 2 x Ulysses 1; 4 GPUs CFG parallel 2 x Ulysses 2; 8 GPUs keeps CFG parallel 2 x Ulysses 4. Tiled VAE patch parallelism spans every GPU from 2 GPUs up."
1290012900
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/5084
12901+
12902+
- config-keys:
12903+
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
12904+
scenario-type:
12905+
- agentic-coding
12906+
description:
12907+
- "Update the image from vllm/vllm-openai-rocm:kimi-k31-6d74a15 (sha256:c470d0de) to vllm/vllm-openai-rocm:kimi-k31-5c14fe8 (sha256:a61996b4), vLLM mum-tracking 5c14fe8de, which adds seungrokj/mum#26 and seungrokj/mum#28 on top of 6d74a152f."
12908+
- "Every point sets VLLM_K3_SP_PREFILL=1 (seungrokj/mum#26): prefill batches of at least 4096 tokens keep the attn-res residual stream token-sharded across TP, replacing each layer's three all-reduces with reduce-scatters and all-gathers over symmetric memory that sum the TP partials in FP32 in a fixed order."
12909+
- "Concurrency 1 and 4 (latency_specialized decode backend) set the seungrokj/mum#28 levers explicitly: VLLM_K31_BEAT_MARKOV, VLLM_K31_BEAT_DRAFTMETA, VLLM_K31_BEAT_K2SYNC, VLLM_K31_BEAT_EAGERPREP, VLLM_K31_BEAT_KDAK1 and VLLM_K3_INDEXER_SAMPLED_TOPK, plus VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER at concurrency 1, where VLLM_KIMI_SD_DSA_PROLOGUE is on. Concurrency 16-48 use the stock decode backend, which ignores these flags."
12910+
- "No change to layouts, draft token counts, capture sizes or the DSpark draft's data types."
12911+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX

0 commit comments

Comments
 (0)