Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ base:
VLLM_ROCM_DSA_MULTISTREAM: '1'
# VRAM reserve for unprofiled DCP prefill transients.
VLLM_ROCM_MEM_HEADROOM_GIB: '8'
# Token-shard the residual stream across TP for prefill batches of >= 4096 tokens.
VLLM_K3_SP_PREFILL: '1'
VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1'
# AITER reads its own variable for the same kernels.
AITER_FLYDSL_SPARSE_MLA: '1'
Expand Down
2 changes: 1 addition & 1 deletion inferencex-e2e/configs/amd-master.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -974,7 +974,7 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark:
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [32, 64], srt-recipe: agentic.yaml }

kimik3.1-fp4-mi355x-vllm-agentic-dspark:
image: vllm/vllm-openai-rocm:kimi-k31-6d74a15@sha256:c470d0de88e8bb664a07006d44e6603a413c9fffd36cc4add4d4b420d4bde5e0
image: vllm/vllm-openai-rocm:kimi-k31-5c14fe8@sha256:a61996b4c270ec09111802f62b44f5c20ad4cb73068be022b2804a287b924b9a
model: moonshotai/Kimi-K3.1-Pro
model-prefix: kimik3.1
runner: cluster:mi355x-amds
Expand Down
11 changes: 11 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12898,3 +12898,14 @@
- "Sweep the B200 Wan2.2-T2V-A14B vLLM-Omni baseline on 1, 2, 4 and 8 GPUs at C1 on the same wan22.yaml release-v1 workload, so the dashboard shows a GPU-count curve."
- "Deployment per count: 1 GPU runs both CFG branches with Ulysses 1 and no VAE patch parallelism; 2 GPUs CFG parallel 2 x Ulysses 1; 4 GPUs CFG parallel 2 x Ulysses 2; 8 GPUs keeps CFG parallel 2 x Ulysses 4. Tiled VAE patch parallelism spans every GPU from 2 GPUs up."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/5084

- config-keys:
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
scenario-type:
- agentic-coding
description:
- "Update the image from vllm/vllm-openai-rocm:kimi-k31-6d74a15 (sha256:c470d0de) to vllm/vllm-openai-rocm:kimi-k31-5c14fe8 (sha256:a61996b4), vLLM mum-tracking 5c14fe8de, which adds seungrokj/mum#26 and seungrokj/mum#28 on top of 6d74a152f."
- "Every point sets VLLM_K3_SP_PREFILL=1 (seungrokj/mum#26): prefill batches of at least 4096 tokens keep the attn-res residual stream token-sharded across TP, replacing each layer's three all-reduces with reduce-scatters and all-gathers over symmetric memory that sum the TP partials in FP32 in a fixed order."
- "Concurrency 1 and 4 (latency_specialized decode backend) pick up the seungrokj/mum#28 levers through their defaults (all 1): VLLM_K31_BEAT_MARKOV, VLLM_K31_BEAT_DRAFTMETA, VLLM_K31_BEAT_K2SYNC, VLLM_K31_BEAT_EAGERPREP, VLLM_K31_BEAT_KDAK1 and VLLM_K3_INDEXER_SAMPLED_TOPK, plus VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER at concurrency 1, where VLLM_KIMI_SD_DSA_PROLOGUE is on. Concurrency 16-48 use the stock decode backend, which ignores these flags."
- "No change to layouts, draft token counts, capture sizes or the DSpark draft's data types."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/5089
Loading