diff --git a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml index 09839520ae..3909562ea0 100644 --- a/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml +++ b/inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp/agentic.yaml @@ -72,6 +72,8 @@ base: VLLM_ROCM_DSA_MULTISTREAM: '1' # VRAM reserve for unprofiled DCP prefill transients. VLLM_ROCM_MEM_HEADROOM_GIB: '8' + # Token-shard the residual stream across TP for prefill batches of >= 4096 tokens. + VLLM_K3_SP_PREFILL: '1' VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1' # AITER reads its own variable for the same kernels. AITER_FLYDSL_SPARSE_MLA: '1' diff --git a/inferencex-e2e/configs/amd-master.yaml b/inferencex-e2e/configs/amd-master.yaml index 4aa163afcf..ee6fbf74c1 100644 --- a/inferencex-e2e/configs/amd-master.yaml +++ b/inferencex-e2e/configs/amd-master.yaml @@ -974,7 +974,7 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark: - { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [32, 64], srt-recipe: agentic.yaml } kimik3.1-fp4-mi355x-vllm-agentic-dspark: - image: vllm/vllm-openai-rocm:kimi-k31-6d74a15@sha256:c470d0de88e8bb664a07006d44e6603a413c9fffd36cc4add4d4b420d4bde5e0 + image: vllm/vllm-openai-rocm:kimi-k31-5c14fe8@sha256:a61996b4c270ec09111802f62b44f5c20ad4cb73068be022b2804a287b924b9a model: moonshotai/Kimi-K3.1-Pro model-prefix: kimik3.1 runner: cluster:mi355x-amds diff --git a/inferencex-e2e/perf-changelog.yaml b/inferencex-e2e/perf-changelog.yaml index 480a19aeeb..d8f719bf8e 100644 --- a/inferencex-e2e/perf-changelog.yaml +++ b/inferencex-e2e/perf-changelog.yaml @@ -12907,3 +12907,14 @@ - "Deployment per count: 1 GPU runs both CFG branches with Ulysses 1 and no VAE patch parallelism; 2 GPUs CFG parallel 2 x Ulysses 1; 4 GPUs CFG parallel 2 x Ulysses 2; 8 GPUs CFG parallel 2 x Ulysses 4. Tiled VAE patch parallelism spans every GPU from 2 GPUs up." - "Each server streams the Wan snapshot once before vLLM-Omni loads it (VIDEO_PREFETCH_MODEL=1): b300-dsxe serves the model from a network-filesystem HF cache, where the engine's multi-thread shard loader alone missed the one-hour readiness window." pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/5082 + +- config-keys: + - kimik3.1-fp4-mi355x-vllm-agentic-dspark + scenario-type: + - agentic-coding + description: + - "Update the image from vllm/vllm-openai-rocm:kimi-k31-6d74a15 (sha256:c470d0de) to vllm/vllm-openai-rocm:kimi-k31-5c14fe8 (sha256:a61996b4), vLLM mum-tracking 5c14fe8de, which adds seungrokj/mum#26 and seungrokj/mum#28 on top of 6d74a152f." + - "Every point sets VLLM_K3_SP_PREFILL=1 (seungrokj/mum#26): prefill batches of at least 4096 tokens keep the attn-res residual stream token-sharded across TP, replacing each layer's three all-reduces with reduce-scatters and all-gathers over symmetric memory that sum the TP partials in FP32 in a fixed order." + - "Concurrency 1 and 4 (latency_specialized decode backend) pick up the seungrokj/mum#28 levers through their defaults (all 1): VLLM_K31_BEAT_MARKOV, VLLM_K31_BEAT_DRAFTMETA, VLLM_K31_BEAT_K2SYNC, VLLM_K31_BEAT_EAGERPREP, VLLM_K31_BEAT_KDAK1 and VLLM_K3_INDEXER_SAMPLED_TOPK, plus VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER at concurrency 1, where VLLM_KIMI_SD_DSA_PROLOGUE is on. Concurrency 16-48 use the stock decode backend, which ignores these flags." + - "No change to layouts, draft token counts, capture sizes or the DSpark draft's data types." + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/5089