Skip to content

Commit ddf781d

Browse files
cquil11billishyahaoclaudeseungrokj
authored
[K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 (#4806)
* feat(kimik3.1): add Kimi K3.1 Pro vLLM AgentX on MI355X Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification, following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs. GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60. * chore(kimik3.1): link the MI355X vLLM changelog entry to its PR * fix(kimik3.1): keep only documented or common recipe settings Write the per-concurrency points as plain override variants instead of zip overrides. * fix(kimik3.1): leave GPU headroom for pinned scratch on MI355X The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38 needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X recipe did on the same nodes. * perf(kimik3.1): match the apex-1104 sweep's points and batching The MI355X AgentX lane and the reference apex-1104 sweep differed in the points they ran and in batching, so a comparison between them could not attribute anything to the engine build. Align the parts that shape the workload: seven points instead of ten (drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs below concurrency as that sweep sets it, max-num-batched-tokens 16384 except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of host DRAM for the Simple CPU offload. Keep this lane's own precision and speculative settings: 5 DSpark tokens with block rejection at every point, the published Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state, BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and no FP8 indexer override or KDA input-projection quantization. Those are deliberate here, and copying them would change what the runs measure -- synthetic acceptance in particular reports a modelled throughput and makes the output nondeterministic by construction. The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% -- the recipe's cpu_bytes_to_use_per_rank wins. Say so instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(kimik3.1): resolve the MI355X DRAM budget to the recipe's 2577 GB The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to 2577 GB, within the 3 TB per-server limit. Align comments with the settings they describe and correct the changelog: throughput points still run synthetic acceptance at the golden 3.08. * perf(kimik3.1): use 7 DSpark tokens and the specialized decode at conc 1 The conc-1 latency point runs the reference sweep's latency_specialized decode backend with 7 draft tokens; the throughput points stay at 5. K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only and breakable graphs are the only thing that would add the prefill sizes. Turn them off and leave scratch reclaim on, as the image sets it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(kimik3.1): prefill with the AITER FA MLA backend at conc 48 only The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves conc 1-32 on the default. Move it out of base into that point's override. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(kimik3.1): set breakable cudagraphs and scratch reclaim per point The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the conc-1 values into that point's override and leave the base on the sweep's throughput setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(kimik3.1): keep scratch reclaim off on the MI355X DRAM points On this cluster's MEC firmware 38, HSA_NO_SCRATCH_RECLAIM=0 hung RCCL allgathers at 445k-700k context on conc 32 and conc 48, and conc 48 ran out of GPU memory at 0.92. Keep the image's reclaim-off default with 0.90 at conc 4-48 and no async scratch limit there; conc 1 keeps its own settings. * perf(kimik3.1): trim the conc-32 and conc-48 points on MI355X Run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no successful request. Cut the per-step working set at the two largest points: max-num-batched-tokens 10240, gpu-memory-utilization 0.91 and 3 DSpark tokens, with the capture lists rebuilt from those. The other five points keep the reference apex-1104 sweep's settings. Also run MoE stage 2 in MXFP8 as that sweep does, and leave the MLA prefill backend at its default everywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(kimik3.1): keep scratch allocated and step the draft count down MEC firmware 38 on this cluster cannot reclaim RCCL scratch, and the release blocks until the running dispatches finish, which is how the conc-16 and conc-48 points hung on a 600 s allgather. Set HSA_NO_SCRATCH_RECLAIM=1 at every point and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a reclaim that no longer happens. Step the DSpark token count down with concurrency, 7/5/4/3, and rebuild each capture list from it. Size the CPU offload at 224.875 GB per rank, the 1799 GB that dram-utilization 0.6 resolves to. Describe the points on their own terms rather than against another sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers. * fix(kimik3.1-mi355x-vllm): lower gpu-memory-utilization to 0.86 at conc 16-32 With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest. * fix * fix * perf(kimik3.1): refresh the day-0 image and map the offload pool with huge pages THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the 224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55 million of them to register and keep mapped, against 107 thousand 2 MiB pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): sync Kimi-K3.1-Pro MI355X vLLM block with the diff [skip-sweep] - Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2). - TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to). - Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48. - Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens. - Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts. - K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE). - Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real. - No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: refresh PR #4806 for sweep reuse [skip-sweep] --------- Co-authored-by: billishyahao <bill.he@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: seungrokj <seungrok.jung@amd.com>
1 parent 1e7c8c9 commit ddf781d

3 files changed

Lines changed: 231 additions & 0 deletions

File tree

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
base:
2+
schema: 2
3+
name: kimik3.1-fp4-mi355x-vllm-agentic
4+
resources:
5+
gpu_type: mi355x
6+
gpus_per_node: 8
7+
frontend:
8+
type: vllm
9+
enable_multiple_frontends: false
10+
observability:
11+
enabled: false
12+
tachometer:
13+
enabled: false
14+
engine:
15+
type: vllm
16+
connector: null
17+
# A node's first run downloads ~1.56 TB of MXFP4 shards into its local HF cache before loading.
18+
health_check:
19+
interval_seconds: 10
20+
max_attempts: 720
21+
roles:
22+
agg:
23+
nodes: 1
24+
workers: 1
25+
gpus: 8
26+
args:
27+
served-model-name: moonshotai/Kimi-K3.1-Pro
28+
tensor-parallel-size: 8
29+
decode-context-parallel-size: 8
30+
dcp-comm-backend: a2a
31+
cp-kv-cache-interleave-size: 1
32+
trust-remote-code: true
33+
moe-backend: auto
34+
seed: 7
35+
language-model-only: true
36+
load-format: fastsafetensors
37+
enable-auto-tool-choice: true
38+
tool-call-parser: kimi_k3
39+
reasoning-parser: kimi_k3
40+
max-model-len: 1048576
41+
enable-prefix-caching: true
42+
prefix-match-unit: 64
43+
kv-cache-dtype: fp8
44+
# The K3.1 week-1 precision policy requires an FP32 KDA recurrent state.
45+
mamba-ssm-cache-dtype: float32
46+
# The policy's FP4 indexer cache; the image's AITER pads the FP4 indexer pages it needs.
47+
attention-config: '{"indexer_kv_dtype":"mxfp4"}'
48+
additional-config: '{"kda_prefill_backend":"fused"}'
49+
# Prefix-cached prefill chunks end on K3.1's 1152-token packed DSA block, so the budget floors to a multiple of 1152.
50+
max-num-batched-tokens: 16384
51+
# Kernel scratch, up to HSA_SCRATCH_SINGLE_LIMIT_ASYNC per dispatch, is allocated outside this budget, so leave headroom.
52+
gpu-memory-utilization: 0.90
53+
stream-interval: 10
54+
disable-uvicorn-access-log: true
55+
# 224.875 GB per rank; TOTAL_CPU_DRAM_GB (1799) split across the 8 TP ranks.
56+
kv-transfer-config: '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":224875000000,"lazy_offload":false}}'
57+
env:
58+
VLLM_ROCM_USE_AITER: '1'
59+
VLLM_ROCM_USE_AITER_MOE_SITUV2: '1'
60+
SAFETENSORS_FAST_GPU: '1'
61+
VLLM_USE_V2_MODEL_RUNNER: '1'
62+
VLLM_FASTSAFETENSORS_QUEUE_SIZE: '-1'
63+
# K3.1 has no torch.compile submodule, so only breakable graphs capture the prefill
64+
# sizes the throughput points list. The conc-1 override turns this off.
65+
VLLM_USE_BREAKABLE_CUDAGRAPH: '1'
66+
VLLM_ROCM_AITER_MLA_ASM_PADDING: asm
67+
# Verify the draft tokens with the AITER assembly DCP kernel.
68+
VLLM_ROCM_AITER_MLA_DCP_VERIFY: asm
69+
VLLM_ROCM_DCP_AITER_ALLGATHER: '1'
70+
VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA: '1'
71+
# AITER reads its own variable for the same kernels.
72+
AITER_FLYDSL_SPARSE_MLA: '1'
73+
VLLM_ROCM_DSA_DCP_GLOBAL_KEY_PREFILL: '1'
74+
VLLM_ROCM_DSA_DCP_ROWSHARD_PREFILL: '1'
75+
VLLM_USE_DIRECT_DCP_A2A: '0'
76+
VLLM_USE_DIRECT_DCP_Q_GATHER: '0'
77+
VLLM_USE_DIRECT_DCP_KV_GATHER: '0'
78+
# The K3.1 policy and MODELS.md keep all-reduce in BF16, so quick reduce must not quantize.
79+
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: NONE
80+
AITER_FLYDSL_STAGE2_FP8: '0'
81+
AITER_BF16_FP8_MOE_BOUND: '0'
82+
HSA_NO_SCRATCH_RECLAIM: '1'
83+
# Back the offload pool with transparent huge pages. Torch's CPU allocator
84+
# only calls madvise(MADV_HUGEPAGE) when this is set, and the host is in
85+
# madvise mode, so without it the pool is all 4 KiB pages.
86+
THP_MEM_ALLOC_ENABLE: '1'
87+
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: '1200'
88+
PYTHONNOUSERSITE: '1'
89+
VLLM_ENGINE_READY_TIMEOUT_S: '7200'
90+
# Identical prefixes must hash to identical block keys run-to-run.
91+
PYTHONHASHSEED: '42'
92+
benchmark:
93+
env:
94+
AIPERF_HTTP_TCP_USER_TIMEOUT: '900000'
95+
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'vllm:'
96+
97+
# max-num-seqs stays below CONC from c4 to c32, and equals it at c1 and c48. The capture list is
98+
# the decode sizes, multiples of one plus the draft token count, then the prefill ladder.
99+
# Throughput runs replace block rejection with the golden acceptance length.
100+
override_gpu_kv_c1:
101+
roles:
102+
agg:
103+
args:
104+
max-num-seqs: 1
105+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[2,4,6,8,10,12,14,16]}'
106+
additional-config: '{"decode_backend":"latency_specialized","kda_prefill_backend":"fused"}'
107+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":7,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
108+
kv-transfer-config: null
109+
env:
110+
# This point captures only the decode sizes, so breakable graphs buy nothing.
111+
VLLM_USE_BREAKABLE_CUDAGRAPH: '0'
112+
benchmark:
113+
env:
114+
CONC: '1'
115+
KV_OFFLOADING: 'none'
116+
117+
override_dram_c4:
118+
roles:
119+
agg:
120+
args:
121+
max-num-seqs: 2
122+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,16,24,32,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
123+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
124+
benchmark:
125+
env:
126+
CONC: '4'
127+
KV_OFFLOADING: 'dram'
128+
TOTAL_CPU_DRAM_GB: '1799'
129+
130+
override_dram_c10:
131+
roles:
132+
agg:
133+
args:
134+
max-num-seqs: 6
135+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[6,12,18,24,30,36,48,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
136+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":5,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
137+
benchmark:
138+
env:
139+
CONC: '10'
140+
KV_OFFLOADING: 'dram'
141+
TOTAL_CPU_DRAM_GB: '1799'
142+
143+
override_dram_c16:
144+
roles:
145+
agg:
146+
args:
147+
max-num-seqs: 10
148+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,64,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
149+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
150+
benchmark:
151+
env:
152+
CONC: '16'
153+
KV_OFFLOADING: 'dram'
154+
TOTAL_CPU_DRAM_GB: '1799'
155+
156+
override_dram_c24:
157+
roles:
158+
agg:
159+
args:
160+
max-num-seqs: 16
161+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":16384,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240,12288,14336,16384]}'
162+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
163+
benchmark:
164+
env:
165+
CONC: '24'
166+
KV_OFFLOADING: 'dram'
167+
TOTAL_CPU_DRAM_GB: '1799'
168+
169+
override_dram_c32:
170+
roles:
171+
agg:
172+
args:
173+
max-num-seqs: 24
174+
max-num-batched-tokens: 10240
175+
gpu-memory-utilization: 0.91
176+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,160,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
177+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
178+
benchmark:
179+
env:
180+
CONC: '32'
181+
KV_OFFLOADING: 'dram'
182+
TOTAL_CPU_DRAM_GB: '1799'
183+
184+
override_dram_c48:
185+
roles:
186+
agg:
187+
args:
188+
max-num-seqs: 48
189+
max-num-batched-tokens: 10240
190+
gpu-memory-utilization: 0.91
191+
compilation-config: '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","max_cudagraph_capture_size":10240,"custom_ops":["+fused_rms_norm_gated"],"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128,132,136,140,144,148,152,156,160,164,168,172,176,180,184,188,192,224,256,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}'
192+
speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark-preview","num_speculative_tokens":3,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
193+
benchmark:
194+
env:
195+
CONC: '48'
196+
KV_OFFLOADING: 'dram'
197+
TOTAL_CPU_DRAM_GB: '1799'
198+

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -926,6 +926,24 @@ minimaxm31flash-fp4-mi355x-vllm-agentic-dspark:
926926
- { tp: 4, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32], srt-recipe: agentic.yaml }
927927
- { tp: 4, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, spec-decoding: mtp, conc-list: [64], srt-recipe: agentic.yaml:override_tp4_dspark3_c64_offload }
928928

929+
kimik3.1-fp4-mi355x-vllm-agentic-dspark:
930+
image: vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1
931+
model: moonshotai/Kimi-K3.1-Pro
932+
model-prefix: kimik3.1
933+
runner: cluster:mi355x-amds
934+
precision: fp4
935+
framework: vllm
936+
multinode: false
937+
srt-recipe-dir: kimik3.1/vllm/mi355x-fp4-mtp
938+
scenarios:
939+
agentic-coding:
940+
# 0.6 resolves to 1799 GB of host DRAM, which the recipe's Simple CPU offload splits across the 8 TP ranks.
941+
- dram-utilization: 0.6
942+
search-space:
943+
# The ~1.56 TB MXFP4 checkpoint needs all 8 GPUs. c1 keeps the KV cache on the GPUs.
944+
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1], srt-recipe: agentic.yaml }
945+
- { tp: 8, ep: 1, dcp-size: 8, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: vllm-simple }, conc-list: [4, 10, 16, 24, 32, 48], srt-recipe: agentic.yaml }
946+
929947
# MiniMax-M3.1-Flash AgentX on MI355X with SGLang: day-zero image from the SGLang
930948
# cookbook (https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1),
931949
# manifest digest verified 2026-10-24. Serve flags follow the cookbook's

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12035,6 +12035,21 @@
1203512035
- "Add Kimi-K3.1-Pro day-0 vLLM AgentX on B300: TP8 x DCP8 with DSpark (Inferact/Kimi-K3.1-Pro-DSpark-preview) and Simple CPU KV offload, conc 1-70."
1203612036
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4796
1203712037

12038+
- config-keys:
12039+
- kimik3.1-fp4-mi355x-vllm-agentic-dspark
12040+
scenario-type:
12041+
- agentic-coding
12042+
description:
12043+
- "Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2)."
12044+
- "TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to)."
12045+
- "Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48."
12046+
- "Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens."
12047+
- "Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts."
12048+
- "K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE)."
12049+
- "Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real."
12050+
- "No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's."
12051+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4806
12052+
1203812053
- config-keys:
1203912054
- kimik3.1-fp4-mi355x-sglang-agentic-dflash
1204012055
scenario-type:

0 commit comments

Comments
 (0)