Repository navigation
Commit ddf781d
[K3.1-Pro] Day-0 support vLLM on MI355X / [K3.1-Pro] MI355X 上 vLLM 的 Day-0 支持 (#4806)
* feat(kimik3.1): add Kimi K3.1 Pro vLLM AgentX on MI355X
Single-node TP8 x DCP8 with the Inferact DSpark draft and block verification,
following vLLM's MI355X K3.1 recipe without its three reduced-precision knobs.
GPU-resident KV at conc 1 and 4; Simple CPU KV offload at conc 4-60.
* chore(kimik3.1): link the MI355X vLLM changelog entry to its PR
* fix(kimik3.1): keep only documented or common recipe settings
Write the per-concurrency points as plain override variants instead of zip overrides.
* fix(kimik3.1): leave GPU headroom for pinned scratch on MI355X
The image keeps HSA_NO_SCRATCH_RECLAIM=1, which the cluster's MEC firmware 38
needs, so large FP4-indexer prefill scratch stays pinned outside vLLM's memory
budget. At 0.92 the DRAM-offload points ran out of GPU memory in long-context
prefill (HSA_STATUS_ERROR_OUT_OF_RESOURCES). Use 0.90, as the Kimi K3 MI355X
recipe did on the same nodes.
* perf(kimik3.1): match the apex-1104 sweep's points and batching
The MI355X AgentX lane and the reference apex-1104 sweep differed in the
points they ran and in batching, so a comparison between them could not
attribute anything to the engine build.
Align the parts that shape the workload: seven points instead of ten
(drop the GPU-resident conc 4 and the conc 52/60 tail), max-num-seqs
below concurrency as that sweep sets it, max-num-batched-tokens 16384
except 10240 at conc 48, its cudagraph capture sizes, and 2577 GB of
host DRAM for the Simple CPU offload.
Keep this lane's own precision and speculative settings: 5 DSpark tokens
with block rejection at every point, the published
Inferact/Kimi-K3.1-Pro-DSpark-preview draft, FP32 KDA recurrent state,
BF16 all-reduce, BF16 MoE stage 2, the AITER FA MLA prefill backend, and
no FP8 indexer override or KDA input-projection quantization. Those are
deliberate here, and copying them would change what the runs measure --
synthetic acceptance in particular reports a modelled throughput and
makes the output nondeterministic by construction.
The dram-utilization comment claimed 0.6 resolves to 1799 GB. The nodes
report 3000 GB, so the pinned 2577 GB is 86% of host DRAM, not 60% --
the recipe's cpu_bytes_to_use_per_rank wins. Say so instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(kimik3.1): resolve the MI355X DRAM budget to the recipe's 2577 GB
The planner derives TOTAL_CPU_DRAM_GB from dram-utilization and rejected the
recipe's 2577 GB against the 1799 GB that 0.6 resolves to. 0.8592 resolves to
2577 GB, within the 3 TB per-server limit.
Align comments with the settings they describe and correct the changelog:
throughput points still run synthetic acceptance at the golden 3.08.
* perf(kimik3.1): use 7 DSpark tokens and the specialized decode at conc 1
The conc-1 latency point runs the reference sweep's latency_specialized decode
backend with 7 draft tokens; the throughput points stay at 5.
K3.1 has no torch.compile submodule, so FULL_AND_PIECEWISE captures decode only
and breakable graphs are the only thing that would add the prefill sizes. Turn
them off and leave scratch reclaim on, as the image sets it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(kimik3.1): prefill with the AITER FA MLA backend at conc 48 only
The apex-1104 sweep sets mla_prefill_backend on its conc-48 point and leaves
conc 1-32 on the default. Move it out of base into that point's override.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(kimik3.1): set breakable cudagraphs and scratch reclaim per point
The apex-1104 sweep turns both off at conc 1 and on at conc 4-48. Move the
conc-1 values into that point's override and leave the base on the sweep's
throughput setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(kimik3.1): keep scratch reclaim off on the MI355X DRAM points
On this cluster's MEC firmware 38, HSA_NO_SCRATCH_RECLAIM=0 hung RCCL
allgathers at 445k-700k context on conc 32 and conc 48, and conc 48 ran out
of GPU memory at 0.92. Keep the image's reclaim-off default with 0.90 at
conc 4-48 and no async scratch limit there; conc 1 keeps its own settings.
* perf(kimik3.1): trim the conc-32 and conc-48 points on MI355X
Run 41954410041's conc-48 point died on a 600 s RCCL allgather timeout with no
successful request. Cut the per-step working set at the two largest points:
max-num-batched-tokens 10240, gpu-memory-utilization 0.91 and 3 DSpark tokens,
with the capture lists rebuilt from those. The other five points keep the
reference apex-1104 sweep's settings.
Also run MoE stage 2 in MXFP8 as that sweep does, and leave the MLA prefill
backend at its default everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(kimik3.1): keep scratch allocated and step the draft count down
MEC firmware 38 on this cluster cannot reclaim RCCL scratch, and the release
blocks until the running dispatches finish, which is how the conc-16 and
conc-48 points hung on a 600 s allgather. Set HSA_NO_SCRATCH_RECLAIM=1 at
every point and drop HSA_SCRATCH_SINGLE_LIMIT_ASYNC, which only bounds a
reclaim that no longer happens.
Step the DSpark token count down with concurrency, 7/5/4/3, and rebuild each
capture list from it. Size the CPU offload at 224.875 GB per rank, the 1799 GB
that dram-utilization 0.6 resolves to.
Describe the points on their own terms rather than against another sweep.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(synthetic-acceptance): drop Kimi vLLM parsers on synthetic throughput servers
Synthetic acceptance streams unverified draft tokens. The kimi_k3 tool and reasoning parsers stop streaming on some of them, which empties or truncates the response the client sees. Evals and acceptance gates verify for real and keep their parsers.
* fix(kimik3.1-mi355x-vllm): lower gpu-memory-utilization to 0.86 at conc 16-32
With HSA_NO_SCRATCH_RECLAIM=1 the kernel scratch of these points filled the VRAM left outside the 0.90-0.91 budget during warmup and stalled RCCL, even on freshly rebooted nodes. Their GPU KV use peaks well under the smaller pool, and DRAM offload backs the rest.
* fix
* fix
* perf(kimik3.1): refresh the day-0 image and map the offload pool with huge pages
THP_MEM_ALLOC_ENABLE=1 is what makes torch's CPU allocator call
madvise(MADV_HUGEPAGE). The hosts run THP in madvise mode, so without it the
224 GB per-rank offload pool is backed entirely by 4 KiB pages: about 55
million of them to register and keep mapped, against 107 thousand 2 MiB pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(changelog): sync Kimi-K3.1-Pro MI355X vLLM block with the diff [skip-sweep]
- Add Kimi-K3.1-Pro vLLM AgentX on one 8x MI355X node with vLLM's day-0 image vllm/vllm-openai-rocm:kimi-k31@sha256:5f47cc1004d9a7ac65f8a913725edb20c62c99a07e31cbf4ba21741bd0b4f9b1 (vLLM 0.0.0+mum.14d1ec2336, AITER 52846bc2).
- TP8 x DCP8 (a2a DCP backend) with the Inferact/Kimi-K3.1-Pro-DSpark-preview draft: GPU-resident KV at concurrency 1, and SimpleCPUOffloadConnector DRAM offload at concurrency 4, 10, 16, 24, 32 and 48 (224.875 GB per rank, 1799 GB per node, which is what dram-utilization 0.6 resolves to).
- Per point at concurrency 1/4/10/16/24/32/48: max-num-seqs 1/2/6/10/16/24/48 and DSpark draft tokens 7/5/5/4/4/3/3. max-num-batched-tokens is 16384 and gpu-memory-utilization 0.90 through concurrency 24; both change to 10240 and 0.91 at concurrency 32 and 48.
- Every point compiles in mode 3 with FULL_AND_PIECEWISE CUDA graphs. Concurrency 1 runs the latency_specialized decode backend, captures sizes 2-16 and turns breakable CUDA graphs off; the offload points keep VLLM_USE_BREAKABLE_CUDAGRAPH=1 and capture max-num-seqs multiples of (1 + draft tokens) plus a prefill ladder up to max-num-batched-tokens.
- Common settings: FP8 KV cache, FP32 KDA state (mamba-ssm-cache-dtype float32), MXFP4 indexer cache (indexer_kv_dtype mxfp4), fused KDA prefill, prefix caching with prefix-match-unit 64, max-model-len 1048576, fastsafetensors loading, stream-interval 10, AITER ASM DCP verify, AITER DCP allgather, FlyDSL sparse MLA, the image's HSA_NO_SCRATCH_RECLAIM=1, transparent huge pages for the offload pool (THP_MEM_ALLOC_ENABLE=1), and 1200 s execute-model and 7200 s engine-ready timeouts.
- K3.1 week-1 precision policy: linear layers and shared experts stay BF16 (no quantization-config); routed experts run A4W4, MXFP4 weights with FP4 activations, through the SiTUv2 FlyDSL MoE (VLLM_ROCM_USE_AITER_MOE_SITUV2=1, the same as the image's auto default); MoE stage 2 stays BF16 (AITER_FLYDSL_STAGE2_FP8=0); all-reduce stays BF16 (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE).
- Verification is block rejection with probabilistic draft sampling. Throughput runs replace it with the golden acceptance length for the draft length: 3.69 at 7 tokens, 3.41 at 5, 3.19 at 4 and 2.88 at 3. Evals and AL gates verify for real.
- No change to the DSpark draft's precision: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 6a2ef20a) ships all-BF16 weights with no quantization_config, the speculative config sets no quantization, so get_draft_quant_config() returns None and K3DSparkModel loads its layers in BF16 with the target's BF16 embedding and LM head. Its KV cache is FP8, the same as the target's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: refresh PR #4806 for sweep reuse [skip-sweep]
---------
Co-authored-by: billishyahao <bill.he@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: seungrokj <seungrok.jung@amd.com>1 parent 1e7c8c9 commit ddf781d
3 files changed
Lines changed: 231 additions & 0 deletions
File tree
- inferencex-e2e
- benchmarks/single_node/srt-slurm-recipes/kimik3.1/vllm/mi355x-fp4-mtp
- configs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
926 | 926 | | |
927 | 927 | | |
928 | 928 | | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
929 | 947 | | |
930 | 948 | | |
931 | 949 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12035 | 12035 | | |
12036 | 12036 | | |
12037 | 12037 | | |
| 12038 | + | |
| 12039 | + | |
| 12040 | + | |
| 12041 | + | |
| 12042 | + | |
| 12043 | + | |
| 12044 | + | |
| 12045 | + | |
| 12046 | + | |
| 12047 | + | |
| 12048 | + | |
| 12049 | + | |
| 12050 | + | |
| 12051 | + | |
| 12052 | + | |
12038 | 12053 | | |
12039 | 12054 | | |
12040 | 12055 | | |
| |||
0 commit comments