Repository navigation
perf(amd): Kimi-K3.1 MI355X vLLM AgentX: image kimi-k31-5c14fe8, SP prefill on all points, c1/c4 specialized-decode levers / perf(amd):Kimi-K3.1 MI355X vLLM AgentX:镜像 kimi-k31-5c14fe8、所有点启用 SP prefill、c1/c4 specialized decode 优化项 - #5089
Conversation
…refill, c1/c4 decode levers Bump the image to vllm/vllm-openai-rocm:kimi-k31-5c14fe8 (mum-tracking 5c14fe8de: seungrokj/mum#26 and seungrokj/mum#28 on top of 6d74a152f). Enable VLLM_K3_SP_PREFILL on every point, and set the latency_specialized decode levers explicitly at concurrency 1 and 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43717568009 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43717568009 |
|
/use 43717568009 |
|
@seungrokj staged run 43717568009. This run remains available across future |
|
/reuse-sweep-run 43717568009 |
|
recipe: vllm-project/vllm#67817 |
|
Draft data types: no change. The DSpark draft
The image moves from vLLM mum-tracking
The recipe sets no The new code on the draft's path is the specialized-decode fused Markov step ( 中文草稿数据类型:无变化。 DSpark 草稿
镜像从 vLLM mum-tracking
本 recipe 在本 PR 前后的任何点都没有设置 草稿路径上的新代码是 specialized decode 的融合 Markov 步骤( |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://inferencex.semianalysis.com/inference?unofficialRun=43717568009
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://inferencex.semianalysis.com/evaluation?unofficialRun=43717568009
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- recipe: vllm-project/vllm#67817
Signed: billishyahao
❌❌❌ REJECTED ❌❌❌@billishyahao Blocking: there is no proof that any eval ran the new SP-prefill path. Add server-log evidence that it ran during an eval, or rerun an eval that exercises it. ❌ Check 2 (Evals): FAIL — Eval coverage of the conditional
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — Run 43717568009 (attempt 3) on in-PR commit ✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#67817 is MERGED (2026-11-16, ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Latest checklist): PASS — All 20 items of the current template are present and checked. ✅ Check 6 (Upstream images): PASS — The ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No architecture hacks): PASS — No ✅ Check 9 (Chat template): PASS — AgentX replays chat-format agentic traces through the agentx harness, not raw completions. ✅ Check 10 (No engine patches): PASS — The diff and the effective recipe contain no patch files, setup scripts, heredocs, or wheel installs; the pinned image runs as shipped. ✅ Check 11 (Agentic golden AL): PASS — The c32 job log shows ➖ Check 12 (Append-only): N/A — The new changelog entry is not ✅ Check 14 (Pareto coverage): PASS — One curve (kimik3.1 / agentic-coding / mi355x vLLM / fp4 / P90 E2EL, run 43717568009 on ✅ Check 15 (PR description): PASS — The title, body and changelog match the diff and evidence: the image digest, ➖ Check 16 (NVIDIA overview): N/A — The only affected configuration is AMD MI355X. ➖ Check 17 (Metrics publication): N/A — Direct vLLM serving with no Dynamo TRT-LLM path. ✅ Check 18 (Recipe consistency): PASS — The config generates at head, the master entry and recipe agree, and the new digest resolves. All six points are re-tuned, none undominated or removed, and none regress (tput/GPU +1.4% to +6.4%, P90 E2EL -0.9% to -6.3%). Assessed commit: |
|
Re: the Check 2 (Evals) rejection above: is it true that no eval ran the SP-prefill path? "The server-log archive could not be opened": not correct. I downloaded "Only shows configuration, not dispatch": correct. The only SP-related line in both eval server logs ( There is no per-batch log or counter, so the logs cannot show a batch of at least 4096 tokens going through SP. "No proof that any eval ran the path": strictly true, but in practice the path almost certainly ran. Here is what the logs do show:
This is strong inference, not a dispatch log. To close the check, we can either rerun one eval point (c48) on an image that logs or counts SP-prefill dispatches in |
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 8, rawBenchmarkRows 15, coveredByPerJobFiles 7, purgedPoints 0, failedRunRows 2, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 242416, powerPublicationPoints 6, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
MI355X Kimi-K3.1 vLLM AgentX (
kimik3.1-fp4-mi355x-vllm-agentic-dspark), on top of #5043:kimi-k31-6d74a15→kimi-k31-5c14fe8(sha256:a61996b4). This is vLLM mum-tracking5c14fe8de, which adds exactly two changes on top of6d74a152f: sequence-parallel (SP) prefill and the K3.1 specialized-decode levers. The AITER build in the image is unchanged.max-num-seqsand capture sizes are unchanged.VLLM_K3_SP_PREFILL=1(SP prefill). On the TP8 x DCP8 points (c4–c48), prefill batches of at least 4096 tokens keep the attn-res residual stream token-sharded across TP, and each layer's three all-reduces become reduce-scatters and all-gathers over symmetric memory. SP prefill needs DCP8 token sharding, so at c1 (TP8 x DCP1) the server logsKimi-K3 sequence-parallel prefill unsupported by this configurationand keeps the all-reduce path.latency_specializeddecode levers come on through their defaults (all1), so the recipe sets no extra env vars for them:VLLM_K31_BEAT_MARKOV,VLLM_K31_BEAT_DRAFTMETA,VLLM_K31_BEAT_K2SYNC,VLLM_K31_BEAT_EAGERPREP,VLLM_K31_BEAT_KDAK1andVLLM_K3_INDEXER_SAMPLED_TOPK.VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER(also default1) takes effect only at c1, whereVLLM_KIMI_SD_DSA_PROLOGUE=1is already set.Draft data types: no change (evidence).
Inferact/Kimi-K3.1-Pro-DSpark-previewresolves to Hugging Face main187f8f6f, whose weights are identical to6a2ef20aused by #5043: all BF16, noquantization_config. vLLM5c14fe8deloads it unquantized with an FP8 KV cache, as6d74a152fdid. The fused Markov step keeps its BF16 weights with FP32 accumulation.Accuracy-sensitive optimizations: none used.
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE, no quantization overrides, FP32 KDA state cache, and the AITER sampled indexer top-k is exact.Validation: run 43717568009 on
75d74552is green, covering all six points plus GSM8K at c1, c4, c32 and c48.Can not write request bodyerror while the server kept running; attempt 3 on mi355x-amds_05 passed.中文
MI355X Kimi-K3.1 vLLM AgentX(
kimik3.1-fp4-mi355x-vllm-agentic-dspark),基于 #5043:kimi-k31-6d74a15→kimi-k31-5c14fe8(sha256:a61996b4)。即 vLLM mum-tracking5c14fe8de,在6d74a152f之上只新增两项改动:sequence-parallel(SP)prefill 和 K3.1 specialized decode 优化项。镜像中的 AITER 构建不变。max-num-seqs和 capture sizes 均不变。VLLM_K3_SP_PREFILL=1(SP prefill)。在 TP8 x DCP8 的点(c4–c48)上,不少于 4096 token 的 prefill batch 将 attn-res 残差流按 token 在 TP 间切分,每层的三次 all-reduce 改为基于 symmetric memory 的 reduce-scatter 和 all-gather。SP prefill 依赖 DCP8 的 token 切分,因此在 c1(TP8 x DCP1)上服务器日志显示Kimi-K3 sequence-parallel prefill unsupported by this configuration,继续使用 all-reduce 路径。latency_specializeddecode 优化项通过默认值(均为1)启用,recipe 不为其额外设置环境变量:VLLM_K31_BEAT_MARKOV、VLLM_K31_BEAT_DRAFTMETA、VLLM_K31_BEAT_K2SYNC、VLLM_K31_BEAT_EAGERPREP、VLLM_K31_BEAT_KDAK1和VLLM_K3_INDEXER_SAMPLED_TOPK。VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER(默认同为1)仅在 c1 生效,c1 已设置VLLM_KIMI_SD_DSA_PROLOGUE=1。草稿模型数据类型:不变(证据)。
Inferact/Kimi-K3.1-Pro-DSpark-preview解析到 Hugging Face main187f8f6f,其权重与 #5043 使用的6a2ef20a完全相同:全部为 BF16,无quantization_config。vLLM5c14fe8de以非量化方式加载,KV cache 为 FP8,与6d74a152f相同。融合的 Markov 步骤保持 BF16 权重并以 FP32 累加。精度敏感优化:未使用。
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE,无量化覆盖,KDA state cache 为 FP32,AITER 的 sampled indexer top-k 为精确算法。验证: run 43717568009 基于
75d74552,全部通过,覆盖全部六个点以及 c1、c4、c32、c48 的 GSM8K。Can not write request body错误中止,服务器仍在运行;第 3 次尝试在 mi355x-amds_05 上通过。AI model disclosure
claude-opus-5-5(Claude Code): made the recipe, master-config and perf-changelog changes, resolved the image digest, checked that mum-tracking5c14fe8decontains only the SP prefill and specialized-decode lever changes on top of6d74a152f, and wrote this description.claude-opus-5-5(Claude Code, pr-changelog-sync): rewrote the perf-changelog block and this description from the diff, the image and run 43717568009, and checked the draft data types and accuracy-sensitive optimizations in the pinned source.🤖 Generated with Claude Code