Repository navigation
perf(amd): Kimi-K3.1 MI355X vLLM AgentX: image kimi-k31-6d74a15, env flags, c1/c4 specialized decode, c16-c32 prefill cadence, c48 three draft tokens / perf(amd):Kimi-K3.1 MI355X vLLM AgentX:镜像 kimi-k31-6d74a15、环境变量、c1/c4 specialized decode、c16-c32 prefill 节奏、c48 三个草稿 token - #5043
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
f6651b4 to
73c4344
Compare
There was a problem hiding this comment.
Beyond the inline finding, I checked how the bare agentic.yaml c4 row binds. The single-node binder matches variants on CONC and KV_OFFLOADING, so only override_gpu_kv_c4 fits. It keeps the base decode-context-parallel-size: 8, which matches the row's dcp-size: 8. The two CONC=1/none variants would be ambiguous under a bare reference, which is why the c1 rows now name their overrides.
Extended reasoning...
This is a ruled-out note added to the inline finding. The single-node binder in infx/srt_slurm/single_node.py matches each recipe variant on its CONC, KV_OFFLOADING and parallelism fields. That leaves exactly one valid binding for the c4 row and one for each named c1 row.
73c4344 to
64e8bfc
Compare
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43399050596 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43399050596 |
…flags, c1 nst 7/5, c4 GPU KV DCP8 latency_specialized, c48 nst 3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
64e8bfc to
3bec727
Compare
Both concurrency-1 overrides (override_gpu_kv_c1, override_gpu_kv_c1_nst5) run each DSA layer's specialized-decode prologue as one launch (mum PR 3, D1a). It needs DCP1, so the DCP8 c4 override is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove override_gpu_kv_c1_nst5 and its master-config row; c1 runs once with override_gpu_kv_c1 (7 draft tokens), selected by CONC as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cadence Image kimi-k31-6d74a15 (mum-tracking 6d74a152f) adds VLLM_PREFILL_CADENCE_NO_ESCAPE (mum PR 21). c16, c24 and c32 set prefill-schedule-interval 32 with the flag on, since their max-num-seqs is below the concurrency and the capacity escape would drop the hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…refill threshold c48 no longer sets prefill-schedule-interval 8, long-prefill-token-threshold 10240 or long-prefill-token-threshold-adaptive; it uses vLLM's defaults. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
override_dram_c52 and override_dram_c56 copy c48 (TP8 x DCP8, DRAM offload, 3 draft tokens, max-num-batched-tokens 10240, gpu-memory-utilization 0.91) with max-num-seqs equal to the concurrency and decode capture sizes up to max-num-seqs x 4 (208, 224); the DRAM row lists 52 and 56. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
override_dram_c60 copies c48 with max-num-seqs 60 and decode capture sizes up to 240 (max-num-seqs x 4); the DRAM row lists 60. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
c32 no longer sets long-prefill-token-threshold 10240 or long-prefill-token-threshold-adaptive; prefill-schedule-interval 32 and VLLM_PREFILL_CADENCE_NO_ESCAPE=1 stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/use 43399050596 |
|
/reuse-sweep-run 43399050596 |
|
@seungrokj staged run 43399050596. This run remains available across future |
|
recipe: vllm-project/vllm#67487 |
- Update the image from vllm/vllm-openai-rocm:kimi-k31-d73dd529 (sha256:5b54f223) to vllm/vllm-openai-rocm:kimi-k31-6d74a15 (sha256:c470d0de), vLLM mum-tracking 6d74a152f. - Split concurrency 4 out of the DRAM-offload row into its own TP8 x DCP8 row with kv-offloading none (recipe override override_dram_c4 renamed override_gpu_kv_c4). Concurrency 1 (TP8 x DCP1, GPU KV) and 16, 24, 32 and 48 (TP8 x DCP8, Simple CPU DRAM offload) keep their layouts. Remove override_dram_c10, unused since concurrency 10 left the sweep. - Every point sets VLLM_ROCM_DCP_VUP_RS=1 (DCP V-up reduce-scatter, on top of the existing VLLM_ROCM_DCP_AITER_ALLGATHER=1), VLLM_ROCM_DSA_MULTISTREAM=1 and VLLM_ROCM_MEM_HEADROOM_GIB=8. - Concurrency 1 (latency_specialized decode backend) sets VLLM_KIMI_SD_DSA_PROLOGUE=1, which runs each DSA layer's decode prologue in one launch. - Concurrency 4 keeps max-num-seqs 2 and switches to the latency_specialized decode backend, 5 draft tokens (golden AL 3.41, from 7), decode-only capture sizes 6 and 12 and breakable CUDA graphs off. - Concurrency 16, 24 and 32 set prefill-schedule-interval 32 (from 1, 1 and 8) and VLLM_PREFILL_CADENCE_NO_ESCAPE=1. Their max-num-seqs is below the concurrency, so requests always wait; without the flag vLLM drops the prefill hold (capacity escape). Concurrency 32 drops long-prefill-token-threshold 10240 and long-prefill-token-threshold-adaptive. - Concurrency 48 uses 3 draft tokens (golden AL 2.88, from 6) with decode capture sizes 4-192, and drops prefill-schedule-interval 8, long-prefill-token-threshold 10240 and long-prefill-token-threshold-adaptive (back to vLLM defaults). - The new vLLM (mum-tracking d73dd529 to 6d74a152f) zeroes the KV null block after CUDA graph capture in the V2 runner (seungrokj/mum#17) and adds the opt-in VLLM_PREFILL_CADENCE_NO_ESCAPE hold (seungrokj/mum#21), the DCP V-up-first combine and multi-stream DSA decode, VLLM_ROCM_MEM_HEADROOM_GIB and the one-launch DSA decode prologue. It also fuses the MXFP4 indexer decode prep, the q_a/kv_a RMSNorm with the sigmoid gate, the KDA f_b_proj GEMM, the AttnRes register tile and the latent-MoE norm, compiles the FlyDSL sparse MLA at load and sizes the packed DSA KV blocks tightly. - No change to the DSpark draft's data types: speculative-config does not pin a revision, so Inferact/Kimi-K3.1-Pro-DSpark-preview resolves to Hugging Face main 6a2ef20a (current since 2026-11-05), whose weights are all BF16 with no quantization_config. vLLM 6d74a152f loads it unquantized with an FP8 KV cache, as d73dd529 did; VLLM_ROCM_DCP_VUP_RS also combines the draft's DCP attention output, in BF16 with an FP32 LSE. 中文: - 将镜像从 vllm/vllm-openai-rocm:kimi-k31-d73dd529(sha256:5b54f223)更新为 vllm/vllm-openai-rocm:kimi-k31-6d74a15(sha256:c470d0de),即 vLLM mum-tracking 6d74a152f。 - 将并发 4 从 DRAM offload 行拆出,单独成为 kv-offloading none 的 TP8 x DCP8 行(recipe override override_dram_c4 改名为 override_gpu_kv_c4)。并发 1(TP8 x DCP1,KV 在 GPU 上)以及 16、24、32、48(TP8 x DCP8,Simple CPU DRAM offload)的布局不变。删除 override_dram_c10,并发 10 退出扫描后已不再使用。 - 所有点设置 VLLM_ROCM_DCP_VUP_RS=1(DCP V-up reduce-scatter,叠加在已有的 VLLM_ROCM_DCP_AITER_ALLGATHER=1 之上)、VLLM_ROCM_DSA_MULTISTREAM=1 和 VLLM_ROCM_MEM_HEADROOM_GIB=8。 - 并发 1(latency_specialized decode 后端)设置 VLLM_KIMI_SD_DSA_PROLOGUE=1,将每个 DSA 层的 decode prologue 合并为一次 launch。 - 并发 4 保持 max-num-seqs 2,改用 latency_specialized decode 后端、5 个草稿 token(golden AL 3.41,原为 7)、仅 decode 的 capture sizes 6 和 12,并关闭 breakable CUDA graph。 - 并发 16、24、32 设置 prefill-schedule-interval 32(原为 1、1、8)和 VLLM_PREFILL_CADENCE_NO_ESCAPE=1。它们的 max-num-seqs 低于并发数,请求始终在排队;不设此标志时 vLLM 会放弃 prefill 保持(capacity escape)。并发 32 去掉 long-prefill-token-threshold 10240 和 long-prefill-token-threshold-adaptive。 - 并发 48 使用 3 个草稿 token(golden AL 2.88,原为 6),decode capture sizes 为 4-192,并去掉 prefill-schedule-interval 8、long-prefill-token-threshold 10240 和 long-prefill-token-threshold-adaptive(恢复 vLLM 默认值)。 - 新 vLLM(mum-tracking d73dd529 到 6d74a152f)在 V2 runner 中于 CUDA graph capture 之后将 KV null block 清零(seungrokj/mum#17),并新增可选的 VLLM_PREFILL_CADENCE_NO_ESCAPE 保持(seungrokj/mum#21)、DCP V-up-first 合并与多流 DSA decode、VLLM_ROCM_MEM_HEADROOM_GIB 以及单次 launch 的 DSA decode prologue。它还融合了 MXFP4 indexer decode 准备、q_a/kv_a RMSNorm 与 sigmoid gate、KDA f_b_proj GEMM、AttnRes register tile 和 latent-MoE norm,在加载时编译 FlyDSL sparse MLA,并收紧 packed DSA KV block 的大小。 - DSpark 草稿的数据类型无变化:speculative-config 未固定 revision,因此 Inferact/Kimi-K3.1-Pro-DSpark-preview 解析为 Hugging Face main 6a2ef20a(自 2026-11-05 起为当前版本),其权重全部为 BF16,没有 quantization_config。vLLM 6d74a152f 不做量化地加载它,KV cache 为 FP8,与 d73dd529 相同;VLLM_ROCM_DCP_VUP_RS 也用于合并草稿的 DCP attention 输出,使用 BF16 和 FP32 LSE。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Draft data types: no change. The DSpark draft
The image moves from vLLM
The recipe sets no Accuracy-sensitive optimizations: none used. Checked against the accuracy-sensitive optimization policy (#4705, not yet merged) and the Kimi K3.1 Pro week-1 precision policy. The setup is aggregated TP8 (DCP1 at c1, DCP8 elsewhere) with the DSpark draft worker in the same engine. I checked image defaults at vLLM
No optimization on the list is enabled, so the three-arm KVV qualification does not apply. I found no model-trainer veto in the PR or the linked issues. Validation: run 43399050596 on 中文草稿数据类型:无变化。 DSpark 草稿
镜像从 vLLM
本 recipe 在本 PR 前后的任何点都没有设置 精度敏感优化:均未使用。 依据精度敏感优化策略(#4705,尚未合并)和 Kimi K3.1 Pro 第一周精度策略进行检查。部署为聚合式 TP8(c1 为 DCP1,其余为 DCP8),DSpark 草稿 worker 位于同一引擎中。检查范围包括 vLLM
列表中的优化均未启用,因此不需要三组 KVV 资格验证。在 PR 及其关联 issue 中未发现模型训练方否决。验证:run 43399050596,运行于 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Draft data types and accuracy-sensitive optimizations: #5043 (comment)
- Recipes:
Signed: seungrokj
|
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 7, rawBenchmarkRows 12, coveredByPerJobFiles 6, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 257456, powerPublicationPoints 6, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
MI355X Kimi-K3.1 vLLM AgentX (
kimik3.1-fp4-mi355x-vllm-agentic-dspark):kimi-k31-d73dd529→kimi-k31-6d74a15(vLLM mum-tracking6d74a152f: KV null-block fix seungrokj/mum#17, prefill-cadence hold seungrokj/mum#21, DCP V-up-first combine, multi-stream DSA decode, memory headroom, one-launch DSA decode prologue, and Kimi-K3.1 decode fusions).VLLM_ROCM_DCP_VUP_RS=1,VLLM_ROCM_DSA_MULTISTREAM=1,VLLM_ROCM_MEM_HEADROOM_GIB=8.VLLM_KIMI_SD_DSA_PROLOGUE=1.latency_specializeddecode, 5 draft tokens, no KV offload, breakable graphs off.prefill-schedule-interval,long-prefill-token-thresholdandlong-prefill-token-threshold-adaptive.prefill-schedule-interval: 32+VLLM_PREFILL_CADENCE_NO_ESCAPE=1(seungrokj/mum#21).override_dram_c10.Draft data types: no change.
Inferact/Kimi-K3.1-Pro-DSpark-previewresolves to Hugging Facemain6a2ef20a, whose weights are all BF16 with noquantization_config. vLLM6d74a152floads it unquantized with an FP8 KV cache, asd73dd529did. Evidence: #5043 (comment)Accuracy-sensitive optimizations: none used. Quick reduce is
NONE; the newVLLM_ROCM_DCP_VUP_RSreduce-scatter and theVLLM_ROCM_DSA_MULTISTREAMall-gather run in BF16/FP32 without quantization; there is no online or shared-expert quantization and no cross-indexer sharing; the KV cache is FP8 (the FP4 indexer cache is allowed by the K3.1 week-1 policy); and the KDA state cache is FP32. Same comment as above.Validation: run-sweep 43399050596 on
34099327, whose Kimi-K3.1 recipe and master config match the head. All six benchmark points (c1, c4, c16, c24, c32, c48) and the c1, c4, c32 and c48 evals are green. Results were staged with/reuse-sweep-run 43399050596.中文
MI355X Kimi-K3.1 vLLM AgentX(
kimik3.1-fp4-mi355x-vllm-agentic-dspark):kimi-k31-d73dd529→kimi-k31-6d74a15(vLLM mum-tracking6d74a152f:KV null block 修复 seungrokj/mum#17、prefill 节奏保持 seungrokj/mum#21、DCP V-up-first 合并、多流 DSA decode、显存余量、单次 launch 的 DSA decode prologue,以及 Kimi-K3.1 decode 融合)。VLLM_ROCM_DCP_VUP_RS=1、VLLM_ROCM_DSA_MULTISTREAM=1、VLLM_ROCM_MEM_HEADROOM_GIB=8。VLLM_KIMI_SD_DSA_PROLOGUE=1。latency_specializeddecode,5 个草稿 token,无 KV offload,关闭 breakable graphs。prefill-schedule-interval、long-prefill-token-threshold和long-prefill-token-threshold-adaptive。prefill-schedule-interval: 32+VLLM_PREFILL_CADENCE_NO_ESCAPE=1(seungrokj/mum#21)。override_dram_c10。草稿数据类型:无变化。
Inferact/Kimi-K3.1-Pro-DSpark-preview解析为 Hugging Facemain6a2ef20a,其权重全部为 BF16,没有quantization_config。vLLM6d74a152f不做量化地加载它,KV cache 为 FP8,与d73dd529相同。证据:#5043 (comment)精度敏感优化:均未使用。 quick reduce 为
NONE;新增的VLLM_ROCM_DCP_VUP_RSreduce-scatter 和VLLM_ROCM_DSA_MULTISTREAMall-gather 均以 BF16/FP32 运行,不做量化;没有在线量化或 shared expert 量化,也没有跨 indexer 共享;KV cache 为 FP8(K3.1 第一周精度策略允许 FP4 indexer cache);KDA 状态缓存为 FP32。证据同上。验证:run-sweep 43399050596,运行于
34099327,其 Kimi-K3.1 recipe 和 master config 与 head 相同。六个 benchmark 点(c1、c4、c16、c24、c32、c48)以及 c1、c4、c32、c48 的 eval 全部通过。结果通过/reuse-sweep-run 43399050596暂存。AI model disclosure
claude-opus-5-5(Claude Code): synced the perf-changelog block and this description with the diff, and checked the draft data types and accuracy-sensitive optimizations in the pinned source.🤖 Generated with Claude Code