Skip to content

perf(amd): Kimi-K3.1 MI355X vLLM AgentX: image kimi-k31-5c14fe8, SP prefill on all points, c1/c4 specialized-decode levers / perf(amd):Kimi-K3.1 MI355X vLLM AgentX:镜像 kimi-k31-5c14fe8、所有点启用 SP prefill、c1/c4 specialized decode 优化项 - #5089

Merged
seungrokj merged 4 commits into
mainfrom
seungrokj/kimik3.1-mi355x-vllm-5c14fe8
Nov 16, 2026

Conversation

@seungrokj

@seungrokj seungrokj commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

MI355X Kimi-K3.1 vLLM AgentX (kimik3.1-fp4-mi355x-vllm-agentic-dspark), on top of #5043:

  • Image kimi-k31-6d74a15 → kimi-k31-5c14fe8 (sha256:a61996b4). This is vLLM mum-tracking 5c14fe8de, which adds exactly two changes on top of 6d74a152f: sequence-parallel (SP) prefill and the K3.1 specialized-decode levers. The AITER build in the image is unchanged.
  • No point is added or removed, and draft token counts, max-num-seqs and capture sizes are unchanged.
  • All points: the recipe sets VLLM_K3_SP_PREFILL=1 (SP prefill). On the TP8 x DCP8 points (c4–c48), prefill batches of at least 4096 tokens keep the attn-res residual stream token-sharded across TP, and each layer's three all-reduces become reduce-scatters and all-gathers over symmetric memory. SP prefill needs DCP8 token sharding, so at c1 (TP8 x DCP1) the server logs Kimi-K3 sequence-parallel prefill unsupported by this configuration and keeps the all-reduce path.
  • c1 and c4: the new latency_specialized decode levers come on through their defaults (all 1), so the recipe sets no extra env vars for them: VLLM_K31_BEAT_MARKOV, VLLM_K31_BEAT_DRAFTMETA, VLLM_K31_BEAT_K2SYNC, VLLM_K31_BEAT_EAGERPREP, VLLM_K31_BEAT_KDAK1 and VLLM_K3_INDEXER_SAMPLED_TOPK.
    • VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER (also default 1) takes effect only at c1, where VLLM_KIMI_SD_DSA_PROLOGUE=1 is already set.
    • c16–c48 use the stock decode backend, which ignores these flags.

Draft data types: no change (evidence). Inferact/Kimi-K3.1-Pro-DSpark-preview resolves to Hugging Face main 187f8f6f, whose weights are identical to 6a2ef20a used by #5043: all BF16, no quantization_config. vLLM 5c14fe8de loads it unquantized with an FP8 KV cache, as 6d74a152f did. The fused Markov step keeps its BF16 weights with FP32 accumulation.

Accuracy-sensitive optimizations: none used.

  • SP prefill moves BF16 partials and sums them in FP32 in a fixed order.
  • The routed MoE input is MXFP4-quantized as in the stock routed-input path (MX-style MoE dispatch quantization is allowed for K3.1); SP only quantizes each rank's own rows before the one all-gather.
  • VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE, no quantization overrides, FP32 KDA state cache, and the AITER sampled indexer top-k is exact.

Validation: run 43717568009 on 75d74552 is green, covering all six points plus GSM8K at c1, c4, c32 and c48.

Conc #5043 (run 43399050596) intvty p50 / p90, tok/s/GPU This PR intvty p50 / p90, tok/s/GPU tok/s/GPU
1 239.0 / 231.4, 2033 263.6 / 255.1, 2164 +6.4%
4 180.7 / 164.0, 3076 189.1 / 169.7, 3137 +2.0%
16 102.4 / 81.7, 8901 103.0 / 85.3, 9089 +2.1%
24 81.4 / 66.0, 13198 83.2 / 69.6, 13387 +1.4%
32 66.2 / 56.6, 15331 68.1 / 59.5, 15850 +3.4%
48 38.7 / 23.4, 19201 40.6 / 26.9, 20319 +5.8%
  • c4 attempt 1 (mi355x-amds_01) measured 147.2 / 127.7, 2815 tok/s/GPU (−8.5%); attempt 2 (mi355x-amds_06) is the row above.
  • c24 attempts 1 and 2 on mi355x-amds_02 aborted in warmup with a client Can not write request body error while the server kept running; attempt 3 on mi355x-amds_05 passed.
  • GSM8K strict-match (two attempts each): c1 0.9795 / 0.9803, c4 0.9788 / 0.9818, c32 0.9795 / 0.9795, c48 0.9795 / 0.9788.
中文

MI355X Kimi-K3.1 vLLM AgentX(kimik3.1-fp4-mi355x-vllm-agentic-dspark),基于 #5043:

  • 镜像 kimi-k31-6d74a15 → kimi-k31-5c14fe8(sha256:a61996b4)。即 vLLM mum-tracking 5c14fe8de,在 6d74a152f 之上只新增两项改动:sequence-parallel(SP)prefill 和 K3.1 specialized decode 优化项。镜像中的 AITER 构建不变。
  • 不新增或删除任何点,草稿 token 数、max-num-seqs 和 capture sizes 均不变。
  • 所有点: recipe 设置 VLLM_K3_SP_PREFILL=1(SP prefill)。在 TP8 x DCP8 的点(c4–c48)上,不少于 4096 token 的 prefill batch 将 attn-res 残差流按 token 在 TP 间切分,每层的三次 all-reduce 改为基于 symmetric memory 的 reduce-scatter 和 all-gather。SP prefill 依赖 DCP8 的 token 切分,因此在 c1(TP8 x DCP1)上服务器日志显示 Kimi-K3 sequence-parallel prefill unsupported by this configuration,继续使用 all-reduce 路径。
  • c1 和 c4: 新增的 latency_specialized decode 优化项通过默认值(均为 1)启用,recipe 不为其额外设置环境变量:VLLM_K31_BEAT_MARKOV、VLLM_K31_BEAT_DRAFTMETA、VLLM_K31_BEAT_K2SYNC、VLLM_K31_BEAT_EAGERPREP、VLLM_K31_BEAT_KDAK1 和 VLLM_K3_INDEXER_SAMPLED_TOPK。
    • VLLM_KIMI_SD_DSA_PROLOGUE_INDEXER(默认同为 1)仅在 c1 生效,c1 已设置 VLLM_KIMI_SD_DSA_PROLOGUE=1。
    • c16–c48 使用原有 decode backend,会忽略这些变量。

草稿模型数据类型:不变(证据)。Inferact/Kimi-K3.1-Pro-DSpark-preview 解析到 Hugging Face main 187f8f6f,其权重与 #5043 使用的 6a2ef20a 完全相同:全部为 BF16,无 quantization_config。vLLM 5c14fe8de 以非量化方式加载,KV cache 为 FP8,与 6d74a152f 相同。融合的 Markov 步骤保持 BF16 权重并以 FP32 累加。

精度敏感优化:未使用。

  • SP prefill 传输 BF16 部分和,并以 FP32、按固定顺序求和。
  • routed MoE 输入与原有 routed-input 路径一样做 MXFP4 量化(K3.1 允许 MX 风格的 MoE dispatch 量化);SP 只是在唯一一次 all-gather 之前对各 rank 自身的行做量化。
  • VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE,无量化覆盖,KDA state cache 为 FP32,AITER 的 sampled indexer top-k 为精确算法。

验证: run 43717568009 基于 75d74552,全部通过,覆盖全部六个点以及 c1、c4、c32、c48 的 GSM8K。

并发 #5043(run 43399050596)intvty p50 / p90,tok/s/GPU 本 PR intvty p50 / p90,tok/s/GPU tok/s/GPU
1 239.0 / 231.4,2033 263.6 / 255.1,2164 +6.4%
4 180.7 / 164.0,3076 189.1 / 169.7,3137 +2.0%
16 102.4 / 81.7,8901 103.0 / 85.3,9089 +2.1%
24 81.4 / 66.0,13198 83.2 / 69.6,13387 +1.4%
32 66.2 / 56.6,15331 68.1 / 59.5,15850 +3.4%
48 38.7 / 23.4,19201 40.6 / 26.9,20319 +5.8%
  • c4 第 1 次尝试(mi355x-amds_01)测得 147.2 / 127.7,2815 tok/s/GPU(−8.5%);上表为第 2 次尝试(mi355x-amds_06)。
  • c24 在 mi355x-amds_02 上的第 1、2 次尝试在 warmup 阶段因客户端 Can not write request body 错误中止,服务器仍在运行;第 3 次尝试在 mi355x-amds_05 上通过。
  • GSM8K strict-match(各两次):c1 0.9795 / 0.9803,c4 0.9788 / 0.9818,c32 0.9795 / 0.9795,c48 0.9795 / 0.9788。
AI model disclosure
  • claude-opus-5-5 (Claude Code): made the recipe, master-config and perf-changelog changes, resolved the image digest, checked that mum-tracking 5c14fe8de contains only the SP prefill and specialized-decode lever changes on top of 6d74a152f, and wrote this description.
  • claude-opus-5-5 (Claude Code, pr-changelog-sync): rewrote the perf-changelog block and this description from the diff, the image and run 43717568009, and checked the draft data types and accuracy-sensitive optimizations in the pinned source.

🤖 Generated with Claude Code

…refill, c1/c4 decode levers

Bump the image to vllm/vllm-openai-rocm:kimi-k31-5c14fe8 (mum-tracking
5c14fe8de: seungrokj/mum#26 and seungrokj/mum#28 on top of 6d74a152f).
Enable VLLM_K3_SP_PREFILL on every point, and set the latency_specialized
decode levers explicitly at concurrency 1 and 4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This review covers commit e85978a, which is no longer the latest commit on this pull request; later commits are not covered by it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@seungrokj seungrokj added the non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim) label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@seungrokj

Copy link
Copy Markdown
Collaborator Author

/use 43717568009

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

@seungrokj staged run 43717568009.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@seungrokj

Copy link
Copy Markdown
Collaborator Author

/reuse-sweep-run 43717568009

@seungrokj

Copy link
Copy Markdown
Collaborator Author

recipe: vllm-project/vllm#67817

@seungrokj

seungrokj commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Draft data types: no change. The DSpark draft Inferact/Kimi-K3.1-Pro-DSpark-preview is not pinned to a revision in speculative-config, so it resolves to Hugging Face main. That is now revision 187f8f6f (2026-11-14, a model-card edit). Its model.safetensors (LFS oid 610afd80871f) and config.json are identical to 6a2ef20a, which #5043 used. It is a standalone 5-layer K3DSparkModel checkpoint with no quantization_config, and all 67 tensors in its safetensors header are BF16:

Part Data type
Attention projections (q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj) BF16
Dense MLP (gate_proj, up_proj, down_proj) BF16
context_proj, Markov head, confidence head BF16
Norms BF16
Embedding and LM head not in the draft checkpoint; shared from the target, whose lm_head is on the compressed-tensors ignore list and stays BF16
KV cache FP8, from speculative-config kv_cache_dtype: fp8, the same as the target's kv-cache-dtype: fp8

The image moves from vLLM mum-tracking 6d74a152f79c14ff366f546476b9d542adc290a2 to 5c14fe8de8465efd08ccbe5cde31300381d1094f (kimi-k31-5c14fe8; the image label runtime.mum.commit matches). That range adds only SP prefill and the K3.1 specialized-decode levers, and the AITER overlay commit in the image (cbe3c8f1) is unchanged. These commits live in a non-public repository, so there are no public permalinks. Paths are at 5c14fe8de. None of the draft load-path files below changes between the two commits:

  • vllm/config/speculative.py:1308-1345: the draft inherits the target's quantization only when speculative-config gives no model. Every point names the draft repo, so quantization stays unset. :1495 builds the draft ModelConfig, and :1620 keeps K3DSparkModel out of the DeepSeek DSpark branch that copies the target's quantization (:1636).
  • vllm/config/vllm.py:937: with no quantization, the quant config is None, so get_draft_quant_config() (vllm/model_executor/models/utils.py:1062-1084) returns None.
  • vllm/models/kimi_k3/nvidia/dspark_mla.py:72,145: every draft linear layer takes that None config and loads in BF16. :434-443 shares the target's embed_tokens and lm_head. vllm/model_executor/models/registry.py:702-705 maps K3DSparkModel to this class on ROCm.
  • vllm/config/speculative.py:421-430: only speculative-config's own attention_backend (ROCM_AITER_MLA) and kv_cache_dtype (fp8) override the draft's attention backend and KV dtype.

The recipe sets no quantization-config, --quantization or draft quantization at any point, before or after this PR, so the draft runs as shipped: BF16 weights and activations with an FP8 KV cache.

The new code on the draft's path is the specialized-decode fused Markov step (VLLM_K31_BEAT_MARKOV, c1 only; the c4 log shows off: more than 1 requests). It requires BF16 w1/w2 (vllm/models/kimi_k3/amd/ops/markov_sample.py:253), accumulates in FP32 (:127-136), rounds the bias to BF16 (:138) and adds it in BF16 (:140). That is the same arithmetic as the unfused path; only the summation order of the 512-term dot product differs. SP prefill does not touch the draft model.

中文

草稿数据类型:无变化。 DSpark 草稿 Inferact/Kimi-K3.1-Pro-DSpark-preview 在 speculative-config 中没有固定 revision,因此解析为 Hugging Face main,目前为 revision 187f8f6f(2026-11-14,仅修改 model card)。其 model.safetensors(LFS oid 610afd80871f)和 config.json 与 #5043 所用的 6a2ef20a 完全相同。它是独立的 5 层 K3DSparkModel checkpoint,没有 quantization_config,其 safetensors header 中的 67 个张量全部为 BF16:

部分 数据类型
注意力投影(q_a_proj、q_b_proj、kv_a_proj_with_mqa、kv_b_proj、o_proj) BF16
Dense MLP(gate_proj、up_proj、down_proj) BF16
context_proj、Markov head、confidence head BF16
Norm BF16
Embedding 和 LM head 不在草稿 checkpoint 中;与目标模型共享,目标模型的 lm_head 在 compressed-tensors 的 ignore 列表中,保持 BF16
KV cache FP8,来自 speculative-config 的 kv_cache_dtype: fp8,与目标模型的 kv-cache-dtype: fp8 相同

镜像从 vLLM mum-tracking 6d74a152f79c14ff366f546476b9d542adc290a2 更新为 5c14fe8de8465efd08ccbe5cde31300381d1094f(kimi-k31-5c14fe8,镜像标签 runtime.mum.commit 一致)。该范围只新增 SP prefill 和 K3.1 specialized decode 优化项,镜像中的 AITER overlay commit(cbe3c8f1)不变。这些 commit 位于非公开仓库,因此没有公开的永久链接。以下路径均基于 5c14fe8de,且下列草稿加载路径文件在两个 commit 之间均无改动:

  • vllm/config/speculative.py:1308-1345:只有在 speculative-config 未指定 model 时,草稿才会继承目标模型的量化。每个点都指定了草稿仓库,因此 quantization 保持未设置。:1495 构建草稿的 ModelConfig,:1620 将 K3DSparkModel 排除在会复制目标模型量化的 DeepSeek DSpark 分支(:1636)之外。
  • vllm/config/vllm.py:937:没有量化时量化配置为 None,因此 get_draft_quant_config()(vllm/model_executor/models/utils.py:1062-1084)返回 None。
  • vllm/models/kimi_k3/nvidia/dspark_mla.py:72,145:草稿的每个线性层都使用这个 None 配置,以 BF16 加载。:434-443 共享目标模型的 embed_tokens 和 lm_head。vllm/model_executor/models/registry.py:702-705 在 ROCm 上将 K3DSparkModel 映射到这个类。
  • vllm/config/speculative.py:421-430:只有 speculative-config 自身的 attention_backend(ROCM_AITER_MLA)和 kv_cache_dtype(fp8)会覆盖草稿的注意力后端和 KV dtype。

本 recipe 在本 PR 前后的任何点都没有设置 quantization-config、--quantization 或草稿量化,因此草稿按发布形式运行:BF16 权重和激活,FP8 KV cache。

草稿路径上的新代码是 specialized decode 的融合 Markov 步骤(VLLM_K31_BEAT_MARKOV,仅 c1;c4 日志显示 off: more than 1 requests)。它要求 w1/w2 为 BF16(vllm/models/kimi_k3/amd/ops/markov_sample.py:253),以 FP32 累加(:127-136),将 bias 舍入为 BF16(:138)并以 BF16 相加(:140)。这与非融合路径的运算相同,只有 512 项点积的求和顺序不同。SP prefill 不涉及草稿模型。

@billishyahao billishyahao left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://inferencex.semianalysis.com/inference?unofficialRun=43717568009
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://inferencex.semianalysis.com/evaluation?unofficialRun=43717568009
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

Signed: billishyahao

@github-actions

Copy link
Copy Markdown
Contributor

❌❌❌ REJECTED ❌❌❌

@billishyahao Blocking: there is no proof that any eval ran the new SP-prefill path. Add server-log evidence that it ran during an eval, or rerun an eval that exercises it.

❌ Check 2 (Evals): FAIL — Eval coverage of the conditional VLLM_K3_SP_PREFILL=1 path could not be verified. The path engages only for batches of at least 4096 tokens under DCP8. The GSM8K numbers are fine: on kimi-k31-5c14fe8@sha256:a61996b4, c1 0.9795/0.9803, c4 0.9788/0.9818, c32 0.9795/0.9795 and c48 0.9795/0.9788 are all within 2-sigma of their siblings and the exact cohort (43399050596, 0.9788). However, the c32 eval job log (job 132976534590) shows only that the env var was set, and the eval server-log archive (srt-single-node-logs.tar.gz) could not be opened in this verification environment. The PR and the upstream recipe cite only the startup "sequence-parallel prefill enabled" line, which shows configuration, not dispatch. Fix: cite the c32 or c48 eval server-log lines from run 43717568009 showing SP prefill running on a batch of at least 4096 tokens (or an equivalent counter), or rerun an eval that demonstrably triggers it.

⚠️ Check 13 (Draft precision): WARN — The sign-off's additional detail section does not record the draft's shipped and effective precision; it contains only the recipe link. Independently verified: Inferact/Kimi-K3.1-Pro-DSpark-preview (unpinned, resolves to 187f8f6f) is all BF16 with no quantization_config. No speculative-config sets a quantization or dtype field. The draft's FP8 KV cache matches the target's kv-cache-dtype: fp8. There is no SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE (this is the vLLM path). The pinned image's default draft load path (mum-tracking 5c14fe8de) is non-public, so only the author's comment attests to it. Human review needed; link that evidence in the sign-off.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — billishyahao is a listed owner of inferencex-e2e/configs/amd-master.yaml and of /inferencex-e2e/benchmarks/*_node/srt-slurm-recipes/**/mi355x-*/; perf-changelog.yaml falls to the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — Run 43717568009 (attempt 3) on in-PR commit 75d74552 has all six agentic / points and four agentic eval / jobs successful. The only later commit, merge bdfa086b, leaves the Kimi recipe and master entry byte-identical, keeps this PR's changelog entry, and brings in only unrelated Wan2.2 B300 video configs and docs.

✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#67817 is MERGED (2026-11-16, 1bf414e9), and recipes.vllm.ai syncs vllm-project/vllm recipes/ (sources.lock.json pins 3494bb04, which contains it). The published mi355x_agentx preset matches every point: pinned digest, VLLM_K3_SP_PREFILL=1, DCP8/a2a, offload, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=NONE, per-point draft counts, max-num-seqs, capture sizes, latency_specialized at c1/c4, and prefill cadence. Synthetic acceptance is documented as throughput-only.

✅ Check 4 (Reuse command): PASS — seungrokj (permission write, role write) posted /use 43717568009 and /reuse-sweep-run 43717568009, each on its own line; that is the Check 1 run.

✅ Check 5 (Latest checklist): PASS — All 20 items of the current template are present and checked.

✅ Check 6 (Upstream images): PASS — The framework: vllm entry uses vllm/vllm-openai-rocm:kimi-k31-5c14fe8@sha256:a61996b4 from the upstream vLLM Docker Hub org, and engine-first ordering does not apply to a vLLM entry. The image is a temporary model-enablement build (non-public mum-tracking 5c14fe8de) for Kimi-K3.1. That model was added 2026-11-04 and is 12 days old, so the new-model transition window covers it.

✅ Check 7 (Deprecated models): PASS — kimik3.1 agentic coding is active in MODELS.md (added 2026-11-04, no retirement).

✅ Check 8 (No architecture hacks): PASS — No hf-overrides or config edits. This non-grandfathered model sets VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: NONE, and no other quick-reduce control is set. SP prefill replaces the all-reduces with BF16 reduce-scatter/all-gather summed in FP32, and its MXFP4 routed-input quantization is MoE dispatch, which the K3.1 week-1 policy allows.

✅ Check 9 (Chat template): PASS — AgentX replays chat-format agentic traces through the agentx harness, not raw completions.

✅ Check 10 (No engine patches): PASS — The diff and the effective recipe contain no patch files, setup scripts, heredocs, or wheel installs; the pinned image runs as shipped.

✅ Check 11 (Agentic golden AL): PASS — The c32 job log shows rejection_sample_method: synthetic with synthetic_acceptance_length: 2.88 for 3 tokens, which signoff_facts golden-al confirms against kimi-k3.1-pro__dspark__kimi-k3.1-pro-dspark-preview__probabilistic-block.yaml (thinking_on). The 7, 5 and 4 token points get 3.69, 3.41 and 3.19 from the same unchanged launcher path.

➖ Check 12 (Append-only): N/A — The new changelog entry is not append-only: true.

✅ Check 14 (Pareto coverage): PASS — One curve (kimik3.1 / agentic-coding / mi355x vLLM / fp4 / P90 E2EL, run 43717568009 on 75d74552, image kimi-k31-5c14fe8) has 6/6 frontier points per pareto_coverage: (22.16 s, 2164), (25.19, 3137), (27.86, 9089), (40.19, 13387), (47.27, 15850), (60.93, 20319) tok/s/GPU.

✅ Check 15 (PR description): PASS — The title, body and changelog match the diff and evidence: the image digest, VLLM_K3_SP_PREFILL=1 on every point, the default-on c1/c4 levers, unchanged points, the validation run, and the per-point throughput figures.

➖ Check 16 (NVIDIA overview): N/A — The only affected configuration is AMD MI355X.

➖ Check 17 (Metrics publication): N/A — Direct vLLM serving with no Dynamo TRT-LLM path.

✅ Check 18 (Recipe consistency): PASS — The config generates at head, the master entry and recipe agree, and the new digest resolves. All six points are re-tuned, none undominated or removed, and none regress (tput/GPU +1.4% to +6.4%, P90 E2EL -0.9% to -6.3%). stream-interval is 10 on every point.

Assessed commit: bdfa086b85607c39fc4c574dfa80c221bb96372f.

@seungrokj

Copy link
Copy Markdown
Collaborator Author

Re: the Check 2 (Evals) rejection above: is it true that no eval ran the SP-prefill path?

"The server-log archive could not be opened": not correct. I downloaded eval_server_logs_* for c32 (artifact 14183329005) and c48 (artifact 14184395977) from run 43717568009, and srt-single-node-logs.tar.gz extracts fine.

"Only shows configuration, not dispatch": correct. The only SP-related line in both eval server logs (mia1-p01-g16_agg_w0.out for c32, mia1-p01-g09_agg_w0.out for c48) is the startup message:

[sequence_parallel.py:515] Kimi-K3 sequence-parallel prefill enabled (batches of >= 4096 tokens).

There is no per-batch log or counter, so the logs cannot show a batch of at least 4096 tokens going through SP.

"No proof that any eval ran the path": strictly true, but in practice the path almost certainly ran. Here is what the logs do show:

  • GSM8K prompt size: lm-eval ran 5-shot multiturn chat with num_concurrent=32 (c32) and num_concurrent=48 (c48). Prompt-throughput totals over the GSM8K window divided by 1319 questions give about 1,167 uncached prompt tokens per request on both servers.
  • c48 (max-num-seqs 48, max-num-batched-tokens 10240): the first 48 requests queue about 56k prefill tokens, so the first several steps should be full 10,240-token batches. Any 4 new prefills that land together already reach 4096.
  • c32 (max-num-seqs 20): 20 slots × about 1,167 tokens ≈ 23k tokens at startup. prefill-schedule-interval 32 could throttle admission, so this case is less certain than c48.
  • Peak prompt throughput (10 s averages): about 4,400 tok/s at c32 and about 7,420 tok/s at c48.

This is strong inference, not a dispatch log. To close the check, we can either rerun one eval point (c48) on an image that logs or counts SP-prefill dispatches in sequence_parallel.py, or accept the scheduler math above. Separately, the GSM8K scores match the #5043 cohort (c48 0.9795 / 0.9788 vs 0.9788 in run 43399050596).

@seungrokj
seungrokj merged commit 67c3ed2 into main Nov 16, 2026
38 checks passed
@seungrokj
seungrokj deleted the seungrokj/kimik3.1-mi355x-vllm-5c14fe8 branch November 16, 2026 05:11
@github-actions

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 6 0 9 0 0
eval 4 12 8 0 0
trace 6 0 0 0 0
power 6 0 0 2 0

benchmarkFiles 8, rawBenchmarkRows 15, coveredByPerJobFiles 7, purgedPoints 0, failedRunRows 2, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 5276, gpuMetricSamplesNew 242416, powerPublicationPoints 6, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Fridge003 added a commit that referenced this pull request Nov 16, 2026
…ernel-c1

Resolve the perf-changelog.yaml tail conflict: keep main's new entries
(#5043, #5081, #5084, #5082, #5089) byte for byte and append this PR's
Muse Spark GB300 megakernel entry after them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

non-canary-full-sweep-enabled Run the full sweep without the canary gate (full search space, no trim)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants