Repository navigation
[AMD][AgentX] DeepSeek-V4.1-Flash MI355X vLLM: add DP2 concurrency 128-384, with UMBP KV offloading at 256 and 384 / [AMD][AgentX] DeepSeek-V4.1-Flash MI355X vLLM:新增 DP2 并发 128–384 点,256、384 启用 UMBP KV offloading - #5036
shen-shanshan wants to merge 16 commits into
Conversation
|
Thanks for the contribution!
中文感谢你的贡献!
|
c0f85e6 to
506b46d
Compare
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=43861172833 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=43861172833 |
c3680f0 to
3a4e067
Compare
…7189 with fp8 KV Move the UMBP key to vllm/vllm-openai-rocm:nightly-be47189f37 (2026-11-12), which adds vllm-project/vllm#66036 on top of #64715, and serve both DP2 points with kv-cache-dtype fp8_e4m3: per-tensor fp8 KV rows for the AITER sparse MLA instead of the block-scaled fp8_ds_mla record. The DSpark draft layers share the target's cache dtype; draft weights are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…oading Carry over concurrency 128 from the closed #4992 into the UMBP key: the same DP2 engine and router settings, prefill cadence 11 and fp8_e4m3 KV cache on the same nightly, with no KV offloading. Local runs showed UMBP does not help at 128, where the GPU KV cache holds the working set, so the point runs without it while 256 and 384 keep UMBP. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
013edc0 to
86b7d7d
Compare
|
Let's sweep with this PR vllm-project/vllm#67377 included before merging |
…16ae The 2026-11-15 ROCm nightly includes vllm-project/vllm#67377 (AITER FlyDSL dense MXFP8 GEMMs for DeepSeek-V4.1-Flash) on top of #64715 and #66036. Co-authored-by: Cursor <cursoragent@cursor.com>
…h-umbp Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # inferencex-e2e/perf-changelog.yaml
The c256 UMBP point landed on that node twice. Slurm exclude keeps the rerun off it. Co-authored-by: Cursor <cursoragent@cursor.com>
| partition: compute | ||
| exclusive: true | ||
| # mia1-p01-g11 is a known bad node. | ||
| exclude: [mia1-p01-g11] |
There was a problem hiding this comment.
@functionstackx @chunfangamd please take this node down, this one is broken with very bad performance across different frameworks.
…d6dd The 2026-11-16 ROCm nightly is 50 commits ahead of nightly-9bc16ae and still carries vllm-project/vllm#64715, #66036, and #67377. Co-authored-by: Cursor <cursoragent@cursor.com>
…h-umbp Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # inferencex-e2e/perf-changelog.yaml
Co-authored-by: Cursor <cursoragent@cursor.com>
| partition: compute | ||
| exclusive: true | ||
| # Known bad nodes. | ||
| exclude: [mia1-p01-g09, mia1-p01-g11] |
| # The mi355x-amds host setup releases hugepages, so the pool uses 4 KiB pages. | ||
| UMBP_DRAM_USE_HUGEPAGES: '0' |
There was a problem hiding this comment.
if u wanna use hugepages, cant u use madvise like mooncake?
madvise is enabled on the cluster
|
/use 43861172833 |
|
@chunfangamd staged run 43861172833. This run remains available across future |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
Checklist items
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43861172833
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/43861172833
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Check 3 (Upstream recipe merged and complete): vllm-project/recipes#1254 and vllm-project/vllm#67378
- Check 2 (Evals pass, accuracy): Upstream vLLM #64715 reports a two-pass GSM8K test with GPU prefix-cache reset between passes. The restore pass achieved 92.95% accuracy with 92.7% of prompt tokens restored from UMBP, versus 92.12% without the connector. This provides supplementary restore-correctness evidence; the upstream TP2 test does not directly establish restore coverage for this PR’s DP2/
fp8_e4m3evaluation.
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd Blocking: no eval shows that the UMBP KV-restore path ran. That path is this PR's conditional backend change, and the eval in run 43861172833 cannot show it was exercised. ❌ Check 2 (Evals pass): FAIL — No eval covers the changed conditional path (UMBP restore of offloaded
Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep + evals): PASS — In-PR commit ✅ Check 3 (Upstream recipe): PASS — vllm#67378 (merged 2026-11-13) adds the ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — Every current ✅ Check 6 (Upstream images): PASS — ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No architecture hacks): PASS — No ✅ Check 9 (Chat template): PASS — AgentX replay drives the server through the chat endpoint with the ✅ Check 10 (No engine patches): PASS — No patches. The ✅ Check 11 (Golden AL): PASS — Benchmark launch uses ➖ Check 12 (Append-only): N/A — The new changelog entry has no ✅ Check 15 (PR description): PASS — Title, body and changelog match the pinned config: image, DP2/TP1 topology, c128 without offload, UMBP at c256/c384 with a 449 GB pool, ➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X. ➖ Check 17 (Metrics publication): N/A — No Dynamo or TRT-LLM serving path is affected. ✅ Check 18 (Recipe consistency): PASS — All 3 points generate. Master Assessed commit: |
…h-umbp # Conflicts: # inferencex-e2e/perf-changelog.yaml
Summary
The MI355X DeepSeek-V4.1-Flash vLLM AgentX curve stops at concurrency 64. Above that, the GPU KV cache stops holding every live session's prefix. Evicted prefixes are then recomputed, and prefill crowds out decode. vllm-project/vllm#64715 (merged 2026-11-10) adds UMBP standalone support for DeepSeek-V4.1-Flash on ROCm. With it, the engine offloads KV blocks to a node-local host-DRAM pool owned by MORI's
umbp_standalone_serverand restores them on a prefix hit instead of recomputing.This PR adds a new config key,
dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp, with three points: concurrency 128 without KV offloading, and 256 and 384 with UMBP. Evals run at c384. The c128 point is carried over from the closed #4992 unchanged; UMBP is off there because it did not help at 128 in local runs (see below).vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d(2026-11-16, digestsha256:67a3a0fb7989…). It carries #64715 (UMBP standalone for DeepSeek-V4.1-Flash), #66036 (per-tensor fp8 KV for sparse MLA on ROCm) and #67377 (AITER FlyDSL dense MXFP8 GEMMs for DeepSeek-V4.1-Flash), and its MORI v1.2.3.post1 is built with UMBP. The existing TP4/TP2 key and the TP8 c1 key are untouched and are not rerun.fp8_e4m3(#66036): plain per-tensor fp8 KV rows for the AITER sparse MLA instead of the default block-scaledfp8_ds_mlarecord. It applies to all three points.tensor-parallel-size 1,data-parallel-size 2, no expert parallelism). vllm-router 0.1.14 fronts the ranks withconsistent_hash, so each session keeps its rank and its prefix. Every point uses the settings of the closed [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992's DP2 points: prefill cadence (prefill-schedule-interval 11) and no--monokernel. The UMBP points differ from c128 only by UMBP.override_dp2_c128(no offloading) andoverride_dp2_umbp_c{256,384}.umbp_standalone_serveras a critical srt-slurmgenericservice that starts before the workers and becomes ready once it logs[StandaloneServer] listening.dram-utilization0.60 of the two-GPU node share, on 4 KiB pages. Both DP ranks share it./tmpis not the workers'. The recipe mounts the node-local/dev/shminto every step at/umbp-host, and the gRPC and fd sockets live in/umbp-host/umbp/. A new server rebinds over socket files an earlier job left behind.UMBPStoreConnector(mode: standalone,kv_load_failure_policy: recompute).mtp.0-2), loaded with the checkpoint's FP8 config. Its attention layers share the target's KV cache dtype, so target and draft both use the per-tensorfp8_e4m3cache.vLLM recipe PR:
Why: local no-offload (B0) vs UMBP (S1) runs
Local AgentX replays on one MI355X node (GPUs 0-1) with this recipe's engine settings before the fp8 KV change (default
fp8_ds_mlaKV): thenightly-96710245container with vLLM main595b1514e8Python code. B0 is #4992's DP2 configuration; S1 adds UMBP with the 449 GB pool. Each point is a single InferenceX agentx-fast run (1200 s profile, prefix cache cold at the start), not a CI run. As in CI, UMBP turns off the extensible KV cache, so S1 has 4.2% less GPU KV than B0 (26.71M vs 27.89M tokens per rank).VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS(300 s) expired and the engine died. In the rerun, with a partly warm Triton cache, the same stall lasted about 75 s. Every run, including B0, logs runtime JIT warnings, so this is not specific to UMBP.nightly-083fbac02. The B0 numbers above come from shorter, cold-cache fast runs, so compare UMBP against B0 here rather than against that absolute figure.Accuracy: GSM8K. (copy from vllm-project/vllm#64715)
lm_evalwith 5-shot, all 1319 questions,local-completions,num_concurrent=64,max_gen_toks=2048./reset_prefix_cache, then pass 2 restores the same prompts from the pool.Risks
/tmp: the workers could not attach to the server (cannot attach to the UMBP standalone server at unix:///tmp/umbp/standalone.grpc.sock). The host-directory mount above fixes the path. The server and the workers also exchange GPU memory handles over HIP IPC; enroot shares the host IPC and PID namespaces, so this should work, but the next sweep is the first confirmation.nightly-60bd6ddandfp8_e4m3: they used thenightly-96710245image's native binaries with newer vLLM Python code, the defaultfp8_ds_mlaKV cache, and fast mode. The fp8 KV layout changes the GPU KV capacity and UMBP block size and was not measured locally; the sweep is its first end-to-end run.AI model disclosure
claude-opus-5-5[1m](Claude Opus 5.5, 1M context), via Claude Code.perf-changelog.yamlentry and this description; ran matrix generation, recipe rendering with the pinned srtctl,infx.matrix.changedand the sweep planner; ran and summarized the local B0/S1 AgentX benchmarks. No delegated agents. A human reviewed the plan and chose the topology (DP2), the concurrencies and the UMBP mode.vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d(digestsha256:67a3a0fb7989…), which includes [Perf][ROCm][DSv4.1] DeepSeek-V4.1-Flash: dense MXFP8 GEMMs and wo_a on AITER's tuned FlyDSL kernels, and a faster MXFP8 activation quant (MI355X) vllm-project/vllm#67377, and updated the changelog entry and this description.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.vLLM recipe PR:
中文
MI355X 上 DeepSeek-V4.1-Flash 的 vLLM AgentX 曲线目前只到并发 64。并发更高时,GPU KV cache 放不下所有活跃会话的前缀,被淘汰的前缀只能重新计算,prefill 会挤占 decode。vllm-project/vllm#64715(2026-11-10 合入)在 ROCm 上为 DeepSeek-V4.1-Flash 支持了 UMBP standalone 模式:引擎把 KV block 卸载到节点本地的主机内存池(由 MORI 的
umbp_standalone_server管理),前缀命中时直接恢复,不再重算。本 PR 新增配置键
dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp,包含三个点:c128 不开 KV offloading,c256 和 c384 开 UMBP,eval 在 c384 上跑。c128 这个点原样搬自已关闭的 #4992;本地测试中 UMBP 在 c128 没有收益(见下文),所以这个点不开 offloading。vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d(2026-11-16,digestsha256:67a3a0fb7989…),包含 #64715(DSv4.1-Flash 的 UMBP standalone 支持)、#66036(ROCm 上 sparse MLA 的 per-tensor fp8 KV)和 #67377(DSv4.1-Flash 的 AITER FlyDSL dense MXFP8 GEMM),其中的 MORI v1.2.3.post1 编译时开启了 UMBP。已有的 TP4/TP2 键和 TP8 c1 键不变,也不会重跑。fp8_e4m3(#66036),即给 AITER sparse MLA 用的 per-tensor fp8 KV,而不是默认的块缩放fp8_ds_mla。三个点都使用这种格式。tensor-parallel-size 1、data-parallel-size 2,不开 expert parallelism)。前端是 vllm-router 0.1.14,策略为consistent_hash,同一会话固定落在同一个 rank 上,前缀也留在那里。所有点都沿用已关闭的 [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992 中 DP2 点的参数:prefill cadence 为 11(prefill-schedule-interval 11),不开--monokernel。两个 UMBP 点和 c128 唯一的区别就是开了 UMBP。override_dp2_c128(不开 offloading),以及override_dp2_umbp_c{256,384}。umbp_standalone_server作为 srt-slurm 的generic服务启动,标记为 critical,在 worker 之前启动,日志出现[StandaloneServer] listening即视为就绪。dram-utilization0.60,使用 4 KiB 页,两个 DP rank 共用。/tmp和 worker 看到的不是同一个。所以 recipe 把节点本地的/dev/shm挂载到每个 step 的/umbp-host,gRPC socket 和 fd socket 都放在/umbp-host/umbp/下。之前作业留下的 socket 文件不影响新 server 重新绑定。UMBPStoreConnector接入(mode: standalone,kv_load_failure_policy: recompute)。mtp.0-2),按 checkpoint 的 FP8 配置加载。它的 attention 层和 target 使用同一种 KV cache 类型,所以 target 和 draft 都用 per-tensorfp8_e4m3KV。依据:本地「不开 offloading(B0)」与「UMBP(S1)」的对比。
fp8_ds_mlaKV)跑 AgentX 回放。容器是nightly-96710245,Python 代码是 vLLM main595b1514e8。B0 就是 [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 with prefill cadence / [AMD] DSv4.1-Flash MI355X vLLM DP2 AgentX c128/c256/c384 启用 prefill 节奏 #4992 的 DP2 配置,S1 在 B0 基础上加 UMBP,池子 449 GB。结果(每 GPU 总吞吐,B0 → S1):
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS(300 s)后引擎崩溃。重跑时 Triton cache 已部分预热,同样的停顿约 75 s 后恢复。所有运行(包括 B0)都有运行时 JIT 的警告,所以这不是 UMBP 特有的问题。nightly-083fbac02,结果为 310,739 tok/s/GPU。上表 B0 是较短、开始时 cache 为冷的 fast 运行,所以 UMBP 的收益应以表中的 B0 为基准,不要和这个绝对值比。风险:
/tmp下,worker 连不上 server(cannot attach to the UMBP standalone server at unix:///tmp/umbp/standalone.grpc.sock)。上面的宿主机目录挂载已修复了这个路径问题。server 和 worker 之间还要通过 HIP IPC 交换 GPU 显存句柄;enroot 共享宿主机的 IPC 和 PID namespace,这一步应该没问题,但要等下一次 sweep 才能确认。nightly-60bd6dd和fp8_e4m3之前做的:用的是nightly-96710245镜像的 native 部分加上更新的 vLLM Python 代码,KV 是默认的fp8_ds_mla,而且是 fast 模式。fp8 KV 会改变 GPU KV 容量和 UMBP 每个 block 的大小,本地没有测过,这次 sweep 是它第一次端到端运行。AI 模型使用说明:使用
claude-opus-5-5[1m](Claude Opus 5.5,1M context),通过 Claude Code。工作内容:阅读现有配置、recipe 和 srt-slurm 服务代码;起草 master config 键、recipe override、perf-changelog.yaml记录和本描述;运行 matrix 生成、用固定版本的 srtctl 渲染 recipe、infx.matrix.changed和 sweep planner;运行本地 B0 / S1 的 AgentX 测试并汇总结果。没有委派给其他 agent。方案经人工审阅,拓扑(DP2)、并发点和 UMBP 模式由人工决定。随后使用 Grok 4.7(通过 Cursor)把镜像更新到vllm/vllm-openai-rocm:nightly-60bd6dd2edd175a1c55fd3488f871e4c3d2ef27d(包含 vllm-project/vllm#67377),并同步 changelog 和本描述。改动类型:配置变更。
vLLM recipe PR:
🤖 Generated with Claude Code