Repository navigation
[ROCm][Docs] Add DeepSeek-V4.1-Flash MI355X DP2 AgentX preset with UMBP KV offload - #67378
Merged
esmeetu merged 2 commits intoNov 13, 2026
Conversation
shen-shanshan
temporarily deployed
to
recipes-preview
November 13, 2026 04:16 — with
GitHub Actions
Inactive
Contributor
5 of 11 tasks
shen-shanshan
force-pushed
the
recipes-dsv41flash-mi355x-umbp-offload
branch
from
November 13, 2026 06:52
45b3afe to
c7ba160
Compare
shen-shanshan
temporarily deployed
to
recipes-preview
November 13, 2026 06:52 — with
GitHub Actions
Inactive
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #102400 for commit |
…BP KV offload Add the mi355x_agentx_dp2_umbp preset for InferenceX vllm-project#5036 (dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp): two GPUs per server with attention DP2 and the MoE tensor-parallel across both ranks, behind the consistent-hash vLLM Router, at concurrency 256 and 384. It selects the UMBP KV Offload option (UMBPStoreConnector, standalone server) and serves a per-tensor fp8_e4m3 KV cache with prefill cadence 11 on the nightly that carries vllm-project#64715 and vllm-project#66036. The preset restates the MI355X kernels the TP rows take from strategy_overrides.single_node_tp, since single_node_dep does not inherit them, and drops --enable-expert-parallel. kv_offload_support records UMBP as verified for this model. The `umbp` KV Offload option is added to the renderer's taxonomy in vllm-project/recipes; this recipe validates only once that lands. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: shen-shanshan <467638484@qq.com>
…28 without offloading InferenceX vllm-project#5036 now also runs DP2 at concurrency 128, carried over from the closed vllm-project#4992, with no KV offloading: UMBP did not help there in local runs, since the GPU KV cache holds the working set. Add that point to the mi355x_agentx_dp2_umbp preset with `kv_offload: off`; 256 and 384 keep UMBP. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: shen-shanshan <467638484@qq.com>
shen-shanshan
force-pushed
the
recipes-dsv41flash-mi355x-umbp-offload
branch
from
November 13, 2026 10:13
c7ba160 to
ad77d06
Compare
Contributor
Author
|
/ci run |
|
✅ Triggered Buildkite CI #102456 for commit |
github-actions Bot
added a commit
to vllm-project/recipes
that referenced
this pull request
Nov 13, 2026
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
esmeetu
approved these changes
Nov 13, 2026
esmeetu
temporarily deployed
to
recipes-preview
November 13, 2026 14:10 — with
GitHub Actions
Inactive
vllm-recipes-publisher Bot
pushed a commit
to vllm-project/recipes
that referenced
this pull request
Nov 13, 2026
vllm-project/vllm@main:recipes c410543d9fff → 10a854ed6b3b [latest.vllm] vllm-project/vllm@c410543...10a854e Commits: - vllm-project/vllm@10a854ed6b3b [ROCm][Docs] Add DeepSeek-V4.1-Flash MI355X DP2 AgentX preset with UMBP KV offload (vllm-project/vllm#67378) Recipes: - deepseek-ai/DeepSeek-V4.1-Flash (modified): https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot
added a commit
to wombat/recipes
that referenced
this pull request
Nov 13, 2026
…mni@2c148ceb1373, vllm-project/vllm-ascend@775899991f96 vllm-project/vllm@main:recipes c410543d9fff → cfd86e432065 [latest.vllm] vllm-project/vllm@c410543...cfd86e4 Commits: - vllm-project/vllm@10a854ed6b3b [ROCm][Docs] Add DeepSeek-V4.1-Flash MI355X DP2 AgentX preset with UMBP KV offload (vllm-project/vllm#67378) Recipes: - deepseek-ai/DeepSeek-V4.1-Flash (modified): https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash vllm-project/vllm-omni@main:recipes 4dee28cac13d → 2c148ceb1373 [latest.vllm-omni] vllm-project/vllm-omni@4dee28c...2c148ce Commits: - vllm-project/vllm-omni@6c8f69303faf [Bugfix][Qwen-Image-2.1] Fix decode graph pool lifetime and stream recovery (vllm-project/vllm-omni#10268) - vllm-project/vllm-omni@b5d54010ed06 [Doc][NPU] Add Atlas 800T A3 performance to the MiniMax-H3-Image recipe (vllm-project/vllm-omni#10292) vllm-project/vllm-ascend@main:recipes 02ac27a19cb7 → 775899991f96 [latest.vllm-ascend] vllm-project/vllm-ascend@02ac27a...7758999 Commits: - vllm-project/vllm-ascend@931bfd948e04 [Doc][Recipe] Add Qwen3-VL-Reranker-2B Ascend recipe (vllm-project/vllm-ascend#20817) - vllm-project/vllm-ascend@5e9559edb202 [Doc][Recipe] Add Qwen3-Embedding-0.6B Ascend recipe (vllm-project/vllm-ascend#20816) - vllm-project/vllm-ascend@61d127541ffd [BugFix][Recipes] Fix Ascend recipe TP/DP configuration (vllm-project/vllm-ascend#21808) - vllm-project/vllm-ascend@63134f7c6b99 [Doc][Recipe] Add Ascend overlay for Qwen3-VL-235B-A22B-Instruct (vllm-project/vllm-ascend#21553) - vllm-project/vllm-ascend@0d8b1f8ba171 [Doc][Recipe] Add Ascend overlay for Qwen3-Next-80B-A3B-Instruct (vllm-project/vllm-ascend#21552) - vllm-project/vllm-ascend@7e18331d207f [Doc][Recipe] Add historical Qwen3-Reranker Ascend deployment (vllm-project/vllm-ascend#20994) - vllm-project/vllm-ascend@3d3a5c038bdd [Doc][Recipes] Add MiniMax-M3 Ascend overlay (vllm-project/vllm-ascend#21065) - vllm-project/vllm-ascend@548ee4c116f0 [Doc][Recipe] Add Ascend overlay for Qwen3.6-35B-A3B (vllm-project/vllm-ascend#20808) Recipes: - Qwen/Qwen3-Embedding-0.6B (added): https://recipes.vllm.ai/Qwen/Qwen3-Embedding-0.6B - Qwen/Qwen3-Reranker-0.6B (added): https://recipes.vllm.ai/Qwen/Qwen3-Reranker-0.6B - Qwen/Qwen3-VL-Reranker-2B (added): https://recipes.vllm.ai/Qwen/Qwen3-VL-Reranker-2B - MiniMaxAI/MiniMax-M3 overlay (added): https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3 - Qwen/Qwen3-Next-80B-A3B-Instruct overlay (added): https://recipes.vllm.ai/Qwen/Qwen3-Next-80B-A3B-Instruct - Qwen/Qwen3-VL-235B-A22B-Instruct overlay (added): https://recipes.vllm.ai/Qwen/Qwen3-VL-235B-A22B-Instruct - Qwen/Qwen3.5-397B-A17B overlay (modified): https://recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B - Qwen/Qwen3.6-27B overlay (modified): https://recipes.vllm.ai/Qwen/Qwen3.6-27B - Qwen/Qwen3.6-35B-A3B overlay (added): https://recipes.vllm.ai/Qwen/Qwen3.6-35B-A3B - deepseek-ai/DeepSeek-V4-Flash overlay (modified): https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds the
mi355x_agentx_dp2_umbppreset torecipes/models/deepseek-ai/DeepSeek-V4.1-Flash.yamlfor InferenceX #5036 (dsv41flash-fp4-mi355x-vllm-agentic-dspark-umbp), and records UMBP as verified for the model.Claims
--data-parallel-size 2), attention DP2 with the MoE tensor-parallel across both ranks (--enable-expert-parallelremoved), behind the consistent-hash vLLM Router;UMBPStoreConnector, standalone server tab);--kv-cache-dtype fp8_e4m3([ROCm][DSv4/4.1] Enable Tensor scale FP8 KV Cache for Sparse MLA #66036),--prefill-schedule-interval 11, Engram tables in host memory, 16384-token prefill budget, cudagraph capture up to 1024;kv_offload: off, as InferenceX runs it without offloading.kv_offload_support: { umbp: verified }makes the opt-in UMBP option available for this model.Validation
From a vllm-project/recipes checkout with the
umbpoption,node scripts/validate-source.mjs --source vllm --dir <this branch>/recipespasses:✓ vllm recipes … are valid against the renderer.The rendered command matches the InferenceX recipe flag for flag: the per-point
vllm serve/docker run, the UMBP Server companion (socket address equal to the connectorendpoint) and the vLLM Router (--intra-node-data-parallel-size 2). At 128 the command has no--kv-transfer-configand no UMBP Server tab; 256 and 384 have both.pnpm test:presetsandpnpm test:kv-offloadpass with this branch as the vLLM source.Local MI355X AgentX replays (agentx-fast, 1200 s, single runs) of this configuration on the default
fp8_ds_mlaKV and an earlier nightly, against the same configuration without offloading:The
fp8_e4m3KV cache and the pinned nightly are qualified by the InferenceX sweep of [Kernel][RFC] Refactor the punica kernel based on Triton #5036.Details
single_node_depdoes not inherit the MI355X kernels the TP rows take fromstrategy_overrides.single_node_tp(MXFP4 indexer, block size 128, AITER sparse MLA), so the preset restates them.--monokernelstays off, since mono decode rejects DP.default: true: it is a benchmark lane for high concurrency, where the working set outgrows GPU KV. At concurrency 128 UMBP did not help, so that point turns the KV Offload row off while keeping the rest of the lane.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity. (Recipe YAML only; reuses the DEP strategy, its router companion and the preset delta fields.)
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance: drafted with
claude-opus-5-5[1m](Claude Opus 5.5, 1M context) via Claude Code, which read the recipe and renderer, wrote the preset, ran the validation and wrote this description; reviewed by a human.🤖 Generated with Claude Code