Repository navigation
[AgentX] DeepSeek-V4.1-Flash GB300 vLLM: move to vLLM main a2bcb565 with specialized decode and retune TP4/DEP4 / [AgentX] DeepSeek-V4.1-Flash GB300 vLLM:升级到 vLLM main a2bcb565,启用 specialized decode 并重新调优 TP4/DEP4 - #4844
Conversation
…565 with specialized decode and retune TP4/DEP4 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the contribution!
中文感谢你的贡献!
|
There was a problem hiding this comment.
Besides the inline findings, I checked a few other things and found no problems:
- Every TP4 zip list has 7 entries and every DEP4 zip list has 10. Both CONC lists match conc-list in nvidia-master.yaml.
- The kv-cache-memory byte values convert exactly to the 24–128 GiB steps given in the changelog.
- The perf-changelog.yaml entry is only appended at the end, and no existing bytes changed.
Extended reasoning...
This change only touches benchmark config: one srt-slurm recipe, one nvidia-master.yaml entry and an appended perf-changelog.yaml block. It touches no security-sensitive code. I'm deferring because the inline findings are already posted. This note only records the extra consistency checks that passed.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| kv-cache-memory: [25769803776, 30064771072, 34359738368, 42949672960, 51539607552, 60129542144, 77309411328, 103079215104, 120259084288, 137438953472] | ||
| # A cold long prompt on one DP rank would otherwise stall the other ranks' decode for a full budget step. | ||
| long-prefill-token-threshold: 8192 | ||
| compilation-config: '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}' | ||
| max-cudagraph-capture-size: 8190 | ||
| # Binds each DP rank's EngineCore and worker to its GPU's NUMA node. | ||
| numa-bind: true | ||
| numa-bind-nodes: [[0, 0, 1, 1]] |
There was a problem hiding this comment.
🔴 GB300 DeepSeek-V4.1-Flash operatorx layer cases that fall back to this recipe would get server args that serve_argv cannot render or was meant to drop. That happens if vLLM types --numa-bind-nodes as an int list. serve_argv (operatorx/recipes.py:129) sends numa-bind-nodes as one JSON token, which argparse would reject. kv-cache-memory also gets past the kv-cache-memory-bytes exclusion. Fix: add kv-cache-memory, numa-bind and numa-bind-nodes to _NOT_APPLIED, or spell the recipe key kv-cache-memory-bytes. Also have serve_argv emit list values as separate tokens.
Why this was flagged
The operatorx CI calls find_recipe (operatorx/ci.py:213) for each GB300 case whose sources name deepseek-ai/DeepSeek-V4.1-Flash. _variants expands this recipe with no selector, and zip keys are sorted (synthetic_acceptance.py:376), so zip_override_dep4[0] comes first. A case whose split matches no variant exactly, such as a single-device or tp2 MoE case, falls back to pool[0] (recipes.py:114-120) and gets the DEP4 c136 args. serve_argv (recipes.py:133-137) turns numa-bind-nodes: [0, 0, 1, 1] into --numa-bind-nodes "[0, 0, 1, 1]". engine.py:59 passes that to vLLM's parse_known_args. If vLLM declares the flag as a list[int] with nargs and type=int, argparse exits on an invalid int value and the shard dies. On the base branch the fallback was zip_override_dep2[0], which set neither key. Neither key has an exclusion in _NOT_APPLIED.
Verification: Severity: normal. Trigger: the recipe image's vLLM declares --numa-bind-nodes as a list[int] flag (nargs="+", type=int). serve_argv (recipes.py:133-138) adds --numa-bind-nodes and then json.dumps(v) as one token, "[0, 0, 1, 1]". Before this change, the same slot held dep2[0], which had no numa or kv-cache-memory args.
| engram-config: '{"cpu_offload":true,"use_thp":true}' | ||
| max-num-seqs: [8, 8, 16, 32] | ||
| kernel-config: '{"enable_flashinfer_autotune":true}' | ||
| # Specialized decode: the SM103 latency-specialized TP4 decode executor. |
There was a problem hiding this comment.
🟡 nit (optional): The comment above additional-config only repeats what the next line says, which AGENTS.md's comment rules forbid. Fix: delete the comment, or swap it for the reason TP4 needs this executor (for example, a measured decode-latency gain or an SM103-only constraint).
Why this was flagged
agentic.yaml:59 adds # Specialized decode: the SM103 latency-specialized TP4 decode executor. directly above additional-config: '{"decode_backend":"latency_specialized"}' at line 60. Root CLAUDE.md points to AGENTS.md. Under its mandatory comment rules, a comment may only briefly explain a specific line or block, such as why a value was chosen or a hardware quirk, and must never restate the code. This comment only names what the value already says, so it adds a convention violation the base recipe did not have. There is no runtime impact.
Verification: nit. Line 59 of the new agentic.yaml in zip_override_tp4 is # Specialized decode: the SM103 latency-specialized TP4 decode executor., directly above line 60, additional-config: '{"decode_backend":"latency_specialized"}'. AGENTS.md lines 54-58 say to "never restate the code." Only "SM103" adds a detail, and it gives no reason. Nothing at runtime is affected.
- operatorx: do not apply numa-bind, numa-bind-nodes or numa-bind-cpus to layer cases. They place a long-running server's processes; numa-bind-nodes was also passed as one JSON token, which vLLM's list[int] parser rejects. - Recipe: spell the KV pool flag kv-cache-memory-bytes. It previously relied on argparse prefix matching, so operatorx's kv-cache-memory-bytes exclusion missed it. - Recipe comments: give the reason for NUMA binding and drop the comment that restated additional-config. - perf-changelog: pr-link #4844. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#4844) vLLM parses list-typed flags (for example --cudagraph-capture-sizes) with nargs="+", so a single JSON token such as "[1, 2, 4]" fails its int conversion. Dict values stay JSON. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the perf-changelog conflict: keep main's entries and append this PR's entry last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42241164555 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42241164555 |
…eep] Keep this PR's GB300 image (vLLM main a2bcb565) over #4558's nightly-00b6c4c5, which is 862 commits older. Resolve the perf-changelog conflict with main's entries kept and this PR's entry last, and name nightly-00b6c4c5 as the image this entry replaces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
/use 42241164555 |
|
@cquil11 staged run 42241164555. This run remains available across future |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Ingest report — ✅ Results ingested into the Ingest workflow · Merge Ingest run · Benchmark source run
benchmarkFiles 18, rawBenchmarkRows 34, coveredByPerJobFiles 17, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 381328, powerPublicationPoints 17, changelogEntriesWritten 1 Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun. 中文Ingest 报告 — ✅ 结果已写入 上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。 |
This updates
dsv41flash-fp4-gb300-vllm-agentic-dspark(GB300 DeepSeek-V4.1-Flash, vLLM, AgentX).The image moves from
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842bto the vLLM main nightlyvllm/vllm-openai:cu134-nightly-a2bcb5657a881a002e79d0adab3c6b5e39436121. It is pinned to indexsha256:76a533475156ffdb3656ccc389d0c71f5cec16897c0f490ac43e4f379c4b4f0d; the arm64 manifest issha256:24541db891c590e3a4022c7206c3ebb00e4be35442f54c20f2edf12eddfef4bf.The search space is replaced with seventeen single-node points, each one aggregated vLLM engine on 4 GPUs.
TP4 with the specialized decode executor, behind vLLM's own frontend, at concurrency 1, 4, 10, 16, 24, 32 and 48 (
zip_override_tp4).additional-config {"decode_backend":"latency_specialized"}runs decode steps on vLLM main's DeepSeek-V4.1 latency-specialized executor for SM103 at TP4.max-num-active-seqs1–8 keeps decode batches small;max-num-seqsstill sizes the runner and graphs. c1 runs one sequence (max-num-seqs1).prefill-schedule-interval(1 at c1, 16 at c4, 96 above) admits prefill only every N steps.DEP4 (TP1 × DP4 + EP4) behind the consistent-hash vLLM Router at concurrency 136, 170, 200, 256, 320, 384, 512, 640, 768 and 1024 (
zip_override_dep4).numa-bind).long-prefill-token-threshold), so a cold long prompt on one DP rank does not stall the other ranks' decode for a whole budget-sized step.kv-cache-memory) grows with concurrency: 24 GiB at c136, then 28, 32, 40, 48, 56, 72, 96, 112 and 128 GiB.max-num-seqsis 128 up to c384, then 192, 224, 256 and 320.Every point uses gpu-memory-utilization 0.93 and five-token DSpark with adaptive verification off, so throughput runs use the golden acceptance length (3.51, from the srt driver). TP4 at concurrency 8 and the DEP2 points (concurrency 8–192) are removed. Each removed point is dominated by a new point or lies under the new curve.
中文
本 PR 更新
dsv41flash-fp4-gb300-vllm-agentic-dspark(GB300 DeepSeek-V4.1-Flash,vLLM,AgentX):镜像由
vllm/vllm-openai:nightly-00b6c4c565a8732c131e1f452d4a17803bb0842b升级为 vLLM main nightlyvllm/vllm-openai:cu134-nightly-a2bcb5657a881a002e79d0adab3c6b5e39436121,固定到 indexsha256:76a533475156ffdb3656ccc389d0c71f5cec16897c0f490ac43e4f379c4b4f0d(arm64 manifestsha256:24541db891c590e3a4022c7206c3ebb00e4be35442f54c20f2edf12eddfef4bf)。搜索空间替换为十七个单节点点,每个点为 4 GPU 上的一个聚合 vLLM 引擎。
TP4,启用 specialized decode 执行器,使用 vLLM 自带前端,并发 1、4、10、16、24、32、48(
zip_override_tp4)。additional-config {"decode_backend":"latency_specialized"}使 decode 步运行在 vLLM main 中面向 SM103、TP4 的 DeepSeek-V4.1 latency-specialized 执行器上。max-num-active-seqs1–8 使 decode 批保持较小,而max-num-seqs仍决定 runner 与 CUDA graph 的规模;c1 只运行一个序列(max-num-seqs为 1)。prefill-schedule-interval(c1 为 1,c4 为 16,其余为 96)每 N 步才接纳一次 prefill。DEP4(TP1 × DP4 + EP4),位于一致性哈希 vLLM Router 之后,并发 136、170、200、256、320、384、512、640、768、1024(
zip_override_dep4)。numa-bind)。long-prefill-token-threshold),使某个 DP rank 上的冷启动长 prompt 不会让其他 rank 的 decode 停顿整整一个预算步。kv-cache-memory)随并发增大:c136 为 24 GiB,其后依次为 28、32、40、48、56、72、96、112、128 GiB;max-num-seqs在 c384 及以下为 128,其后依次为 192、224、256、320。所有点均使用 gpu-memory-utilization 0.93 与五 token DSpark,并关闭 adaptive verification,使吞吐测试采用 golden acceptance length(3.51,由 srt driver 提供)。移除 TP4 并发 8 与 DEP2 点(并发 8–192);每个被移除的点都被新点支配,或位于新曲线之下。
AI model disclosure
🤖 Generated with Claude Code