Repository navigation
[NVIDIA][AgentX] Add GB300 DSV4.1-Flash Dynamo + TensorRT-LLM disagg / [NVIDIA][AgentX] 新增 GB300 DSV4.1-Flash Dynamo + TensorRT-LLM 分离式配置 - #4991
Open
jh-nv wants to merge 5 commits into
Conversation
Twelve AgentX points for nvidia/DeepSeek-V4.1-Flash-NVFP4 on GB300 with Dynamo 1.7.0.dev20261110 (first nightly with ai-dynamo/dynamo#18170) over the TensorRT-LLM feat/ds-v4.1-bringup release image: the seven points of the trtllm-serve AgentX Pareto plus c11718, c2286, c1283, c180 and c114. Metrics and power on, decode stream_interval 20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The seven DEP points set the draft MoE backend to MEGAMOE_CUTEDSL, which re-encodes the checkpoint's MXFP4 draft experts as NVFP4 with static activation scales. CUTLASS keeps their default W4A8_MXFP4_MXFP8 path. The changelog entry now also names the FP8 KV cache and the FP8 MoE combine on decode, and a recipe comment no longer says workers publish per-request metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jh-nv
requested review from
Ankur-singh,
RohitNagraj,
kedarpotdar-nv,
nvpohanh and
wzhao18
as code owners
November 10, 2026 23:05
…namo-trt-agentx-sa # Conflicts: # inferencex-e2e/perf-changelog.yaml
jh-nv
marked this pull request as draft
November 10, 2026 23:19
functionstackx
requested changes
Nov 10, 2026
|
|
||
| dsv41flash-fp4-gb300-dynamo-trt-agentic-dspark-disagg: | ||
| image: nvcr.io/nvidia/tensorrt-llm/release:feat-ds-v4.1-bringup-1.4.0a0.dev202610310238@sha256:960406744410b9601d134334d2b4f468f8fee5875f9194aaad1dccea5d7880e7 | ||
| model: nvidia/DeepSeek-V4.1-Flash-NVFP4 |
Collaborator
There was a problem hiding this comment.
do u have evals for this on terminalbench 4.0? showing that nvfp4 quant is just as good as mxfp4?
Collaborator
There was a problem hiding this comment.
logs, manual reprod scripts, etc.
functionstackx
requested changes
Nov 10, 2026
| tokens_per_block: 128 | ||
| max_seq_len: 996595 | ||
| moe_config: | ||
| use_low_precision_moe_combine: true |
Collaborator
There was a problem hiding this comment.
do u have tb4.0 and deepswe evals (along with manual reprod scripts & logs) as discussed with @xinli-sw about lower precision reductions & nvfp4 dsv4.1 flash
The image is now nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc4.dev202611160007 (main commit ab2dfeba8e69), which includes NVIDIA/TensorRT-LLM#21438, so the recipes no longer need the feat/ds-v4.1-bringup branch. Decode no longer sets use_low_precision_moe_combine, so the MoE combine stays BF16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…namo-trt-agentx-sa # Conflicts: # inferencex-e2e/configs/nvidia-master.yaml # inferencex-e2e/perf-changelog.yaml
jh-nv
marked this pull request as ready for review
November 16, 2026 20:14
Contributor
|
Thanks for the contribution!
中文感谢你的贡献!
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
dsv41flash-fp4-gb300-dynamo-trt-agentic-dspark-disagg: DeepSeek-V4.1-Flash NVFP4 (nvidia/DeepSeek-V4.1-Flash-NVFP4) AgentX on GB300 with Dynamo + TensorRT-LLM disaggregated serving. Seven points reuse the topologies, engine settings and concurrencies the TensorRT-LLM team tuned fortrtllm-serveAgentX on this model. Five more points (c11718, c2286, c1283, c180, c114) fill the regions that Pareto leaves empty.nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc4.dev202611160007(pinned by digest), a TensorRT-LLMmainnightly built from commitab2dfeba8e69. It includes the DeepSeek-V4.1-Flash optimizations of NVIDIA/TensorRT-LLM#21438 (merged 2026-11-14), which these recipes use: MegaMoE shared-expert fusion,gen_prompt_tail_tokens, EPLBwarmup_placement,host_cache_bind_to_gpu_numa_node, strict conversation affinity,max_gen_init_admissions_per_passandenable_client_io_during_gpu_wait.1.7.0.dev20261110wheel, installed at job start. It is the first nightly with ai-dynamo/dynamo#18170.Each DEP2 prefill rank keeps a native host KV tier on its GPU's NUMA node: 125 GiB at the DEP8/DEP16 points, 111 GiB at the DEP32 points and 80 GiB at the TEP8/TP4 points. Every decode worker streams every 20 iterations (
stream_interval: 20); prefill keeps TensorRT-LLM's default of 1. Both roles use an FP8 KV cache (kv_cache_config.dtype: fp8). The MoE combine stays BF16 (TensorRT-LLM's default;use_low_precision_moe_combineis not set).The DSpark draft runs as it ships. Its routed experts are MXFP4 in the checkpoint and run with MXFP8 activations (
W4A8_MXFP4_MXFP8, the image default). At the DEP points the draft's MoE runs on CUTLASS (speculative_config.moe_backend: CUTLASS): left unset, the draft would inherit the target's MegaMoE backend, which serves NVFP4 experts only, and setting MegaMoE for the draft explicitly re-encodes them as NVFP4. At the TEP8/TP4 points the draft inherits the target's TRTLLM backend, which keeps the same MXFP4 path.Dynamo settings
DYN_ENGINE_CONV_AFFINITY_DP_RANK_SOURCE=dynamo). This puts a subagent's first turn on the rank that holds its parent's prefix; in our runs it raised the first-turn prefix hit at c7600 from 0.29 to 0.42 and output throughput at c7000 by 4.9%. The AgentX client runs on a dedicated node.DYN_TOKEN_IDS_AS_BYTES=1). Workers are bound to their GPU's NUMA node for CPUs (numa_cpu_bind) and prefer it for memory (numa_memory_bind: preferred).Metrics and power
Every worker publishes TensorRT-LLM iteration and KV-cache metrics (
engine.publish_metrics: true, no KV events).iter_perf_stats_intervalandkv_cache_config.iteration_stats_intervalare 40 on prefill and TEP8/TP4 decode and 400 on DEP decode. The client requirestrtllm_kv_cache_utilization(AIPERF_REQUIRED_SERVER_METRIC_PREFIX). Every point setspower: true.Per-request perf metrics stay off, which is the default since ai-dynamo/dynamo#18170 (
--publish-per-request-metrics). On earlier Dynamo builds,--publish-metricsalso turned them on. TensorRT-LLM'srecord_statsthen rebuilt each finished ~130K-token prompt as a Python list in the decode handler and saturated it: total tok/s/GPU dropped 39.5% at c7900, 30.9% at c7600, 18.8% at c5100 and about 13% at c7000. NVIDIA/TensorRT-LLM#21674 makes those metrics cheap for anyone who turns them on. This config does not turn them on.Validation. All twelve points ran end to end on an internal NVIDIA GB300 NVL72 cluster (no public run links) with metrics and power on, on the
feat/ds-v4.1-bringuprelease image that #21438 was merged from (TensorRT-LLM054a88929a1). In those runs the DEP points' draft MoE was still on MegaMoE. c7900, c5100 and c7600 were within 1% of the same recipes with metrics and power off. c7000 was 3.3% lower over the part of its window it measured; that cause is not isolated. The TEP8 points were 1–3% lower, which is what the shared DCGM telemetry block (--collect-interval=100) costs these small-batch points; the cost goes away at 1000 ms. Those runs also used an FP8 MoE combine on decode. Moving the DEP points' draft to CUTLASS, as in this PR, costs another 1–2% output throughput, and returning the combine to BF16 costs at most 0.5% (short runs). Short runs of this PR's exact config on themainnightly match the earlier image: output throughput +0.7% at c7900, +0.1% at c7600, +1.2% at c84 and +2.5% at c180.Validation details
The runs used srt-slurm at this repository's pin with a 60-minute AgentX window and 3 warmup requests per lane; cluster scheduler settings were added at submission. Dynamo was
1.7.0.dev20261109with #18170's Python change applied (the two files it touches), because no nightly carried #18170 yet;1.7.0.dev20261110ships that same change. Apart from that, the image, the draft MoE backend and the MoE combine, the bound recipes are identical to this PR; the short runs below cover those three differences.Median tok/s/user is 1000 / p50 ITL. The baseline is the same recipe with metrics and power off; in those runs the seven reused points were within 1% of the TensorRT-LLM team's own
trtllm-servemeasurements in output tok/s/GPU.use_low_precision_moe_combine: true). With this PR's BF16 combine, short runs of the pinned config (900 s window) against the same config with the FP8 combine:mainnightly1.4.0rc4.dev202611160007, Dynamo1.7.0.dev20261110, CUTLASS draft, BF16 combine), short runs (900 s window) against the same config on the earlier image:c180's earlier run also used the FP8 combine and the patched
1.7.0.dev20261109.Metrics collection (Check 17)
1.7.0.dev20261110(dynamo.source.wheelandrouter.version; not inferred from the container tag). Itsdynamo/trtllm/backend_args.pyhas--publish-per-request-metrics, andworkers/llm_worker.pysetsreturn_perf_metrics = config.publish_per_request_metrics. It contains #12162, so--publish-metricsand KV events are independent controls.engine.publish_metrics: truemakes srtctl pass--publish-metricsto every prefill and decode worker. KV events stay off (event_buffer_max_size: 0). Iteration stats are on (enable_iter_perf_stats,iter_perf_stats_interval,kv_cache_config.iteration_stats_interval)./metricsexposedtrtllm_kv_cache_utilization. The client requires that metric, so a point without it fails.AI model disclosure
claude-opus-5-5), in Claude Code; one delegated agent on the same model..github/codeowner-signoff-verify-prompt.md.PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
新增
dsv41flash-fp4-gb300-dynamo-trt-agentic-dspark-disagg:DeepSeek-V4.1-Flash NVFP4(nvidia/DeepSeek-V4.1-Flash-NVFP4)在 GB300 上使用 Dynamo + TensorRT-LLM 分离式部署运行 AgentX。其中七个点沿用 TensorRT-LLM 团队为trtllm-serveAgentX 在该模型上调优的拓扑、引擎设置和并发;另外五个点(c11718、c2286、c1283、c180、c114)补齐该 Pareto 未覆盖的区域。镜像为
nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc4.dev202611160007(按 digest 固定),即基于提交ab2dfeba8e69构建的 TensorRT-LLMmainnightly。它包含 NVIDIA/TensorRT-LLM#21438(2026-11-14 合入)中的 DeepSeek-V4.1-Flash 优化,这些配方用到其中的 MegaMoE shared expert 融合、gen_prompt_tail_tokens、EPLBwarmup_placement、host_cache_bind_to_gpu_numa_node、严格会话亲和、max_gen_init_admissions_per_pass与enable_client_io_during_gpu_wait。Dynamo 在作业启动时安装1.7.0.dev20261110wheel,这是第一个包含 ai-dynamo/dynamo#18170 的 nightly。配置点见上方表格:c11718(10×DEP2 + 1×DEP16,batch 256)、c7000(5×DEP2 + 1×DEP8,batch 320)、c7900(7×DEP2 + 1×DEP16,batch 128)、c5100(5×DEP2 + 1×DEP16,batch 64)、c7600(7×DEP2 + 1×DEP32,batch 25)、c2286(3×DEP2 + 1×DEP32,batch 8)、c1283(2×DEP2 + 1×DEP32,batch 4),目标模型 MoE 均使用 MegaMoE CuTe DSL;c240、c140、c84(1×DEP2 + 4×TEP8,batch 4/2/1)与 c180、c114(1×DEP2 + 6×TP4,batch 2/1)使用 TRTLLM MoE。DEP2 预填充 worker 的每个 rank 在其 GPU 所在 NUMA 节点上保留原生主机 KV 层:DEP8/DEP16 点为 125 GiB,DEP32 点为 111 GiB,TEP8/TP4 点为 80 GiB。所有解码 worker 每 20 次迭代输出一次(
stream_interval: 20),预填充使用 TensorRT-LLM 默认值 1。两个角色均使用 FP8 KV cache(kv_cache_config.dtype: fp8)。MoE combine 保持 BF16(TensorRT-LLM 默认值,不设置use_low_precision_moe_combine)。DSpark 草稿模型按发布时的精度运行:其路由专家在检查点中为 MXFP4,并以 MXFP8 激活运行(
W4A8_MXFP4_MXFP8,镜像默认路径)。DEP 点的草稿 MoE 使用 CUTLASS(speculative_config.moe_backend: CUTLASS):若不设置,草稿会继承目标模型的 MegaMoE 后端,而 MegaMoE 只支持 NVFP4 专家;若为草稿显式设置 MegaMoE,则会把这些专家重新编码为 NVFP4。TEP8/TP4 点的草稿继承目标模型的 TRTLLM 后端,保持同样的 MXFP4 路径。Dynamo 设置:DEP 点使用带会话亲和的 KV 路由,预填充 worker 让新会话使用路由器选定的 attention-DP rank(
DYN_ENGINE_CONV_AFFINITY_DP_RANK_SOURCE=dynamo),使子代理的首轮请求落在持有其父会话前缀的 rank 上,AgentX 客户端运行在独立节点上;TEP8 与 TP4 点的解码按最小负载路由,不使用会话亲和。所有点的 frontend 与 worker 以打包字节传递 token ID(DYN_TOKEN_IDS_AS_BYTES=1),worker 绑定 GPU 所在 NUMA 节点的 CPU,并优先使用该节点内存。指标与功耗:所有 worker 发布 TensorRT-LLM 迭代与 KV cache 指标(
engine.publish_metrics: true,不启用 KV 事件),预填充与 TEP8/TP4 解码的iter_perf_stats_interval和kv_cache_config.iteration_stats_interval为 40,DEP 解码为 400;客户端要求trtllm_kv_cache_utilization指标;所有点设置power: true。逐请求性能指标保持关闭,这是 ai-dynamo/dynamo#18170(--publish-per-request-metrics)之后的默认行为。在更早的 Dynamo 中,--publish-metrics会同时开启逐请求指标,TensorRT-LLM 的record_stats随后会在解码 handler 中把每个约 130K token 的 prompt 重建为 Python 列表并使其饱和:total tok/s/GPU 在 c7900 下降 39.5%、c7600 下降 30.9%、c5100 下降 18.8%、c7000 约下降 13%。NVIDIA/TensorRT-LLM#21674 让开启这些指标的用户开销很低;本配置不开启它们。验证:全部十二个点在一个内部 NVIDIA GB300 NVL72 集群(无公开运行链接)上使用本 PR 绑定后的配方端到端运行(开启指标与功耗),当时 DEP 点的草稿 MoE 仍使用 MegaMoE。c7900、c5100、c7600 与关闭指标和功耗的同一配方相差不超过 1%;c7000 在其测得的窗口内低 3.3%,原因尚未定位;TEP8 点低 1–3%,即共享 DCGM 遥测配置(
--collect-interval=100)对这些小 batch 点造成的代价,改为 1000 ms 后该代价消失。这些运行的解码还使用了 FP8 MoE combine。按本 PR 将 DEP 点的草稿改为 CUTLASS 后,输出吞吐再下降 1–2%(c7000、c7900、c7600 的短时运行,900 s 窗口,与上述运行的前 900 s 相比:output tok/s/GPU 分别 −1.1%、−1.2%、−1.9%);将 MoE combine 恢复为 BF16 的代价不超过 0.5%(c7900、c7600、c84 的短时运行:output tok/s/GPU 分别 −0.3%、−0.4%、−0.5%)。这些全量运行使用 #21438 所来自的feat/ds-v4.1-bringuprelease 镜像(TensorRT-LLM054a88929a1),Dynamo 为1.7.0.dev20261109并应用 #18170 的 Python 改动(当时尚无包含 #18170 的 nightly);1.7.0.dev20261110包含同样的改动。在mainnightly 上以本 PR 的完整配置进行的短时运行与旧镜像一致:output tok/s/GPU 在 c7900 为 +0.7%、c7600 为 +0.1%、c84 为 +1.2%、c180 为 +2.5%。各点数据见上方 Validation details 表格。AI 模型使用说明:使用 Claude Code 中的 Claude Opus 5.5(
claude-opus-5-5)编写配方、master 配置与 changelog 条目,运行验证并分析结果,撰写本说明;另有一个同模型的委派 agent 按.github/codeowner-signoff-verify-prompt.md独立审查了改动。改动类型:配置改动(Configuration change)。
🤖 Generated with Claude Code