Skip to content

[NV] dsr1 fp4 b200 trt agg update - #620

Merged
camiloamoreno merged 6 commits into
mainfrom
nv/dsr1-fp4-b200-agg-260202
Feb 3, 2026
Merged

camiloamoreno merged 6 commits into
mainfrom
nv/dsr1-fp4-b200-agg-260202

Conversation

@camiloamoreno

Copy link
Copy Markdown
Contributor

This PR contains below updates:

  • Update to the scripts to use latest TRTLLM release container
  • Fine-tune choice of parallelism in nvidia-master for various cases
  • Enable a few optimizations offered by TensorRT LLM on specific cases (MOE_BACKEND and Piecewise cuda graphs)

Near the top of the benchmark script (L26-28), we use flags to enable specific optimizations for specific ISL/OSL/CONC combinations according to the chosen TP/EP/DP-ATTENTION in nvidia-master and to what is best for balancing ctx vs gen workload and achieving good utilization.

Further down the script, from L47 onwards, we key off these flags to generate the necessary yaml for enabling these features in the runtime config file.

Piecewise Cuda Graphs (https://nvidia.github.io/TensorRT-LLM/features/torch_compile_and_piecewise_cuda_graph.html) enables some components to execute thorugh cuda graphs while other components are run eagerly, to gain benefit with lower overhead. We use the formula from the documentation to generate a capture_num_tokens list dynamically as it depends on MAX_NUM_TOKENS which is not constant across cases.

"cuda graph max batch size" is mostly equal to CONC as the natural limit on the expected size of a batch, but is configured as batch_size/4 when DP_ATTENTION is enabled.

@functionstackx

Copy link
Copy Markdown
Collaborator

hi @camiloamoreno thanks for the PR, can u please update the trtllm recipes about this?

https://docs.nvidia.com/llm-inference-quick-start-recipes/index.html
https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/deployment-guide/deployment-guide-for-deepseek-r1-on-trtllm.md

@claude can u create an ticket about updating the deepseek trtllm recipe and assigning @kedarpotdar-nv and @camiloamoreno to it

@Klaud-Cold

Klaud-Cold commented Feb 2, 2026 •

Copy link
Copy Markdown
Collaborator

Claude encountered an error —— View job


I'll analyze this and get back to you.

Comment thread .github/configs/nvidia-master.yaml Resolved
Comment thread benchmarks/dsr1_fp4_b200_trt.sh Resolved
@cquil11

cquil11 commented Feb 3, 2026

Copy link
Copy Markdown
Collaborator

@pr-claude review

@claude

claude Bot commented Feb 3, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @cquil11's task in 1m 37s —— View job


PR Review Complete

  • Gather context and understand PR changes
  • Review .github/configs/nvidia-master.yaml changes
  • Review benchmarks/dsr1_fp4_b200_trt.sh changes
  • Review perf-changelog.yaml changes
  • Validate master config + perf-changelog consistency ✅
  • Validate container image accessibility ✅ (nvcr.io#nvidia/tensorrt-llm/release:1.2.0rc6.post2)
  • Post review summary

Summary

Overall the PR looks good. The container image update to TRTLLM 1.2.0rc6.post2 is valid, and the perf-changelog.yaml entry correctly documents the changes. I found 2 minor issues:

  1. 🟡 Overlapping search space at conc=32 for ISL=8192/OSL=1024 with TP=4 - could result in duplicate benchmark runs (see inline comment)

  2. 🟡 Documentation clarification - the perf-changelog mentions "TEP8 and CONC64" for piecewise CUDA graphs, but the config also runs TP8/EP8 at higher concurrencies with dp-attn (see inline comment)

No blocking issues found. The benchmark script logic is sound and the edge case for CONC < 4 with dp-attn was already addressed per the review comments.


Comment thread .github/configs/nvidia-master.yaml Resolved
fi
if [[ "$ISL" == "1024" && "$OSL" == "1024" ]]; then
if [[ "$TP" == "8" && "$EP_SIZE" == "8" ]]; then
PIECEWISE_CUDA_GRAPHS="true"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: Comment for piecewise CUDA graphs is incomplete/misleading

The comment mentions "TEP8 and CONC64" but looking at the nvidia-master.yaml config:

  • Line 842: { tp: 8, ep: 8, conc-start: 64, conc-end: 64 } - matches (no dp-attn)
  • Line 843: { tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256 } - also has TP=8/EP=8 but with dp-attn

Why it matters: When dp-attn: true, the script sets PIECEWISE_CUDA_GRAPHS="true" AND then overwrites CUDA_GRAPH_MAX_BATCH_SIZE to CONC/4. The piecewise cuda graphs config includes a capture_num_tokens list that's generated from MAX_NUM_TOKENS. This interaction isn't documented.

Fix: Update the perf-changelog description or add a comment here clarifying which cases actually use piecewise CUDA graphs (TP=8, EP=8 at any concurrency for 1k1k, regardless of dp-attn).

@cquil11 cquil11 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@camiloamoreno
camiloamoreno merged commit 6eecc0e into main Feb 3, 2026
6 of 30 checks passed
@camiloamoreno
camiloamoreno deleted the nv/dsr1-fp4-b200-agg-260202 branch February 3, 2026 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants