Repository navigation
[NV] dsr1 fp4 b200 trt agg update - #620
Conversation
|
hi @camiloamoreno thanks for the PR, can u please update the trtllm recipes about this? https://docs.nvidia.com/llm-inference-quick-start-recipes/index.html @claude can u create an ticket about updating the deepseek trtllm recipe and assigning @kedarpotdar-nv and @camiloamoreno to it |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
|
@pr-claude review |
|
Claude finished @cquil11's task in 1m 37s —— View job PR Review Complete
SummaryOverall the PR looks good. The container image update to TRTLLM
No blocking issues found. The benchmark script logic is sound and the edge case for |
| fi | ||
| if [[ "$ISL" == "1024" && "$OSL" == "1024" ]]; then | ||
| if [[ "$TP" == "8" && "$EP_SIZE" == "8" ]]; then | ||
| PIECEWISE_CUDA_GRAPHS="true" |
There was a problem hiding this comment.
🟡 WARNING: Comment for piecewise CUDA graphs is incomplete/misleading
The comment mentions "TEP8 and CONC64" but looking at the nvidia-master.yaml config:
- Line 842:
{ tp: 8, ep: 8, conc-start: 64, conc-end: 64 }- matches (no dp-attn) - Line 843:
{ tp: 8, ep: 8, dp-attn: true, conc-start: 128, conc-end: 256 }- also has TP=8/EP=8 but with dp-attn
Why it matters: When dp-attn: true, the script sets PIECEWISE_CUDA_GRAPHS="true" AND then overwrites CUDA_GRAPH_MAX_BATCH_SIZE to CONC/4. The piecewise cuda graphs config includes a capture_num_tokens list that's generated from MAX_NUM_TOKENS. This interaction isn't documented.
Fix: Update the perf-changelog description or add a comment here clarifying which cases actually use piecewise CUDA graphs (TP=8, EP=8 at any concurrency for 1k1k, regardless of dp-attn).
This PR contains below updates:
Near the top of the benchmark script (L26-28), we use flags to enable specific optimizations for specific ISL/OSL/CONC combinations according to the chosen TP/EP/DP-ATTENTION in nvidia-master and to what is best for balancing ctx vs gen workload and achieving good utilization.
Further down the script, from L47 onwards, we key off these flags to generate the necessary yaml for enabling these features in the runtime config file.
Piecewise Cuda Graphs (https://nvidia.github.io/TensorRT-LLM/features/torch_compile_and_piecewise_cuda_graph.html) enables some components to execute thorugh cuda graphs while other components are run eagerly, to gain benefit with lower overhead. We use the formula from the documentation to generate a capture_num_tokens list dynamically as it depends on MAX_NUM_TOKENS which is not constant across cases.
"cuda graph max batch size" is mostly equal to CONC as the natural limit on the expected size of a batch, but is configured as batch_size/4 when DP_ATTENTION is enabled.