Repository navigation
[TRTLLM-16622][feat] DeepSeek-V4.1: Perf optimizations for DS-V4.1-Flash-NVFP4 - #21438
Conversation
|
/bot run --disable-fail-fast |
|
PR_Github #81973 [ run ] triggered by Bot. Commit: |
|
PR_Github #81973 [ run ] completed with state
|
1812cb1 to
ff6ef97
Compare
sunnyqgg
left a comment
There was a problem hiding this comment.
Requesting changes. This review has 41 blockers, 32 possibly-blocking items and 18 required items: 87 are inline, and the 4 below have no anchor in the diff. The most urgent are the TRTLLMINF-435 TEMPORARY CI override, stages and test lists, which must not reach main, the version regression to 1.4.0a0, and the shared-default changes that alter behavior for models other than DS-V4.1 without data. Some areas got only a light pass: the inner pipelines of the large CuTe DSL kernels (CSA2 decode/prefill, MegaMoE, mHC); the ~28k lines of new tests, checked for registration, whether they can fail, hangs and env leaks but not test logic; and deepgemm.patch, not compared line by line with the pinned upstream source. Nothing was run on a GPU, and CI had not reported for PR_Github #81973 when this was written.
-
[Possibly blocking]
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/indexConversionKernels.cu:160(outside the diff). The PR changes the SWA scratch rule to "a block in the scratch range that already has a regular page keeps it". It updates four page-table builders:PageIndexConverter(kvCacheManager.cpp:74);- the Python table (kv_cache_manager_v2.py:1485);
- both DeepSeek-V4 block-table kernels (deepseekV4BlockTable.cu:205/284-285);
- the CSA2 converter (csa2/kernel.py:444).
It does not update
indexConversionScratchKernel, whichBatch::_convertIndicesuses (batch.cpp:882-896); that kernel still maps every block in [scratchBeg, scratchEnd) to scratch. The two can disagree whenever a speculative reservation has kept a regular page in the scratch range. A caller of the publictensorrt_llm.runtime.kv_cache_manager_v2.Batchindex tables with SWA scratch reuse then gets a page table that differs from the manager's own converter: KV goes to scratch while the persistent page goes stale. This is latent today, because the PyTorch executor doesn't build aBatchwith index tables. Separately, the reference in tests/microbenchmarks/dsv4_block_table_perf.py:77-78 still uses the old rule and fails its first comparison with the default--bad-ratio 0.05. Please addraw[block] == kBadPageIndex &&to the scratch predicate, update the microbenchmark reference, and consider one shared helper for the five copies of the rule. -
[Required] The PR title, description and scope don't follow the repo rules:
- Scope. The title says "Perf optimizations for DS-V4.1-Flash-NVFP4", but the PR ports the whole bring-up branch: 389 files, +76k/-7.7k. It also includes two
[fix]commits for KV cache manager v2 that the description doesn't mention. - Ticket. It is
[None]. CONTRIBUTING.md:91 says: "For NVIDIA developers, please include the JIRA number or NVBUG ID in the PR title whenever possible." - Testing. Test Coverage is empty, and the description says the full CI run "has not been re-run on this commit".
- Undescribed shared changes. Several changes affect all models:
- logits processors are rejected for every one-model speculative mode;
- attached frontends start with
CUDA_VISIBLE_DEVICES=""; - decode graphs with ≤256 tokens are captured on a priority stream;
- every BF16 embedding on SM100 runs a new CuTe kernel;
- every TRTLLM-backend model gets a
HostUploader; - the autotuner adds a per-profile cross-rank sync and cold-L2 profiling;
- MNNVL all-reduce returns None as a fallback;
- the DeepGEMM
sm100_mqa_logitspatch.
- Reviewability. Owners of the shared code (executor, sampler, serve, autotuner, embedding, distributed) can't evaluate their parts. Once squashed, any regression bisects to a single 76k-line commit.
- Move plus rewrite. dspark_rmsnorm_rope.py → dspark/rmsnorm_rope.py moves the file and rewrites it in the same change (+184/-100).
AGENTS.md: "One concern per PR — avoid scope creep. If a PR touches unrelated areas, split it."
Please add a JIRA ticket and split the PR into:
- infra;
- each shared default change, with non-DS-V4.1 perf/accuracy data;
- the generic disaggregated-serving features;
- the KV cache manager v2 fixes;
- the DS-V4.1 model work.
Then fill in Description and Test Coverage, and re-run CI on the final commit.
- Scope. The title says "Perf optimizations for DS-V4.1-Flash-NVFP4", but the PR ports the whole bring-up branch: 389 files, +76k/-7.7k. It also includes two
-
[Required] The user docs contradict the code, and new features are undocumented. The docs/source files are outside the diff.
- Encoder replay.
docs/source/features/sparse-attention.md:353-368anddocs/source/features/feature-combination-matrix.md:7-15still say Encoder replay is opt-in ("Encoder replay is opt-in viaTRTLLM_V41_ENCODER_REPLAY=1"). - Hardware. Those two pages and the CSA2 README hardware table (csa2/README.md:21-23/43/110-112/352) still list SM90 and SM120/121. Candidate-consumer layers now need SM100.
- SM103 decode path. The CSA2 README leaves out the SM103
gen_fmhadecode path (backend.py:354-355/453, gen_fmha.py:28), and two of its statements are not true for that path. - New knobs. None of these has a docs/source page:
- the env var
TRTLLM_V41_BOUNDED_REPLAY_TOKENS; prompt_token_cache_config,gen_prompt_tail_tokensandenable_client_io_during_gpu_wait;max_gen_init_admissions_per_passandhost_cache_bind_to_gpu_numa_node;megamoe_fuse_shared_expertandwarmup_placement.
- the env var
- Supported models. supported-models.md has no V4.1 hardware note.
As a result, operators will:
- assume approximate Encoder recovery is off;
- deploy checkpoints with candidate layers on Hopper or SM120, where they fail at warmup;
- look at
csa2_sparse_decodewhile debugging GB300 large-batch decode, when an undocumented third kernel is running.
Please:
- update sparse-attention.md, the feature matrix and the CSA2 README (encoder replay default, the SM100 requirement for candidate layers, the gen_fmha path);
- document the replay env var or move it into the config;
- document the new knobs;
- add the V4.1 hardware footnote to supported-models.md.
- Encoder replay.
-
[Required]
tests/unittest/llmapi/test_trtllm_serve_config_schemas.pyis unchanged, although this PR adds serving YAML fields and constraints:gen_prompt_tail_tokens(ge=256);- the
MoeWarmupPlacementConfigbounds; PromptTokenCacheConfig;- the
least_prefillvalue; - CSA2's
_validate_gvr_emission.
AGENTS.md: "Changes to serving YAML fields, aliases, defaults, types, or constraints ... must regenerate and inspect the editor JSON Schemas ... and add regression coverage for the affected accepted/rejected values."
Without those tests:
- the schema and the runtime can drift apart silently;
- the rejection list in
validate_gen_prompt_tail_tokenshas no unit test; test_load_balancer_schema_matches_runtimedoesn't coverwarmup_placement.
Please add accepted and rejected schema cases for each new field and validator, and extend
test_load_balancer_schema_matches_runtimeto coverwarmup_placement.
|
Thanks for the review. The fixes are in fbe5b2a, and each inline thread has a reply. On the items in the review body: Scope, JIRA, description, split We'll keep this as one PR: splitting the bring-up into five dependent PRs would serialize their reviews and CI and push the merge out much further. To keep the shared code reviewable, the CI/infra changes are reverted, and the shared-default changes you listed now keep main's behavior for other models and apply only to DeepSeek-V4/V4.1 (see the individual threads). The title now carries TRTLLM-16622, the description now lists the behavior changes, the answers to your questions and the Test Coverage, and CI will be re-run on the final commit.
Fixed: User docs vs code Thanks, the docs now follow the code. sparse-attention.md and the feature matrix say that Encoder replay is on by default whenever Decoder bounded replay runs with CED precompute, with
Added schema regression cases in |
|
/bot run --disable-fail-fast |
…world size The coordinator now reads the distributed world size to decide whether a rank may post ready generation receives first; main's diagnostics tests build it from a stub without one. Test: tests/unittest/disaggregated/test_disagg_transfer_diagnostics_wiring.py. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…te the coordinator reads The coordinator waits for the KV manager's receive readiness before posting a V2 receive, and reports completion only for receives it started. Main's diagnostics tests stubbed neither. Test: tests/unittest/disaggregated/test_disagg_transfer_diagnostics_wiring.py. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…the QA list The test runs in the GB300 multi-GPU pre-merge list; QA's single-node list now runs it too. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…apacities to a storage policy SmallVector is generic, so it now lives in tensorrt_llm/common. The KV cache manager's inline capacities (four life cycles, one beam, two finish events) and the typed container aliases built on them move to kv_cache_manager_v2/storagePolicy.h, which keeps storage tuning apart from the cache algorithms. No behavior change. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…t out of the host tier The host tier no longer carries NUMA placement. KVCacheManagerConfig.host_mem_backing (HostMemBackingConfig: numa_node, allow_remote_numa_fallback) says how the host tier's memory is backed; the storage manager turns it into HostMemBacking options, where the policy is applied and its failures are reported, and the tier never reads it. HostCacheTierConfig.allow_remote_numa_fallback stays as a deprecated input that can only make the placement stricter. host_cache_bind_to_gpu_numa_node now fills the backing config. No behavior change. Test: test_kvcm2_integration.py::test_host_tier_bound_to_gpu_numa_node. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…y in iteration stats Several KV decision points left no trace in iteration stats, logs or Prometheus. InflightBatchingStats now reports, for the iterations since the previous record: context chunks whose KV growth was given back after a skipped forward or a drop, and their tokens; the most disaggregated generation receives waiting at once to be posted, and the time blocked on their KV admission work; generation-init requests left by the admission cap or by the KV-transfer budget; and decoder-tail hold passes and held requests. KvCacheStats now reports the host bytes of live page slabs and the sparse page-table conversion paths (reused, row updates, full builds) with the full builds' host time, which also get an NVTX range. Attention-DP rank rows carry each rank's counters, and Prometheus exports them per rank. Test: test_adp_router.py (wire schema). Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ats round - NoopDisaggCoordinator overrides take_gen_receive_counters: main's test requires the no-op coordinator to override every entry point. - test_deepseek_v41_ced reads the scratch slots and range through get_scratch_slots / get_swa_scratch_range, which replaced get_scratch_desc on main. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ecode - PyExecutor: under the overlap headroom, a retiring context-and-generation request keeps its seat until its last response is sent, so an aggregated server no longer prefills the next request before that response. Aggregated TP8 batch-1 (DSpark k=5, concurrency 10): 1000 / P90 ITL 1,015.9 -> 1,353.0. - CSA2: a request that joins a decode batch of the same size and query lengths keeps the steady prepare: it maps to row -1, its start bases are uploaded, the page tables re-read and its pages checked. Prepared inputs and page tables are unchanged. - Disaggregated GEN: the overlap loop publishes a request's first token once; a request with a batch index has already run its first step. Batch-1 GEN-only disaggregated proxy (TP4 x MoE EP2, DSpark k=5): first token to second chunk 6.76 -> 6.15 ms p50. Test: test_steady_page_tables.py has new requests take finished requests' rows (steady vs full prepare); the first-token request stubs gain py_batch_idx. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…on inputs in one launch The paged DSpark draft attention reads each request's rolling window through two aliases of one page. Its block tables, positions, valid lengths and capacities came from five PyTorch ops per decode step; one Triton kernel now writes all four. CPU inputs keep the PyTorch path. GB300, CUDA-graph replay, per decode step: the metadata takes 1.5-1.9 us instead of 7.3-7.9 us at the batch-1, batch-25 and batch-320 decode shapes. Test: the existing DSpark attention tests (test_dspark_cute_dsl_attention.py, hw_agnostic/test_dspark_attention.py) on GB300. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ntry The B300 unittest/_torch/attention aggregate already takes about 87 of its 90 minutes on main; this PR's CSA2 tests pushed it past the timeout. The CSA2 directory now has its own entry. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…he deferred-receive test Main's NVIDIA#21909 removed DisaggTransferCoordinator's force_terminate_ctx_for_partial_reuse argument; the deferred GEN receive test still passed it. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…mmy test Main's test_adp_dummy_holds_paired_draft_kv calls PyExecutor._revert_gen_alloc, which this PR renamed to _revert_unexecuted_alloc when it also gave back the growth of unexecuted context chunks. The test's scheduled batch holds only the dummy generation request, so the renamed method does the same revert there. Test: tests/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py on GB300. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ggregate Running CSA2 as its own B300 entry changed the attention aggregate's test id, so pytest-split no longer found its recorded duration and estimated it at the per-test average (about 2 min instead of about 77 min). Both B300 shards were estimated equal, but the second one carried about 70 more minutes of tests and hit the stage time limit. Record the new test id with its measured B300 duration. Replaying the split with measured per-test times gives about 224 and 219 minutes of tests per shard, in line with the B300 shards of other PRs. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…nt test The idle fetch path now flushes staged responses before it blocks, and main's fetch-hint test calls it on a SimpleNamespace stub without that method. The stub gets a no-op flush, as no responses are staged there. Test: tests/unittest/_torch/pyexecutor/test_fetch_request_hint.py. Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
|
/bot run --disable-fail-fast |
|
PR_Github #84853 [ run ] triggered by Bot. Commit: |
|
PR_Github #84847 [ run ] completed with state |
|
PR_Github #84853 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
1 similar comment
|
/bot run --disable-fail-fast |
|
PR_Github #84880 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast |
|
PR_Github #84901 [ run ] triggered by Bot. Commit: |
|
PR_Github #84880 [ run ] completed with state |
|
PR_Github #84901 [ run ] completed with state |
This PR ports the DeepSeek-V4.1 performance optimizations onto main.
It brings:
Shared code keeps main's behavior for other models. Where this PR changes a shared path for DeepSeek-V4/V4.1, the model opts in:
@coderabbitai summary
Description
After applying the optimizations, the AgentX perf is as below.

Behavior changes to know about:
TRTLLM_V41_ENCODER_REPLAY=0keeps the required Encoder state.candidate_block_sizea multiple of 8; model construction fails otherwise.gen_prompt_tail_tokensneeds KV cache manager V2 andtransceiver_runtime: PYTHON(set explicitly;autois rejected).mqa_logitspatch gives bitwise-identical logits for finite inputs; NaN and -inf scores now contribute 0 instead of NaN (DSA prefill: DeepSeek-V3.2, GLM-5).Answers to review questions:
_DENSE_O_B_MIN_ROWS,PROJECTION_CUTE_DSL_ROWS,_N256_MIN_ROWS, the fitted constants in csa2SparseDecode.cu) andHOLD_MAX_ROWSwere tuned with DeepSeek-V4.1 on GB300 (SM103); they were not measured on B200. They are reached only by DeepSeek-V4.1.gen_prompt_tail_tokens >= 256: generation workers read prompt ids only near the end (the Engram lookback and stop sequences that straddle the prompt end); 256 covers both with margin.Test Coverage
New L0 entries:
TestDeepSeekV41::test_gsm8k_dspark[nvfp4-tp4]/[nvfp4-dp4](l0_gb300) andtest_breakable_prefill_cuda_graph.test_disaggregated_deepseek_v3_lite_fp8_attention_dp_overlapnow also runs the disaggregated scheduling fields (gen-init admission cap, client I/O during the GPU wait, prompt-tail trimming, prefill-aware conversation placement) and checks the answer to a long prompt that the generation workers receive trimmed.Run on GB300 (SM103) at dfcfb6e, rebased on main ee1e8e4, with its C++ changes built: the CSA2 kernel, module, decode (with and without PDL), indexer and cache tests; the DS-V4.1 model tests; attention-DP, Engram sharding, MegaMoE e2e and warmup placement; the DSpark, speculative and sharded-sampling tests; GSM8K with DSpark (TP4, DP4); the extended attention-DP overlap disaggregated test; CSA2 tensor parallel; breakable prefill; and Qwen3.8-Flash-Next MMMU. Not run here: SM100 (B200) and the 8-GPU cases (helix CP, world-8 attention DP).
Since then, on GB300 with the C++ changes built:
The head 66f5889 is 9c7e147 rebased onto main d782229.
AgentX on GB300 at 9c7e147, this PR against the bring-up branch it ports (first 30 minutes of each run):
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.