Skip to content

[TRTLLM-16622][feat] DeepSeek-V4.1: Perf optimizations for DS-V4.1-Flash-NVFP4 - #21438

Merged
ziyixiong-nv merged 36 commits into
NVIDIA:mainfrom
ziyixiong-nv:ds-v4.1-perf-opt
Nov 14, 2026
Merged

ziyixiong-nv merged 36 commits into
NVIDIA:mainfrom
ziyixiong-nv:ds-v4.1-perf-opt

Conversation

@ziyixiong-nv

@ziyixiong-nv ziyixiong-nv commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

This PR ports the DeepSeek-V4.1 performance optimizations onto main.

It brings:

  • DeepSeek-V4.1 on the PyTorch backend: compressed KV, mHC, Engram and the DSpark draft;
  • CSA2 sparse attention: decode and prefill kernels, the indexer and exact Top-K;
  • bounded encoder and decoder replay, with SWA scratch reuse;
  • MegaMoE: the one-round push router, the fused shared expert, eager launches without PDL, and warmup expert placement (moe_config.load_balancer.warmup_placement, standard mode only);
  • GB300 (SM103) and SM107 tuning for disaggregated serving, with the executor, KV-cache and test changes these need.

Shared code keeps main's behavior for other models. Where this PR changes a shared path for DeepSeek-V4/V4.1, the model opts in:

  • RuntimeHooks set by DeepseekV4ForCausalLM: stage_decode_inputs, prioritize_decode_graphs, and loop_on_execution_stream with speculative decoding;
  • use_fp8_small_m_gemm for the FP8 small-M GEMM race;
  • DeepSeek-V4 routing for the NVLink one-sided pull combine, one-round MegaMoE dispatch, CuteDSL FC1 zeroing and the spread tuning profile;
  • caller flags for the wide Top-K configurations (CSA2 only).

@coderabbitai summary

Description

After applying the optimizations, the AgentX perf is as below.
image

Behavior changes to know about:

  • DeepSeek-V4.1 runs Encoder replay by default when Decoder bounded replay runs with CED precompute; TRTLLM_V41_ENCODER_REPLAY=0 keeps the required Encoder state.
  • CSA2 candidate-consumer layers need an SM100-family GPU, DeepGEMM's sparse MQA-logits kernels, 32 index heads of 128 dims and candidate_block_size a multiple of 8; model construction fails otherwise.
  • gen_prompt_tail_tokens needs KV cache manager V2 and transceiver_runtime: PYTHON (set explicitly; auto is rejected).
  • The DeepGEMM mqa_logits patch gives bitwise-identical logits for finite inputs; NaN and -inf scores now contribute 0 instead of NaN (DSA prefill: DeepSeek-V3.2, GLM-5).

Answers to review questions:

  • The CSA2 cutover constants (_DENSE_O_B_MIN_ROWS, PROJECTION_CUTE_DSL_ROWS, _N256_MIN_ROWS, the fitted constants in csa2SparseDecode.cu) and HOLD_MAX_ROWS were tuned with DeepSeek-V4.1 on GB300 (SM103); they were not measured on B200. They are reached only by DeepSeek-V4.1.
  • GVR self-sampling decode on SM107, against the exact decode Top-K, DeepSeek-V4.1 TP8 synthetic-KV generation throughput: C1 -1.27%, C256 -5.95%, C8192 -6.69%. SM107 decode therefore keeps the exact Top-K; prefill uses self-sampling.
  • gen_prompt_tail_tokens >= 256: generation workers read prompt ids only near the end (the Engram lookback and stop sequences that straddle the prompt end); 256 covers both with margin.

Test Coverage

New L0 entries:

  • l0_b200 (SM100): DS-V4.1 bounded replay, snapshot and decoder-global model tests; DSpark draft step, acceptance, argmax chain and sampling buffers; fused DS-V4 route+quant (TRTLLM, MegaMoE); FP8 small-M MLP; vocab-slice embedding; LL all-gather; deferred gen-init receive.
  • l0_b300 (SM103): all of test_mhc.py and test_engram.py, replay model tests, DSpark CuTe DSL attention, draft step, argmax chain, WKV projection, DS-V4 route+quant.
  • l0_gb200_multi_gpus: MegaMoE e2e, warmup placement and combine tactics; DSpark Markov chain and forked draft prep; peer buffers; mHC all-reduce boundary; MNNVL RMSNorm; attention-DP decoder-pass graph and Engram sharding; autotuner cache disagreement; CSA2 two-rank TP.
  • l0_gb300: DSpark Markov chain and forked draft prep. l0_gb300_multi_gpus (post-merge): MegaMoE e2e and placement, attention-DP decoder/encoder-pass graphs, breakable prefill graphs against eager prefill.
  • l0_gr100 (SM107, post-merge): DSpark CuTe DSL attention and RMSNorm/RoPE.
  • Accuracy: TestDeepSeekV41::test_gsm8k_dspark[nvfp4-tp4] / [nvfp4-dp4] (l0_gb300) and test_breakable_prefill_cuda_graph.
  • l0_dgx_h100 (existing entry): test_disaggregated_deepseek_v3_lite_fp8_attention_dp_overlap now also runs the disaggregated scheduling fields (gen-init admission cap, client I/O during the GPU wait, prompt-tail trimming, prefill-aware conversation placement) and checks the answer to a long prompt that the generation workers receive trimmed.

Run on GB300 (SM103) at dfcfb6e, rebased on main ee1e8e4, with its C++ changes built: the CSA2 kernel, module, decode (with and without PDL), indexer and cache tests; the DS-V4.1 model tests; attention-DP, Engram sharding, MegaMoE e2e and warmup placement; the DSpark, speculative and sharded-sampling tests; GSM8K with DSpark (TP4, DP4); the extended attention-DP overlap disaggregated test; CSA2 tensor parallel; breakable prefill; and Qwen3.8-Flash-Next MMMU. Not run here: SM100 (B200) and the 8-GPU cases (helix CP, world-8 attention DP).

Since then, on GB300 with the C++ changes built:

  • at 15e5184 (main 1399e99): the PyExecutor, LLM API and API-stability unit tests; main's new engine, sampler, speculative and QSA tests; the MXFP8 and MiniMax-M3 dispatch tests; GSM8K with DSpark (TP4);
  • at 9c7e147 (main 7a06809): the mHC, DSv4 compressor, DSpark engine-runtime, decoder-runner, DS-V4.1 model, disaggregated extractor, mixed-precision KV and serve-schema tests; GSM8K with DSpark (TP4).

The head 66f5889 is 9c7e147 rebased onto main d782229.

AgentX on GB300 at 9c7e147, this PR against the bring-up branch it ports (first 30 minutes of each run):

  • hi (c7000): +1.3 to +1.8% throughput, +1.6 to +2.6% tokens/s per user;
  • mid (c7600): +0.5 to +3.7% throughput, +1.0 to +5.2% tokens/s per user;
  • ll (TP8, c84): +0.0 to +0.2% throughput, +0.3 to +0.4% tokens/s per user.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #81973 [ run ] triggered by Bot. Commit: 12faa4c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #81973 [ run ] completed with state SUCCESS. Commit: 12faa4c
/LLM/main/L0_MergeRequest_PR pipeline #67615 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

  • Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
  • If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
    Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@sunnyqgg sunnyqgg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. This review has 41 blockers, 32 possibly-blocking items and 18 required items: 87 are inline, and the 4 below have no anchor in the diff. The most urgent are the TRTLLMINF-435 TEMPORARY CI override, stages and test lists, which must not reach main, the version regression to 1.4.0a0, and the shared-default changes that alter behavior for models other than DS-V4.1 without data. Some areas got only a light pass: the inner pipelines of the large CuTe DSL kernels (CSA2 decode/prefill, MegaMoE, mHC); the ~28k lines of new tests, checked for registration, whether they can fail, hangs and env leaks but not test logic; and deepgemm.patch, not compared line by line with the pinned upstream source. Nothing was run on a GPU, and CI had not reported for PR_Github #81973 when this was written.

  • [Possibly blocking] cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/indexConversionKernels.cu:160 (outside the diff). The PR changes the SWA scratch rule to "a block in the scratch range that already has a regular page keeps it". It updates four page-table builders:

    • PageIndexConverter (kvCacheManager.cpp:74);
    • the Python table (kv_cache_manager_v2.py:1485);
    • both DeepSeek-V4 block-table kernels (deepseekV4BlockTable.cu:205/284-285);
    • the CSA2 converter (csa2/kernel.py:444).

    It does not update indexConversionScratchKernel, which Batch::_convertIndices uses (batch.cpp:882-896); that kernel still maps every block in [scratchBeg, scratchEnd) to scratch. The two can disagree whenever a speculative reservation has kept a regular page in the scratch range. A caller of the public tensorrt_llm.runtime.kv_cache_manager_v2.Batch index tables with SWA scratch reuse then gets a page table that differs from the manager's own converter: KV goes to scratch while the persistent page goes stale. This is latent today, because the PyTorch executor doesn't build a Batch with index tables. Separately, the reference in tests/microbenchmarks/dsv4_block_table_perf.py:77-78 still uses the old rule and fails its first comparison with the default --bad-ratio 0.05. Please add raw[block] == kBadPageIndex && to the scratch predicate, update the microbenchmark reference, and consider one shared helper for the five copies of the rule.

  • [Required] The PR title, description and scope don't follow the repo rules:

    • Scope. The title says "Perf optimizations for DS-V4.1-Flash-NVFP4", but the PR ports the whole bring-up branch: 389 files, +76k/-7.7k. It also includes two [fix] commits for KV cache manager v2 that the description doesn't mention.
    • Ticket. It is [None]. CONTRIBUTING.md:91 says: "For NVIDIA developers, please include the JIRA number or NVBUG ID in the PR title whenever possible."
    • Testing. Test Coverage is empty, and the description says the full CI run "has not been re-run on this commit".
    • Undescribed shared changes. Several changes affect all models:
      • logits processors are rejected for every one-model speculative mode;
      • attached frontends start with CUDA_VISIBLE_DEVICES="";
      • decode graphs with ≤256 tokens are captured on a priority stream;
      • every BF16 embedding on SM100 runs a new CuTe kernel;
      • every TRTLLM-backend model gets a HostUploader;
      • the autotuner adds a per-profile cross-rank sync and cold-L2 profiling;
      • MNNVL all-reduce returns None as a fallback;
      • the DeepGEMM sm100_mqa_logits patch.
    • Reviewability. Owners of the shared code (executor, sampler, serve, autotuner, embedding, distributed) can't evaluate their parts. Once squashed, any regression bisects to a single 76k-line commit.
    • Move plus rewrite. dspark_rmsnorm_rope.py → dspark/rmsnorm_rope.py moves the file and rewrites it in the same change (+184/-100).

    AGENTS.md: "One concern per PR — avoid scope creep. If a PR touches unrelated areas, split it."

    Please add a JIRA ticket and split the PR into:

    • infra;
    • each shared default change, with non-DS-V4.1 perf/accuracy data;
    • the generic disaggregated-serving features;
    • the KV cache manager v2 fixes;
    • the DS-V4.1 model work.

    Then fill in Description and Test Coverage, and re-run CI on the final commit.

  • [Required] The user docs contradict the code, and new features are undocumented. The docs/source files are outside the diff.

    • Encoder replay. docs/source/features/sparse-attention.md:353-368 and docs/source/features/feature-combination-matrix.md:7-15 still say Encoder replay is opt-in ("Encoder replay is opt-in via TRTLLM_V41_ENCODER_REPLAY=1").
    • Hardware. Those two pages and the CSA2 README hardware table (csa2/README.md:21-23/43/110-112/352) still list SM90 and SM120/121. Candidate-consumer layers now need SM100.
    • SM103 decode path. The CSA2 README leaves out the SM103 gen_fmha decode path (backend.py:354-355/453, gen_fmha.py:28), and two of its statements are not true for that path.
    • New knobs. None of these has a docs/source page:
      • the env var TRTLLM_V41_BOUNDED_REPLAY_TOKENS;
      • prompt_token_cache_config, gen_prompt_tail_tokens and enable_client_io_during_gpu_wait;
      • max_gen_init_admissions_per_pass and host_cache_bind_to_gpu_numa_node;
      • megamoe_fuse_shared_expert and warmup_placement.
    • Supported models. supported-models.md has no V4.1 hardware note.

    As a result, operators will:

    • assume approximate Encoder recovery is off;
    • deploy checkpoints with candidate layers on Hopper or SM120, where they fail at warmup;
    • look at csa2_sparse_decode while debugging GB300 large-batch decode, when an undocumented third kernel is running.

    Please:

    • update sparse-attention.md, the feature matrix and the CSA2 README (encoder replay default, the SM100 requirement for candidate layers, the gen_fmha path);
    • document the replay env var or move it into the config;
    • document the new knobs;
    • add the V4.1 hardware footnote to supported-models.md.
  • [Required] tests/unittest/llmapi/test_trtllm_serve_config_schemas.py is unchanged, although this PR adds serving YAML fields and constraints:

    • gen_prompt_tail_tokens (ge=256);
    • the MoeWarmupPlacementConfig bounds;
    • PromptTokenCacheConfig;
    • the least_prefill value;
    • CSA2's _validate_gvr_emission.

    AGENTS.md: "Changes to serving YAML fields, aliases, defaults, types, or constraints ... must regenerate and inspect the editor JSON Schemas ... and add regression coverage for the affected accepted/rejected values."

    Without those tests:

    • the schema and the runtime can drift apart silently;
    • the rejection list in validate_gen_prompt_tail_tokens has no unit test;
    • test_load_balancer_schema_matches_runtime doesn't cover warmup_placement.

    Please add accepted and rejected schema cases for each new field and validator, and extend test_load_balancer_schema_matches_runtime to cover warmup_placement.

Comment thread jenkins/L0_Test.groovy Resolved
Comment thread jenkins/L0_MergeRequest.groovy Resolved
Comment thread tensorrt_llm/version.py Resolved
Comment thread tensorrt_llm/llmapi/llm_args.py Resolved
@ziyixiong-nv ziyixiong-nv changed the title [None][feat] DeepSeek-V4.1: Perf optimizations for DS-V4.1-Flash-NVFP4 [TRTLLM-16622][feat] DeepSeek-V4.1: Perf optimizations for DS-V4.1-Flash-NVFP4 Nov 3, 2026
@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. The fixes are in fbe5b2a, and each inline thread has a reply. On the items in the review body:

Scope, JIRA, description, split

We'll keep this as one PR: splitting the bring-up into five dependent PRs would serialize their reviews and CI and push the merge out much further. To keep the shared code reviewable, the CI/infra changes are reverted, and the shared-default changes you listed now keep main's behavior for other models and apply only to DeepSeek-V4/V4.1 (see the individual threads). The title now carries TRTLLM-16622, the description now lists the behavior changes, the answers to your questions and the Test Coverage, and CI will be re-run on the final commit.

indexConversionKernels.cu (SWA scratch predicate)

Fixed: indexConversionScratchKernel now takes a scratch slot only for a block whose raw entry has no regular page, the same rule as PageIndexConverter and the other three builders, and the microbenchmark reference applies it too. The existing test that compares a Batch's converted table with PageIndexConverter now also runs with a speculative reservation whose regular page lies inside the next chunk's scratch range. The five copies live in C++, two CUDA kernels in different libraries, torch and CuTe DSL, so a shared helper would cover only two of them; each keeps the same one-line predicate.

User docs vs code

Thanks, the docs now follow the code. sparse-attention.md and the feature matrix say that Encoder replay is on by default whenever Decoder bounded replay runs with CED precompute, with TRTLLM_V41_ENCODER_REPLAY=0 to keep required Encoder state and the fallbacks for other layouts and small max_num_tokens; they document TRTLLM_V41_BOUNDED_REPLAY_TOKENS (default: 140, or the required window plus one) and limit SM90 and SM120/121 to checkpoints without candidate-consumer layers, and supported-models.md has a matching DeepSeek-V4.1 footnote. The CSA2 README's routing table and text now cover the SM103 gen_fmha decode path for large ratio-2 generation tiles and no longer say that every 64-head generation tile skips staging for the fused decode. The new knobs are documented next to their features: the prompt token cache with the prefix-tokenization cache, client I/O with the overlap scheduler, prompt-tail transport and the admission bound with disaggregated serving, host-tier NUMA binding with KV cache offloading, warmup placement with EPLB, the fused shared expert in the MoE performance guide, and TRTLLM_ENGRAM_EVICT_PAGE_CACHE in the model footnote. The replay switches remain environment variables in this PR, like the existing TRTLLM_V41_* switches.

test_trtllm_serve_config_schemas.py coverage

Added schema regression cases in test_trtllm_serve_config_schemas.py. test_config_validation covers gen_prompt_tail_tokens (256 accepted, 255 rejected, also under generation_servers), PromptTokenCacheConfig, least_prefill and the CSA2 field set, and test_load_balancer_schema_matches_runtime now covers warmup_placement (accepted configs, the required start_step, the interval_steps/decay/max_moved_fraction bounds, an unknown planner and per-layer dynamic mode). Rules that span fields, which the editor schema does not express (validate_gen_prompt_tail_tokens, CSA2 _validate_gvr_emission, warmup placement with online EPLB), have a new test that validates each YAML against the schema and then builds TorchLlmArgs from it. The checked-in schema snapshots are regenerated.

@ziyixiong-nv
ziyixiong-nv marked this pull request as ready for review November 3, 2026 07:50
@ziyixiong-nv
ziyixiong-nv requested review from a team and peihu-nv as code owners November 3, 2026 07:50
@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@ziyixiong-nv
ziyixiong-nv requested review from a team as code owners November 3, 2026 07:50
…world size

The coordinator now reads the distributed world size to decide whether a rank may
post ready generation receives first; main's diagnostics tests build it from a stub
without one. Test: tests/unittest/disaggregated/test_disagg_transfer_diagnostics_wiring.py.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…te the coordinator reads

The coordinator waits for the KV manager's receive readiness before posting a V2
receive, and reports completion only for receives it started. Main's diagnostics
tests stubbed neither. Test: tests/unittest/disaggregated/test_disagg_transfer_diagnostics_wiring.py.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…the QA list

The test runs in the GB300 multi-GPU pre-merge list; QA's single-node list now runs
it too.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…apacities to a storage policy

SmallVector is generic, so it now lives in tensorrt_llm/common. The KV cache
manager's inline capacities (four life cycles, one beam, two finish events) and the
typed container aliases built on them move to kv_cache_manager_v2/storagePolicy.h,
which keeps storage tuning apart from the cache algorithms. No behavior change.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…t out of the host tier

The host tier no longer carries NUMA placement. KVCacheManagerConfig.host_mem_backing
(HostMemBackingConfig: numa_node, allow_remote_numa_fallback) says how the host tier's
memory is backed; the storage manager turns it into HostMemBacking options, where the
policy is applied and its failures are reported, and the tier never reads it.
HostCacheTierConfig.allow_remote_numa_fallback stays as a deprecated input that can only
make the placement stricter. host_cache_bind_to_gpu_numa_node now fills the backing
config. No behavior change.

Test: test_kvcm2_integration.py::test_host_tier_bound_to_gpu_numa_node.
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…y in iteration stats

Several KV decision points left no trace in iteration stats, logs or Prometheus.
InflightBatchingStats now reports, for the iterations since the previous record:
context chunks whose KV growth was given back after a skipped forward or a drop,
and their tokens; the most disaggregated generation receives waiting at once to be
posted, and the time blocked on their KV admission work; generation-init requests
left by the admission cap or by the KV-transfer budget; and decoder-tail hold passes
and held requests. KvCacheStats now reports the host bytes of live page slabs and
the sparse page-table conversion paths (reused, row updates, full builds) with the
full builds' host time, which also get an NVTX range. Attention-DP rank rows carry
each rank's counters, and Prometheus exports them per rank.

Test: test_adp_router.py (wire schema).
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ats round

- NoopDisaggCoordinator overrides take_gen_receive_counters: main's test
  requires the no-op coordinator to override every entry point.
- test_deepseek_v41_ced reads the scratch slots and range through
  get_scratch_slots / get_swa_scratch_range, which replaced
  get_scratch_desc on main.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ecode

- PyExecutor: under the overlap headroom, a retiring context-and-generation
  request keeps its seat until its last response is sent, so an aggregated
  server no longer prefills the next request before that response. Aggregated
  TP8 batch-1 (DSpark k=5, concurrency 10): 1000 / P90 ITL 1,015.9 -> 1,353.0.
- CSA2: a request that joins a decode batch of the same size and query lengths
  keeps the steady prepare: it maps to row -1, its start bases are uploaded,
  the page tables re-read and its pages checked. Prepared inputs and page
  tables are unchanged.
- Disaggregated GEN: the overlap loop publishes a request's first token once;
  a request with a batch index has already run its first step.

Batch-1 GEN-only disaggregated proxy (TP4 x MoE EP2, DSpark k=5): first token
to second chunk 6.76 -> 6.15 ms p50.

Test: test_steady_page_tables.py has new requests take finished requests' rows
(steady vs full prepare); the first-token request stubs gain py_batch_idx.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…on inputs in one launch

The paged DSpark draft attention reads each request's rolling window through
two aliases of one page. Its block tables, positions, valid lengths and
capacities came from five PyTorch ops per decode step; one Triton kernel now
writes all four. CPU inputs keep the PyTorch path.

GB300, CUDA-graph replay, per decode step: the metadata takes 1.5-1.9 us
instead of 7.3-7.9 us at the batch-1, batch-25 and batch-320 decode shapes.

Test: the existing DSpark attention tests (test_dspark_cute_dsl_attention.py,
hw_agnostic/test_dspark_attention.py) on GB300.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ntry

The B300 unittest/_torch/attention aggregate already takes about 87 of its
90 minutes on main; this PR's CSA2 tests pushed it past the timeout. The CSA2
directory now has its own entry.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…he deferred-receive test

Main's NVIDIA#21909 removed DisaggTransferCoordinator's
force_terminate_ctx_for_partial_reuse argument; the deferred GEN receive test
still passed it.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…mmy test

Main's test_adp_dummy_holds_paired_draft_kv calls PyExecutor._revert_gen_alloc,
which this PR renamed to _revert_unexecuted_alloc when it also gave back the
growth of unexecuted context chunks. The test's scheduled batch holds only the
dummy generation request, so the renamed method does the same revert there.

Test: tests/unittest/_torch/pyexecutor/kv_cache/test_kvcm2_integration.py on GB300.
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…ggregate

Running CSA2 as its own B300 entry changed the attention aggregate's
test id, so pytest-split no longer found its recorded duration and
estimated it at the per-test average (about 2 min instead of about
77 min). Both B300 shards were estimated equal, but the second one
carried about 70 more minutes of tests and hit the stage time limit.

Record the new test id with its measured B300 duration. Replaying the
split with measured per-test times gives about 224 and 219 minutes of
tests per shard, in line with the B300 shards of other PRs.

Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
…nt test

The idle fetch path now flushes staged responses before it blocks, and
main's fetch-hint test calls it on a SimpleNamespace stub without that
method. The stub gets a no-op flush, as no responses are staged there.

Test: tests/unittest/_torch/pyexecutor/test_fetch_request_hint.py.
Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com>
@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84853 [ run ] triggered by Bot. Commit: d18635e Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84847 [ run ] completed with state ABORTED. Commit: 3dabd38

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84853 [ run ] completed with state FAILURE. Commit: d18635e
/LLM/main/L0_MergeRequest_PR pipeline #70214 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

1 similar comment
@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84880 [ run ] triggered by Bot. Commit: d18635e Link to invocation

@ziyixiong-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84901 [ run ] triggered by Bot. Commit: d18635e Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84880 [ run ] completed with state ABORTED. Commit: d18635e

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84901 [ run ] completed with state SUCCESS. Commit: d18635e
/LLM/main/L0_MergeRequest_PR pipeline #70259 completed with status: 'SUCCESS'

CI Report

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Agent Code Review: High Risk api-compatible Accepted LLM API contract change that is backwards-compatible ci: full pre-merge approved

Projects

None yet

Development

Successfully merging this pull request may close these issues.