Skip to content

[Do NOT Merge][Preview branches] DeepSeek-V4.1-Flash performance optimization - #21017

Closed
zhenhuaw-me wants to merge 1 commit into
NVIDIA:mainfrom
zhenhuaw-me:preview/dsv4.1-flash
Closed

zhenhuaw-me wants to merge 1 commit into
NVIDIA:mainfrom
zhenhuaw-me:preview/dsv4.1-flash

Conversation

@zhenhuaw-me

@zhenhuaw-me zhenhuaw-me commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Description

Preview branch: do not merge. This draft PR publishes the DeepSeek-V4.1-Flash
performance optimization work as a single commit for technical reference only. It is not
intended to be merged into main and will stay in draft.

  • Base: 16df7eaf25 ("[TRTLLM-16621][feat] Support DeepSeek-V4.1-Flash ([TRTLLM-16621][feat] Support DeepSeek-V4.1-Flash #19727)"), the
    main commit the work was last synced to. Rebasing onto the latest main conflicts in about 78
    files (the cute_dsl_kernels/ split, KV cache manager V2, and the speculative-decoding runners).
    That integration is left to the product merge.
  • Main areas:
    • CSA2 sparse attention: single-path indexer, native FP8 staging and context attention, a fused
      sparse-context kernel, and CuTe DSL decode kernels.
    • mHC, Engram and DSpark: fused CuTe DSL kernels, host-NUMA hugepage Engram tables with head
      sharding, and DSpark draft preparation with sharded sampling.
    • MoE and GEMM: MegaMoE at non-512-aligned sizes, shared-expert overlap, MXFP8 Swap-AB decode
      projections, and a low-M FP8 GEMM.
    • Runtime: bounded encoder/decoder replay, an FP8 KV cache by default, KV cache manager V2
      host-memory changes, and SM107 enablement.
  • Left out: CI pipeline changes that only applied to the internal development branch (Jenkins
    stage overrides and the DS-V4.1-only test-db lists).
  • Authorship: the main contributor is Ziyi (Fernando) Xiong. Every contributor is credited in
    the commit.

Test Coverage

Verified on 4×GB200. This branch's own perf-sanity and TestDeepSeekV41 accuracy cases are gated
to GB300.

  • Pre-commit over the full diff: all hooks pass, including mypy with the compiled bindings.
  • Source build (-a 100-real, Release): passes.
  • DS-V4.1 single-GPU unit tests (CSA2 backend, indexer and sparse context; Engram; CED, encoder
    and decoder replay; breakable CUDA graphs): 269 passed, 2 skipped, 2 failed.
    • Known failure on GB200:
      unittest/_torch/modeling/test_deepseek_v41_ced.py::test_final_window_mixed_batch_keeps_finished_context_and_generation[auto|fp8].
    • About 1.5% of elements exceed the test's atol=0.02, with a max absolute difference of 0.043.
    • The failure is deterministic across runs and GPUs, and the test passes on B200.
    • It looks like a platform-dependent numerical difference under a tight tolerance; GB200 has
      152 SMs and B200 has 148.
  • accuracy/test_llm_api_pytorch.py::TestDeepSeekV41FlashDSpark::test_gsm8k_tp4[bounded_replay=True-attention_dp=False]
    passes. This is full GSM8K with TP4/EP4, DSpark, an FP8 KV cache and bounded replay. Scores are
    95.38 (flexible-extract) and 94.77 (strict-match), against a reference of 93.33.

To reproduce the low-latency perf cases on GB300×4:
perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_v41_flash_grace_blackwell-v41_flash_tep4_8k1k].
The v41_flash_tep4_dspark_8k1k and v41_flash_dep4_8k1k cases run the same way.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

@zhenhuaw-me zhenhuaw-me changed the title [Do NOT Merge][Preview branches] DeepSeek-V4.1-Flash low-latency opti… [Do NOT Merge][Preview branches] DeepSeek-V4.1-Flash performance optimization Oct 26, 2026
…mization

Preview of the DeepSeek-V4.1-Flash performance optimization, squashed onto
16df7ea ("Support DeepSeek-V4.1-Flash", NVIDIA#19727). It is for technical
reference only and is not intended to be merged into main.

Main areas:
- CSA2 sparse attention: single-path indexer, native FP8 staging and context
  attention, a fused sparse-context kernel, and CuTe DSL decode kernels.
- mHC, Engram and DSpark: fused CuTe DSL kernels, host-NUMA hugepage Engram
  tables with head sharding, and DSpark draft preparation with sharded sampling.
- MoE and GEMM: MegaMoE at non-512-aligned sizes, shared-expert overlap, MXFP8
  Swap-AB decode projections, and a low-M FP8 GEMM.
- Runtime: bounded encoder/decoder replay, an FP8 KV cache by default, KV cache
  manager V2 host-memory changes, and SM107 enablement.

Main contributor: Ziyi (Fernando) Xiong.

Co-authored-by: Ziyi Xiong <219238287+ziyixiong-nv@users.noreply.github.com>
Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com>
Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com>
Co-authored-by: Qi Zhang <10434017+Tracin@users.noreply.github.com>
Co-authored-by: Alessio Netti <26897207+AlessioNetti@users.noreply.github.com>
Co-authored-by: Linquan Hua <34030136+kris1025@users.noreply.github.com>
Co-authored-by: Zihao Kong <161090975+zihaok@users.noreply.github.com>
Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com>
Co-authored-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com>
Co-authored-by: Xianjie Qiao <5410381+qiaoxj07@users.noreply.github.com>
Signed-off-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com>
@zhenhuaw-me

Copy link
Copy Markdown
Member Author

Suppressed by #21438

@zhenhuaw-me zhenhuaw-me closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant