Repository navigation
[Do NOT Merge][Preview branches] DeepSeek-V4.1-Flash performance optimization - #21017
Closed
zhenhuaw-me wants to merge 1 commit into
Closed
zhenhuaw-me wants to merge 1 commit into
zhenhuaw-me wants to merge 1 commit into
Conversation
zhenhuaw-me
force-pushed
the
preview/dsv4.1-flash
branch
from
October 26, 2026 11:58
feadc58 to
327877f
Compare
…mization Preview of the DeepSeek-V4.1-Flash performance optimization, squashed onto 16df7ea ("Support DeepSeek-V4.1-Flash", NVIDIA#19727). It is for technical reference only and is not intended to be merged into main. Main areas: - CSA2 sparse attention: single-path indexer, native FP8 staging and context attention, a fused sparse-context kernel, and CuTe DSL decode kernels. - mHC, Engram and DSpark: fused CuTe DSL kernels, host-NUMA hugepage Engram tables with head sharding, and DSpark draft preparation with sharded sampling. - MoE and GEMM: MegaMoE at non-512-aligned sizes, shared-expert overlap, MXFP8 Swap-AB decode projections, and a low-M FP8 GEMM. - Runtime: bounded encoder/decoder replay, an FP8 KV cache by default, KV cache manager V2 host-memory changes, and SM107 enablement. Main contributor: Ziyi (Fernando) Xiong. Co-authored-by: Ziyi Xiong <219238287+ziyixiong-nv@users.noreply.github.com> Co-authored-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com> Co-authored-by: Qi Zhang <10434017+Tracin@users.noreply.github.com> Co-authored-by: Alessio Netti <26897207+AlessioNetti@users.noreply.github.com> Co-authored-by: Linquan Hua <34030136+kris1025@users.noreply.github.com> Co-authored-by: Zihao Kong <161090975+zihaok@users.noreply.github.com> Co-authored-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com> Co-authored-by: Mingyang Hao <200044211+mingyangHao@users.noreply.github.com> Co-authored-by: Xianjie Qiao <5410381+qiaoxj07@users.noreply.github.com> Signed-off-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com>
zhenhuaw-me
force-pushed
the
preview/dsv4.1-flash
branch
from
October 26, 2026 12:15
327877f to
668f64b
Compare
Member
Author
|
Suppressed by #21438 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
16df7eaf25("[TRTLLM-16621][feat] Support DeepSeek-V4.1-Flash ([TRTLLM-16621][feat] Support DeepSeek-V4.1-Flash #19727)"), themaincommit the work was last synced to. Rebasing onto the latestmainconflicts in about 78files (the
cute_dsl_kernels/split, KV cache manager V2, and the speculative-decoding runners).That integration is left to the product merge.
sparse-context kernel, and CuTe DSL decode kernels.
sharding, and DSpark draft preparation with sharded sampling.
projections, and a low-M FP8 GEMM.
host-memory changes, and SM107 enablement.
stage overrides and the DS-V4.1-only test-db lists).
the commit.
Test Coverage
Verified on 4×GB200. This branch's own perf-sanity and
TestDeepSeekV41accuracy cases are gatedto GB300.
-a 100-real, Release): passes.and decoder replay; breakable CUDA graphs): 269 passed, 2 skipped, 2 failed.
unittest/_torch/modeling/test_deepseek_v41_ced.py::test_final_window_mixed_batch_keeps_finished_context_and_generation[auto|fp8].atol=0.02, with a max absolute difference of 0.043.152 SMs and B200 has 148.
accuracy/test_llm_api_pytorch.py::TestDeepSeekV41FlashDSpark::test_gsm8k_tp4[bounded_replay=True-attention_dp=False]passes. This is full GSM8K with TP4/EP4, DSpark, an FP8 KV cache and bounded replay. Scores are
95.38 (flexible-extract) and 94.77 (strict-match), against a reference of 93.33.
To reproduce the low-latency perf cases on GB300×4:
perf/test_perf_sanity.py::test_e2e[aggr_upload-deepseek_v41_flash_grace_blackwell-v41_flash_tep4_8k1k].The
v41_flash_tep4_dspark_8k1kandv41_flash_dep4_8k1kcases run the same way.PR Checklist