Repository navigation
Conversation
fungaren
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
OrangeRedeng,
Qiaolin-Yu,
b8zhong,
ch-wan,
hebiao064,
ispobock,
merrymercy and
mmangkad
as code owners
September 21, 2026 09:51
fungaren
force-pushed
the
sm89-deepseek-v4-flash
branch
from
September 21, 2026 09:54
90432bc to
f88fa36
Compare
|
lgtm |
fungaren
force-pushed
the
sm89-deepseek-v4-flash
branch
from
September 21, 2026 11:32
f88fa36 to
c18024b
Compare
…ace) for MXFP4 Marlin MoE + sparse MLA Triton fallback
fungaren
force-pushed
the
sm89-deepseek-v4-flash
branch
from
September 21, 2026 13:05
c18024b to
e557b08
Compare
Author
|
/tag-run-ci-label |
Author
|
/rerun-failed-ci |
This was referenced Nov 4, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is rebased version of #33937
Motivation
SGLang's MXFP4 Marlin MoE path and DSV4 sparse MLA backend were gated to SM90 (Hopper) and SM120 (Blackwell), leaving Ada Lovelace GPUs (SM89 — RTX 4090 / L4 /L40S / L20) unable to run DeepSeek-V4-Flash via the W4A16 Marlin path. The only fallback on SM89 was load-time FP4→FP8 dequant, which materializes a full FP8 weight copy on every TP shard and OOMs at TP8 on a single 8× RTX 4090 (24 GB) node.
This PR relaxes the software hardware-gate so SM89 takes the existing Marlin + Triton sparse MLA fallback path. No new kernels are introduced; we enable pre-existing code paths that the original authors restricted to SM90/SM120 purely for testing scope, not a hard hardware limitation. The underlying Marlin MoE kernel (moe_wna16_marlin.cuh) only guards on CUDA_ARCH >= 800, and SM89 (8.9) passes that guard.
Modifications
python/sglang/srt/utils/common.py— Addis_sm89_supported(), returning True only for device capability(8, 9). It deliberately avoids_check_cuda_device_version, which matches on the major version alone and would wronglyinclude Ampere SM80.
python/sglang/srt/runtime_context.py— Registeris_sm89as a platform probe.python/sglang/srt/layers/quantization/mxfp4_marlin_moe.py— Relax the MXFP4 Marlin MoE hardware gate fromSM90 or SM120toSM89 or SM90 or SM120(safe since the Marlin kernel only requiresCUDA_ARCH >= 800).python/sglang/srt/layers/attention/deepseek_v4_backend.py— Treat SM89 like SM120 in three places:_create_flashmla_metadataskipssgl_kernel.flash_mla;flash_mla_sparse_fwd);flash_mla_with_kvcache_sm120) is selected on SM89 as well.python/sglang/srt/layers/attention/dsv4/indexer.py— Pick the SM120 Torch fallback(
fp8_paged_mqa_logits_torch_sm120) for multi-dim seq_lens on SM89.python/sglang/srt/arg_groups/model_hook.py— For DeepSeek-V4-Flash on SM89, disable Hopper/Blackwell-onlyfeatures (
SGLANG_OPT_FP8_WO_A_GEMM, topk-v2, TileLang MHC pre, DeepGEMM HC prenorm; TileLang indexer stays off as it isSM120-only) and enable
SGLANG_FP8_PAGED_MQA_LOGITS_TORCH.test/registered/unit/utils/test_common.py— Add unit tests foris_sm89_supported: True on(8, 9), False on(9, 0),(8, 0), and without CUDA (probes mocked so they run on CPU CI).Serving command
L20 (48G) * 8, TP=8
Accuracy Tests
Speed Tests and Profiling
DeepSeek-V4-Flash (Preview, FP8), L20 (48G) * 8, TP=8, Input=3500, Output=1500
DeepSeek-V4-Flash-0731 (Official, FP8), L20 (48G) * 8, TP=8, Input=3500, Output=1500
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #35603375206
Latest PR Test (Extra): ❌ Run #35603374816
Latest PR Test (AMD ROCm 10): ❌ Run #35603375448