Skip to content

[feat] Support DeepSeek-V4-Flash / DeepSeek-V4-Flash-0731 on SM89 - #40566

Open
fungaren wants to merge 1 commit into
sgl-project:mainfrom
fungaren:sm89-deepseek-v4-flash
Open

fungaren wants to merge 1 commit into
sgl-project:mainfrom
fungaren:sm89-deepseek-v4-flash

Conversation

@fungaren

@fungaren fungaren commented Sep 21, 2026 •

Copy link
Copy Markdown

This PR is rebased version of #33937

Motivation

SGLang's MXFP4 Marlin MoE path and DSV4 sparse MLA backend were gated to SM90 (Hopper) and SM120 (Blackwell), leaving Ada Lovelace GPUs (SM89 — RTX 4090 / L4 /L40S / L20) unable to run DeepSeek-V4-Flash via the W4A16 Marlin path. The only fallback on SM89 was load-time FP4→FP8 dequant, which materializes a full FP8 weight copy on every TP shard and OOMs at TP8 on a single 8× RTX 4090 (24 GB) node.

This PR relaxes the software hardware-gate so SM89 takes the existing Marlin + Triton sparse MLA fallback path. No new kernels are introduced; we enable pre-existing code paths that the original authors restricted to SM90/SM120 purely for testing scope, not a hard hardware limitation. The underlying Marlin MoE kernel (moe_wna16_marlin.cuh) only guards on CUDA_ARCH >= 800, and SM89 (8.9) passes that guard.

Modifications

  • python/sglang/srt/utils/common.py — Add is_sm89_supported(), returning True only for device capability (8, 9). It deliberately avoids _check_cuda_device_version, which matches on the major version alone and would wrongly
    include Ampere SM80.

  • python/sglang/srt/runtime_context.py — Register is_sm89 as a platform probe.

  • python/sglang/srt/layers/quantization/mxfp4_marlin_moe.py — Relax the MXFP4 Marlin MoE hardware gate from SM90 or SM120 to SM89 or SM90 or SM120 (safe since the Marlin kernel only requires CUDA_ARCH >= 800).

  • python/sglang/srt/layers/attention/deepseek_v4_backend.py — Treat SM89 like SM120 in three places:

    • _create_flashmla_metadata skips sgl_kernel.flash_mla;
    • the sparse-prefill path stays dense (the Triton fallback handles prefill and decode without flash_mla_sparse_fwd);
    • the SM120 decode kernel (flash_mla_with_kvcache_sm120) is selected on SM89 as well.
  • python/sglang/srt/layers/attention/dsv4/indexer.py — Pick the SM120 Torch fallback
    (fp8_paged_mqa_logits_torch_sm120) for multi-dim seq_lens on SM89.

  • python/sglang/srt/arg_groups/model_hook.py — For DeepSeek-V4-Flash on SM89, disable Hopper/Blackwell-only
    features (SGLANG_OPT_FP8_WO_A_GEMM, topk-v2, TileLang MHC pre, DeepGEMM HC prenorm; TileLang indexer stays off as it is
    SM120-only) and enable SGLANG_FP8_PAGED_MQA_LOGITS_TORCH.

  • test/registered/unit/utils/test_common.py — Add unit tests for is_sm89_supported: True on (8, 9), False on
    (9, 0), (8, 0), and without CUDA (probes mocked so they run on CPU CI).

Serving command

L20 (48G) * 8, TP=8

export SGLANG_ENABLE_JIT_DEEPGEMM=0
export SGLANG_DSV4_FP4_EXPERTS=0
export SGLANG_DSV4_FP4_DEQUANT=1
export SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16
export SGLANG_DEFAULT_THINKING=1
export SGLANG_SM120_FLASHMLA_BACKEND=triton

python3 -m sglang.launch_server --attention-backend dsv4 --chunked-prefill-size 8192 \
--context-length 131072 --cuda-graph-backend-prefill disabled --cuda-graph-bs-decode 1 8 16 32 \
--cuda-graph-max-bs-decode 32 --enable-metrics --host 0.0.0.0 --max-running-requests 32 \
--mem-fraction-static 0.87 --model-path /dsv4flash --port 8080 --reasoning-parser deepseek-v4 \
--served-model-name DeepSeek-V4-Flash-0731 --tool-call-parser deepseekv4 --tp 8 --trust-remote-code

Accuracy Tests

Model Dataset MaxTokens Config Accuracy
DeepSeek-V4-Flash-0731 gsm8k 8000 8-shot, temperature 0.1, thinking + reasoning_effort=max 97.04% (1280/1319)
DeepSeek-V4-Flash Preview gsm8k 8000 8-shot, temperature 0.1, thinking + reasoning_effort=max 95.6% (1261/1319)

Speed Tests and Profiling

DeepSeek-V4-Flash (Preview, FP8), L20 (48G) * 8, TP=8, Input=3500, Output=1500

============ Serving Benchmark Result ============
Successful requests:                     32        
Failed requests:                         0         
Maximum request concurrency:             16        
Request rate configured (RPS):           1.00      
Benchmark duration (s):                  560.00    
Total input tokens:                      112000    
Total generated tokens:                  48000     
Request throughput (req/s):              0.06      
Output token throughput (tok/s):         85.71     
Peak output token throughput (tok/s):    112.00    
Peak concurrent requests:                32.00     
Total token throughput (tok/s):          285.71    
---------------Time to First Token----------------
Mean TTFT (ms):                          19182.23  
Median TTFT (ms):                        18336.09  
P50 TTFT (ms):                           18336.09  
P99 TTFT (ms):                           33134.28  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          171.31    
Median TPOT (ms):                        171.14    
P50 TPOT (ms):                           171.14    
P99 TPOT (ms):                           183.19    
---------------Inter-token Latency----------------
Mean ITL (ms):                           171.31    
Median ITL (ms):                         164.82    
P50 ITL (ms):                            164.82    
P99 ITL (ms):                            166.44    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          275981.93 
Median E2EL (ms):                        279584.23 
P50 E2EL (ms):                           279584.23 
P99 E2EL (ms):                           279697.12 
==================================================

DeepSeek-V4-Flash-0731 (Official, FP8), L20 (48G) * 8, TP=8, Input=3500, Output=1500

============ Serving Benchmark Result ============
Successful requests:                     32        
Failed requests:                         0         
Maximum request concurrency:             16        
Request rate configured (RPS):           1.00      
Benchmark duration (s):                  561.73    
Total input tokens:                      112000    
Total generated tokens:                  48000     
Request throughput (req/s):              0.06      
Output token throughput (tok/s):         85.45     
Peak output token throughput (tok/s):    112.00    
Peak concurrent requests:                32.00     
Total token throughput (tok/s):          284.84    
---------------Time to First Token----------------
Mean TTFT (ms):                          19186.95  
Median TTFT (ms):                        18342.09  
P50 TTFT (ms):                           18342.09  
P99 TTFT (ms):                           33146.04  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          171.89    
Median TPOT (ms):                        171.92    
P50 TPOT (ms):                           171.92    
P99 TPOT (ms):                           183.56    
---------------Inter-token Latency----------------
Mean ITL (ms):                           171.89    
Median ITL (ms):                         165.43    
P50 ITL (ms):                            165.43    
P99 ITL (ms):                            166.94    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          276843.50 
Median E2EL (ms):                        280524.80 
P50 E2EL (ms):                           280524.80 
P99 E2EL (ms):                           280765.74 
==================================================

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #35603375206
Latest PR Test (Extra): ❌ Run #35603374816
Latest PR Test (AMD ROCm 10): ❌ Run #35603375448

@SilverNarcissus

Copy link
Copy Markdown

lgtm

@fungaren
fungaren force-pushed the sm89-deepseek-v4-flash branch from f88fa36 to c18024b Compare September 21, 2026 11:32
…ace) for MXFP4 Marlin MoE + sparse MLA Triton fallback
@fungaren
fungaren force-pushed the sm89-deepseek-v4-flash branch from c18024b to e557b08 Compare September 21, 2026 13:05
@fungaren

Copy link
Copy Markdown
Author

/tag-run-ci-label

@fungaren

Copy link
Copy Markdown
Author

/rerun-failed-ci

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants