Skip to content

[feat] Support DeepSeek-V4-Flash-0731 on sm89 - #33937

Closed
xs-alt wants to merge 1 commit into
sgl-project:mainfrom
xs-alt:feat/sm89-deepseek-v4-flash-0731
Closed

xs-alt wants to merge 1 commit into
sgl-project:mainfrom
xs-alt:feat/sm89-deepseek-v4-flash-0731

Conversation

@xs-alt

@xs-alt xs-alt commented Aug 7, 2026 •

Copy link
Copy Markdown

SGLang's MXFP4 Marlin MoE path and DSV4 sparse MLA backend were gated to SM90 (Hopper) and SM120 (Blackwell), leaving Ada Lovelace GPUs (SM89 — RTX 4090 / L4 /L40S / L20) unable to run DeepSeek-V4-Flash via the W4A16 Marlin path. The only fallback on SM89 was load-time FP4→FP8 dequant, which materializes a full FP8 weight copy on every TP shard and OOMs at TP8 on a single 8× RTX 4090 (24 GB) node.

This PR relaxes the software hardware-gate so SM89 takes the existing Marlin + Triton sparse MLA fallback path. No new kernels are introduced; we enable pre-existing code paths that the original authors restricted to SM90/SM120 purely for testing scope, not a hard hardware limitation. The underlying Marlin MoE kernel (moe_wna16_marlin.cuh) only guards on CUDA_ARCH >= 800, and SM89 (8.9) passes that guard.

Modifications

Six files, +94 / −10 lines:

  • python/sglang/srt/utils/common.py (+9): Add is_sm89_supported() helper next to is_sm90_supported. Uses torch.cuda.get_device_capability() == (8, 9) rather than _check_cuda_device_version because the major-only match would wrongly include Ampere SM80, which lacks the FP8 throughput this gate ultimately guards.
  • python/sglang/srt/layers/quantization/mxfp4_marlin_moe.py (+10/−2): Relax the Mxfp4MarlinMoEMethod.
    _build_mega_moe_weights gate from is_sm90_supported() OR is_sm120_supported() to also accept is_sm89_supported(). Update the import block and the RuntimeError message.
  • python/sglang/srt/layers/attention/deepseek_v4_backend.py (+9/−5): Cache _is_sm89 at module load; route SM89 through the same Triton-fallback path as SM120 — _create_flashmla_metadata returns None on SM89 (no compiled sgl_kernel.flash_mla), the sparse_prefill_fwd skip applies on SM89, and flash_mla_with_kvcache_sm120 is dispatched on SM89 the same way as SM120.
  • python/sglang/srt/layers/attention/dsv4/indexer.py (+2/−2): Enable fp8_paged_mqa_logits_torch_sm120 on SM89 in the C4IndexerBackendMixin logits fallback path (same multi-dim seq_lens handling as SM120).
  • python/sglang/srt/server_args.py (+11): Add an elif is_sm89_supported(): branch in ServerArgs._init_sm_specific_envs() mirroring the SM120 branch — disables Hopper-only flags (SGLANG_OPT_FP8_WO_A_GEMM, SGLANG_OPT_USE_TOPK_V2, SGLANG_OPT_USE_TILELANG_MHC_PRE, SGLANG_OPT_DEEPGEMM_HC_PRENORM) and enables SGLANG_FP8_PAGED_MQA_LOGITS_TORCH. TileLang indexer is left off because it is SM120-only.
  • test/registered/unit/utils/test_common.py (+50): Add TestIsSm89Supported with 4 cases — SM89 true, SM90 false, SM80 false (guards against the major==8 trap), no-CUDA false. Each test patches is_cuda and torch.cuda.get_device_capability and clears the lru_cache in setUp/tearDown.

Accuracy Tests

Hardware: 8 × RTX 4090 (24 GB, SM89), --tp-size 8 --moe-runner-backend marlin. CUDA graph capture bs=[1,2,4].

  • Smoke test (/generate): "What is 2 + 2? Answer with just the number." → "4" — numerically correct, no NaN / divergent output.
  • To verify numerical accuracy and functional correctness on SM89, we evaluated the model against the already-supported RTX Pro 5000 (SM120) baseline using identical datasets:
Benchmark Evaluation Setup RTX 4090 (8× 24GB, SM89) RTX Pro 5000 (SM120) Parity
GSM8K 5-shot, 200 questions 98.0% 98.0% Exact Match ($\pm 0.0%$)
HellaSwag 20-shot, 200 questions 77.0% 77.0% Exact Match ($\pm 0.0%$)

Speed Tests and Profiling

Hardware: 8 × RTX 4090 (24 GB, SM89), --tp-size 8 --moe-runner-backend marlin --cuda-graph-max-bs-decode 4 --mem-fraction-static 0.91, Triton sparse MLA via SGLANG_SM120_FLASHMLA_BACKEND=triton.

  • Single-concurrent decode (256 tokens): 11.38–12.42 tok/s
  • 4-concurrent decode aggregate (100 tokens): 21.7 tok/s (5.4 tok/s per request)
  • Memory: per-card weight 20.45 GB (Marlin keeps FP4 packed, no FP8 dequant copy), versus 22.88 GB for the FP4→FP8 dequant path which OOMs at TP8 on a single 8× 4090 node. KV pool 0.73 GB, CUDA graph 1.36 GB.

SGLANG_SM120_FLASHMLA_BACKEND=triton
python3 -m sglang.launch_server
--model-path deepseek-ai/DeepSeek-V4-Flash-0731
--tp-size 8
--moe-runner-backend marlin
--host 0.0.0.0 --port 30000
--mem-fraction-static 0.91
--cuda-graph-max-bs-decode 4
--max-running-requests 12
--trust-remote-code


CI States

Latest PR Test (Base): ❌ Run #31145732652
Latest PR Test (Extra): ❌ Run #31145732474

…ace) for MXFP4 Marlin MoE + sparse MLA Triton fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Thanks @xs-alt. Closing this because it has had no updates in 91 days.

Reopen it if the work is still relevant.

Some directories moved recently, so an older branch may need retargeting:
sgl-kernel/ -> python/sglang/kernels/aot/, python/sglang/jit_kernel/
-> python/sglang/kernels/jit/, docs/ -> docs/docs/ (.mdx),
bench_serving.py -> benchmark/serving.py, test/srt/ -> test/registered/.

@github-actions github-actions Bot closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant