Skip to content

[DSV4.1][PD][5/N] Decode node support DP attention - #40177

Merged
Oasis-Git merged 6 commits into
sgl-project:mainfrom
SYChen123:v4.1/dp-attn
Oct 8, 2026
Merged

Oasis-Git merged 6 commits into
sgl-project:mainfrom
SYChen123:v4.1/dp-attn

Conversation

@SYChen123

@SYChen123 SYChen123 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Issue #39700. Currently, decode cannot use dp attention in PD disaggregation. This PR supports dp attn in PD.

Modifications

Remove the restricts and reuse the dp attn and kv transfer path of dsv4.

Accuracy Tests

image

Speed Tests and Profiling

We test the performance gains on 1P1D on a single node B200. Prefill is CP4 and Decode is DP4. On both E2E and output throughput, this PR achieves 10% performance gains.

Setup TTFT / TPOT (ms) E2E (ms) Output throughput (token/s)
TP basline 9689 / 3.01 11190 3948
With this PR 8912 / 2.30 10061 4440.26

Baseline
image

With this PR
image

Launch command

# prefill
CUDA_VISIBLE_DEVICES=0,1,2,3 \
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true \
MC_INTRANODE_NVLINK=true \
SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
        sglang serve --trust-remote-code --served-model-name llm --model-path ${MODEL_PATH} --tp 4 --ep 4 --moe-runner-backend flashinfer_mxfp4 --enable-deepseek-v4-fp4-indexer --enable-prefill-cp --cp-strategy interleave --chunked-prefill-size 16384 --max-prefill-tokens 16384 --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.8 --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-write-policy write_through --detokenizer-worker-num 8 --tokenizer-worker-num 8 --reasoning-parser auto --tool-call-parser auto --enable-strict-thinking --enable-cache-report --enable-metrics --disaggregation-mode prefill --disaggregation-transfer-backend mooncake  --host ${ip} --port 9091 > prefill.log 2>&1 & 

# decode
CUDA_VISIBLE_DEVICES=4,5,6,7 \
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true \
MC_INTRANODE_NVLINK=true \
SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
      sglang serve  --trust-remote-code --served-model-name llm --model-path deepseek-ai/DeepSeek-V4.1-Flash/ --tp 4 --ep 4 --dp 4 --enable-dp-attention --enable-dp-lm-head --moe-runner-backend flashinfer_mxfp4 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-deepseek-v4-fp4-indexer --max-running-requests 256 --swa-full-tokens-ratio 0.05 --mem-fraction-static 0.87 --detokenizer-worker-num 8 --tokenizer-worker-num 8 --reasoning-parser auto  --tool-call-parser auto --enable-strict-thinking --dist-init-addr $d_ip0:5757 --nnodes ${D_NNODES} --node-rank ${di} --host ${ip} --port 9095 > decode.log 2>&1 & 

# benchmark
python3 -m sglang.bench_serving --model deepseek-ai/DeepSeek-V4.1-Flash --flush-cache --host "${HOST}" --port "${PORT}" --backend sglang --dataset-name generated-shared-prefix --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 512 --gsp-num-groups 1 --gsp-prompts-per-group 512 -gsp-system-prompt-len 61440 --gsp-question-len 4096 --gsp-output-len 500 --max-concurrency 96 --warmup-requests 10 > "benchmark.log" 2>&1 

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #37844962128
Latest PR Test (Extra): ❌ Run #37844961572
Latest PR Test (AMD ROCm 10): ⏳ Run #37844962075

nvpohanh and others added 2 commits October 6, 2026 18:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

nvpohanh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/server_args/test_platform_prefill_cp_deprecation.py test/registered/disaggregation/test_disaggregation_dsv4.py test/registered/disaggregation/test_disaggregation_dp_attention.py test/registered/unit/disaggregation/test_dsv4_unified_fp8_pd.py test/registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/e2e/models/test_deepseek_v4_flash_fp4_b200.py test/registered/e2e/models/test_deepseek_v4_flash_fp8_h200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/server_args/test_platform_prefill_cp_deprecation.py
cd test/ && python3 registered/unit/disaggregation/test_dsv4_unified_fp8_pd.py

🚀 8-gpu-h200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dsv4.py
cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp8_h200.py

🚀 8-gpu-h20 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dp_attention.py

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

@nvpohanh

nvpohanh commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

@Oasis-Git could you review this?

@nvpohanh

nvpohanh commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/unit/server_args/test_platform_prefill_cp_deprecation.py test/registered/disaggregation/test_disaggregation_dsv4.py test/registered/disaggregation/test_disaggregation_dp_attention.py test/registered/unit/disaggregation/test_dsv4_unified_fp8_pd.py test/registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/e2e/models/test_deepseek_v4_flash_fp4_b200.py test/registered/e2e/models/test_deepseek_v4_flash_fp8_h200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/server_args/test_platform_prefill_cp_deprecation.py
cd test/ && python3 registered/unit/disaggregation/test_dsv4_unified_fp8_pd.py

🚀 8-gpu-h200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dsv4.py
cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp8_h200.py

🚀 8-gpu-h20 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dp_attention.py

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/e2e/models/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

@Oasis-Git Oasis-Git added the run-ci CI: run the baseline test suite on this PR label Oct 8, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator

UT CI pass

@Oasis-Git
Oasis-Git merged commit b0e0635 into sgl-project:main Oct 8, 2026
108 of 169 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants