Skip to content

[Perf] [AMD] dsv4: Add a separate MegaMoE decode instance - #44657

Merged
HaiShaw merged 4 commits into
sgl-project:mainfrom
RolaoDenthu:dsv4/megamoe-decode-mtpr
Oct 16, 2026
Merged

HaiShaw merged 4 commits into
sgl-project:mainfrom
RolaoDenthu:dsv4/megamoe-decode-mtpr

Conversation

@RolaoDenthu

@RolaoDenthu RolaoDenthu commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Decode currently shares a MegaMoEV2 instance sized for prefill. Even with a small decode batch, the large MTPR keeps it on the large-instance dispatch/combine path, preventing it from benefiting from kernel configurations optimized for small token counts.

Introduce a separate, smaller decode instance to enable the small-MTPR execution path while retaining the main instance for prefill.

Modification

Add SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR (default: 0, disabled). When enabled, maintain two cached MegaMoEV2 instances:

  • Main instance: uses SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR for steps containing prefill.
  • Decode instance: uses the smaller MTPR SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR for decode, target verify, and draft extend in decode-only steps.

Both instances are constructed together in a fixed order across EP ranks before graph capture. They own separate communication buffers and synchronization state while sharing model weights through per-layer rebinding.

Instance selection follows the globally synchronized step type, ensuring all EP ranks use the same instance regardless of local batch size or graph/eager execution. Startup validation checks that the decode MTPR covers the maximum per-rank workload.

Accuracy Tests

gsm8k 1319: 0.939

Speed Tests and Profiling

kernel level

MegaMoEV2 per-layer time in µs, max across the 8 ranks (MI355X, EP8, DeepSeek-V4-Pro shapes, a8w4, uniform routing, CUDA-graph):

Tokens per rank 512 (fixed-slot) 8192 (compact) 16384 (compact)
7 254 (1.31x faster) 331 333
56 343 (1.31x faster) 454 450
112 362 (1.30x faster) 472 469
224 449 (1.28x faster) 566 573
448 647 (2% slower) 632 633
672 does not fit 797 790

The speedups are relative to the 16384 column.

e2e test

~3% TTT improvement in cc=256, agentx benchmark

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #38882210507
Latest PR Test (Extra): ❌ Run #38882210274
Latest PR Test (AMD ROCm 10): ⏳ Run #38882210435

SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR (default 0 = off) adds a smaller
MegaMoEV2 instance next to the SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR one. Every
forward of a decode step runs on it: decode, target verify, draft extend,
and the idle forward of a rank with no requests. A step with an extend on
any DP rank keeps the large capacity for all of its forwards.

The capacity picks the dispatch wire format, so all EP ranks must agree,
and per-rank token counts cannot decide it: DP token counts are only
gathered under require_mlp_tp_gather (not, e.g., for EP8 / DP8 with dense
TP1 and the DP LM head), decode graphs then replay the bucket of each
rank's own batch, and a rank past the largest bucket runs eager beside
graph replays. The choice is made only from what the group agrees on:
is_extend_in_batch, max-reduced over the DP group, for eager forwards,
and the forward mode at capture, since decode-step graphs replay only
when no rank extends (can_run_decode_cuda_graph is min-reduced).

Because every rank runs the decode capacity whatever its batch, startup
checks that it holds the largest decode-step forward on one rank: max
running requests per DP rank x max(draft tokens, top-k). It needs
--max-running-requests and is not supported with attention context
parallelism. A forward that still overflows fails the existing per-rank
capacity assert instead of moving that rank alone to the large capacity.

Both capacities are built together on the first eager call, in the same
order on every rank, before CUDA-graph capture.

The decode capacity must be 0 or a power of two no larger than the main
one. Other values are rejected; an equal value runs one capacity and warns
once.
@RolaoDenthu RolaoDenthu added amd run-ci CI: run the baseline test suite on this PR labels Oct 16, 2026
@github-actions

This comment has been minimized.

@HaiShaw
HaiShaw enabled auto-merge (squash) October 16, 2026 09:49
@HaiShaw
HaiShaw disabled auto-merge October 16, 2026 09:49
@HaiShaw HaiShaw changed the title [Perf][AMD] Add a separate MegaMoE decode instance for DeepSeek V4 [Perf][AMD] DSV4: Add a separate MegaMoE decode instance Oct 16, 2026
@sgl-project sgl-project deleted a comment from github-actions Bot Oct 16, 2026
@HaiShaw
HaiShaw marked this pull request as draft October 16, 2026 10:05
@HaiShaw
HaiShaw marked this pull request as ready for review October 16, 2026 10:05
@github-actions

This comment has been minimized.

@HaiShaw HaiShaw changed the title [Perf][AMD] DSV4: Add a separate MegaMoE decode instance [Perf] [AMD] dsv4: Add a separate MegaMoE decode instance Oct 16, 2026
@HaiShaw
HaiShaw merged commit 4afae55 into sgl-project:main Oct 16, 2026
104 of 151 checks passed
detain pushed a commit to detain/sglang that referenced this pull request Oct 29, 2026
…project#44901)

cherry-pick -x 3a059de. PORT ORDER: landed FIRST in this family because
its merged mainline chronology makes it the strict precondition of sgl-project#44904
(main merges: sgl-project#44901 16:54 -> sgl-project#44904 16:56 -> sgl-project#44912 16:57 -> sgl-project#44913 16:59;
sgl-project#44904's diff preimage literally consumes fused_shared_expert_weight /
is_routed_scaling_factor_fused_in_topk introduced here, and sgl-project#44913's preimage
deletes the helpers sgl-project#44912 introduces -- the only cherry-pickable order is the
mainline one; later commits in this family build on earlier ones).

Conflict resolutions (2):
  * models/deepseek_v2.py: the pick's hunk anchors on `class
    MergedMoEOutput(MoEOutput)` as CONTEXT, but that class is upstream
    sgl-project#44657 content this fork never took (review-record skip family), so the
    region conflicted empty-vs-context. Kept ONLY sgl-project#44901's own addition:
    _scale_routed_and_add_shared (verified byte-identical to the upstream
    post-image function text) plus all nine routed_scaling_factor_fused_in_topk
    sites (two __init__ sets, the aiter shared-scaling compensation block,
    two non-CUDA post-MoE-multiply guards, and the forward_deepep/state-based
    forward replacements). Nothing in this tree references MergedMoEOutput;
    did not smuggle it in.
  * test/registered/unit/layers/moe/test_mega_moe_decode_capacity.py:
    modify/delete conflict (the file was created by sgl-project#44657, absent here;
    sgl-project#44901 merely touched 1 line inside it). Kept it deleted.

Fidelity: 7/12 python files are BYTE-IDENTICAL to the upstream post-image
(topk.py, hash_topk.py, waterfill.py, moe_ep_setup.py, glm4_moe.py,
glm4_moe_lite.py, vl_routing.py). mega_moe/mega_moe_flydsl/bailing_moe_v3/
dots3 landed inside files carrying nemotron-era (sgl-project#44518) upstream drift we do
not have; their fold changes were diff-reviewed line by line (old
`should_fuse_routed_scaling_factor_in_topk or _use_aiter` predicates replaced
by the new config-level predicate; residual old-form reads match upstream's
own post-image -- legitimately kept). WaterfillBalancer's routed_scaling_factor
kwarg became shared_weight (one production constructor, moe_ep_setup, updated
by the pick; no other call sites in the tree).

Battery (CVD=9) vs base 0df18e9: test_fused_shared_expert_scaling.py
6P -> 12P/25sub (upstream's +343-line fold matrix, green); test_hash_topk.py
3P -> 4P; test_jit_grouped_topk 10S -> 12S (CUDA-gated adds); every pin held
(standard_dispatcher_runner_backend 12P, smallm 28P/66sub, b12x 67P/46sub,
kernels_namespace 76P, model_overrides 111P/1S/179sub, modelopt suites exact).
Pre-existing classes unchanged (trtllm-gen 8F, fused_gate 46F, round-robin 37F,
triton-fused-moe 96F, mem_pool_ep 1F, mega_moe_deepgemm 1F).
ruff --select=F401,F821,F823,UP037 + ruff format clean on all 18 files.

Fold-predicate e2e on real NVFP4 weights stays GPU-window debt (the standalone
CPU contract probe follows this family as its own commit).

(cherry picked from commit 3a059de)

Co-authored-by: port-moe-contracts subagent
(cherry picked from commit c4511d515077ea018200eeccfff256cc25ff18e5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants