Skip to content

fix(rocm): DeepSeek-V4 asymmetric SwiGLU clamp in the HIP Triton MoE path - #41358

Open
xiaohong42 wants to merge 1 commit into
sgl-project:mainfrom
xiaohong42:gfx942/triton-swiglu-clamp
Open

xiaohong42 wants to merge 1 commit into
sgl-project:mainfrom
xiaohong42:gfx942/triton-swiglu-clamp

Conversation

@xiaohong42

@xiaohong42 xiaohong42 commented Sep 26, 2026 •

Copy link
Copy Markdown

Motivation

DeepSeek-V4 clamps gate and up asymmetrically before SiLU (gate at max only, up at both ends). In the Triton MoE runner the fused silu_and_mul_clamp that implements this is CUDA/XPU-only, and the unfiltered-expert path guards it with assert _is_cuda or _is_xpu, so on HIP any DSV4 MoE that reaches that path (expert parallelism off, e.g. --ep-size 1) fails on the assertion.

Modifications

Apply the same clamp explicitly on HIP before silu_and_mul. The CUDA/XPU branch and its assert are unchanged; filter_expert=True (EP > 1) already passed the limit through and is untouched.

Accuracy Tests

test/registered/amd/test_dsv4_triton_swiglu.py runs fused_experts_impl(..., filter_expert=False, swiglu_limit=10) on HIP and compares against silu(gate.clamp(max=10)) * up.clamp(-10, 10). The earlier revision of the test patched fused_moe.get_tp_group, which does not exist on main, so it raised before reaching the code under test. It now patches whichever TP-group accessor the module has (get_parallel on main, get_tp_group on sglang-miles). Passes on MI308X (gfx942) against the sglang-miles-based integration tree.

Speed Tests and Profiling

Only the HIP unfiltered-expert path changes, which previously failed on an assertion, so there is no prior speed to compare against. The clamp is two elementwise ops before silu_and_mul.

Checklist

  • Formatted with the repo's pre-commit hooks (isort 7.0.0, ruff 0.15.1, ruff-format, codespell)
  • Unit tests added or fixed (see Accuracy Tests)
  • Documentation: not needed (no user-facing option)
  • Accuracy results above; speed see above

Scope

Optional for the miles 2-node recipe, which runs with --ep-size 4 and never takes this path. It matters for single-GPU or EP=1 DeepSeek-V4 serving on ROCm.

Supersedes #40579.

DeepSeek-V4 clamps gate and up asymmetrically before SiLU: gate at `max` only,
up at both ends. The fused `silu_and_mul_clamp` kernel that implements this is
CUDA/XPU-only, and the unfiltered-expert path guards it with
`assert _is_cuda or _is_xpu`, so on HIP any DSV4 MoE that reaches that path
(expert parallelism off, e.g. --ep-size 1) failed on the assertion.

Apply the same clamp explicitly on HIP before `silu_and_mul`. The
CUDA/XPU assert is kept: the new branch catches HIP ahead of it, so it still
holds, and it keeps documenting which targets own the fused path.

Only the unfiltered path changes; `filter_expert` already passed the limit
through to Triton.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant