Skip to content

[Fix] [AMD] triton-moe: Clamp SwiGLU on HIP - #44543

Open
yuankaichen-amd wants to merge 3 commits into
sgl-project:mainfrom
yuankaichen-amd:amd/hip-triton-swiglu-clamp
Open

yuankaichen-amd wants to merge 3 commits into
sgl-project:mainfrom
yuankaichen-amd:amd/hip-triton-swiglu-clamp

Conversation

@yuankaichen-amd

@yuankaichen-amd yuankaichen-amd commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

A model with swiglu_limit set (DeepSeek V4, GLM-5.3-Flash) crashes on the Triton MoE runner when every expert is local (TP only, no expert parallelism):

AssertionError: fused silu_and_mul_clamp kernel is CUDA/XPU only; HIP must disable SWIGLU_CLAMP_FUSION

silu_and_mul_clamp exists only for CUDA and XPU. The expert-filtered path is unaffected, because act_and_mul_triton applies the limit itself. HIP has no way through the non-filtered branch.

Related: #41136 carries this same clamp together with a different mHC change. This PR is only the clamp.

Modifications

On HIP, and on the expert-filtered path, _fused_moe_kernel_sequence now calls act_and_mul_triton with swiglu_limit. With no expert ids, that kernel clamps the gate from above and the up half to ±limit before SiLU, over every row. The CUDA/XPU path still calls silu_and_mul_clamp. The stale SGLANG_OPT_SWIGLU_CLAMP_FUSION comment is removed.

Accuracy Tests

On GLM-5.3-Flash BF16 RL, the same 4 steps with this clamp (Triton MoE) and without it (aiter MoE, which does not apply the clamp) agreed to three decimals on train_rollout_logprob_abs_diff: 0.087, 0.085, 0.061, 0.094. Pre-activations almost never exceeded the limit of 10 on this checkpoint, so the clamp is required for the kernel contract and did not move the logprob gap.

Speed Tests and Profiling

act_and_mul_triton replaces the two in-place clamps plus silu_and_mul on the HIP path that previously asserted. On MI350X, with DeepSeek-V4-Flash shapes and inputs beyond ±10, that single kernel matched the clamp-then-silu_and_mul outputs and was 4.3–6.0× faster at decode sizes (13.7 → 2.6 µs for 48×4096) and 3.4–5.7× at prefill sizes. The RL recipe keeps aiter MoE; this change is so the Triton runner can serve the same model on HIP.

Checklist

Made with Cursor


CI States

Latest PR Test (Base): ❌ Run #42662183159
Latest PR Test (Extra): ❌ Run #42662182881
Latest PR Test (AMD ROCm 10): ❌ Run #42662182767

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

The fused silu_and_mul_clamp kernel is CUDA/XPU only, and the HIP path
asserted instead of applying swiglu_limit. Clamp the gate and up halves in
place, then run the regular HIP silu_and_mul.
@yuankaichen-amd yuankaichen-amd changed the title [AMD] Clamp SwiGLU in the Triton MoE runner on HIP [Fix] [AMD] triton-moe: Clamp SwiGLU on HIP Oct 16, 2026
@XinyuJiangCMU

Copy link
Copy Markdown
Contributor

cc @michaelzhang-ai: #41136 has the same HIP clamp for the non-filtered Triton MoE path. Could you take a look?

The clamp itself looks right to me: gate is capped at the limit only, up is clipped to ±limit, and both happen before SiLU, matching the CUDA fused kernel.

One suggestion: the filtered branch already uses act_and_mul_triton, which supports swiglu_limit and also runs without the expert filter (topk_ids=None, expert_ids=None). Using it here would replace the two clamp_ calls plus silu_and_mul with a single kernel. In an activation-only benchmark on MI350X (BF16, DeepSeek-V4-Flash shapes, inputs beyond ±10), the outputs matched the current path exactly, and the single kernel was 4.3–6.0× faster at decode sizes (e.g. 13.7 → 2.6 µs for 48×4096) and 3.4–5.7× at prefill sizes.

@michaelzhang-ai

Copy link
Copy Markdown
Collaborator

The clamp contract is right: the gate is capped from above, up is clipped to ±limit, and it matches act_and_mul_kernel. Editing intermediate_cache1 in place is safe here.

I agree with the suggestion above to use act_and_mul_triton on this path. It already runs on HIP in the filtered branch, it applies swiglu_limit itself, and it works without the expert filter. That also removes the in-place mutation and the swiglu_limit_for_* locals:

if filter_expert or _is_hip:
    # HIP has no silu_and_mul_clamp; act_and_mul_triton applies the clamp itself.
    act_and_mul_triton(
        intermediate_cache1.view(-1, N),
        intermediate_cache2,
        config,
        topk_ids if filter_expert else None,
        expert_ids if filter_expert else None,
        down_moe_use_tma,
        activation,
        swiglu_limit=swiglu_limit,
    )
else:
    silu_and_mul_clamp(intermediate_cache1.view(-1, N), intermediate_cache2, swiglu_limit)

Two smaller things:

  • The comment block above this branch still describes a SGLANG_OPT_SWIGLU_CLAMP_FUSION fusion=False path that no longer exists.
  • The RL comparison doesn't exercise the clamp, since pre-activations rarely exceed 10. A small HIP unit test on the non-filtered path, with inputs beyond ±10 checked against swiglu_clamped, would cover it.

On the overlap with #41136: this PR should land first. Once it merges, I'll drop the fused_moe.py hunk from #41136 so it carries only the mHC change.

CI hasn't run yet; only the gate failed.

yuankaichen-amd and others added 2 commits November 9, 2026 08:57
…ton on HIP

HIP has no silu_and_mul_clamp. The non-filtered path was two in-place clamps
plus silu_and_mul. act_and_mul_triton already applies swiglu_limit and runs
on every row when no expert ids are passed.
@yuankaichen-amd

Copy link
Copy Markdown
Contributor Author

Thanks for the review @XinyuJiangCMU @michaelzhang-ai Updated accordingly to your suggestions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants