Repository navigation
[Fix] [AMD] triton-moe: Clamp SwiGLU on HIP - #44543
yuankaichen-amd wants to merge 3 commits into
Conversation
The fused silu_and_mul_clamp kernel is CUDA/XPU only, and the HIP path asserted instead of applying swiglu_limit. Clamp the gate and up halves in place, then run the regular HIP silu_and_mul.
|
cc @michaelzhang-ai: #41136 has the same HIP clamp for the non-filtered Triton MoE path. Could you take a look? The clamp itself looks right to me: gate is capped at the limit only, up is clipped to ±limit, and both happen before SiLU, matching the CUDA fused kernel. One suggestion: the filtered branch already uses |
|
The clamp contract is right: the gate is capped from above, up is clipped to ±limit, and it matches I agree with the suggestion above to use if filter_expert or _is_hip:
# HIP has no silu_and_mul_clamp; act_and_mul_triton applies the clamp itself.
act_and_mul_triton(
intermediate_cache1.view(-1, N),
intermediate_cache2,
config,
topk_ids if filter_expert else None,
expert_ids if filter_expert else None,
down_moe_use_tma,
activation,
swiglu_limit=swiglu_limit,
)
else:
silu_and_mul_clamp(intermediate_cache1.view(-1, N), intermediate_cache2, swiglu_limit)Two smaller things:
On the overlap with #41136: this PR should land first. Once it merges, I'll drop the CI hasn't run yet; only the gate failed. |
…ton on HIP HIP has no silu_and_mul_clamp. The non-filtered path was two in-place clamps plus silu_and_mul. act_and_mul_triton already applies swiglu_limit and runs on every row when no expert ids are passed.
|
Thanks for the review @XinyuJiangCMU @michaelzhang-ai Updated accordingly to your suggestions. |
Motivation
A model with
swiglu_limitset (DeepSeek V4, GLM-5.3-Flash) crashes on the Triton MoE runner when every expert is local (TP only, no expert parallelism):silu_and_mul_clampexists only for CUDA and XPU. The expert-filtered path is unaffected, becauseact_and_mul_tritonapplies the limit itself. HIP has no way through the non-filtered branch.Related: #41136 carries this same clamp together with a different mHC change. This PR is only the clamp.
Modifications
On HIP, and on the expert-filtered path,
_fused_moe_kernel_sequencenow callsact_and_mul_tritonwithswiglu_limit. With no expert ids, that kernel clamps the gate from above and the up half to±limitbefore SiLU, over every row. The CUDA/XPU path still callssilu_and_mul_clamp. The staleSGLANG_OPT_SWIGLU_CLAMP_FUSIONcomment is removed.Accuracy Tests
On GLM-5.3-Flash BF16 RL, the same 4 steps with this clamp (Triton MoE) and without it (aiter MoE, which does not apply the clamp) agreed to three decimals on
train_rollout_logprob_abs_diff: 0.087, 0.085, 0.061, 0.094. Pre-activations almost never exceeded the limit of 10 on this checkpoint, so the clamp is required for the kernel contract and did not move the logprob gap.Speed Tests and Profiling
act_and_mul_tritonreplaces the two in-place clamps plussilu_and_mulon the HIP path that previously asserted. On MI350X, with DeepSeek-V4-Flash shapes and inputs beyond ±10, that single kernel matched the clamp-then-silu_and_muloutputs and was 4.3–6.0× faster at decode sizes (13.7 → 2.6 µs for 48×4096) and 3.4–5.7× at prefill sizes. The RL recipe keeps aiter MoE; this change is so the Triton runner can serve the same model on HIP.Checklist
Made with Cursor
CI States
Latest PR Test (Base): ❌ Run #42662183159
Latest PR Test (Extra): ❌ Run #42662182881
Latest PR Test (AMD ROCm 10): ❌ Run #42662182767
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: