You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FlyDSL] [MoE] fused_moe: add a deterministic mode for MXFP4 MoE - #6583
fused_moe does not give the same output for the same token. For MXFP4 MoE with wiGLU or SiLU (gpt-oss, MiniMax-M3), three things make a token's result depend on scheduling or on its batch:
Atomic top-k sum: the down projection adds each token's top-k expert outputs into its output row with atomic adds, in whatever order the workgroups finish.
Split-K: split-K in the first stage also accumulates with atomic adds.
Configuration by token count: the tuned configurations change the activation dtype (bf16 below 256 tokens, MXFP4 or MXFP8 from 256), the stage-1 K waves and the stage-2 kernel family and tile_k with the token count. Each of these changes how a row's sums are split.
Serving frameworks that offer deterministic inference need bitwise the same output for a token whatever else is in the batch; SGLang's --enable-deterministic-inference is one. With gpt-oss and MiniMax-M3 on MI350X, the MoE layer was one of the layers whose output changed while its input was bitwise equal.
Modifications
fused_moe(..., deterministic=False): with True, MXFP4 MoE runs one configuration with a fixed reduction order:
MXFP8 activations at every token count, quantized per 1x32 block within a token's row: the precision the default uses for gate/up-interleaved weights from 256 tokens, and as accurate as the default end to end (below). MXFP4 activations are not offered: they cost 5-11 AIME points.
One K structure. Stage 1 is the FlyDSL a8w4 kernel with tile_k 256 (_w2, gate/up-interleaved); stage 2 is the v1 kernel with the reduce epilogue (unique [token*topk + slot] rows, then moe_reduce sums the top-k slots in a fixed order), no split-K. Only the tile height grows with the token count (64 up to 2,048 tokens, then 128), which does not change a row's reduction. It starts at tile_m 64 because the a8w4 stage 1 at tile_m 32 with tile_n 128 is not repeatable from run to run on gfx950 (below).
Supported: gfx950, MXFP4 weights (QuantType.per_1x32) preshuffled gate/up-interleaved, SwiGLU with or without clamp and per-expert biases, bf16 output.
Refused: everything else raises NotImplementedError naming what is unsupported: separated weights, w1 and w2 not both preshuffled, other activation dtypes, other activations, expert parallelism, shared experts, prequantized activations, other block sizes, hidden sizes not divisible by 512, hidden padding without a stage-2 bias, intermediate padding with a stage-2 bias but no stage-1 bias.
Default unchanged. The kernels are existing registered FlyDSL kernels; no kernel changes.
Tests:op_tests/test_moe_mxfp4_deterministic.py. op_tests/test_fhmoe.py's signature test lists deterministic after the existing parameters.
Test results
MI350X. Op tests and per-call latency: image rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030 with this PR's files applied to its AITER. End-to-end results: images 20261021 / 20261030, the same way.
Op tests:op_tests/test_moe_mxfp4_deterministic.py, 13 passed. Shapes: gpt-oss (3072 x 3072, 32 and 128 experts, top-4) and MiniMax-M3 per GPU at TP4 / TP2, SwiGLU with clamp and biases, 2,304 tokens, gate/up-interleaved weights. Outputs are equal from run to run, and a token's output is the same alone, in the full batch and in prefixes across the default's activation switch (256) and the tile-height switch (2,048). Accuracy against a bf16 torch reference is at least the default's.
Accuracy against today's default, end to end in SGLang (AIME25, thinking on, one sample per seed, differences paired by seed; GSM8K, 1,319 questions, temperature 0):
model
AIME25, deterministic vs default
GSM8K, deterministic (default)
gpt-oss-20b
67.9% vs 70.0%, -2.1 ± 1.3 points (48 seeds)
94.31% (94.84%)
gpt-oss-120b
78.7% vs 78.4%, +0.3 ± 1.1 (40 seeds)
96.06%
MiniMax-M3, TP4
76.3% vs 78.1% (16 seeds, within seed noise)
96.21% (95.98%)
For gpt-oss only the MoE call differs; for MiniMax-M3 the whole deterministic mode is compared with normal mode (SGLang loads its experts gate/up-interleaved in deterministic mode). MXFP4 activations, which an earlier head of this PR used, cost gpt-oss-20b 5.4 ± 2.0 AIME points and MiniMax-M3 11.3 ± 2.5 (66.7% vs 77.9%, GSM8K 92.87%), which is why they were dropped. Kernel error against a bf16 reference (relative L2): 0.043, the same as the default's MXFP8 path (0.04); MXFP4 activations 0.21.
SiLU with a clamp and no bias at three shapes: accuracy no worse than the default, run to run, batch invariance.
Latency per call (HIP-graph replay, median of 50 calls), deterministic time / default time (below 1 is faster):
shape
1 token
8
32
128
256
1,024
4,096
8,192
gpt-oss-20b (32 experts, top-4)
0.95
1.12
1.06
0.81
0.80
0.78
0.70
0.73
gpt-oss-120b (128 experts, top-4)
0.95
1.08
1.14
1.12
1.17
1.09
0.97
1.07
MiniMax-M3 TP4 (129 experts, 6144 x 768, top-6)
0.92
1.09
1.10
1.14
1.07
0.94
0.99
1.12
End to end in SGLang, output throughput as a ratio to the default (same server, only the MoE call changes):
1k/1k, 1 request
16
64
8k/1k, 1 request
16
gpt-oss-20b
0.99
0.93
0.98
1.00
0.99
gpt-oss-120b
0.99
0.89
0.92
1.00
0.93
Determinism end to end in SGLang (test_deterministic single / prefix / prefix logprobs / radix cache, plus a bitwise batch-size check of 1,275 responses, token IDs and logprobs compared exactly): every check passes for gpt-oss-20b (TP1, TP2), gpt-oss-120b (TP1, TP4, TP8) and MiniMax-M3 (TP4, TP8); gpt-oss also with 64k and 120k-token prompts. Per call, 0 of 20 runs differ at 17 to 4,096 tokens (32 and 128 experts, allocator filled with NaN between runs).
Notes
Layout: callers shuffle MXFP4 weights with shuffle_weight_a16w4(w, 16, True) (gate/up-interleaved) and the scales with shuffle_scale_a16w4(s, E, True). Models stored with separated gate/up (MiniMax-M3) can be shuffled interleaved at load time; SGLang does this in deterministic mode.
a8w4 kernel finding:flydsl_moe1_afp8_wfp4_bf16_t32x128x256*_gui (every waves-per-EU variant) gives a different result from run to run on gfx950 with identical inputs (up to 5 of 5 runs at 300 tokens, also with NaN-filled memory); tile_n 64 at tile_m 32, and tile_m 64, are repeatable. The deterministic path avoids it; worth a look by the FlyDSL owners.
AOT: AOT precompilation covers the kernels named in the tuned configurations; the deterministic configuration's kernels that are not named there are JIT-compiled on first use.
Other activations: SiLU (GLM-5, DeepSeek-V4 routed experts) can follow once its accuracy is checked against its default; SiTUv2 likewise.
fused_moe gets a deterministic=False keyword. With True, MXFP4 MoE (gfx950, gate/up-interleaved weights, SwiGLU, bf16 output) runs one fixed configuration: MXFP8 activations at every token count, FlyDSL a8w4 / v1 kernels with one K structure, no split-K, and the non-atomic reduce epilogue in stage 2, so a token's output is bitwise the same from run to run and whatever else is in the batch. Unsupported configurations raise NotImplementedError. With False (the default) nothing changes.
The existing behavior is insufficient for deterministic inference: the stage-2 atomic top-k sum depends on workgroup order, stage-1 split-K adds atomically too, and the tuned configuration (activation dtype, K waves, stage-2 kernel family, tile_k) changes with the token count. In SGLang's deterministic mode on MI350X this made gpt-oss and MiniMax-M3 give different outputs for identical requests depending on their batchmates.
Validation and unverified scope
Validated on MI350X (gfx950), image rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030, with this revision's Python files applied to the image's AITER:
python3 -m pytest -q op_tests/test_fhmoe.py -k "signature or positional or deterministic": 1 passed (the signature test).
Per call: latency as in the table above. Run to run (earlier revisions with the same kernels and tiles): 20 runs with the allocator filled with NaN between runs, 17 to 4,096 tokens, 32 and 128 experts: 0 runs differ, prefixes match the full batch.
End to end in SGLang deterministic mode: python3 -m sglang.test.test_deterministic (single, prefix, prefix with logprobs, radix cache) plus a 1,275-response bitwise batch-size check pass for gpt-oss-20b (TP1, TP2), gpt-oss-120b (TP1, TP4, TP8) and MiniMax-M3 (TP4†, TP8); 64k / 120k-token batch invariance and chunked-prefill boundaries for gpt-oss; GSM8K and AIME25 as in the accuracy table. gpt-oss-20b TP1 and the runs marked † are on this revision, the rest on the previous one, which runs the same kernels for interleaved weights.
The merge of main (conflicts only in parameter lists, the HERD metadata check and fused_moe_2stages' num_rows) was checked with black, ruff and byte-compilation; the GPU results above ran on the branch before the merge, applied to the image's AITER.
Not verified: other GPUs (the mode refuses anything but gfx950; MI355X not run), AITER's full op-test suite (only the files above), AOT precompilation of the deterministic kernels (they JIT on first use), activations other than SwiGLU (refused), expert parallelism and shared experts (refused). Not root-caused: why the a8w4 stage 1 at tile_m 32 x tile_n 128 varies from run to run (avoided here).
PR-specific question
Before this PR every test in op_tests/test_moe_mxfp4_deterministic.py fails at the call, since fused_moe has no deterministic keyword. With the default path in its place, the bitwise assertions fail: repeated calls on the same batch differ (atomic top-k sum), and the first k tokens differ from the same tokens in the full batch at the activation-dtype switch (256 tokens) and the tile switches. The rejection test fails because nothing is rejected. Existing MoE op tests compare against a reference within a tolerance for one batch; none checks bitwise equality across runs or across batch compositions, which is the property this mode adds.
Changed op tests
op_tests/test_moe_mxfp4_deterministic.py (new): covers fused_moe(deterministic=True) with gate/up-interleaved weights at gpt-oss and MiniMax-M3 shapes, SwiGLU clamp and biases (accuracy at least the default's, bitwise run to run, bitwise across batch prefixes and a single token), zero routing weights, num_local_tokens, and every refused configuration (including separated weights and MXFP4 activations). Not redundant: no existing test exercises the deterministic mode or checks batch invariance.
op_tests/test_fhmoe.py (one line added): test_clamp_shared_preserves_existing_positional_argument_order lists fused_moe's parameters from quant_type_a to the end; deterministic is appended after the HERD parameters, so the test still guards existing positional callers. Necessary because the assertion would fail on this PR without it; no other test covers the signature order.
ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm
vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all
All standard extended tests (excludes ci:atom_full)
Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6583 --add-label <label>
One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.
PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
On layouts: let's support what is most needed, more support could be done later.
On block_m: using only one for determinism could make sense, I think, but could be worth to also try larger blocks and see if it retains the determinism
On layouts: let's support what is most needed, more support could be done later. On block_m: using only one for determinism could make sense, I think, but could be worth to also try larger blocks and see if it retains the determinism
Thanks! Layouts: agreed, separated only for now.
Block size: I tried 16, 64 and 128 against 32 on gpt-oss-20b's shape (3072 x 3072, 32 experts, top-4, with biases) on MI350X. Every block size gives bitwise the same output as 32 at 1-1,024 tokens, and each is deterministic and batch-invariant on its own. So the reduction order doesn't depend on the block size, as expected. They are slower than 32 at every size, though (ms per call):
tokens
32
16
64
128
1
0.186
0.189
0.204
0.297
256
0.274
0.290
0.352
0.626
1,024
0.475
0.582
0.658
0.936
4,096
1.393
1.869
1.835
2.365
So I kept 32 for every token count. Since the results don't depend on the block size, a token-count heuristic would stay deterministic if a later kernel change makes larger blocks faster.
…4 weights
The a16w-mix stage1/stage2 kernels (bf16 activations x MXFP4/int4 weights)
take an optional per-expert f32 bias: stage1 adds it to gate and up before
the activation ([E, 2*inter_dim], gate | up), stage2 adds it before the
routing weight ([E, model_dim]). Models with expert biases, such as
gpt-oss, previously could not use these kernels.
The non-atomic stage2 epilogue (unique [token*topk + slot] rows, then
moe_reduce summing the top-k slots in a fixed order) was enabled only for
int4 weights; it now also runs for MXFP4 weights when the stage2 kernel
name selects "reduce". The default stays atomic.
fused_moe(deterministic=True) gives bitwise the same output for a token
whatever it is batched with, and from run to run, as deterministic
inference in serving frameworks requires. For bf16 activations with MXFP4
weights (QuantType.per_1x32, separated gate/up, preshuffled) it runs the
FlyDSL a16w4 kernels with:
- the non-atomic stage-2 epilogue: per-(token, slot) rows, then moe_reduce
sums the top-k slots in a fixed order, instead of atomic adds whose order
depends on scheduling;
- no split-K;
- bf16 activations at every token count (the default switches to fp4/fp8
activations from 256 tokens);
- one block size (32) for every token count.
Per-expert biases (gpt-oss) are supported. Unsupported configurations
(other weight types, interleaved gate/up, expert parallelism, shared
experts, prequantized activations) raise NotImplementedError. The default
(deterministic=False) is unchanged.
stage2_uses_route_reduce now reports the a16w4 reduce epilogue, so the
output buffer is not zeroed for it.
op_tests/test_flydsl_moe_a16wfp4.py: SwiGLU with clamp and per-expert
biases against the torch reference, run-to-run equality, and the same
tokens alone and in batches of 1-300 compared bitwise.
github-actionsBot
changed the title
[MoE] fused_moe: add a deterministic mode for bf16 x MXFP4 MoE
[FlyDSL] [MoE] fused_moe: add a deterministic mode for bf16 x MXFP4 MoE
Oct 19, 2026
Review follow-ups for fused_moe(deterministic=True):
- reject SiLU: both bf16 activation-quantization bypasses cover only
SwiGLU and SiTUv2, so SiLU would feed MXFP4-quantized activations to
the bf16 kernels;
- reject non-bf16 output: the reduce epilogue stores bf16 only;
- reject weights tagged gate/up-interleaved: the deterministic kernels
read the separated preshuffle;
- allow per-expert biases past the older a16w4 validation, which
refused them for SiTUv2 and rerouted SwiGLU shapes;
- check the live GPU with get_gfx_runtime();
- AOT precompile builds the reduce epilogue for MXFP4 too, as the
runtime does.
The test compares against the torch reference elementwise with
checkAllclose and a tighter cosine bound, and a new test checks that
the rejected configurations raise.
Review follow-ups for fused_moe(deterministic=True):
- _reduce_bf16_epilog checked only token_id < M. With num_local_tokens
below the batch capacity, the sort's padding rows carry token_id =
num_local_tokens and slot = topk, so they were stored at row
(num_local_tokens + 1) * topk, past the [M * topk, N] buffer when
num_local_tokens = M - 1. The store now also requires slot < topk.
- the interleaved-weight check uses _flydsl_mxfp4_layout_is_compatible,
as the other MXFP4 dispatch paths do;
- AOT a16w4 precompile builds the has_bias variants when the job asks for
them, and moe_reduce after the reduce epilogue, as the runtime does;
- the docstring says the deterministic kernels are JIT-compiled on first
use (they are not in the tuned configs that AOT reads).
A new test runs the reduce epilogue with a guard region after its buffer
and num_local_tokens below the capacity: the guard stays untouched, and
the local tokens match the same tokens run alone.
The standard CI runner selects this module but does not execute its tests: .github/scripts/aiter_test.sh:91–104 uses python3 "$file", and this path is not a pytest exception. With no script entry point, the module exits without checking determinism, unsupported configurations, or buffer bounds. Add this file to the runner's existing pytest dispatch cases so CI collects and runs these regressions.
refused with NotImplementedError: SiLU, non-bf16 output, gate/up-interleaved weights (via _flydsl_mxfp4_layout_is_compatible);
per-expert biases now reach the deterministic path;
the reduce epilogue skips padding slots (slot < topk), fixing the out-of-bounds write with num_local_tokens;
AOT builds the MXFP4 reduce epilogue, the bias variants and moe_reduce.
On MI350X, op_tests/test_flydsl_moe_a16wfp4.py: 15 passed, including the deterministic tests, a test that the refused configurations raise, and a guard-region test for the epilogue (which fails with the previous gemm2.py in your two cases). The 4 situv2_e2e failures are pre-existing: they fail the same way without this PR. black and ruff clean.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The architecture gates can skip all correctness tests on supported hardware, and the latency sweep can report success despite out-of-tolerance results because mismatch limits are not enforced.
r4490120184: fixed in 6caf18cd0. The docstring now describes a single K wave (k_wave 1; the _w2 in the kernel name is waves_per_eu 2, an occupancy setting). The comment at the hidden-size check no longer explains the multiple-of-512 restriction with the two-wave reading. I could not give a verified kernel reason for 512, so it now only states what is known: only multiples of 512 have been validated with the deterministic kernels. The restriction itself is unchanged.
The 18 Copilot threads: I've replied and resolved each.
The skip marker and main() used get_gfx(), which follows GPU_ARCHS, so a
multi-arch GPU_ARCHS such as "gfx950;gfx942" skipped every test on gfx950.
Use get_gfx_runtime(), as fused_moe's deterministic check does.
The latency sweep reported each candidate's mismatch ratio without checking
it, so a sweep shape or token count the pytest checks do not cover could pass
with a wrong result. Assert it against _ERR_MAX, as the pytest checks do. On
MI350X the largest ratio in the default sweep is 3.3e-4 (deterministic) and
1.5e-4 (default), against 1e-3.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The changed reduction kernel needs gfx950 correctness and latency validation on the merged revision, but reported GPU tests predate the merge and this environment cannot rerun them.
CI failure on 220eaa8: shard 5, test_flydsl_hgemm.py
The MI35X shard-5 failure is in op_tests/test_flydsl_hgemm.py, which this PR does not change: graph replay mismatch for bf16 with split_k=9, k_stagger=127 (1 of 2,280 elements differs by 0.015625 in the changed-input CUDA-graph check). The new test file in this PR moved that file into shard 5; it passes on main in another shard, so it looks intermittent. Could someone re-run the failed job?
fused_moe(deterministic=True) accepted only SwiGLU. Routed experts with plain
SiLU (clamped through swiglu_limit, no bias), as in GLM-5 and DeepSeek-V4,
raised NotImplementedError. The deterministic path itself does not depend on
the activation: the FlyDSL stage-1 kernels already implement SiLU, and the
fixed tiles, the single activation dtype and the non-atomic stage-2 reduce
apply unchanged.
Allow ActivationType.Silu next to Swiglu. New op test: SiLU with a clamp and
no bias at three shapes (GLM-5 and DeepSeek-V4 per GPU at TP4 among them),
checked for accuracy against the bf16 reference (no worse than the default
path), run-to-run equality and batch invariance.
Small addition in 2aaf04295: deterministic=True now also accepts SiLU experts (clamped, no bias), as used by GLM-5 and DeepSeek-V4. The deterministic path itself is unchanged: the FlyDSL stage-1 kernels already implement SiLU, and the fixed tiles, the single activation dtype and the non-atomic stage-2 reduce apply as they are; only the support check changes. A new op test runs SiLU at three shapes (GLM-5 and DeepSeek-V4 routed experts per GPU at TP4 among them): accuracy against the bf16 reference no worse than the default path, run-to-run equality and batch invariance. On MI350X the op tests pass (19 passed). SGLang uses it for GLM-5.3 (tracking: sgl-project/sglang#); keeping it in this PR means one AITER update for SGLang instead of two.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The new SiLU accuracy test lacks the per-element reference-tolerance check required by the shared convention, allowing localized errors to pass its aggregate cosine metric.
… tolerance
The SiLU accuracy test compared only the aggregate cosine difference. Assert
the checkAllclose mismatch ratio against _ERR_MAX as the SwiGLU test does, and
keep the cosine comparison.
CI. The failing job is shard 2 of the standard tests: op_tests/test_flydsl_moe_aux.py fails with NameError: name 'DROPPED_ROUTE_ROW' is not defined in aiter/ops/flydsl/kernels/moe_contiguous_psum.py. It comes from main, not from this PR: main fixed the missing import in [FlyDSL] Fix main CI: missing DROPPED_ROUTE_ROW import in moe_contiguous_psum #7583 (a97e999228), after the main you merged (8e0000db). Merging current main (no conflicts as of 07933277) and rerunning should clear it.
Open conversations.
r4490120184: fused_moe.py (around lines 4103 and 4160) still says stage 1 runs two K waves. _w2 is waves_per_eu=2, and the kernels run with k_wave=1. Please describe the single K wave, and either give the verified reason for the multiple-of-512 restriction or drop that explanation.
18 other review threads, all Copilot's from earlier revisions, have no reply yet; three of them point at files this PR no longer changes. Please resolve the ones later commits addressed and reply to any that still apply.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The bitwise-determinism and batch-invariance guarantees require human validation on the final gfx950 revision because supplied GPU results predate the merge and correctness and latency checks cannot be run here.
1. **CI.** The failing job is shard 2 of the standard tests: `op_tests/test_flydsl_moe_aux.py` fails with `NameError: name 'DROPPED_ROUTE_ROW' is not defined` in `aiter/ops/flydsl/kernels/moe_contiguous_psum.py`. It comes from main, not from this PR: main fixed the missing import in [[FlyDSL] Fix main CI: missing DROPPED_ROUTE_ROW import in moe_contiguous_psum #7583](https://github.com/ROCm/aiter/pull/7583) (`a97e999228`), after the main you merged (`8e0000db`). Merging current main (no conflicts as of `07933277`) and rerunning should clear it.
2. **Open conversations.**
* [r4490120184](https://github.com/ROCm/aiter/pull/6583#discussion_r4490120184): `fused_moe.py` (around lines 4103 and 4160) still says stage 1 runs two K waves. `_w2` is `waves_per_eu=2`, and the kernels run with `k_wave=1`. Please describe the single K wave, and either give the verified reason for the multiple-of-512 restriction or drop that explanation.
* 18 other review threads, all Copilot's from earlier revisions, have no reply yet; three of them point at files this PR no longer changes. Please resolve the ones later commits addressed and reply to any that still apply.
r4490120184: fixed in 6caf18cd0 and unchanged since; on the current head the docstring describes a single K wave (fused_moe.py around line 4240: "k_wave 1; the _w2 in the kernel name is waves_per_eu 2"), and the hidden-size check around line 4183 only states that multiples of 512 are the validated ones, without a kernel explanation. I've resolved the thread.
Copilot threads: every thread has a reply and is resolved, including the three on files this PR no longer changes. If one still needs more, point me to it and I'll follow up there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
fused_moedoes not give the same output for the same token. For MXFP4 MoE with wiGLU or SiLU (gpt-oss, MiniMax-M3), three things make a token's result depend on scheduling or on its batch:Serving frameworks that offer deterministic inference need bitwise the same output for a token whatever else is in the batch; SGLang's
--enable-deterministic-inferenceis one. With gpt-oss and MiniMax-M3 on MI350X, the MoE layer was one of the layers whose output changed while its input was bitwise equal.Modifications
fused_moe(..., deterministic=False): withTrue, MXFP4 MoE runs one configuration with a fixed reduction order:_w2, gate/up-interleaved); stage 2 is the v1 kernel with the reduce epilogue (unique[token*topk + slot]rows, thenmoe_reducesums the top-k slots in a fixed order), no split-K. Only the tile height grows with the token count (64 up to 2,048 tokens, then 128), which does not change a row's reduction. It starts at tile_m 64 because the a8w4 stage 1 at tile_m 32 with tile_n 128 is not repeatable from run to run on gfx950 (below).QuantType.per_1x32) preshuffled gate/up-interleaved, SwiGLU with or without clamp and per-expert biases, bf16 output.NotImplementedErrornaming what is unsupported: separated weights, w1 and w2 not both preshuffled, other activation dtypes, other activations, expert parallelism, shared experts, prequantized activations, other block sizes, hidden sizes not divisible by 512, hidden padding without a stage-2 bias, intermediate padding with a stage-2 bias but no stage-1 bias.op_tests/test_moe_mxfp4_deterministic.py.op_tests/test_fhmoe.py's signature test listsdeterministicafter the existing parameters.Test results
MI350X. Op tests and per-call latency: image
rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030with this PR's files applied to its AITER. End-to-end results: images20261021/20261030, the same way.Op tests:
op_tests/test_moe_mxfp4_deterministic.py, 13 passed. Shapes: gpt-oss (3072 x 3072, 32 and 128 experts, top-4) and MiniMax-M3 per GPU at TP4 / TP2, SwiGLU with clamp and biases, 2,304 tokens, gate/up-interleaved weights. Outputs are equal from run to run, and a token's output is the same alone, in the full batch and in prefixes across the default's activation switch (256) and the tile-height switch (2,048). Accuracy against a bf16 torch reference is at least the default's.Accuracy against today's default, end to end in SGLang (AIME25, thinking on, one sample per seed, differences paired by seed; GSM8K, 1,319 questions, temperature 0):
For gpt-oss only the MoE call differs; for MiniMax-M3 the whole deterministic mode is compared with normal mode (SGLang loads its experts gate/up-interleaved in deterministic mode). MXFP4 activations, which an earlier head of this PR used, cost gpt-oss-20b 5.4 ± 2.0 AIME points and MiniMax-M3 11.3 ± 2.5 (66.7% vs 77.9%, GSM8K 92.87%), which is why they were dropped. Kernel error against a bf16 reference (relative L2): 0.043, the same as the default's MXFP8 path (0.04); MXFP4 activations 0.21.
SiLU with a clamp and no bias at three shapes: accuracy no worse than the default, run to run, batch invariance.
Latency per call (HIP-graph replay, median of 50 calls), deterministic time / default time (below 1 is faster):
End to end in SGLang, output throughput as a ratio to the default (same server, only the MoE call changes):
Determinism end to end in SGLang (
test_deterministicsingle / prefix / prefix logprobs / radix cache, plus a bitwise batch-size check of 1,275 responses, token IDs and logprobs compared exactly): every check passes for gpt-oss-20b (TP1, TP2), gpt-oss-120b (TP1, TP4, TP8) and MiniMax-M3 (TP4, TP8); gpt-oss also with 64k and 120k-token prompts. Per call, 0 of 20 runs differ at 17 to 4,096 tokens (32 and 128 experts, allocator filled with NaN between runs).Notes
shuffle_weight_a16w4(w, 16, True)(gate/up-interleaved) and the scales withshuffle_scale_a16w4(s, E, True). Models stored with separated gate/up (MiniMax-M3) can be shuffled interleaved at load time; SGLang does this in deterministic mode.flydsl_moe1_afp8_wfp4_bf16_t32x128x256*_gui(every waves-per-EU variant) gives a different result from run to run on gfx950 with identical inputs (up to 5 of 5 runs at 300 tokens, also with NaN-filled memory); tile_n 64 at tile_m 32, and tile_m 64, are repeatable. The deterministic path avoids it; worth a look by the FlyDSL owners.PR Self-Review
Reviewed revision
f1d9917
Behavior and intent
fused_moegets adeterministic=Falsekeyword. WithTrue, MXFP4 MoE (gfx950, gate/up-interleaved weights, SwiGLU, bf16 output) runs one fixed configuration: MXFP8 activations at every token count, FlyDSL a8w4 / v1 kernels with one K structure, no split-K, and the non-atomic reduce epilogue in stage 2, so a token's output is bitwise the same from run to run and whatever else is in the batch. Unsupported configurations raiseNotImplementedError. WithFalse(the default) nothing changes.The existing behavior is insufficient for deterministic inference: the stage-2 atomic top-k sum depends on workgroup order, stage-1 split-K adds atomically too, and the tuned configuration (activation dtype, K waves, stage-2 kernel family, tile_k) changes with the token count. In SGLang's deterministic mode on MI350X this made gpt-oss and MiniMax-M3 give different outputs for identical requests depending on their batchmates.
Validation and unverified scope
Validated on MI350X (gfx950), image
rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030, with this revision's Python files applied to the image's AITER:python3 -m pytest -q op_tests/test_moe_mxfp4_deterministic.py: 13 passed.python3 -m pytest -q op_tests/test_fhmoe.py -k "signature or positional or deterministic": 1 passed (the signature test).python3 -m sglang.test.test_deterministic(single, prefix, prefix with logprobs, radix cache) plus a 1,275-response bitwise batch-size check pass for gpt-oss-20b (TP1, TP2), gpt-oss-120b (TP1, TP4, TP8) and MiniMax-M3 (TP4†, TP8); 64k / 120k-token batch invariance and chunked-prefill boundaries for gpt-oss; GSM8K and AIME25 as in the accuracy table. gpt-oss-20b TP1 and the runs marked † are on this revision, the rest on the previous one, which runs the same kernels for interleaved weights.The merge of main (conflicts only in parameter lists, the HERD metadata check and
fused_moe_2stages'num_rows) was checked with black, ruff and byte-compilation; the GPU results above ran on the branch before the merge, applied to the image's AITER.Not verified: other GPUs (the mode refuses anything but gfx950; MI355X not run), AITER's full op-test suite (only the files above), AOT precompilation of the deterministic kernels (they JIT on first use), activations other than SwiGLU (refused), expert parallelism and shared experts (refused). Not root-caused: why the a8w4 stage 1 at tile_m 32 x tile_n 128 varies from run to run (avoided here).
PR-specific question
Before this PR every test in
op_tests/test_moe_mxfp4_deterministic.pyfails at the call, sincefused_moehas nodeterministickeyword. With the default path in its place, the bitwise assertions fail: repeated calls on the same batch differ (atomic top-k sum), and the first k tokens differ from the same tokens in the full batch at the activation-dtype switch (256 tokens) and the tile switches. The rejection test fails because nothing is rejected. Existing MoE op tests compare against a reference within a tolerance for one batch; none checks bitwise equality across runs or across batch compositions, which is the property this mode adds.Changed op tests
op_tests/test_moe_mxfp4_deterministic.py(new): coversfused_moe(deterministic=True)with gate/up-interleaved weights at gpt-oss and MiniMax-M3 shapes, SwiGLU clamp and biases (accuracy at least the default's, bitwise run to run, bitwise across batch prefixes and a single token), zero routing weights,num_local_tokens, and every refused configuration (including separated weights and MXFP4 activations). Not redundant: no existing test exercises the deterministic mode or checks batch invariance.op_tests/test_fhmoe.py(one line added):test_clamp_shared_preserves_existing_positional_argument_orderlistsfused_moe's parameters fromquant_type_ato the end;deterministicis appended after the HERD parameters, so the test still guards existing positional callers. Necessary because the assertion would fail on this PR without it; no other test covers the signature order.