Skip to content

[FlyDSL] [MoE] fused_moe: add a deterministic mode for MXFP4 MoE - #6583

Merged
Bernard-Liu merged 29 commits into
ROCm:mainfrom
aarnetalman:moe-a16w4-deterministic
Nov 14, 2026
Merged

Bernard-Liu merged 29 commits into
ROCm:mainfrom
aarnetalman:moe-a16w4-deterministic

Conversation

@aarnetalman

@aarnetalman aarnetalman commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Motivation

fused_moe does not give the same output for the same token. For MXFP4 MoE with wiGLU or SiLU (gpt-oss, MiniMax-M3), three things make a token's result depend on scheduling or on its batch:

  • Atomic top-k sum: the down projection adds each token's top-k expert outputs into its output row with atomic adds, in whatever order the workgroups finish.
  • Split-K: split-K in the first stage also accumulates with atomic adds.
  • Configuration by token count: the tuned configurations change the activation dtype (bf16 below 256 tokens, MXFP4 or MXFP8 from 256), the stage-1 K waves and the stage-2 kernel family and tile_k with the token count. Each of these changes how a row's sums are split.

Serving frameworks that offer deterministic inference need bitwise the same output for a token whatever else is in the batch; SGLang's --enable-deterministic-inference is one. With gpt-oss and MiniMax-M3 on MI350X, the MoE layer was one of the layers whose output changed while its input was bitwise equal.

Modifications

fused_moe(..., deterministic=False): with True, MXFP4 MoE runs one configuration with a fixed reduction order:

  • MXFP8 activations at every token count, quantized per 1x32 block within a token's row: the precision the default uses for gate/up-interleaved weights from 256 tokens, and as accurate as the default end to end (below). MXFP4 activations are not offered: they cost 5-11 AIME points.
  • One K structure. Stage 1 is the FlyDSL a8w4 kernel with tile_k 256 (_w2, gate/up-interleaved); stage 2 is the v1 kernel with the reduce epilogue (unique [token*topk + slot] rows, then moe_reduce sums the top-k slots in a fixed order), no split-K. Only the tile height grows with the token count (64 up to 2,048 tokens, then 128), which does not change a row's reduction. It starts at tile_m 64 because the a8w4 stage 1 at tile_m 32 with tile_n 128 is not repeatable from run to run on gfx950 (below).
  • Supported: gfx950, MXFP4 weights (QuantType.per_1x32) preshuffled gate/up-interleaved, SwiGLU with or without clamp and per-expert biases, bf16 output.
  • Refused: everything else raises NotImplementedError naming what is unsupported: separated weights, w1 and w2 not both preshuffled, other activation dtypes, other activations, expert parallelism, shared experts, prequantized activations, other block sizes, hidden sizes not divisible by 512, hidden padding without a stage-2 bias, intermediate padding with a stage-2 bias but no stage-1 bias.
  • Default unchanged. The kernels are existing registered FlyDSL kernels; no kernel changes.
  • Tests: op_tests/test_moe_mxfp4_deterministic.py. op_tests/test_fhmoe.py's signature test lists deterministic after the existing parameters.

Test results

MI350X. Op tests and per-call latency: image rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030 with this PR's files applied to its AITER. End-to-end results: images 20261021 / 20261030, the same way.

Op tests: op_tests/test_moe_mxfp4_deterministic.py, 13 passed. Shapes: gpt-oss (3072 x 3072, 32 and 128 experts, top-4) and MiniMax-M3 per GPU at TP4 / TP2, SwiGLU with clamp and biases, 2,304 tokens, gate/up-interleaved weights. Outputs are equal from run to run, and a token's output is the same alone, in the full batch and in prefixes across the default's activation switch (256) and the tile-height switch (2,048). Accuracy against a bf16 torch reference is at least the default's.

Accuracy against today's default, end to end in SGLang (AIME25, thinking on, one sample per seed, differences paired by seed; GSM8K, 1,319 questions, temperature 0):

model AIME25, deterministic vs default GSM8K, deterministic (default)
gpt-oss-20b 67.9% vs 70.0%, -2.1 ± 1.3 points (48 seeds) 94.31% (94.84%)
gpt-oss-120b 78.7% vs 78.4%, +0.3 ± 1.1 (40 seeds) 96.06%
MiniMax-M3, TP4 76.3% vs 78.1% (16 seeds, within seed noise) 96.21% (95.98%)

For gpt-oss only the MoE call differs; for MiniMax-M3 the whole deterministic mode is compared with normal mode (SGLang loads its experts gate/up-interleaved in deterministic mode). MXFP4 activations, which an earlier head of this PR used, cost gpt-oss-20b 5.4 ± 2.0 AIME points and MiniMax-M3 11.3 ± 2.5 (66.7% vs 77.9%, GSM8K 92.87%), which is why they were dropped. Kernel error against a bf16 reference (relative L2): 0.043, the same as the default's MXFP8 path (0.04); MXFP4 activations 0.21.

SiLU with a clamp and no bias at three shapes: accuracy no worse than the default, run to run, batch invariance.

Latency per call (HIP-graph replay, median of 50 calls), deterministic time / default time (below 1 is faster):

shape 1 token 8 32 128 256 1,024 4,096 8,192
gpt-oss-20b (32 experts, top-4) 0.95 1.12 1.06 0.81 0.80 0.78 0.70 0.73
gpt-oss-120b (128 experts, top-4) 0.95 1.08 1.14 1.12 1.17 1.09 0.97 1.07
MiniMax-M3 TP4 (129 experts, 6144 x 768, top-6) 0.92 1.09 1.10 1.14 1.07 0.94 0.99 1.12

End to end in SGLang, output throughput as a ratio to the default (same server, only the MoE call changes):

1k/1k, 1 request 16 64 8k/1k, 1 request 16
gpt-oss-20b 0.99 0.93 0.98 1.00 0.99
gpt-oss-120b 0.99 0.89 0.92 1.00 0.93

Determinism end to end in SGLang (test_deterministic single / prefix / prefix logprobs / radix cache, plus a bitwise batch-size check of 1,275 responses, token IDs and logprobs compared exactly): every check passes for gpt-oss-20b (TP1, TP2), gpt-oss-120b (TP1, TP4, TP8) and MiniMax-M3 (TP4, TP8); gpt-oss also with 64k and 120k-token prompts. Per call, 0 of 20 runs differ at 17 to 4,096 tokens (32 and 128 experts, allocator filled with NaN between runs).

Notes

  • Layout: callers shuffle MXFP4 weights with shuffle_weight_a16w4(w, 16, True) (gate/up-interleaved) and the scales with shuffle_scale_a16w4(s, E, True). Models stored with separated gate/up (MiniMax-M3) can be shuffled interleaved at load time; SGLang does this in deterministic mode.
  • a8w4 kernel finding: flydsl_moe1_afp8_wfp4_bf16_t32x128x256*_gui (every waves-per-EU variant) gives a different result from run to run on gfx950 with identical inputs (up to 5 of 5 runs at 300 tokens, also with NaN-filled memory); tile_n 64 at tile_m 32, and tile_m 64, are repeatable. The deterministic path avoids it; worth a look by the FlyDSL owners.
  • AOT: AOT precompilation covers the kernels named in the tuned configurations; the deterministic configuration's kernels that are not named there are JIT-compiled on first use.
  • Other activations: SiLU (GLM-5, DeepSeek-V4 routed experts) can follow once its accuracy is checked against its default; SiTUv2 likewise.

PR Self-Review

Reviewed revision

f1d9917

Behavior and intent

fused_moe gets a deterministic=False keyword. With True, MXFP4 MoE (gfx950, gate/up-interleaved weights, SwiGLU, bf16 output) runs one fixed configuration: MXFP8 activations at every token count, FlyDSL a8w4 / v1 kernels with one K structure, no split-K, and the non-atomic reduce epilogue in stage 2, so a token's output is bitwise the same from run to run and whatever else is in the batch. Unsupported configurations raise NotImplementedError. With False (the default) nothing changes.

The existing behavior is insufficient for deterministic inference: the stage-2 atomic top-k sum depends on workgroup order, stage-1 split-K adds atomically too, and the tuned configuration (activation dtype, K waves, stage-2 kernel family, tile_k) changes with the token count. In SGLang's deterministic mode on MI350X this made gpt-oss and MiniMax-M3 give different outputs for identical requests depending on their batchmates.

Validation and unverified scope

Validated on MI350X (gfx950), image rocm/sgl-dev:v0.5.23-rocm724-mi35x-20261030, with this revision's Python files applied to the image's AITER:

  • python3 -m pytest -q op_tests/test_moe_mxfp4_deterministic.py: 13 passed.
  • python3 -m pytest -q op_tests/test_fhmoe.py -k "signature or positional or deterministic": 1 passed (the signature test).
  • Per call: latency as in the table above. Run to run (earlier revisions with the same kernels and tiles): 20 runs with the allocator filled with NaN between runs, 17 to 4,096 tokens, 32 and 128 experts: 0 runs differ, prefixes match the full batch.
  • End to end in SGLang deterministic mode: python3 -m sglang.test.test_deterministic (single, prefix, prefix with logprobs, radix cache) plus a 1,275-response bitwise batch-size check pass for gpt-oss-20b (TP1, TP2), gpt-oss-120b (TP1, TP4, TP8) and MiniMax-M3 (TP4†, TP8); 64k / 120k-token batch invariance and chunked-prefill boundaries for gpt-oss; GSM8K and AIME25 as in the accuracy table. gpt-oss-20b TP1 and the runs marked † are on this revision, the rest on the previous one, which runs the same kernels for interleaved weights.

The merge of main (conflicts only in parameter lists, the HERD metadata check and fused_moe_2stages' num_rows) was checked with black, ruff and byte-compilation; the GPU results above ran on the branch before the merge, applied to the image's AITER.

Not verified: other GPUs (the mode refuses anything but gfx950; MI355X not run), AITER's full op-test suite (only the files above), AOT precompilation of the deterministic kernels (they JIT on first use), activations other than SwiGLU (refused), expert parallelism and shared experts (refused). Not root-caused: why the a8w4 stage 1 at tile_m 32 x tile_n 128 varies from run to run (avoided here).

PR-specific question

Before this PR every test in op_tests/test_moe_mxfp4_deterministic.py fails at the call, since fused_moe has no deterministic keyword. With the default path in its place, the bitwise assertions fail: repeated calls on the same batch differ (atomic top-k sum), and the first k tokens differ from the same tokens in the full batch at the activation-dtype switch (256 tokens) and the tile switches. The rejection test fails because nothing is rejected. Existing MoE op tests compare against a reference within a tolerance for one batch; none checks bitwise equality across runs or across batch compositions, which is the property this mode adds.

Changed op tests

  • op_tests/test_moe_mxfp4_deterministic.py (new): covers fused_moe(deterministic=True) with gate/up-interleaved weights at gpt-oss and MiniMax-M3 shapes, SwiGLU clamp and biases (accuracy at least the default's, bitwise run to run, bitwise across batch prefixes and a single token), zero routing weights, num_local_tokens, and every refused configuration (including separated weights and MXFP4 activations). Not redundant: no existing test exercises the deterministic mode or checks batch invariance.
  • op_tests/test_fhmoe.py (one line added): test_clamp_shared_preserves_existing_positional_argument_order lists fused_moe's parameters from quant_type_a to the end; deterministic is appended after the HERD parameters, so the test still guards existing positional callers. Necessary because the assertion would fail on this PR without it; no other test covers the signature order.

@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:gfx1250-ffm-triton Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355 Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu Aiter multi-GPU tests on the 8-GPU runner
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6583 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

samremes
samremes previously approved these changes Oct 15, 2026

@samremes samremes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

On layouts: let's support what is most needed, more support could be done later.
On block_m: using only one for determinism could make sense, I think, but could be worth to also try larger blocks and see if it retains the determinism

Comment thread op_tests/test_flydsl_moe_a16wfp4.py Resolved
@aarnetalman

Copy link
Copy Markdown
Member Author

LGTM.

On layouts: let's support what is most needed, more support could be done later. On block_m: using only one for determinism could make sense, I think, but could be worth to also try larger blocks and see if it retains the determinism

Thanks! Layouts: agreed, separated only for now.

Block size: I tried 16, 64 and 128 against 32 on gpt-oss-20b's shape (3072 x 3072, 32 experts, top-4, with biases) on MI350X. Every block size gives bitwise the same output as 32 at 1-1,024 tokens, and each is deterministic and batch-invariant on its own. So the reduction order doesn't depend on the block size, as expected. They are slower than 32 at every size, though (ms per call):

tokens 32 16 64 128
1 0.186 0.189 0.204 0.297
256 0.274 0.290 0.352 0.626
1,024 0.475 0.582 0.658 0.936
4,096 1.393 1.869 1.835 2.365

So I kept 32 for every token count. Since the results don't depend on the block size, a token-count heuristic would stay deterministic if a later kernel change makes larger blocks faster.

…4 weights

The a16w-mix stage1/stage2 kernels (bf16 activations x MXFP4/int4 weights)
take an optional per-expert f32 bias: stage1 adds it to gate and up before
the activation ([E, 2*inter_dim], gate | up), stage2 adds it before the
routing weight ([E, model_dim]). Models with expert biases, such as
gpt-oss, previously could not use these kernels.

The non-atomic stage2 epilogue (unique [token*topk + slot] rows, then
moe_reduce summing the top-k slots in a fixed order) was enabled only for
int4 weights; it now also runs for MXFP4 weights when the stage2 kernel
name selects "reduce". The default stays atomic.
fused_moe(deterministic=True) gives bitwise the same output for a token
whatever it is batched with, and from run to run, as deterministic
inference in serving frameworks requires. For bf16 activations with MXFP4
weights (QuantType.per_1x32, separated gate/up, preshuffled) it runs the
FlyDSL a16w4 kernels with:
- the non-atomic stage-2 epilogue: per-(token, slot) rows, then moe_reduce
  sums the top-k slots in a fixed order, instead of atomic adds whose order
  depends on scheduling;
- no split-K;
- bf16 activations at every token count (the default switches to fp4/fp8
  activations from 256 tokens);
- one block size (32) for every token count.
Per-expert biases (gpt-oss) are supported. Unsupported configurations
(other weight types, interleaved gate/up, expert parallelism, shared
experts, prequantized activations) raise NotImplementedError. The default
(deterministic=False) is unchanged.

stage2_uses_route_reduce now reports the a16w4 reduce epilogue, so the
output buffer is not zeroed for it.

op_tests/test_flydsl_moe_a16wfp4.py: SwiGLU with clamp and per-expert
biases against the torch reference, run-to-run equality, and the same
tokens alone and in batches of 1-300 compared bitwise.
@aarnetalman
aarnetalman force-pushed the moe-a16w4-deterministic branch from ae32a4c to f67b87b Compare October 19, 2026 11:17
@aarnetalman
aarnetalman marked this pull request as ready for review October 19, 2026 11:17
@aarnetalman
aarnetalman requested review from a team and a balanced review from Copilot October 19, 2026 11:17
@github-actions github-actions Bot changed the title [MoE] fused_moe: add a deterministic mode for bf16 x MXFP4 MoE [FlyDSL] [MoE] fused_moe: add a deterministic mode for bf16 x MXFP4 MoE Oct 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Deterministic dispatch accepts incompatible activation, output-dtype, and weight-layout configurations while rejecting some supported biased calls.

7 open issues
What changed in this PR

Adds opt-in deterministic bf16 × MXFP4 execution to fused_moe, targeting repeatable, batch-independent inference.

Changes:

  • Selects fixed bf16 kernels without split-K and with ordered top-k reduction.
  • Adds per-expert biases to FlyDSL a16w4 kernels.
  • Tests reference accuracy, repeatability, and batch invariance.
File Description
op_tests/​test_flydsl_moe_a16wfp4.py Adds deterministic SwiGLU tests with biases.
aiter/​ops/​flydsl/​moe_kernels.py Connects bias arguments and MXFP4 reduction mode.
aiter/​ops/​flydsl/​kernels/​moe_2stage_a16wmix/​utils.py Adds an f32 bias-load helper.
aiter/​ops/​flydsl/​kernels/​moe_2stage_a16wmix/​gemm2.py Adds down-projection bias before routing weights.
aiter/​ops/​flydsl/​kernels/​moe_2stage_a16wmix/​gemm1.py Adds gate/up bias before activation.
aiter/​ops/​flydsl/​kernels/​moe_2stage_a16wmix/​__init__.py Normalizes and forwards bias tensors.
aiter/​fused_moe.py Adds deterministic dispatch and support checks.

Review effort: Balanced

Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/ops/flydsl/moe_kernels.py Resolved
Comment thread op_tests/test_flydsl_moe_a16wfp4.py Resolved
Review follow-ups for fused_moe(deterministic=True):
- reject SiLU: both bf16 activation-quantization bypasses cover only
  SwiGLU and SiTUv2, so SiLU would feed MXFP4-quantized activations to
  the bf16 kernels;
- reject non-bf16 output: the reduce epilogue stores bf16 only;
- reject weights tagged gate/up-interleaved: the deterministic kernels
  read the separated preshuffle;
- allow per-expert biases past the older a16w4 validation, which
  refused them for SiTUv2 and rerouted SwiGLU shapes;
- check the live GPU with get_gfx_runtime();
- AOT precompile builds the reduce epilogue for MXFP4 too, as the
  runtime does.

The test compares against the torch reference elementwise with
checkAllclose and a tighter cosine bound, and a new test checks that
the rejected configurations raise.
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/ops/flydsl/moe_kernels.py Resolved
Comment thread aiter/fused_moe.py Resolved
Comment thread aiter/ops/flydsl/moe_kernels.py Resolved
Comment thread aiter/fused_moe.py Resolved
Review follow-ups for fused_moe(deterministic=True):
- _reduce_bf16_epilog checked only token_id < M. With num_local_tokens
  below the batch capacity, the sort's padding rows carry token_id =
  num_local_tokens and slot = topk, so they were stored at row
  (num_local_tokens + 1) * topk, past the [M * topk, N] buffer when
  num_local_tokens = M - 1. The store now also requires slot < topk.
- the interleaved-weight check uses _flydsl_mxfp4_layout_is_compatible,
  as the other MXFP4 dispatch paths do;
- AOT a16w4 precompile builds the has_bias variants when the job asks for
  them, and moe_reduce after the reduce epilogue, as the runtime does;
- the docstring says the deterministic kernels are JIT-compiled on first
  use (they are not in the tuned configs that AOT reads).

A new test runs the reduce epilogue with a guard region after its buffer
and num_local_tokens below the capacity: the guard stays untouched, and
the local tokens match the same tokens run alone.
Copilot AI balanced review requested due to automatic review settings October 21, 2026 04:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shuffle validation accepts incompatible weight pairs, and standard CI does not execute the new regression tests.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Run this module's tests in the standard CI runner

op_tests/​test_flydsl_moe_a16wfp4.py:136

The standard CI runner selects this module but does not execute its tests: .github/scripts/aiter_test.sh:91–104 uses python3 "$file", and this path is not a pytest exception. With no script entry point, the module exits without checking determinism, unsupported configurations, or buffer bounds. Add this file to the runner's existing pytest dispatch cases so CI collects and runs these regressions.

6 resolved since last review

Review effort: Balanced

Comment thread aiter/fused_moe.py Resolved
@aarnetalman

Copy link
Copy Markdown
Member Author

@Bernard-Liu thanks for the thorough review. All six points are addressed in 2d544d3 and d6572e6:

  • refused with NotImplementedError: SiLU, non-bf16 output, gate/up-interleaved weights (via _flydsl_mxfp4_layout_is_compatible);
  • per-expert biases now reach the deterministic path;
  • the reduce epilogue skips padding slots (slot < topk), fixing the out-of-bounds write with num_local_tokens;
  • AOT builds the MXFP4 reduce epilogue, the bias variants and moe_reduce.

On MI350X, op_tests/test_flydsl_moe_a16wfp4.py: 15 passed, including the deterministic tests, a test that the refused configurations raise, and a guard-region test for the epilogue (which fails with the previous gemm2.py in your two cases). The 4 situv2_e2e failures are pre-existing: they fail the same way without this PR. black and ruff clean.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The architecture gates can skip all correctness tests on supported hardware, and the latency sweep can report success despite out-of-tolerance results because mismatch limits are not enforced.

4 open issues
1 resolved since last review

Review effort: Balanced

@aarnetalman

Copy link
Copy Markdown
Member Author

Thanks @Bernard-Liu. Done in the latest push:

  • CI: merged current main (d02f9d058, includes [FlyDSL] Fix main CI: missing DROPPED_ROUTE_ROW import in moe_contiguous_psum #7583); no conflicts.
  • r4490120184: fixed in 6caf18cd0. The docstring now describes a single K wave (k_wave 1; the _w2 in the kernel name is waves_per_eu 2, an occupancy setting). The comment at the hidden-size check no longer explains the multiple-of-512 restriction with the two-wave reading. I could not give a verified kernel reason for 512, so it now only states what is known: only multiples of 512 have been validated with the deterministic kernels. The restriction itself is unchanged.
  • The 18 Copilot threads: I've replied and resolved each.

The skip marker and main() used get_gfx(), which follows GPU_ARCHS, so a
multi-arch GPU_ARCHS such as "gfx950;gfx942" skipped every test on gfx950.
Use get_gfx_runtime(), as fused_moe's deterministic check does.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 09:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The latency sweep never enforces the _ERR_MAX bound, allowing incorrect outputs for shapes not covered by pytest to pass without failing the script.

No open issues 🎉

4 resolved since last review

Review effort: Balanced

The latency sweep reported each candidate's mismatch ratio without checking
it, so a sweep shape or token count the pytest checks do not cover could pass
with a wrong result. Assert it against _ERR_MAX, as the pytest checks do. On
MI350X the largest ratio in the default sweep is 3.3e-4 (deterministic) and
1.5e-4 (default), against 1e-3.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 11:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The changed reduction kernel needs gfx950 correctness and latency validation on the merged revision, but reported GPU tests predate the merge and this environment cannot rerun them.

No open issues 🎉

Review effort: Balanced

@aarnetalman

Copy link
Copy Markdown
Member Author

CI failure on 220eaa8: shard 5, test_flydsl_hgemm.py
The MI35X shard-5 failure is in op_tests/test_flydsl_hgemm.py, which this PR does not change: graph replay mismatch for bf16 with split_k=9, k_stagger=127 (1 of 2,280 elements differs by 0.015625 in the changed-input CUDA-graph check). The new test file in this PR moved that file into shard 5; it passes on main in another shard, so it looks intermittent. Could someone re-run the failed job?

fused_moe(deterministic=True) accepted only SwiGLU. Routed experts with plain
SiLU (clamped through swiglu_limit, no bias), as in GLM-5 and DeepSeek-V4,
raised NotImplementedError. The deterministic path itself does not depend on
the activation: the FlyDSL stage-1 kernels already implement SiLU, and the
fixed tiles, the single activation dtype and the non-atomic stage-2 reduce
apply unchanged.

Allow ActivationType.Silu next to Swiglu. New op test: SiLU with a clamp and
no bias at three shapes (GLM-5 and DeepSeek-V4 per GPU at TP4 among them),
checked for accuracy against the bf16 reference (no worse than the default
path), run-to-run equality and batch invariance.
Copilot AI balanced review requested due to automatic review settings November 11, 2026 14:33
@aarnetalman

Copy link
Copy Markdown
Member Author

Small addition in 2aaf04295: deterministic=True now also accepts SiLU experts (clamped, no bias), as used by GLM-5 and DeepSeek-V4. The deterministic path itself is unchanged: the FlyDSL stage-1 kernels already implement SiLU, and the fixed tiles, the single activation dtype and the non-atomic stage-2 reduce apply as they are; only the support check changes. A new op test runs SiLU at three shapes (GLM-5 and DeepSeek-V4 routed experts per GPU at TP4 among them): accuracy against the bf16 reference no worse than the default path, run-to-run equality and batch invariance. On MI350X the op tests pass (19 passed). SGLang uses it for GLM-5.3 (tracking: sgl-project/sglang#); keeping it in this PR means one AITER update for SGLang instead of two.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new SiLU accuracy test lacks the per-element reference-tolerance check required by the shared convention, allowing localized errors to pass its aggregate cosine metric.

1 open issue

Review effort: Balanced

… tolerance

The SiLU accuracy test compared only the aggregate cosine difference. Assert
the checkAllclose mismatch ratio against _ERR_MAX as the SwiGLU test does, and
keep the cosine comparison.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 10:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Exact merged-head gfx950 validation is needed because reported GPU runs predate the merge, leaving bitwise batch invariance and latency unconfirmed.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@Bernard-Liu

Copy link
Copy Markdown
Contributor

Could you take care of these before merge:

  1. CI. The failing job is shard 2 of the standard tests: op_tests/test_flydsl_moe_aux.py fails with NameError: name 'DROPPED_ROUTE_ROW' is not defined in aiter/ops/flydsl/kernels/moe_contiguous_psum.py. It comes from main, not from this PR: main fixed the missing import in [FlyDSL] Fix main CI: missing DROPPED_ROUTE_ROW import in moe_contiguous_psum #7583 (a97e999228), after the main you merged (8e0000db). Merging current main (no conflicts as of 07933277) and rerunning should clear it.

  2. Open conversations.

    • r4490120184: fused_moe.py (around lines 4103 and 4160) still says stage 1 runs two K waves. _w2 is waves_per_eu=2, and the kernels run with k_wave=1. Please describe the single K wave, and either give the verified reason for the multiple-of-512 restriction or drop that explanation.
    • 18 other review threads, all Copilot's from earlier revisions, have no reply yet; three of them point at files this PR no longer changes. Please resolve the ones later commits addressed and reply to any that still apply.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The bitwise-determinism and batch-invariance guarantees require human validation on the final gfx950 revision because supplied GPU results predate the merge and correctness and latency checks cannot be run here.

No open issues 🎉

Review effort: Balanced

@aarnetalman

Copy link
Copy Markdown
Member Author

Could you take care of these before merge:

1. **CI.** The failing job is shard 2 of the standard tests: `op_tests/test_flydsl_moe_aux.py` fails with `NameError: name 'DROPPED_ROUTE_ROW' is not defined` in `aiter/ops/flydsl/kernels/moe_contiguous_psum.py`. It comes from main, not from this PR: main fixed the missing import in [[FlyDSL] Fix main CI: missing DROPPED_ROUTE_ROW import in moe_contiguous_psum #7583](https://github.com/ROCm/aiter/pull/7583) (`a97e999228`), after the main you merged (`8e0000db`). Merging current main (no conflicts as of `07933277`) and rerunning should clear it.

2. **Open conversations.**
   
   * [r4490120184](https://github.com/ROCm/aiter/pull/6583#discussion_r4490120184): `fused_moe.py` (around lines 4103 and 4160) still says stage 1 runs two K waves. `_w2` is `waves_per_eu=2`, and the kernels run with `k_wave=1`. Please describe the single K wave, and either give the verified reason for the multiple-of-512 restriction or drop that explanation.
   * 18 other review threads, all Copilot's from earlier revisions, have no reply yet; three of them point at files this PR no longer changes. Please resolve the ones later commits addressed and reply to any that still apply.

Thanks @Bernard-Liu.

  1. CI: merged current main (5c1489c24, which includes the [FlyDSL] Fix main CI: missing DROPPED_ROUTE_ROW import in moe_contiguous_psum #7583 import fix) in cd9ee6537; no conflicts. The shard 2 DROPPED_ROUTE_ROW failure should be gone on the rerun.
  2. r4490120184: fixed in 6caf18cd0 and unchanged since; on the current head the docstring describes a single K wave (fused_moe.py around line 4240: "k_wave 1; the _w2 in the kernel name is waves_per_eu 2"), and the hidden-size check around line 4183 only states that multiples of 512 are the validated ones, without a kernel explanation. I've resolved the thread.
  3. Copilot threads: every thread has a reply and is resolved, including the three on files this PR no longer changes. If one still needs more, point me to it and I'll follow up there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants