Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
cbdc00a
feat(qwen3-next): MoE route/combine contract with a batch-invariant g…
fusheng-ji Oct 9, 2026
362b80b
docs(qwen3-next): MoE route/combine contract, prior-art and TP4 evide…
fusheng-ji Oct 9, 2026
dc79953
test(qwen3-next): measure Megatron-core/TE and SGLang in the MoE prio…
fusheng-ji Oct 9, 2026
53dce6e
docs(qwen3-next): MoE prior art with Megatron-core/TE and SGLang meas…
fusheng-ji Oct 9, 2026
f01ab18
Merge test-qwennext refactor and migrate Qwen3-Next code and tests
fusheng-ji Oct 10, 2026
7317e5a
style(tools): mark the Qwen3-Next TP gate imports after sys.path setu…
fusheng-ji Oct 10, 2026
52b97e3
fix(qwen3-next): type annotations for mypy in the MoE prior-art runner
fusheng-ji Oct 11, 2026
4daf3c4
Merge the latest #469 branch into the MoE route branch
fusheng-ji Oct 14, 2026
57216db
refactor(qwen3-next): MoE route branch to the reviewed layout
fusheng-ji Oct 14, 2026
8943c14
docs: move the measurement outputs out of the pull request
fusheng-ji Oct 19, 2026
bc43113
docs: point commit hashes at the rewritten history
fusheng-ji Oct 23, 2026
68240ec
Merge #469 after #467 landed in test-qwennext
fusheng-ji Oct 26, 2026
cfbd101
Merge #469 review fixes
fusheng-ji Oct 26, 2026
d53ac95
ci(qwen3-next): trigger the provider job on validation/common
fusheng-ji Oct 26, 2026
9800e1a
Merge #469 second-round review fixes
fusheng-ji Oct 26, 2026
d5253a5
Merge branch 'test-qwennext' into feat/ws1-qwen3-next-moe-route-combine
maxiaosong1124 Oct 30, 2026
e514d42
fix(qwen3-next): fixed-order route-weight gradient in combine_routes
fusheng-ji Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ jobs:
run: |
python -m pytest tests/runtime/test_dispatch.py -v
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest tests/ops/attention/test_attention_correctness.py -q -rs
python -m pytest tests/validation/operators/test_forward_invariance.py tests/contracts/test_tolerance_contract.py tests/validation/operators/test_ws1_workload.py tests/validation/operators/test_gradient_invariance.py tests/validation/operators/test_elementwise_inventory.py tests/validation/operators/test_four_judgment_matrix.py tests/validation/operators/test_op_checks.py tests/validation/operators/test_operator_inputs.py tests/benchmarks/test_profiler.py tests/validation/operators/test_kv_consistency.py tests/models/qwen3/test_ws1_qwen3_dense.py tests/validation/models/test_ws1_chain_integration.py tests/models/qwen3_next/test_qwen3_next_norm.py tests/models/qwen3_next/test_gdn_state_contract.py -q
python -m pytest tests/validation/operators/test_forward_invariance.py tests/contracts/test_tolerance_contract.py tests/validation/operators/test_ws1_workload.py tests/validation/operators/test_gradient_invariance.py tests/validation/operators/test_elementwise_inventory.py tests/validation/operators/test_four_judgment_matrix.py tests/validation/operators/test_op_checks.py tests/validation/operators/test_operator_inputs.py tests/benchmarks/test_profiler.py tests/validation/operators/test_kv_consistency.py tests/models/qwen3/test_ws1_qwen3_dense.py tests/validation/models/test_ws1_chain_integration.py tests/models/qwen3_next/test_qwen3_next_norm.py tests/models/qwen3_next/test_gdn_state_contract.py tests/validation/common/test_tensor_identity.py tests/models/qwen3_next/test_qwen3_next_forward_contract.py tests/models/qwen3_next/test_qwen3_next_tp_blocks.py -q

- name: Run Cross-Configuration Contract Tests (CPU-safe)
run: |
Expand Down
15 changes: 12 additions & 3 deletions .github/workflows/qwen3-next-provider-gpu.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# RFC #428 Qwen3-Next provider comparisons: the C1/C3/C4 norm providers and the C6
# GDN decode-step goldens, each checked against the real vLLM 0.30.0 provider.
# RFC #428 Qwen3-Next provider comparisons: the C1/C3/C4 norm providers, the C6
# GDN decode-step goldens and the shared GEMM/MoE route-combine provider, each
# checked against the real vLLM 0.30.0 provider.
#
# These check_ files import vLLM, so the default pytest collection never runs them
# (tests/integrations/common/test_framework_operator_integrations.py asserts vLLM is not imported). This
Expand All @@ -23,10 +24,15 @@ on:
paths:
- "rl_engine/backends/**"
- "rl_engine/reference/**"
- "rl_engine/models/qwen3_next/**"
- "rl_engine/integrations/engines/train/vllm/qwen3_next*"
- "rl_engine/validation/models/qwen3_next*"
- "tools/validation/models/*qwen3_next*"
- "rl_engine/validation/models/qwen3_next/**"
- "tools/validation/models/qwen3_next/**"
- "tests/models/qwen3_next/**"
- "rl_engine/config/**"
- "rl_engine/validation/common/**"
- ".github/workflows/qwen3-next-provider-gpu.yml"
workflow_dispatch:
inputs:
Expand Down Expand Up @@ -118,10 +124,13 @@ jobs:
print("rl_engine._C:", loaded)
PY
- name: Check pinned providers in their own process
# The FP32 router needs IEEE Triton dots from process start.
env:
TRITON_F32_DEFAULT: ieee
run: |
set -o pipefail
log="$RUNNER_TEMP/provider-pytest.log"
python3 -m pytest -q -rs tests/models/qwen3_next/check_qwen3_next_norm_providers.py tests/models/qwen3_next/check_gdn_recurrent_golden.py | tee "$log"
python3 -m pytest -q -rs tests/models/qwen3_next/check_qwen3_next_norm_providers.py tests/models/qwen3_next/check_gdn_recurrent_golden.py tests/models/qwen3_next/check_qwen3_next_forward.py | tee "$log"
# Fail closed: these modules skip as a whole when CUDA or vLLM is missing, and pytest
# exits 0 for an all-skipped run. A skipped provider check is not provider evidence.
if grep -Eq '^SKIPPED|[0-9]+ skipped' "$log"; then
Expand Down
1 change: 1 addition & 0 deletions docs/operators/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,5 @@ Every operator page should include:
- [Token Embedding](embedding.md)
- [Qwen3-Next RMSNorm (zero-centred)](qwen3-next-rms-norm.md)
- [Qwen3-Next Gated RMSNorm](qwen3-next-rms-norm-gated.md)
- [Qwen3-Next MoE route / combine](qwen3-next-moe-route-combine.md)
- [Operator Doc Template](../contributing/operator-doc-template.md)
153 changes: 153 additions & 0 deletions docs/operators/qwen3-next-moe-route-combine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# Qwen3-Next MoE route / combine contract

## Summary

Qwen3-Next's sparse MoE block routes every token to 10 of 512 experts (expert
width 512, TP4-local width 128) and adds a gated shared expert. RFC #428 row
`moe_route_combine_contract` asks for a routed MoE whose output for a token is a
pure function of that token: the same bits whatever else is in the batch, in
training replay and in rollout. This page states the contract and how
`rl_engine.models.qwen3_next.qwen3_next_forward.shared_moe` meets it.

Upstream references: `transformers` `Qwen3NextSparseMoeBlock` (5.17.0) and vLLM
0.30.0 `fused_topk` + `fused_experts`.

## Entry Point

```python
from rl_engine.models.qwen3_next.qwen3_next_forward import shared_moe, stable_top10_routes
from rl_engine.models.qwen3_next.qwen3_next_tp_blocks import TP4MoE

routed, routes = shared_moe(x, router_weight, gate_up, down) # TP-local, not reduced
block = TP4MoE(group=tp_group, device="cuda") # routed + shared, reduced
block.load_hf_weights(hf_layer_mlp_weights)
y = block(hidden)
```

`TRITON_F32_DEFAULT=ieee` must be set before the process starts: the FP32 router
GEMM refuses to run on TF32 Triton dots.

## Contract

| Stage | Order and dtype | Atomics |
| --- | --- | --- |
| Router | BF16 `x`, `W` upcast to FP32; one pinned vLLM batch-invariant GEMM (IEEE FP32) | none |
| Softmax | max subtraction, `exp`, row sum by a fixed pairwise tree (`fixed_order_row_sum`) | none |
| Selection | `argsort(logits, descending, stable)[:, :10]`: score descending, equal scores to the lower expert id | none |
| Weights | the ten selected FP32 probabilities divided by their fixed-tree sum | none |
| Experts | per (token, route) row: `gate_up` GEMM (BF16 in, FP32 accumulate, BF16 out), SwiGLU in FP32, BF16 cast, `down` GEMM | none: one writer per row |
| Combine | `out = e0*w0`, then `out += e_k*w_k` for k = 1..9 in route order, FP32; one BF16 cast | none |
| Shared expert (TP4 block) | `routed + shared * sigmoid(gate)` in BF16, then one TP all-reduce | collective as configured |

Payload dtype between stages: BF16 expert outputs, FP32 route weights, BF16
block output. The expert GEMMs are vLLM 0.30.0's `fused_moe` Triton kernel with
the tile vLLM itself uses under `VLLM_BATCH_INVARIANT=1`
(`BLOCK_SIZE_M/N/K = 64/64/32`, `GROUP_SIZE_M = 8`, `SPLIT_K = 1`), passed
explicitly so no environment variable, tuned-config file or token count can
change it. The routed weight is *not* multiplied inside the kernel and the
kernel does not sum over routes; that stays in the combine above.

Fail-closed checks: CUDA tensors only (no CPU fallback, no ROCm), BF16 weights of
the exact `[512, 2*I, H]` / `[512, H, I]` shapes, finite FP32 router logits,
vLLM exactly 0.30.0, `VLLM_TRITON_USE_TD` off (the tensor-descriptor path is a
different instruction stream), and IEEE Triton FP32.

### Backward

`_RoutedExperts.backward` recomputes the `[gate, up]` projection with the same
grouped kernel, then visits experts in ascending id. Each expert's `dgate_up`
and `ddown` slice is one pinned GEMM written once into a dense buffer; `dx` is
accumulated per token in ascending expert order (a token's ten experts are
distinct, so each launch has unique rows). `combine_routes` has its own backward:
the expert-output gradient is `grad * weight` per route, and the route-weight
gradient sums `grad * expert_output` over H with `fixed_order_row_sum`, because
`torch.sum` picks its reduction order from the batch shape and made that gradient
batch-variant from 16 rows up. Router-weight gradients then flow through the
selected probabilities; the indices are discrete.

## Reuse decision

| Implementation | Routes BI | Output BI | Backward | Why not reused as is |
| --- | --- | --- | --- | --- |
| vLLM `fused_topk` + `fused_experts`, `VLLM_BATCH_INVARIANT=0` | no | no | none | tuned configs and the BF16 router GEMM depend on the token count |
| vLLM, `VLLM_BATCH_INVARIANT=1` | yes | yes | none | no VJP for training; BF16 router logits (Qwen3-Next and VIME route in FP32); routed weight applied in the expert GEMM epilogue on the BF16 output, then routes summed by `moe_sum` |
| HF `Qwen3NextExperts` (eager) | yes | yes | autograd | `dx`/`dW` change with batch size (cuBLAS shape heuristics); BF16 router; `index_add_` combine in BF16 |
| FlashInfer `cutlass_fused_moe` | (torch routing) | no | none | output changes with batch size; no VJP |
| Megatron-core 0.16 `MoELayer` + TE 2.16 grouped GEMM, as VIME configures Qwen3-Next | no (256+) | no (1024) | autograd, not BI | FP32 router routes like FP64, but the routing, output, dx and dW change with batch size |
| SGLang 0.5.21 Triton `fused_moe`, default | no (256+) | no (256+) | none | tuned configs and the BF16 router GEMM depend on the token count |
| SGLang, deterministic inference | yes | yes | none | inference only; BF16 router logits; its deterministic tile (64/64/32) is the one vLLM uses in batch-invariant mode |

What RL-Kernel reuses: vLLM's batch-invariant GEMM for the router and the
backward, and vLLM's `fused_moe` Triton kernel for the forward expert GEMMs,
which is batch-invariant per route at the fixed tile. What it implements: the
FP32 fixed-order routing, the stable tie rule, the atomic-free FP32 combine, the
deterministic backward and the TP4 boundaries. The grouped kernel's per-route
output is bitwise equal to the per-expert pinned GEMM loop it replaced
(`tests/models/qwen3_next/check_qwen3_next_forward.py::test_grouped_*`), so this is a speed change
with no change in bits.

## Results

Measured on B200 with the prior-art runner of the Qwen3-Next benchmark PR
(`benchmarks/models/qwen3_next/qwen3_next_moe_prior_art.py`) and this PR's TP4 gate
`tools/validation/models/qwen3_next/qwen3_next_tp_moe_check.py`. The summary, tested
commits and commands are in
[`reports/experiments/qwen3_next/qwen3-next-moe-route-b200.md`](https://github.com/fusheng-ji/RL-Kernel/blob/feat/ws1-qwen3-next-moe-route-combine/reports/experiments/qwen3_next/qwen3-next-moe-route-b200.md);
the raw reports and the figure are a
[ZIP release attachment](https://github.com/fusheng-ji/RL-Kernel/releases/download/qwen3-next-rfc428-evidence/qwen3-next-moe-route-b200.zip).

![Qwen3-Next routed MoE prior art](https://github.com/fusheng-ji/RL-Kernel/releases/download/qwen3-next-rfc428-evidence/qwen3-next-moe-route-b200.png)

TP1 shape (H=2048, 512 experts, top-10, width 512), random weights, BF16:

| | routes / output BI (16-1024 tokens) | dx / dW BI | rel. L2 vs FP64 | tokens routed unlike FP64 (of 256) | forward, 1 / 64 / 1024 / 4096 tokens (ms) | fwd + bwd, 64 / 1024 (ms) |
| --- | --- | --- | --- | --- | --- | --- |
| **RL-Kernel `shared_moe`** | **yes / yes** | **yes / yes** | **3.9e-3** | **0** | 1.39 / 1.69 / 2.04 / 3.05 | 114 / 169 |
| Megatron-core + TE (VIME config) | no / no | no / no (1024) | 4.6e-3 | 0 | 3.8 / 9.7 / 11.7 / 12.1 | 45 / 51 |
| vLLM BI=1 | yes / yes | no backward | 7.0e-2 | 11 | 0.39 / 0.66 / 0.87 / 1.05 | - |
| SGLang deterministic | yes / yes | no backward | 7.0e-2 | 11 | 0.35 / 0.66 / 0.84 / 1.37 | - |
| vLLM BI=0 | no / no | no backward | 7.0e-2 | 11 | 0.38 / 0.66 / 0.87 / 1.06 | - |
| SGLang default | no / no | no backward | 7.0e-2 | 11 | 0.34 / 0.61 / 0.83 / 1.35 | - |
| FlashInfer CUTLASS | no / no | no backward | 7.0e-2 | 11 | 0.37 / 0.72 / 0.90 / 1.12 | - |
| HF eager | yes / yes | no / no (1024) | 7.0e-2 | 11 | 2.1 / 62 / 89 / 87 | 877 / 1284 |

* The accuracy gap is the router: every candidate with BF16 router logits sends
11 of 256 tokens to a different expert set than FP64 routing does. RL-Kernel
and Megatron-core (VIME's training path) route in FP32 and select the same
experts as FP64.
* The only batch-invariant candidates are RL-Kernel and the inference-only
modes of vLLM and SGLang; of these, only RL-Kernel has a backward, so only it
can run the same forward on the training and the rollout side.
* RL-Kernel's forward is 2-4x slower than vLLM's. About 0.5 ms of it, at any
token count, is the FP32 IEEE router GEMM, which is part of the contract.
The expert GEMMs themselves run in vLLM's kernel. The per-expert GEMM loop
they replace (same bits) took 47 / 66 / 71 ms at 64 / 1024 / 4096 tokens in
a local B200 run of the same runner; that run is not part of this evidence.
* The backward is still a per-expert loop of pinned GEMMs (512 experts x 4
GEMMs): 169 ms at 1024 tokens, against 51 ms for Megatron-core + TE, which is
not batch-invariant. It dominates a training step's MoE time.

TP4 gate on the real layer-0 weights (`tp4-moe/rank-*.json`): all twelve cases
pass on all four ranks, including the round trip of all 512 experts, identical
routes on every rank, full == chunked and reordered batches at 8-1024 tokens,
and identical replicated router gradients. The TP4 block forward (routed +
shared expert + all-reduce) takes 1.9-2.4 ms at 8-1024 tokens.

## Tests

| File | Device | Covers |
| --- | --- | --- |
| `tests/models/qwen3_next/test_qwen3_next_forward_contract.py` | CPU | CPU rejection, vLLM pin, fixed-order sum/softmax row independence |
| `tests/models/qwen3_next/test_qwen3_next_tp_blocks.py` | CPU | HF shard/assemble round trips, replica drift rejection, TP4 parameter ownership |
| `tests/validation/common/test_tensor_identity.py` | CPU | raw-bit identity (signed zero, NaN/Inf, dtype) |
| `tests/models/qwen3_next/check_qwen3_next_forward.py` | CUDA + vLLM | GEMM batch/chunk/reorder, route ties, combine order, MoE VJP vs FP64, route and output batch invariance, grouped == per-expert bitwise, fail-closed TD path |
| `tools/validation/models/qwen3_next/qwen3_next_tp_moe_check.py` | 4 x CUDA | real layer-0 weights: HF round trip, replicated routes, chunk/reorder, training forward, gradients |

The `check_` file imports vLLM, so it runs in the `Qwen3-Next-provider-GPU`
workflow rather than the default collection.

## Scope

TP4 with EP1 (all 512 experts on every rank) on CUDA B200. ROCm, EP > 1 and
other TP sizes are separate claims. Model-level parity uses this block but is
the `full_model_chain` row.
57 changes: 57 additions & 0 deletions reports/experiments/qwen3_next/qwen3-next-moe-route-b200.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Qwen3-Next routed MoE: prior art and TP4 gate (B200)

[Download raw results (ZIP)](https://github.com/fusheng-ji/RL-Kernel/releases/download/qwen3-next-rfc428-evidence/qwen3-next-moe-route-b200.zip).

ZIP SHA-256: `9fcc704291bf697274edfe16d6958ca799134008acd9d5f452767b9a92758cb0`.

The archive holds `report.json`, `figure.png`, the TP4 gate's
`tp4-moe/rank-{0..3}.json` and the original README, byte for byte as measured,
plus `SHA256SUMS`. The prior art was measured at commit `dc79953` from a clean
clone on one B200 (torch 2.13.0+cu130, Triton 3.7.1); the TP4 gate at
`cbdc00a`, whose MoE code is identical, on four B200. The prior art uses
random weights (scale 0.02) at the TP1 shape (H=2048, 512 experts, top-10,
width 512); the TP4 gate reads layer 0 of the official
`Qwen/Qwen3-Next-80B-A3B-Instruct` revision
`9c7f2fbe84465e40164a94cc16cd30b6999b0cc7`.

```bash
# prior art (benchmark PR, benchmarks/models/qwen3_next/), training environment
TRITON_F32_DEFAULT=ieee python benchmarks/models/qwen3_next/qwen3_next_moe_prior_art.py \
--only rl_kernel_cuda,hf_transformers,vllm_bi0,vllm_bi1,flashinfer_cutlass,megatron_te --out main.json
# SGLang environment
TRITON_F32_DEFAULT=ieee python benchmarks/models/qwen3_next/qwen3_next_moe_prior_art.py \
--only sglang_triton,sglang_deterministic --merge main.json --out report.json
python benchmarks/models/qwen3_next/plot_qwen3_next_moe_prior_art.py report.json
# TP4 gate on real layer-0 weights (this PR)
TRITON_F32_DEFAULT=ieee torchrun --nproc-per-node 4 tools/validation/models/qwen3_next/qwen3_next_tp_moe_check.py \
--checkpoint <checkpoint> --output tp4-moe
```

At `dc79953` / `cbdc00a` the runners lived in `tools/validation/models/` and
`rl_engine/validation/models/`; the commands above use the current layout.

| Candidate | Forward batch invariant | Backward batch invariant | Different expert set (of 256 tokens) | Max abs vs FP64 | Forward, 1024 tokens |
| --- | --- | --- | --- | --- | --- |
| **RL-Kernel `shared_moe`** | **yes** | **yes** | **0** | 2.1e-3 | 2.04 ms |
| HF transformers `Qwen3NextExperts` (eager) | yes | no (1024) | 11 | 1.1e-1 | 89.3 ms |
| vLLM `fused_experts`, `VLLM_BATCH_INVARIANT=0` | no (256, 1024) | n/a (inference) | 11 | 1.1e-1 | 0.87 ms |
| vLLM `fused_experts`, `VLLM_BATCH_INVARIANT=1` | yes | n/a (inference) | 11 | 1.1e-1 | 0.87 ms |
| FlashInfer `cutlass_fused_moe` | no (256, 1024) | n/a (inference) | 11 | 1.1e-1 | 0.90 ms |
| Megatron-core `MoELayer` + TE grouped GEMM | no (1024) | no (1024) | 0 | 2.2e-3 | 11.7 ms |
| SGLang `fused_moe` (Triton), default | no (256, 1024) | n/a (inference) | 11 | 1.1e-1 | 0.83 ms |
| SGLang `fused_moe`, deterministic inference | yes | n/a (inference) | 11 | 1.1e-1 | 0.84 ms |

- Batch invariance: 8 probe tokens computed alone and placed first and last in
batches of 16, 64, 256 and 1024 tokens; the token counts in parentheses are
the batches that differ.
- Accuracy: 256 tokens against the HF formula in FP64 with FP64 routing. The
candidates with a BF16 router pick a different expert set for 11 of 256
tokens, which dominates their error.
- SGLang's compiled `sgl_kernel` 0.3.21 cannot load next to torch 2.13, so its
two kernels on this path are SGLang's own replacements (Triton
`moe_sum_reduce`, JIT `moe_align_block_size`); `environments` in
`report.json` records both Python environments.
- TP4 gate (`tp4-moe/rank-*.json`): HF round trip, replicated routes, chunk and
reorder invariance, training forward and gradients pass on every rank.

Site paths, host names and the scheduler do not appear in any file.
1 change: 1 addition & 0 deletions rl_engine/models/qwen3_next/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# SPDX-License-Identifier: Apache-2.0
Loading
Loading