[fusheng-ji]

Stack: #466 (merged) → #467 → #468 → #469 → #503 → #507 → #543 → #544 → #549 → this PR → benchmarks (#551), base test-qwennext; merges after #549. This PR's own diff. RFC #428 row distributed_logprob_parity, CUDA, EP part; the TP part is #549.

What

• EP4 MoE layer (rl_engine/models/qwen3_next/qwen3_next_ep.py): expert parallel 4 over the TP4 group, expert tensor parallel 1. Rank r owns experts [128r, 128r + 128) at full width; attention, GDN, embedding, LM head and the shared expert stay TP4.
  • Dispatch is a local selection, since the MoE input is already replicated and every rank computes the same routes.
  • Each rank evaluates its own routes with the fixed-config grouped GEMM of [WS1][CUDA][Qwen3-Next] MoE route/combine contract with a batch-invariant grouped expert GEMM (RFC #428 stack 5/11) #503. The rows are exchanged with the fixed-order collective's all_gather, which is a byte copy with no arithmetic.
  • Every rank then applies combine_routes (FP32, route order) locally.
  • Backward: weight gradients only on the owning rank; per-route input gradients exchanged and summed in ascending expert order, as EP1 does.
  • A route slot owned by no rank, or by two, fails closed.
• vLLM rollout: enable_expert_parallel with linear placement; SharedVLLMMoE uses the EP path and checks that vLLM's expert_map is exactly [128r, 128r + 128).
• VIME replay: Megatron expert_model_parallel_size 4, expert_tensor_parallel_size 1. SharedVIMEEPMoE asserts that the expert-parallel rank equals the TP rank, and the HF loader returns only local experts.
• Launcher and gates: qwen3_next_run.py --ep {1,4} records the topology in run.json, and every later stage checks it. The layers stage runs the new EP4 MoE gate (moe_ep) in place of the TP4 MoE gate.
• Fail closed: both engines, the scheduler gate and the L2 report accept only TP4 / PP1 / CP1 with EP 1 (expert tensor parallel 4) or EP 4 (expert tensor parallel 1). The training actor and HF export reject EP4.
• Contract (docs/operators/qwen3-next-distributed-logprob.md): new EP section, one row per cross-rank point.
• Chain report: names the measured EP size in its disclaimer and backend profile.
• Real tests, not stand-ins: the vLLM model contract builds a real VllmConfig from vLLM's own config classes (tests/models/qwen3_next/vllm_real.py), and check_qwen3_next_vllm_real_model.py builds RL-Kernel's wrapper around vLLM's actual Qwen3NextForCausalLM on four real ranks at EP1 and EP4 (weights allocated, not loaded): drift rejected before any model memory is allocated; 48 layers hold the shared classes and shapes; the routed-expert parameters keep vLLM's native weight loaders; at EP4 vLLM's own expert_map equals [128r, 128r + 128) and every expert is held once. Both run in preflight's device guards, which previously ran only the multi-GPU guard.
• Layout as reviewed on [WS1][CUDA][Qwen3-Next] TP distributed logprob parity: fixed reduction order, selected-token logprob exact across NCCL settings (RFC #428 stack 9/11) #549 from the start: runners under tools/validation/models/qwen3_next/, the comparison benchmark in [WS1][CUDA][Qwen3-Next] Comparison benchmarks and plots for the Qwen3-Next stack: MoE, attention, logprob, EP (RFC #428 stack 11/11) #551, raw evidence as a release ZIP.

Prior art & reuse decision

Measured on the same layer-0 weights and replicated input, EP4 under 13 NCCL settings and EP1, 4 × B200, 1024 tokens (summary, runner in #551, measured at 45719cb3):

Library	Implementation	Fwd EP4 == EP1	Fwd NCCL-setting invariant	Fwd batch invariant	Backward: batch invariant / EP4 == EP1	Max abs vs FP64	EP4 fwd / bwd latency
RL-Kernel	qwen3_next_ep (this PR, both engines)	yes	yes (1 result)	yes	yes / yes	1.8e-5	2.27 / 51 ms
vLLM 0.30.0	fused_experts + expert_map + vLLM's all-reduce (default or VLLM_BATCH_INVARIANT=1)	no	yes (1)	yes	no backward	8.7e-4	0.48 / – ms
vLLM 0.30.0	same expert kernel + NCCL all_reduce	no	no (5)	no	no backward	8.7e-4	0.52 / – ms
FlashInfer 0.6.18.post1 / CUTLASS	cutlass_fused_moe with ep_size/ep_rank + vLLM's all-reduce	no	yes (1)	yes	no backward	8.7e-4	0.39 / – ms
SGLang 0.5.21	FusedMoE, EP ranks combined by tensor_model_parallel_all_reduce (fused_moe_triton/layer.py:1597)				no backward		not measured: sgl_kernel does not load next to torch 2.13, and the only machine available for the follow-up check has 2 B200 (EP4 needs 4); same structure as vLLM
Megatron-core 0.16 (Vime's training path)	MoELayer, alltoall / allgather dispatcher	no	no (13)	no	no / no	2.5e-5 - 4.2e-5	13-15 / 37-41 ms
Megatron-core 0.16	same under torch.use_deterministic_algorithms(True), alltoall	yes	yes (1)	yes	no / no	4.2e-5	15.4 / 42 ms
Megatron-core 0.16	same, allgather	no	yes (1)	yes	no / no	2.5e-5	14.3 / 40 ms
Miles (radixark/miles@23d3d38)	miles/kernels/moe/fused_experts{,_bwd}.py (SGLang forward, own backward), imported unmodified; layer-0 weights at EP1, 20 launches				no, measured: dx, dW gate/up, dW down equal the first in 1 / 20 launches (BF16 tl.atomic_add); dx probe rows 0 / 10 batch cells		10 / – ms (EP1)
verl (volcengine/verl@e5a627c)	no MoE kernel (searched verl/); Megatron-core, HF/FSDP or VeOmni for training, vLLM/SGLang for rollout						covered by those rows
FlashAttention	—						N/A: no MoE

Batch invariance: 8 probe tokens alone vs inside batches of 8 / 64 / 256 / 1024 at the first, middle and last position, on the output and (backward) the input gradient.

Decision: write our own EP4 path, reusing the #503 MoE contract and RL-Kernel's fixed-order collective.

• vLLM and FlashInfer are batch- and NCCL-setting-invariant when they use vLLM's own all-reduce, but they are inference-only: training needs the same bits with a backward, and neither has one. They also route in BF16, 8.7e-4 away from the FP32-routed candidates, and EP4 differs from their own EP1.
• Megatron-core, VIME's training path, is the only measured stack with a backward. Under deterministic algorithms its alltoall forward passes every check, but its backward input gradient changes with the batch and differs from EP1. That is a non-performance reason not to reuse it. It would also only cover the training side; vLLM workers cannot run Megatron's dispatcher.
• Miles' backward accumulates in BF16 with tl.atomic_add, and over 20 identical launches on the real weights dx, dW gate/up and dW down equal the first in only 1 of 20 (measured, EP1 as Miles has no EP kernel path); at EP1 it is 15× faster than our EP1 path (10 vs 158 ms forward + backward, 1024 tokens); RL-Kernel's EP4 forward + backward is about 53 ms (2.3 + 51).
• Ours passes every forward and backward check at EP4 and equals its own EP1 bit for bit. Our EP4 forward is 4.7× slower than vLLM's at 1024 tokens (2.27 vs 0.48 ms); the backward is 1.2× Megatron's (51 vs 42 ms).

Results (4 × B200, official checkpoint, TP4 × EP4)

EP4 evidence summary and raw results (ZIP) (SHA-256 c7530d29…ba53). Measured at 1c4ae265 from clean clones in the same environment as the TP4 × EP1 chain.

Stage	Checks	Result
preflight	operator gates, device guards, TP4 NCCL probe, checkpoint identity, rollout smoke	passed
layers	GDN, attention, adapters, head TP4 gates; EP4 MoE gate on layer 0; native scheduler on 16 cases	passed
l2	vLLM rollout (EP4) vs VIME recompute (EP4): tokens, BF16 logits, FP32 logprobs, 48 layer, sub-node and state digests	16 / 16 raw-bit

Chain-gate report: config_fingerprint.ep 4, 16 cells, passed: true, first_drift: null.

EP4 MoE gate, every rank, bitwise:

Case	Checks
hf-load-export-each-expert-once	each of the 512 experts loaded and exported by exactly one rank, equal to the checkpoint
ep4-vs-ep1-chunk-reorder-train-{8,64,256,1024}	identical routes on every rank; routed rows, input gradient and owned experts' weight gradients equal all 512 experts on one GPU (EP1); full == chunked; reordered tokens; training forward == no-grad
empty-rank-participation	a batch in which one rank owns no route equals EP1

EP4 is not TP4 × EP1.

• Layer 0: the EP4 MoE block differs from the TP4 block by up to 4.9e-4, from one K = 512 GEMM per expert instead of four K = 128 partials.
• End to end: against [WS1][CUDA][Qwen3-Next] Full-model chain: vLLM rollout vs VIME recompute, raw-bit through two updates and a reload (RFC #428 stack 7/11) #543's TP4 × EP1 L2, all 16 cases generate the same tokens, but selected logprobs differ by up to 0.32 and logits by up to 7.2.
• Where it starts: the embedding and layer 0's recurrent state are equal, so the first difference is after the layer-0 mixer.
• Consequence: EP4 is its own certified profile (qwen3-next-moe-ep4-v1), and parity means rollout and replay both run it.

Status

• Head 88c4f02e, on top of [WS1][CUDA][Qwen3-Next] TP distributed logprob parity: fixed reduction order, selected-token logprob exact across NCCL settings (RFC #428 stack 9/11) #549 (3f7cf6ec). CI is green on this head: lint, unit-tests, docs, DCO and the bot review pass; provider and ws1-gtest skip because no self-hosted runner is attached.
• Ready for review for EP4 up to the full-model L2.
• Not covered: training on EP4 (Megatron expert-parallel optimizer state, checkpoint export and reload, the train* stages). The training actor and export fail closed on EP4; follow-up PR.
• Hopper (@Flink-ddd's change request of 2026-10-19): the four-GPU preflight and gpu_probe accept exactly four B200, H100 or H200 GPUs (a705d287; H200 accepted, not tested). On 4 × H100, at a705d287 (not rerun at the head 88c4f02e), the Qwen3-Next test_*.py and check_*.py suites (814 passed) and the six gates passed. The full-model EP4 chain has not been run on H100 with this PR's code yet. See "Results (4 × H100; no full-model run)" at the end of this description.
• Rerun at the aligned stack top (clean clone of 90c09164, 4 × B200): preflight, layers (with moe_ep) and l2 passed, 16/16 raw-bit, and the 32 L2 artifacts are bitwise identical to the 1c4ae265 run; the standalone EP4 MoE gate passed on all four ranks. The full batch-invariance suite then ran at EP4: 97 / 97 cells bitwise (same cell definitions as at EP1), raw results ZIP, SHA-256 b3012ab6…fe4d. EP4 training stays fail-closed, so no EP4 training stage is claimed. At 17ba4a4c the EP4 preflight passed again with the new device guards (27 tests, 0 skipped, 0 failed).
• The EP4 gate checks EP4 against EP1 bit for bit and its exchange uses no NCCL arithmetic; the prior-art run above additionally measured it under the 13 NCCL settings of [WS1][CUDA][Qwen3-Next] TP distributed logprob parity: fixed reduction order, selected-token logprob exact across NCCL settings (RFC #428 stack 9/11) #549 (one result).

History

• 2026-10-27 (description): the H100 results now name commits on public branches: the tests and gates at a705d287 and the comparison at 58aa50c7. The raw files record 2a8f18e4 and 77d59a59, the ids these commits had before the branches were rewritten; each pair has the same tree. An earlier version also reported a full-model EP4 chain run on H100 that was made with local changes that are not part of this PR; that result is withdrawn, its raw files are removed from the pr_assets branch (927b93d6), and the chain is listed as not run on H100. The FlashInfer CUTLASS row of the H100 comparison, which showed [WS1][CUDA][Qwen3-Next] MoE route/combine contract with a batch-invariant grouped expert GEMM (RFC #428 stack 5/11) #503's single-GPU (non-EP) numbers instead of an EP4 measurement, is now marked as not measured. The RL-Kernel backward batch-invariance cell has footnote ¹: the stock combine_routes backward was not batch invariant from batch 16 upward in [WS1][CUDA][Qwen3-Next] MoE route/combine contract with a batch-invariant grouped expert GEMM (RFC #428 stack 5/11) #503's H100 diagnostics.
• 2026-10-27 (provider path filter): the provider workflow's pull_request paths were missing Qwen3-Next modules that its checks import; merged the additions of [WS1][CUDA][Qwen3-Next] Full-model chain: vLLM rollout vs VIME recompute, raw-bit through two updates and a reload (RFC #428 stack 7/11) #543 and [WS1][CUDA][Qwen3-Next] Batch-invariance matrix for the full model, 97/97 cells bitwise (RFC #428 stack 8/11) #544; this PR's own logprob-kernel and rollout-bridge entries are kept once (88c4f02e). Workflow-only change; shared infrastructure such as rl_engine/ops and rl_engine/platforms stays out of the filter.
• 2026-10-26 (review): two OpenCodeReview rounds on the stack (the bot re-reviews every PR's whole diff after each push); every finding of these two rounds is answered in its own thread or comment, most as repeats of earlier answers or as not applicable with the code reference. Fixes in this PR: 6dc9d410 triggers the provider job on csrc/cuda/logprob/** and the vLLM rollout bridge. The fixes of the PRs below it are merged in; new head 65eaae4d. Kernel arithmetic is unchanged, so the measured results stand.
• 2026-10-26: merged test-qwennext after [WS1][CUDA][Qwen3-Next] C1: zero-centred RMSNorm references and CUDA kernel (RFC #428 stack 2/4) #467 landed (c382dd4b), through [WS1][CUDA][Qwen3-Next] TP distributed logprob parity: fixed reduction order, selected-token logprob exact across NCCL settings (RFC #428 stack 9/11) #549's new head fe6ed386, in 85297004. One conflict, in the provider workflow: the pinned-provider step takes [WS1][CUDA][Qwen3-Next] C6: GDN decode-step goldens and recurrent/conv state contract (RFC #428 stack 4/4) #469's fail-closed skip check, and this PR's four-GPU vLLM model step follows it unchanged.

Tests

python -m pytest -q tests/models/qwen3_next/test_qwen3_next_ep.py tests/models/qwen3_next/test_qwen3_next_chain_report.py
TRITON_F32_DEFAULT=ieee torchrun --nproc-per-node 4 tools/validation/models/qwen3_next/qwen3_next_ep_moe_check.py \
    --checkpoint <ckpt> --output <dir>
python tools/validation/models/qwen3_next/qwen3_next_run.py --stage all --until l2 --ep 4 \
    --runs-root <runs> --checkpoint <ckpt> --cuda-home <cuda-13.0> --python <venv>/bin/python

Check	Result
Qwen3-Next CPU list in ci.yml (includes the new test_qwen3_next_ep.py) + digest tests	463 passed (99 CUDA-only skips)
evidence redaction test	80 passed
vLLM model contract + real four-rank model check, EP1 and EP4, 4 × B200	21 passed
preflight device guards at EP4 (multi-GPU + contract + real model)	27 tests, 0 skipped, 0 failed
pre-commit on all files, mkdocs build --strict	clean

Scope

CUDA, TP4 × EP4 × ETP1 × CP1, one node of four B200, rollout and replay up to L2. EP training, other EP sizes, ROCm and multi-node are out of scope.

Results (4 × H100; no full-model run)

4 × NVIDIA H100 (driver 580.173.02 in the gates run), torch 2.13.0+cu130, vLLM 0.30.0, NCCL 2.29.7. The comparison run does not record the Megatron-core version.

Tests and gates, at a705d287 on this PR's branch (not rerun at the head 88c4f02e). gates/info.txt and the EP4 MoE, TP4 attention, TP4 GDN and TP4 MoE gate files record 2a8f18e4, the id of this commit before the branch was rewritten. Both ids have the tree 4102963e2673c15b8a5e807f7ea4223173630370; git rev-parse a705d287^{tree} gives it. The GDN engine adapter and TP4 embedding / LM head gate files record no commit. gates/pytest.log is a quiet-mode log (progress marks and the final line) and does not record the command line.

Check	Result	Raw data
tests/models/qwen3_next/test_*.py and check_*.py	814 passed, 0 failed	gates/pytest.log, gates/info.txt
GDN engine adapter, EP4 MoE, TP4 attention, TP4 GDN, TP4 embedding / LM head and TP4 MoE gates	all six passed on all four ranks	adapter, ep_moe, tp_attention, tp_gdn, tp_head, tp_moe (each gate also has rank-1.json to rank-3.json and a .log in the same folder)

The EP4 MoE gate (script at a705d287; its rank files record the pass status and output hashes, not the individual comparisons) checks bit for bit, at 8, 64, 256 and 1024 tokens, that the EP4 routed-expert rows (before combine_routes), their input gradient dx and each rank's own experts' dW equal the same experts evaluated on one GPU (EP1) on the same routes. For the EP4 block alone it checks that the full batch equals the concatenated chunks, that reordered tokens give the reordered output, that the first row alone matches, and that the training forward equals the no-grad forward. It does not compare the combined block output with EP1, and it does not differentiate through combine_routes or the router.

Not run on H100 yet: the full-model EP4 chain (preflight, layers, l2: vLLM rollout against VIME recompute) with this PR's code.

Comparison with existing implementations (same runner and columns as "Prior art & reuse decision" above). Layer-0 MoE with the checkpoint's layer-0 weights on one fixed random 1024-token input: EP4 on four GPUs under each of the 13 NCCL settings of #549, and EP1 on one GPU. Measured at 58aa50c7, which is on the #551 branch (that branch contains this PR's head 88c4f02e), with #551's EP benchmark runner (moe_ep_candidates.py, qwen3_next_moe_ep_prior_art.py). The report records 77d59a59, the id of this commit before the branch was rewritten. Both ids have the tree 67a6c5c8a30c1351b05beb9b9f509976f8dff45c; git rev-parse 58aa50c7^{tree} gives it. The EP4 == EP1 and backward dx / dW equalities of RL-Kernel appear as flags in ep_matrix/report.json under candidates.rl_kernel_ep (ep4_equals_ep1, backward.dx_ep4_equals_ep1, backward.dw_ep4_equals_ep1).

In the "Fwd NCCL-setting invariant" column, the number in parentheses is the count of distinct EP4 outputs across the 13 NCCL settings.

Implementation	Fwd EP4 == EP1	Fwd NCCL-setting invariant	Fwd batch invariant	Backward: batch invariant / EP4 == EP1	Max abs vs FP64	EP4 fwd / fwd + bwd latency
RL-Kernel qwen3_next_ep	yes	yes (1)	yes	yes¹ / yes	1.8e-5	2.77 / 94 ms
vLLM fused_experts + vLLM all-reduce	no	yes (1)	no	no backward	8.7e-4	0.88 / – ms
vLLM fused_experts + vLLM all-reduce, VLLM_BATCH_INVARIANT=1 (identical outputs)	no	yes (1)	no	no backward	8.7e-4	0.99 / – ms
vLLM fused_experts + NCCL all_reduce	no	no (8)	no	no backward	8.7e-4	0.90 / – ms
FlashInfer CUTLASS	not measured²	–	–	–	–	–
Megatron-core MoELayer, alltoall	no	no (13)	no	no / no	3.3e-5	32 / 87 ms
Megatron-core MoELayer, allgather	no	no (13)	no	no / no	2.5e-5	30 / 81 ms
Megatron-core MoELayer, alltoall, torch.use_deterministic_algorithms(True)	yes	yes (1)	yes	no / no	4.2e-5	33.8 / 91 ms
Megatron-core MoELayer, allgather, torch.use_deterministic_algorithms(True)	no	yes (1)	yes	no / no	2.5e-5	31.9 / 86 ms

¹ Measured on one fixed random input with the stock combine_routes, which is unchanged from 58aa50c7 to the head. Its autograd backward reduces the route-weight gradient over the hidden dimension with torch.sum. In #503's H100 diagnostics (results.txt), on synthetic inputs, that reduction was not batch invariant from batch 16 upward and made dx differ. This cell therefore holds for the measured input only.
² The JIT build of FlashInfer 0.6.18.post1's fused_moe_90 module failed at link time on the H100 machine (/usr/bin/ld: cannot find -lnvrtc, Ninja build failed; see candidates.flashinfer_cutlass_ep.unavailable in ep_matrix/report.json). #503's H100 run linked it through a libnvrtc.so symlink, which this run did not have. The candidate is therefore recorded as unavailable, and this table has no FlashInfer numbers.

Batch invariance: the probe of 8 tokens is compared alone and inside batches of 8, 64, 256 and 1024 tokens, at the first, middle and last position (cell 1024@508 is the 1024-token batch with the probe at position 508), on the output and, for the backward, on the input gradient. With vLLM's all-reduce (default and VLLM_BATCH_INVARIANT=1), vLLM differs only in the 1024-token batch (cells 1024@0, 1024@508 and 1024@1016); with NCCL all_reduce it differs in every cell except 8@0.

Latency: the benchmark runner takes one warm-up call, then the median of 20 timed forward calls at 1024 tokens and, for the second number, the median of 5 timed forward + backward calls. The EP4 value in the table is the largest of the four ranks' values (latency_us_ep4 in the report), in the run with default NCCL settings. The published run files hold rank 0 only. Report: ep_matrix/report.json; run files: ep_matrix/runs/.

[fusheng-ji]

OpenCodeReview · .github/workflows/qwen3-next-provider-gpu.yml (L130-L130): The stale-extension guard only asserts the loaded _C lives under GITHUB_WORKSPACE.

Same point as 543-S5; not changed. rl_engine is a regular package, so a _C loaded from the checkout means every rl_engine submodule was loaded from it, and actions/checkout fetches exactly the given SHA.

[fusheng-ji]

OpenCodeReview · rl_engine/backends/cuda/norm/rmsnorm.py (L330-L330): Arity mismatch: forward(ctx, x, weight, gate, eps=1e-6, weight_offset=0.0, activation=0) has 5 differentiable inputs (x, weight, gate, eps, weight_offset, activation → 5 non-ctx args that map to grads = x, weight, gate, eps, weight_offset, activation is 6).

Not changed: forward(ctx, x, weight, gate, eps, weight_offset, activation) has six inputs and backward returns six values, dx, dw, dgate, None, None, None.

[fusheng-ji]

Replies to the three OpenCodeReview summary items:

1. run_qwen3_next_chain_gate.sh L26: fixed in 4e2c33af (merge f935bf3a); see the [WS1][CUDA][Qwen3-Next] Full-model chain: vLLM rollout vs VIME recompute, raw-bit through two updates and a reload (RFC #428 stack 7/11) #543 reply.
2. qwen3_next_tp.py L18-L19, collective_for_group can return None: fixed in 4e2c33af. The function returns None only for group is None; _reduce now raises RuntimeError: Qwen3-Next TP reduction needs a process group, as the EP exchange does (qwen3_next_ep.py:99-100), instead of an AttributeError.
3. qwen3_next_tp_blocks.py L107, zero-initialised RMSNorm gain: not changed. These norms are zero-centred: the module applies Qwen3NextRMSNormCudaOp, y = x * rstd * (1 + weight) (rmsnorm.py:178-186, weight_offset = 1.0; used at qwen3_next_tp_blocks.py:240), and the Qwen3-Next checkpoint stores q_norm.weight and k_norm.weight in that form. Zero is the identity here. The GDN block's norm is the gated RMSNorm, which multiplies by the plain weight, so it starts at ones.

[Flink-ddd]

The layout follows #497, but Hopper validation is still incomplete. Please fix the known batch-dependent combine_routes backward reduction, cover the router and combine path in the EP4 gradient tests, and run full EP4 rollout and replay parity on four H100 and H200 GPUs using the submitted code before merging.