[fusheng-ji]

Stack: … → #549 → #557 → this PR, base test-qwennext; merges after #557; kept draft until the validation PRs land. This PR's own diff.

What

Per the layout review on #549, every comparison benchmark and plot of the Qwen3-Next stack lives here, under benchmarks/models/qwen3_next/; the validation PRs keep only checks and CI tests. Code is moved from the validation PRs with imports and paths adjusted, except the new EP benchmark.

Validation PR	Candidates module	Runner	Plot
#503 routed MoE	moe_route_candidates.py	qwen3_next_moe_prior_art.py	plot_qwen3_next_moe_prior_art.py
#507 full attention	attention_candidates.py	qwen3_next_attention_prior_art.py	plot_qwen3_next_attention_prior_art.py
#543 chain	—	—	plot_qwen3_next_chain.py
#544 batch invariance	—	—	plot_qwen3_next_batch_invariance.py
#549 TP logprob	logprob_candidates.py	qwen3_next_logprob_prior_art.py	plot_qwen3_next_distributed_logprob.py
#557	moe_ep_candidates.py	qwen3_next_moe_ep_prior_art.py	plot_qwen3_next_moe_ep.py

Miles and verl are measured, not read: thirdparty_sources.py imports their own kernels unmodified from checkouts of pinned commits (radixark/miles@23d3d38, volcengine/verl@e5a627c) and records the SHA-256 of every file used. New here: qwen3_next_miles_moe_backward.py (Miles' routed-MoE backward, 20 identical launches on the real layer-0 weights, RL-Kernel as the control), qwen3_next_logprob_backward.py (verl, Miles and RL-Kernel logprob backward), miles_fused and verl_fused in the logprob candidates, and plot_qwen3_next_thirdparty.py.

Results summaries (download link, SHA-256, tested commit, commands) are in reports/experiments/qwen3_next/: qwen3-next-logprob-comparison-b200.md, qwen3-next-moe-ep-comparison-b200.md and qwen3-next-thirdparty-comparison-b200.md here; the MoE-route, attention, chain and BI summaries in their validation PRs.

Prior art & reuse decision

N/A: benchmarks only. The reuse decisions are in each validation PR; this PR provides the code that measures them.

Results

Expert-parallel MoE (new, for the EP PR), layer 0 real weights, EP4 under 13 NCCL settings and EP1, 4 × B200, measured at 7db4a14b (summary, raw results ZIP, SHA-256 2375ec36…97c6cf):

Candidate	Fwd EP4 == EP1	Fwd distinct results / 13 NCCL settings	Fwd repeat / batch inv.	Bwd EP4 == EP1 (dx, dW)	Bwd batch inv.	Max abs vs FP64	EP4 fwd 64 / 1024 / 4096 tokens	EP4 bwd
RL-Kernel EP4	yes	1	yes / yes	yes, yes	yes	1.8e-5	1.85 / 2.27 / 4.08 ms	51 ms
vLLM + vLLM all-reduce	no	1	yes / yes	no backward	–	8.7e-4	0.35 / 0.48 / 0.68 ms	–
vLLM, VLLM_BATCH_INVARIANT=1	no	1	yes / yes	no backward	–	8.7e-4	0.41 / 0.51 / 0.81 ms	–
vLLM + NCCL all_reduce	no	5	yes / no	no backward	–	8.7e-4	0.39 / 0.52 / 0.74 ms	–
FlashInfer cutlass_fused_moe (CUTLASS)	no	1	yes / yes	no backward	–	8.7e-4	0.33 / 0.39 / 0.48 ms	–
Megatron alltoall	no	13	no / no	no, no	no	4.2e-5	11.8 / 14.6 / 14.9 ms	41 ms
Megatron allgather	no	13	no / no	no, no	no	2.5e-5	10.8 / 13.2 / 13.6 ms	37 ms
Megatron alltoall, deterministic	yes	1	yes / yes	no, no	no	4.2e-5	12.7 / 15.4 / 15.9 ms	42 ms
Megatron allgather, deterministic	no	1	yes / yes	no, no	no	2.5e-5	11.6 / 14.3 / 15.0 ms	40 ms

Miles and verl (for #503, #549, EP PR), measured at 6ad63611 (MoE backward, logprob forward) and 03a5898e (logprob backward, TP1 and TP4) (summary, raw results ZIP, SHA-256 d0eaa326…f2ff):

Measurement	Miles	verl	RL-Kernel
routed-MoE backward, 20 identical launches: dx / dW gate/up / dW down equal the first	1 / 20 each (BF16 tl.atomic_add)	no MoE kernel	20 / 20
routed-MoE dx of probe rows alone == in batch (10 cells)	0 / 10	–	10 / 10
logprob forward, TP4 == TP1	no	no	yes
logprob forward, distinct TP4 results over 13 NCCL settings	2	2	1
logprob backward, 10 launches equal (TP1)	10 / 10	10 / 10	10 / 10
logprob backward row gradient, probe rows alone == in batch (10 cells; logits of Miles and RL-Kernel from the batch-invariant GEMM)	10 / 10	1 / 10	10 / 10
logprob backward, TP4 == TP1	no	no	yes
logprob backward, distinct TP4 results over 13 NCCL settings (10 launches each)	2	2	1

Miles' MoE backward is not reproducible; both libraries' logprob forward depends on the NCCL algorithm like Megatron's and VIME's. Their logprob backward also depends on the NCCL algorithm at TP4 (it consumes the all-reduced forward statistics) and differs from TP1, while RL-Kernel's does not. Not what the source suggested: verl's backward tl.atomic_add did not change the bits between launches, and its forward atomic_add only runs for reduction sum/mean.

TP logprob (#549): measured in #549 (summary and ZIP there): TP4 median latency 2.55 ms (RL-Kernel), 1.51 ms (vLLM), 0.57 ms (Megatron), 0.56 ms (VIME); invariance as in #549.

The routed-MoE, attention, chain and BI figures are rendered by the moved plot scripts from the ZIPs attached to their PRs (checked: each renders from the downloaded evidence layout).

Figures of the validation PRs, rendered by the plot scripts in this PR from the evidence ZIPs of those PRs:

#503 routed MoE	#507 attention
#543 full-model chain	#544 batch invariance (97 / 97 cells; reruns at EP1 and EP4 in #544 and #557)

Status

• Head 015bec4a, on top of [WS1][CUDA][Qwen3-Next] EP4 distributed logprob parity: expert-parallel MoE, full-model L2 bitwise on TP4 x EP4 (RFC #428 stack 10/11) #557 (88c4f02e). CI is green on this head: lint, unit-tests, docs and DCO pass; provider and ws1-gtest skip because no self-hosted runner is attached.
• Draft until [WS1][CUDA][Qwen3-Next] MoE route/combine contract with a batch-invariant grouped expert GEMM (RFC #428 stack 5/11) #503-[WS1][CUDA][Qwen3-Next] TP distributed logprob parity: fixed reduction order, selected-token logprob exact across NCCL settings (RFC #428 stack 9/11) #549 and the EP PR merge. The moved files keep their measurement logic; the changes in this PR are the new EP benchmark, the Miles/verl runners and their plots.
• The Miles/verl comparison was first drafted from their source; it is now measured, and two source-based expectations did not hold (above).
• Measurement fixes in the EP benchmark before the published run: vLLM's tile-config cache was first cleared around every call (inflated vLLM's default latency; fixed in 3a5263d2); Megatron was first measured only without deterministic algorithms (added in 6e316f4b); v2 adds FlashInfer, vLLM's own all-reduce dispatch, a two-dimensional batch grid and the backward. The first v2 launch crashed in vLLM's parallel-state setup because the job's TMPDIR path exceeded the IPC socket path limit; the published run uses a short node-local TMPDIR (recorded in the summary).

History

• 2026-10-27 (provider path filter): the provider workflow's pull_request paths were missing Qwen3-Next modules that its checks import; merged the additions of [WS1][CUDA][Qwen3-Next] Full-model chain: vLLM rollout vs VIME recompute, raw-bit through two updates and a reload (RFC #428 stack 7/11) #543 and [WS1][CUDA][Qwen3-Next] Batch-invariance matrix for the full model, 97/97 cells bitwise (RFC #428 stack 8/11) #544 (015bec4a). Workflow-only change; shared infrastructure such as rl_engine/ops and rl_engine/platforms stays out of the filter.
• 2026-10-26 (review): two OpenCodeReview rounds on the stack (the bot re-reviews every PR's whole diff after each push); every finding of these two rounds is answered in its own thread or comment, most as repeats of earlier answers or as not applicable with the code reference. The fixes of the PRs below it are merged in; new head cc7dc4bc. Kernel arithmetic is unchanged, so the measured results stand.
• 2026-10-26: merged test-qwennext after [WS1][CUDA][Qwen3-Next] C1: zero-centred RMSNorm references and CUDA kernel (RFC #428 stack 2/4) #467 landed (c382dd4b), through [WS1][CUDA][Qwen3-Next] EP4 distributed logprob parity: expert-parallel MoE, full-model L2 bitwise on TP4 x EP4 (RFC #428 stack 10/11) #557's new head 85297004, in 5364c406. No conflicts.

Tests

python benchmarks/models/qwen3_next/qwen3_next_moe_ep_prior_art.py --help
torchrun --nproc-per-node 4 benchmarks/models/qwen3_next/qwen3_next_moe_ep_prior_art.py --checkpoint <ckpt> --output runs/ep4-default
torchrun --nproc-per-node 1 benchmarks/models/qwen3_next/qwen3_next_moe_ep_prior_art.py --checkpoint <ckpt> --output runs/ep1
python benchmarks/models/qwen3_next/qwen3_next_moe_ep_prior_art.py --merge runs/ep4-default runs/ep4-<...> runs/ep1 --out report.json

Check	Result
every runner --help and every plot rendered from its published evidence	passed
pre-commit on all files	clean

Scope

CUDA, B200. Benchmarks and plots only.

[github-actions]

🔍 OpenCodeReview found 8 issue(s) in this PR.

• ✅ Successfully posted inline: 0 comment(s)
• 📋 Routed to summary by policy: 8 comment(s)

⚠️ 4 warning(s) occurred during review.

---

📄 benchmarks/models/qwen3_next/moe_ep_candidates.py (L135-L139)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

_vllm_batch_invariant mutates os.environ["VLLM_BATCH_INVARIANT"] but never restores the previous value on exit (it is a @contextmanager with no finally). Trigger: run moe_report with MOE_EP_CANDIDATES selecting vllm_ep_bi (or any run where it is measured last); afterwards the whole process keeps VLLM_BATCH_INVARIANT=1, so any later candidate or harness code that does its own batch-invariance qualification silently runs under vLLM's batch-invariant kernel overrides. That can change the numbers reported for candidates measured after this one. Note moe_route_candidates._env correctly saves/restores; mirror that here.

💡 Suggested Change

Before:

    value = "1" if enabled else "0"
    if os.environ.get("VLLM_BATCH_INVARIANT", "0") != value:
        os.environ["VLLM_BATCH_INVARIANT"] = value
        get_moe_configs.cache_clear()
    yield

After:

    value = "1" if enabled else "0"
    previous = os.environ.get("VLLM_BATCH_INVARIANT")
    if previous != value:
        os.environ["VLLM_BATCH_INVARIANT"] = value
        get_moe_configs.cache_clear()
    try:
        yield
    finally:
        if previous is None:
            os.environ.pop("VLLM_BATCH_INVARIANT", None)
        else:
            os.environ["VLLM_BATCH_INVARIANT"] = previous

---

📄 benchmarks/models/qwen3_next/moe_ep_candidates.py (L155-L157)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

For the default "vllm" backend _combine ignores the group argument and calls tensor_model_parallel_all_reduce(), which reduces over vLLM's model-parallel group rather than the EP group passed to candidate_report/_vllm/_flashinfer. If the EP group membership differs from vLLM's TP group (e.g. EP4 with TP1, or the benchmark's group is a subgroup not registered as vLLM's TP group), the partial per-rank expert outputs are combined over the wrong ranks, corrupting the accuracy/BI numbers for vllm_ep, vllm_ep_bi and flashinfer_cutlass_ep. If vLLM's distributed state is not initialized, the helper returns the input unchanged, so no combine happens at all. Use the explicit group (as the nccl branch does) so all candidates reduce over the same ranks, or assert the groups match.

💡 Suggested Change

Before:

    from vllm.distributed import tensor_model_parallel_all_reduce

    return tensor_model_parallel_all_reduce(out)

After:

    from vllm.distributed import tensor_model_parallel_all_reduce

    # Reduce over the same EP group the other candidates use.
    dist.all_reduce(out, group=group)
    return out

---

📄 benchmarks/models/qwen3_next/attention_candidates.py (L219-L220)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category performance)

_vllm_triton.run rebuilds the entire paged KV cache (k_cache/v_cache zero-init + pad/copy, and the block table) from scratch on every invocation. _latency times run per call, so the reported prefill (up to 8192 tokens) and decode latencies for vllm_triton_2d are dominated by cache materialization rather than the attention kernel, while the other candidates time only their kernel. This is not an equivalent workload and overstates this candidate's latency. Build the cached K/V and block table once per (fixed) seqs outside the timed region.

---

📄 benchmarks/models/qwen3_next/plot_qwen3_next_thirdparty.py (L55-L55)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

Unlike the peer prior-art plots in this PR (plot_qwen3_next_attention_prior_art.py, plot_qwen3_next_moe_prior_art.py), this script selects candidates purely by key presence and never filters entries flagged unavailable/failed. Every later access assumes the full result payload (selected_sha256, tp4_equals_tp1, nccl_setting_invariant, batch_bitwise, latency_us_tp4, and the gradients/batch_grid blocks). A run where one implementation is unavailable or failed will therefore either raise KeyError/TypeError or index a diagnostic string, aborting the whole figure — exactly the degraded case the evidence figure needs to document. Filter the candidates the same way the sibling plots do (and skip them in the latency/backward panels).

---

📄 benchmarks/models/qwen3_next/plot_qwen3_next_thirdparty.py (L74-L75)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

The digest heatmap hard-codes digests["tp4-default"] as the reference with no existence check, and also indexes each per-run candidate k unconditionally. The referenced runs come from separate multi-GPU jobs, so a missing/renamed run (runs only guarantees the directories that exist) or a candidate absent from one run's rank-0.json produces an opaque KeyError instead of a diagnosable error. plot_qwen3_next_distributed_logprob.py handles the analogous situation with an explicit SystemExit when the reference run is absent; do the same here and skip candidates missing from a run.

---

📄 benchmarks/models/qwen3_next/plot_qwen3_next_moe_prior_art.py (L74-L74)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

cands only filters unavailable, not failed, and then assumes the first surviving candidate carries an output_rows_bitwise mapping. Two failure modes follow: next(iter(cands)) raises StopIteration if every candidate is filtered out, and KeyError if the first candidate is a failed/inference-only entry that omits that field. The peer attention plot filters "failed" as well; align the filter here and derive sizes from a candidate known to have the field (or from the report metadata) rather than from an arbitrary first element.

---

📄 benchmarks/models/qwen3_next/plot_qwen3_next_moe_prior_art.py (L121-L121)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

max() over the filtered generator has no default, so if no candidate reports an accuracy block (all rel values are nan) the plot aborts with ValueError: max() arg is an empty sequence. Note plot_qwen3_next_attention_prior_art.py guards the identical computation with default=0.0 ... or 1.0; apply the same guard here.

💡 Suggested Change

Before:

    ax.set_xlim(0, 2.2 * max(r for r in rel if r == r))

After:

    ax.set_xlim(0, 2.2 * max((r for r in rel if r == r), default=0.0) or 1.0)

---

📄 benchmarks/models/qwen3_next/plot_qwen3_next_distributed_logprob.py (L106-L106)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category documentation)

When a pair has no max_abs_vs entry the cell value is nan, which is rendered by :.1e as the literal string "nan" — visually indistinguishable from a measured value. Since this panel is meant to show that the max |Δ logprob| is exactly zero between implementations, label missing comparisons explicitly (e.g. "n/a") so an unmeasured pair cannot be read as a measured difference.

💡 Suggested Change

Before:

            text = "0" if delta[i, j] == 0 else f"{delta[i, j]:.1e}"

After:

            text = "not run" if delta[i, j] != delta[i, j] else ("0" if delta[i, j] == 0 else f"{delta[i, j]:.1e}")

---

⚠️ Warnings:

• .github/workflows/ci.yml,.github/workflows/qwen3-next-provider-gpu.yml,ci/scripts/run_qwen3_next_chain_gate.sh,ci/scripts/run_qwen3_next_operator_gates.sh,ci/scripts/run_ws1_gtest.sh (review_limit_reached): reached the aggregate token budget before finishing; the grace round completed the task
• benchmarks/models/qwen3_next/qwen3_next_attention_prior_art.py (token_budget_reached): stopped dispatch: used 626145 tokens + group estimate 269140 = projected 895285 exceeds budget 500000
• benchmarks/models/qwen3_next/__init__.py,benchmarks/models/qwen3_next/attention_candidates.py,benchmarks/models/qwen3_next/logprob_candidates.py,benchmarks/models/qwen3_next/moe_ep_candidates.py,benchmarks/models/qwen3_next/moe_route_candidates.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task
• benchmarks/models/qwen3_next/plot_qwen3_next_attention_prior_art.py,benchmarks/models/qwen3_next/plot_qwen3_next_batch_invariance.py,benchmarks/models/qwen3_next/plot_qwen3_next_chain.py,benchmarks/models/qwen3_next/plot_qwen3_next_distributed_logprob.py,benchmarks/models/qwen3_next/plot_qwen3_next_moe_ep.py,benchmarks/models/qwen3_next/plot_qwen3_next_moe_prior_art.py,benchmarks/models/qwen3_next/plot_qwen3_next_thirdparty.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task

[github-actions]

vocab_parallel_cross_entropy performs its all-reduce over Megatron's global parallel_state tensor-parallel group, not the group argument passed to this factory, and this candidate never initializes parallel_state. At TP4 the call therefore either raises (tensor-parallel group uninitialized) and is recorded as unavailable, or — if a stale/other group happens to exist — reduces the max/target/sum-exp over the wrong ranks. In both cases the Megatron logprob prior-art number is not measuring Megatron's real TP4 reduction and will silently misrepresent it. Initialize parallel_state to match tp (and the rank layout) before calling, or dispatch through an explicit group-aware reduction.

[github-actions]

@fusheng-ji The automated OCR review has completed.

Please review and address all automated review bot findings, then push your updates. Maintainers will begin manual review after the bot findings have been addressed.

[fusheng-ji]

Replies to the six OpenCodeReview summary items. The plot fixes are compiled but I have not re-rendered the figures from the evidence.

1. run_qwen3_next_chain_gate.sh L21-L22, bare assert: fixed in 74a1b1a6 (the script belongs to [WS1][CUDA][Qwen3-Next] Full-model chain: vLLM rollout vs VIME recompute, raw-bit through two updates and a reload (RFC #428 stack 7/11) #543 and reaches this PR through the merge). The checks are explicit now: a malformed report exits 2 with the missing field or schema named, a failed gate still exits 1. Run with PYTHONOPTIMIZE=1 on an empty report it exits 2.
2. plot_qwen3_next_attention_prior_art.py L106, KeyError on accuracy: fixed in 026e0440. A candidate without it gets NaN and no label, as in the MoE sibling.
3. same file L112, all-zero x-limit: fixed in 026e0440. The upper bound ignores NaN and falls back to 1.0 for an all-zero column.
4. plot_qwen3_next_thirdparty.py L160-L161, cell keys from names[0]: fixed in 026e0440. A cell a grid did not run is drawn grey instead of raising KeyError. The columns still come from the Miles MoE grid.
5. plot_qwen3_next_distributed_logprob.py L55-L57, iterdir(): fixed in 026e0440. Only directories count as runs, and a missing default run exits with a message.
6. plot_qwen3_next_moe_ep.py L100, hard-coded ep4-default: fixed in 026e0440. A missing ep4-default launch exits with a message that names it. The name stays: it is the launch the sort key and the reference both use.

[fusheng-ji]

One OpenCodeReview summary item:

1. run_qwen3_next_chain_gate.sh L26-L29, JSON-only keys: fixed in 4e2c33af (merge 2c699e3f). first_drift_family is required explicitly because it is a payload key and not a dataclass field, cells must be an object, and the load no longer leaks the file handle; every failure exits 2. A report without first_drift_family now exits 2 with the field named.

[Flink-ddd]

LGTM for the benchmark changes; please merge after #557 is ready.