Repository navigation
[WS1][CUDA][Qwen3-Next] TP4 GDN, convolution and D=256 attention mixers on real weights (RFC #428 stack 6/11) - #507
Conversation
…rouped expert GEMM RFC RL-Align#428 moe_route_combine_contract (CUDA). - Routing: FP32 router through the pinned vLLM batch-invariant GEMM, a fixed-order softmax, stable descending argsort (ties go to the lower expert id) and FP32 renormalisation of the ten selected weights. - Experts: per-route gate_up and down projections through vLLM 0.30.0's fused_moe Triton kernel with the batch-invariant tile passed explicitly (BLOCK 64/64/32, SPLIT_K=1); no routed-weight multiply and no sum inside the kernel, so every (token, route) row has one writer. Bitwise equal to the previous per-expert GEMM loop at TP4 and TP1 widths; fails closed if VLLM_TRITON_USE_TD is enabled. - Combine: ten ordered FP32 multiply-adds and one BF16 cast; no atomics. - Backward: per-expert pinned GEMMs into one dense gradient buffer; dx is summed in ascending expert order. - TP4 block: HF load/export for all 512 experts, replicated router and shared-expert gate, Megatron-style copy/reduce autograd boundaries moved into qwen3_next_tp.py. - Prior-art runner and plot comparing HF transformers, vLLM fused_moe (BI=0/1) and FlashInfer cutlass_fused_moe; TP4 real-weight gate script. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Prep for RFC RL-Align#428 full_model_chain; serves the gdn_state_bridge row. - GDN: packed recurrent prefill/decode through the pinned vLLM 0.30.0 provider with an explicit state boundary (qwen3_next_gdn.py), the depthwise causal convolution with cache handoff (qwen3_next_conv.py), their combined shared core (qwen3_next_provider.py) and the TP4 block with HF GQA-interleaved weight mapping (qwen3_next_tp_gdn.py), now on the shared TP boundaries in qwen3_next_tp.py. - Attention: the deterministic CUDA attention kernel accepts head dim 256 (ROCm stays 128) and validates backward inputs; shared_attention and TP4FullAttention with the KV head replicated on each rank pair, partial RoPE at absolute positions and explicit per-slot KV state. - Real-weight TP4 gates for GDN (layer 0) and attention (layer 3), sharing scripts/qwen3_next_gate_common.py with the MoE gate. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…nce (B200, 2e950f3) Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Compares RL-Kernel's deterministic attention with torch SDPA, vLLM FlashAttention-2 (num_splits auto and 1), vLLM's Triton unified attention and FlashInfer's ragged prefill: batch, chunked-prefill and decode invariance, backward repeatability, accuracy against FP64 and latency. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…nce (B200, 9e6c4f2/6517077) Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…r-art runner - Megatron-core 0.16 MoELayer with Transformer Engine grouped GEMM and fused permute, configured as VIME runs Qwen3-Next (FP32 softmax router, top-10, all-to-all dispatcher), including its backward. - SGLang 0.5.21 Triton fused_moe in default and deterministic-inference mode. sgl_kernel cannot load next to torch 2.13, so its two kernels on this path are replaced by SGLang's own Triton moe_sum_reduce and JIT moe_align_block_size; any other sgl_kernel call fails loudly. - --only and --merge combine reports from different Python environments. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…ured (B200, c58816b) Every library in the reuse table is now measured: Megatron-core 0.16 + TE 2.16 as VIME configures Qwen3-Next, and SGLang 0.5.21 in default and deterministic-inference mode. The TP4 gate evidence is unchanged (2e950f3, identical MoE code). Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…gatron-core attention Add three engines to the attention prior-art runner: - FlashAttention-4 (CuTe DSL) varlen as vendored by vLLM, forward and backward. - Transformer Engine DotProductAttention on THD-packed sequences, in inference mode (cuDNN fused) and in training mode (on SM100 TE has no fused head-dim-256 backward and falls back to its unfused path). - Megatron-core's local DotProductAttention, one launch per sequence. The backward check now compares the target's dq/dk/dv alone against the same sequence packed first and last, so engines with a packed backward are checked for batch invariance rather than repeatability. A prefill/decode crash is recorded per candidate without skipping its other checks. --only and --merge combine reports from different Python environments. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
The decode and forward+backward latency sections are keyed by candidate directly, the prefill section by size first; merge them at any depth. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…p as E402-exempt The gate scripts insert the repo root into sys.path before importing rl_engine and tools, which flake8 reports as E402. Mark those imports with noqa, as the other tools/ scripts do. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…p as E402-exempt The gate scripts insert the repo root into sys.path before importing rl_engine and tools, which flake8 reports as E402. Mark those imports with noqa, as the other tools/ scripts do. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Annotate the latency samples and accept the default-argument lambdas used as timed calls. No behaviour change. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
… runner Annotate the latency samples and accept the default-argument lambdas used as timed calls. No behaviour change. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Brings in RL-Align#468's caedea5 and the provider path-filter changes of RL-Align#469 and RL-Align#503. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
From this PR the provider modules reach compiled code: the shared GDN core uses the gated RMSNorm kernel (csrc/cuda/norm) and the D=256 attention kernels are bound in csrc/bindings/ops.cpp; deterministic_attn imports rl_engine/contracts. rl_engine/backends/** only covers the Python wrappers, so these paths are added. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Brings in RL-Align#468's caedea5 and the provider path-filter changes of RL-Align#469, RL-Align#503 and RL-Align#507. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| echo "source_sha=${{ github.event.pull_request.head.sha }}" | ||
|
|
||
| provider: | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository |
There was a problem hiding this comment.
pull_request trigger + this if means any same-repo branch (e.g. an unreviewed collaborator branch) executes PR-controlled code (setup.py build_ext, tools/checks/ci_smoke.py, the check_*.py modules) directly on the trusted self-hosted runner rl-kernel-qwen3-next via ref: ... head.sha. The header comment states "Fork PRs never reach the self-hosted runner; a maintainer dispatches the reviewed commit", but a same-repo PR bypasses maintainer dispatch entirely and still gets persistent runner filesystem state (which the cleanup step below exists to work around). This is the exact privileged-execution pattern the workflow claims to avoid. Either drop the pull_request trigger and keep only workflow_dispatch, or gate the job on github.event.pull_request.head.repo.full_name == github.repository && contains(fromJSON('["maintainer","member"]'), github.event.pull_request.author_association).
There was a problem hiding this comment.
Same answer as r4372884187, r4372907792 and 549-I4381852969; not changed. Pushing a branch to this repository already requires write access, the job runs with contents: read and no pull_request_target, and fork PRs never reach the runner. This is the trust model ws1-chain-npu.yml uses; an approval environment or a dispatch-only trigger is a maintainer policy choice for both workflows.
| set -o pipefail | ||
| log="$RUNNER_TEMP/provider-pytest.log" | ||
| python3 -m pytest -q -rs tests/models/qwen3_next/check_qwen3_next_norm_providers.py tests/models/qwen3_next/check_gdn_recurrent_golden.py \ | ||
| tests/models/qwen3_next/check_qwen3_next_forward.py tests/models/qwen3_next/check_qwen3_next_attention.py \ | ||
| tests/models/qwen3_next/check_qwen3_next_gdn_bridge.py tests/models/qwen3_next/check_qwen3_next_gdn_sequence.py \ | ||
| tests/models/qwen3_next/check_qwen3_next_conv_bridge.py tests/models/qwen3_next/check_qwen3_next_core_matrix.py \ | ||
| tests/models/qwen3_next/check_qwen3_next_shared_core.py | tee "$log" |
There was a problem hiding this comment.
set -o pipefail is set inside this script but the step itself is an implicit bash -e, not bash -eo pipefail: pipefail here only affects pipelines within the step, and the python3 -m pytest ... | tee "$log" pipeline's exit status is discarded because the step does not exit on it. The next commands are if grep ... (whose status is consumed by the if) and a final grep -Eq '[0-9]+ passed', so a non-zero pytest exit (e.g. an ImportError or a partially-failing module) is masked as long as some module printed "N passed". With -q, a partial run of 8 modules that fails the 9th still prints "X passed, 1 failed", which the final grep matches. Capture the pytest status explicitly and propagate it, e.g. pytest ... | tee "$log"; status=${PIPESTATUS[0]}; [ "$status" -eq 0 ] || exit "$status", and make the final grep reject failed/error lines too.
There was a problem hiding this comment.
Not changed: set -o pipefail runs in the same shell before the pytest | tee pipeline, so a failing pytest gives the pipeline a non-zero status and bash -e stops the step there, before either grep (bash -e -c 'set -o pipefail; false | cat; echo reached' exits 1 without printing). Same point as 557-I4381857006.
Fixed in b9390f3: the path filter now includes |
Agreed, and nothing to change: both loops call the same |
Agreed, no defect: the contributions use the same |
Brings in RL-Align#468's a203719: the norm manifest's top-level keys, chain semantics and duplicate-op checks, the gate dtype message and the new norm tests. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Brings in RL-Align#468's a203719: the norm manifest's top-level keys, chain semantics and duplicate-op checks, the gate dtype message and the new norm tests. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Brings in RL-Align#468's a203719: the norm manifest's top-level keys, chain semantics and duplicate-op checks, the gate dtype message and the new norm tests. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Not changed. The hook returns per-row contributions on purpose: the C4 harness keys them per logical token, checks the row count, and folds them in fp32 in sorted key order itself, which is the property C4 tests; a pre-folded |
Not changed: |
Same point as 468-S2, 549-S2; not changed. |
Not changed. This file is not the norms' gate: |
Not changed: an explicit |
Not changed; the premise is off. Line 673 is in the ROCm map, and the only dedicated kernel for this op is CUDA: ROCm, MUSA, CPU and NPU all list just the PyTorch reference, on purpose, so the lookup cannot fall through to the generic |
| provider: | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository | ||
| runs-on: [self-hosted, linux, x64, rl-kernel-qwen3-next] |
There was a problem hiding this comment.
The pull_request trigger can reach the self-hosted runner for any same-repo branch, including branches pushed by less-trusted collaborators, and then builds and executes that branch's setup.py/tests (python3 setup.py build_ext --inplace, arbitrary test imports) on a long-lived privileged runner. The header comment claims "a maintainer dispatches the reviewed commit from a trusted branch", but that only holds for workflow_dispatch; the pull_request path bypasses the reviewed-SHA gate entirely (ws1-chain-npu.yml has the same shape, but this new runner is explicitly described as audited and provides no additional protection). Either drop the pull_request trigger, or restrict it to the trusted base (e.g. require github.event.pull_request.head.ref == 'main' / an environment approval / pull_request_target-free label gating) so unreviewed PR code cannot execute here.
There was a problem hiding this comment.
Same answer as 549-I4381852969, 507-I4383081716 and r4372884187; not changed. Pushing a branch here already requires write access, the job runs with contents: read and no pull_request_target, and fork PRs never reach the runner; this is the trust model of ws1-chain-npu.yml.
Not changed. Forward and backward call the same |
Same point as 544-S2, 503-S2; not changed. With |
Same point as 468-I4383081337, 543-S6, 503-S3; not changed. Both gate tools take the pair |
|
Hi, I'm working on Task 20 of RFC #642. Following the maintainer's guidance, I'll first extract the reusable GDN/causal-convolution references from #469 into a small PR targeting I noticed #507 already includes packed sequence execution and state handling. I'll keep the shared CPU reference independent of the CUDA/vLLM provider and avoid duplicating that integration work. Please let me know if there are any interfaces or semantics I should preserve to keep the two efforts compatible. |
|
Hi @hongleng, thanks for checking before building. Keeping the CPU reference independent of the CUDA/vLLM provider is the right split. Here are the contracts in #507 (head Packed sequences (
Recurrent state (
Causal conv (
Properties the gates check, bitwise on all 4 TP ranks
Notes on scope
Questions for you: which dtype and state layout will your shared numerical contract use? If it uses the same |
|
Thanks for the detailed contracts! This is very helpful. I plan to define the shared recurrent-state layout as I'll keep the CPU sequence/chunk reference independent of the CUDA/vLLM provider and use your packed-sequence and state-continuation conventions for the #507 compatibility tests. I'll also make sure provider-specific arithmetic is explicit rather than assuming that vLLM agreement implies compatibility with miles. Thanks for coordinating! |
| weight.contiguous(), | ||
| activation="silu", | ||
| conv_state_indices=indices[active].to(torch.int32).contiguous(), | ||
| out=torch.empty_like(packed), |
There was a problem hiding this comment.
The pinned causal_conv1d_update is called with an out= keyword and a stacked (num_seqs, dim, seq_len) multi-token tensor. Neither the out= keyword nor a grouped multi-length prefill layout is part of the documented single-token/decode signature of the update kernel. If the pinned signature lacks out, this raises TypeError on the very first call (including any prefill chunk of length > 1); if it accepts out but only fills seq_len == 1, longer chunks silently drop positions and leave new_state partially updated. Verify the provider signature and restrict the call to the supported shape/kwargs (e.g. loop over positions for multi-token groups, or use the documented out contract).
There was a problem hiding this comment.
Not changed: the pinned provider documents both. In vLLM 0.30.0 causal_conv1d_update is (x, conv_state, weight, bias=None, activation=None, conv_state_indices=None, ..., out=None) and its docstring says x may be [batch, dim, seqlen] ("single or multiple tokens prediction") and out is an "optional output tensor with the same shape as x" (vllm/model_executor/layers/mamba/ops/causal_conv1d.py:1096-1118, 1148-1149). The kernel loops over seqlen and writes every token (for idx_token in tl.range(seqlen), line 999), and shifts the cache once per call, so a chunk longer than 1 does not drop positions.
The bridge calls it with equal-length groups only (no padding, no speculative cache), and uses the returned tensor, which is out itself. The bridge test covers the case you describe: check_qwen3_next_conv_bridge.py::test_varlen_matches_repeated_decode_and_preserves_inputs runs lengths (64, 8, 0, 16) and (1024, 256, 64, 8) through the grouped call and compares output and final cache bitwise against single-token provider calls made one at a time (lines 31-53); test_chunked_and_reordered_sequences_match checks chunk continuation. At 1fef29b0: 6 passed, 0 skipped on RTX PRO 6000 Blackwell (sm_120), vLLM 0.30.0.
| new_state, | ||
| out, |
There was a problem hiding this comment.
_PackedDecode.forward allocates out as zeros, passes it into the provider, discards the provider's return value, and returns the preallocated out. This silently assumes the pinned fused_recurrent_gated_delta_rule_packed_decode writes into out in place (and that out is the correct positional argument). If the provider instead allocates/returns a fresh tensor (or returns a (out, state) tuple), the returned out stays all zeros, so the whole conv→recurrence→norm chain downstream produces zero output and zero/incorrect gradients with no error. The CPU-only bridge tests cannot catch this because they do not execute the GPU provider. Capture and use the provider's return value (falling back to out only when it writes in place), and assert/verify non-placeholder output.
There was a problem hiding this comment.
Not changed: the pinned provider writes out in place and its return value is the same objects. In vLLM 0.30.0 fused_recurrent_gated_delta_rule_packed_decode(mixed_qkv, a, b, A_log, dt_bias, scale, initial_state, out, ssm_state_indices, use_qk_l2norm_in_kernel=False) takes initial_state and out as its 7th and 8th arguments, which is the bridge's order. It passes o=out, h0=initial_state, ht=initial_state to the Triton kernel and ends with return out, initial_state (vllm/third_party/flash_linear_attention/ops/fused_recurrent.py:343-353, 462-464, 485). The kernel stores zeros for state index <= 0 and the result otherwise, so every element of out is written. vLLM's own decode path calls it the same way, with a preallocated out= and the return value ignored. The bridge only runs when importlib.metadata.version("vllm") == "0.30.0".
Measured on 1fef29b0 (RTX PRO 6000 Blackwell, sm_120, vLLM 0.30.0, Triton 3.7.1): the returned out and state are the same tensors that were passed in (is and data_ptr); an out pre-filled with NaN is fully overwritten; the bridge output agrees with an independent PyTorch reference to 3e-5 (16 key heads, 32 value heads, slots [1, 0, 3, -1, 2]), and packed sequences of length 0-7 agree with a token-by-token loop to 1e-6. The GPU bridge checks are 21 passed.
You are right that the CPU-only tests never reach the provider; the checks that do are the GPU files run by qwen3-next-provider-gpu.yml, which is skipped for fork PRs and needs a maintainer dispatch.
|
Thanks, |
What
rl_engine/integrations/engines/train/vllm/qwen3_next_{gdn,conv,provider}.py): packed recurrent prefill and decode through the pinned vLLM 0.30.0 provider, the depthwise causal convolution with its cache handoff, and one shared core whose prompt state crosses an explicit boundary into decode and into the trainer's gradient path.TP4GDN(rl_engine/models/qwen3_next/qwen3_next_tp_gdn.py): HF GQA-interleaved weight mapping per rank, explicitGDNState, Megatron-style TP boundaries.csrc/cuda/attention/deterministic_attention.cuaccepts head dim 256 on CUDA (ROCm stays 128) and validates backward inputs;shared_attentionandTP4FullAttentionwith the KV head replicated per rank pair, partial RoPE (64 of 256 dims) at absolute positions, per-slot KV state.tools/validation/models/qwen3_next/qwen3_next_tp_{gdn,attention,moe}_check.pyonqwen3_next_gate_common.py.reports/experiments/qwen3_next/.Prior art & reuse decision
GDN: reused as surveyed in #469 (the decode step runs vLLM's own FLA kernels; RL-Kernel supplies the state boundary and the golden).
Attention (one TP4 rank: 4 query heads, 1 KV head, D = 256, causal), measured:
num_splits=1num_splitsautoBatchPrefillWithRaggedKVCacheDotProductAttention¹ Default backward; its
deterministic=Truebackward asserts for head dim 256 on SM100. ² TE's unfused THD path raises a shape error when the query is shorter than the KV. ³ No packed-sequence path: one launch per sequence.Decision: keep RL-Kernel's deterministic attention on both sides in this PR. It covers forward and backward with one documented reduction order and backs the WS1 attention contract; train and rollout must run the same kernel. FlashAttention-4 also passes every check here and is 30-500× faster at 2048-8192 tokens, so it is the follow-up for the slow forward. It is not adopted here because its backward has no deterministic mode at head dim 256 on SM100 (
deterministic=Trueasserts), so the batch-invariant backward measured here is not guaranteed by its contract, and because adopting it means pinning vLLM's vendored CuTe build on both engines. TE fused is invariant for inference only; Megatron-core's local path only by launching per sequence.Results (4 × B200, official checkpoint)
Summary, tested commits and commands:
reports/experiments/qwen3_next/qwen3-next-tp4-mixers-b200.md; raw reports ZIP (SHA-2560976d8e9…e8d4b). Gates measured at9e6c4f2, attention prior art at703c6dc.Status
d849e180, on top of [WS1][CUDA][Qwen3-Next] MoE route/combine contract with a batch-invariant grouped expert GEMM (RFC #428 stack 5/11) #503 (9800e1a4). CI is green on this head: lint, unit-tests, docs, DCO and the bot review pass;providerandws1-gtestskip because no self-hosted runner is attached.f10a271): all three gates passed with every case value and digest identical; only MoE wall-clock timings differ.90c09164, 4 × B200): the standalone GDN, attention and MoE gates passed on all four ranks (7, 10 and 12 cases), and the full-model L2 artifacts are bitwise identical to the pre-move chain.History
0fdbf80band the attention comparison at58aa50c7(the [WS1][CUDA][Qwen3-Next] Comparison benchmarks and plots for the Qwen3-Next stack: MoE, attention, logprob, EP (RFC #428 stack 11/11) #551 branch). The raw files recorded885ed0and77d59a59, the ids these commits had before the branches were rewritten; each pair has the same tree. The H100 gates were not rerun at the current head.b9390f32triggers the provider job oncsrc/cuda/norm/**,csrc/bindings/**andrl_engine/contracts/**. The fixes of the PRs below it are merged in; new headd849e180. Kernel arithmetic is unchanged, so the measured results stand.test-qwennextafter [WS1][CUDA][Qwen3-Next] C1: zero-centred RMSNorm references and CUDA kernel (RFC #428 stack 2/4) #467 landed (c382dd4b), through [WS1][CUDA][Qwen3-Next] MoE route/combine contract with a batch-invariant grouped expert GEMM (RFC #428 stack 5/11) #503's new head68240ec3, inb36b8f6a. One conflict, in the provider workflow's pytest step: [WS1][CUDA][Qwen3-Next] C6: GDN decode-step goldens and recurrent/conv state contract (RFC #428 stack 4/4) #469's fail-closed skip check now also runs this PR's check modules.Tests
python setup.py build_ext --inplace python -m pytest -q tests/models/qwen3_next/test_qwen3_next_tp_gdn.py tests/models/qwen3_next/test_qwen3_next_tp_blocks.py tests/models/qwen3_next/test_qwen3_next_forward_contract.py TRITON_F32_DEFAULT=ieee python -m pytest -q tests/models/qwen3_next/check_qwen3_next_{attention,gdn_bridge,gdn_sequence,conv_bridge,core_matrix,shared_core,forward}.py torchrun --nproc-per-node 4 tools/validation/models/qwen3_next/qwen3_next_tp_gdn_check.py --checkpoint <ckpt> --output <dir> torchrun --nproc-per-node 4 tools/validation/models/qwen3_next/qwen3_next_tp_attention_check.py --checkpoint <ckpt> --output <dir>ci.ymlat this branch9e6c4f2)test_attention_dispatch, also onmainThe
check_files import vLLM and run in theQwen3-Next-provider-GPUworkflow.Scope
CUDA, TP4. ROCm keeps D = 128. No engine adapter or model-level claim; those are
full_model_chain(#543).Results (H100 / Hopper, 4 GPUs, official checkpoint)
Run at
0fdbf80b, this PR's head on 2026-10-19 (it is an ancestor of the current headd849e180). The raw files recorded885ed0, the id of this commit before the branch was rewritten; both ids have the treea1f26d9003bfd8bd71f5245b49b5b81bb9d1556e(git rev-parse 0fdbf80b^{tree}). Not rerun atd849e180: since0fdbf80bthe branch changed the RMSNorm sources (csrc/cuda/norm/rmsnorm.cu,csrc/bindings/ops.cpp,rl_engine/backends/cuda/norm/rmsnorm.py),rl_engine/config/workload.py, the norm and conv references, the workload validation and tests, but none of the three TP4 modules these gates run. 4 × NVIDIA H100 80GB, torch 2.13.0+cu130, vLLM 0.30.0, NCCL 2.29.7, checkpoint revision9c7f2fbe. Raw data:hopper-qwen3-next/pr507.tests/models/qwen3_next/test_*.py+check_*.py(4 GPUs)Attention prior art (head dim 256, one launch per sequence; runner of #551 at
58aa50c7, which imports RL-Kernel's attention from that tree; the raw files record77d59a59, the id of this commit before the branch was rewritten, and both ids have the tree67a6c5c8a30c1351b05beb9b9f509976f8dff45c):dq / dk / dvbatch invariantnum_splits=0num_splits=1All implementations match the FP64 reference to the same max abs error (6.7e-3; Megatron local 1.3e-2). RL-Kernel's attention is the slowest of the ten at all three lengths on H100: it buys invariance with one launch per sequence.