Repository navigation
[WS1][CUDA][Qwen3-Next] TP distributed logprob parity: fixed reduction order, selected-token logprob exact across NCCL settings (RFC #428 stack 9/11) #549
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
fusheng-ji
wants to merge
117
commits into
RL-Align:test-qwennext
Choose a base branch
from
fusheng-ji:feat/ws1-qwen3-next-distributed-logprob-tp
base: test-qwennext
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+28,841
−34
Open
Changes from all commits
Commits
Show all changes
117 commits
Select commit
Hold shift + click to select a range
cbdc00a
feat(qwen3-next): MoE route/combine contract with a batch-invariant g…
fusheng-ji fce346d
feat(qwen3-next): TP4 GDN, convolution and D=256 attention mixers
fusheng-ji 362b80b
docs(qwen3-next): MoE route/combine contract, prior-art and TP4 evide…
fusheng-ji 89b5212
Merge MoE route/combine evidence into the TP4 mixers
fusheng-ji bd13adc
test(qwen3-next): prior-art runner for the TP4 D=256 attention core
fusheng-ji c318687
docs(qwen3-next): TP4 mixer gates and D=256 attention prior-art evide…
fusheng-ji ce07836
Merge TP4 mixer evidence into the full-model chain
fusheng-ji 92e93a4
feat(qwen3-next): full-model chain with first-drift localisation
fusheng-ji d6106c2
feat(qwen3-next): batch-invariance matrix for the full model
fusheng-ji 02c576c
fix(qwen3-next): native state shapes for any prompt that fits the rol…
fusheng-ji 193805d
Merge the evidence publisher fix into the batch-invariance matrix
fusheng-ji f431bad
fix(qwen3-next): evidence publisher redacts every recorded site path
fusheng-ji dc79953
test(qwen3-next): measure Megatron-core/TE and SGLang in the MoE prio…
fusheng-ji 53dce6e
docs(qwen3-next): MoE prior art with Megatron-core/TE and SGLang meas…
fusheng-ji 3c40295
Merge the MoE prior-art update into the TP4 mixers
fusheng-ji a97a6fe
test(qwen3-next): measure FlashAttention-4, Transformer Engine and Me…
fusheng-ji 137d0fb
fix(qwen3-next): merge flat and per-size attention latency sections
fusheng-ji f01ab18
Merge test-qwennext refactor and migrate Qwen3-Next code and tests
fusheng-ji 6b74b80
Merge test-qwennext refactor and migrate Qwen3-Next code and tests
fusheng-ji 7317e5a
style(tools): mark the Qwen3-Next TP gate imports after sys.path setu…
fusheng-ji 5cacf98
style(tools): mark the Qwen3-Next TP gate imports after sys.path setu…
fusheng-ji 52b97e3
fix(qwen3-next): type annotations for mypy in the MoE prior-art runner
fusheng-ji d44704b
Merge the MoE prior-art type fix into the TP4 mixers
fusheng-ji 4d7d3d8
fix(qwen3-next): type annotations for mypy in the attention prior-art…
fusheng-ji c331746
Merge test-qwennext refactor and migrate the Qwen3-Next full-model chain
fusheng-ji 7bf6ef7
Merge test-qwennext refactor and migrate the Qwen3-Next batch-invaria…
fusheng-ji e485f0f
Merge the full-model chain evidence (c67ec14) into the migrated chain
fusheng-ji b66a8d0
build(pre-commit): leave the byte-exact Qwen3-Next chain evidence unt…
fusheng-ji 13ba74a
Merge the full-model chain evidence into the batch-invariance matrix
fusheng-ji 848ea44
build(pre-commit): leave the byte-exact batch-invariance evidence unt…
fusheng-ji 84a7bfe
test(qwen3-next): pin the case-manifest subprocess rootdir to its tem…
fusheng-ji 6684427
Merge the case-manifest test fix into the batch-invariance matrix
fusheng-ji cbb7b3d
docs(qwen3-next): figure for the full-model chain evidence
fusheng-ji e5a6987
Merge the chain evidence figure into the batch-invariance matrix
fusheng-ji 9cc2fbc
docs(qwen3-next): figure for the batch-invariance evidence
fusheng-ji bddabba
test(qwen3-next): digest every TP4 cross-rank point and compare acros…
fusheng-ji 99c244a
test(qwen3-next): require TP4 == TP1 for selected logprob and LSE only
fusheng-ji c5cea58
test(qwen3-next): prior art for the vocabulary-parallel selected-toke…
fusheng-ji da566bb
test(qwen3-next): no raw torch.distributed reduction in Qwen3-Next fo…
fusheng-ji f3a213d
docs(qwen3-next): TP distributed logprob contract and evidence figure
fusheng-ji 682c233
test(qwen3-next): publisher for the distributed logprob parity evidence
fusheng-ji 16171f5
docs(qwen3-next): record the prior-art NCCL-setting results and latency
fusheng-ji 1106e46
refactor(qwen3-next): focus TP review on validation and externalize e…
fusheng-ji a9d0ba8
fix(qwen3-next): hash validation artifacts on Python 3.10
fusheng-ji 722ef1c
docs(qwen3-next): record post-review B200 validation results
fusheng-ji d8333f4
fix(validation): stream artifact SHA-256 on Python 3.10
fusheng-ji b5708e9
fix(validation): retain batch invariance with Python 3.10 hashing
fusheng-ji 7756969
fix(validation): keep standalone evidence runtime free of torch imports
fusheng-ji 6c7c49c
fix(validation): keep standalone evidence runtime free of torch imports
fusheng-ji 5a5aed4
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji 25ded88
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji ffa0d1d
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji 343c8c9
Merge the batch-invariance fixes into the TP logprob PR
fusheng-ji 4746836
test(qwen3-next): drop the duplicate file_sha256 import left by the m…
fusheng-ji 4daf3c4
Merge the latest #469 branch into the MoE route branch
fusheng-ji 57216db
refactor(qwen3-next): MoE route branch to the reviewed layout
fusheng-ji f7338a7
Merge the MoE route layout change; TP4 mixers to the reviewed layout
fusheng-ji 4ceaa22
Merge the TP4 mixers layout change into the full-model chain branch
fusheng-ji 0d04347
refactor(qwen3-next): full-model chain to the reviewed layout
fusheng-ji 0f5266d
Merge the full-model chain layout change into the batch-invariance br…
fusheng-ji 299ea4c
refactor(qwen3-next): batch-invariance branch to the reviewed layout
fusheng-ji d18e0ff
Merge the batch-invariance layout change into the TP logprob branch
fusheng-ji d84fbf3
style(qwen3-next): black-format the relocated runtime path in the dig…
fusheng-ji e7d08a8
Merge the digest-test formatting fix
fusheng-ji c86ce74
Merge the digest-test formatting fix
fusheng-ji 32040a7
docs(qwen3-next): record the full chain rerun on the stack top
fusheng-ji b59f628
Merge branch 'feat/ws1-qwen3-next-full-model-chain' into feat/ws1-qwe…
fusheng-ji 7b71b93
docs(qwen3-next): record the full BI suite rerun on the stack top
fusheng-ji 0165a26
Merge branch 'feat/ws1-qwen3-next-batch-invariance-ci' into feat/ws1-…
fusheng-ji 8943c14
docs: move the measurement outputs out of the pull request
fusheng-ji 0fdbf80
docs: move the measurement outputs out of the pull request
fusheng-ji 46d5108
docs: move the measurement outputs out of the pull request
fusheng-ji 8fd657c
docs: move the measurement outputs out of the pull request
fusheng-ji 3d4fb2e
docs: move the measurement outputs out of the pull request
fusheng-ji bc43113
docs: point commit hashes at the rewritten history
fusheng-ji f9b85d1
docs: point commit hashes at the rewritten history
fusheng-ji 4f61a39
docs: point commit hashes at the rewritten history
fusheng-ji 804f0a4
docs: point commit hashes at the rewritten history
fusheng-ji 78fc14a
docs: point commit hashes at the rewritten history
fusheng-ji 68240ec
Merge #469 after #467 landed in test-qwennext
fusheng-ji b36b8f6
Merge #503 after #467 landed in test-qwennext
fusheng-ji d243839
Merge #507 after #467 landed in test-qwennext
fusheng-ji eb04b15
Merge #543 after #467 landed in test-qwennext
fusheng-ji fe6ed38
Merge #544 after #467 landed in test-qwennext
fusheng-ji cfbd101
Merge #469 review fixes
fusheng-ji d53ac95
ci(qwen3-next): trigger the provider job on validation/common
fusheng-ji c9a0aed
Merge #503 review fixes
fusheng-ji b9390f3
ci(qwen3-next): trigger the provider job on the CUDA sources it builds
fusheng-ji 47b0c78
Merge #507 review fixes
fusheng-ji 73872d5
chore(qwen3-next): executable operator-gate script; say what is check…
fusheng-ji 9220f1a
Merge #543 review fixes
fusheng-ji b3bd4ea
Merge #544 review fixes
fusheng-ji 9800e1a
Merge #469 second-round review fixes
fusheng-ji d849e18
Merge #503 second-round review fixes
fusheng-ji ecafcc2
Merge #507 second-round review fixes
fusheng-ji 349d6ba
Merge #543 second-round review fixes
fusheng-ji e20c29b
Merge #544 second-round review fixes
fusheng-ji 5ffd4c3
ci: run test_file_digest.py once in the CPU job
fusheng-ji d78b22b
ci(qwen3-next): trigger the provider job on the code this PR's checks…
fusheng-ji dfb0832
Merge #543 provider path-filter fix
fusheng-ji db1eb37
ci(qwen3-next): trigger the provider job on the distributed collectives
fusheng-ji 3f7cf6e
Merge #544 provider path-filter fixes
fusheng-ji 62cc122
fix(workload): sync operator case ids with pytest collection
fusheng-ji fd87c0e
feat(vime): load the Megatron model without DDP or an optimizer for r…
fusheng-ji f2e9aa9
Merge #543 operator-case ids and no-DDP recompute into #544
fusheng-ji 7849c9c
Merge #544 operator-case ids and no-DDP recompute into #549
fusheng-ji 1c68102
fix(vime): pass skip_load_to_model_and_opt to the VIME checkpoint loader
fusheng-ji 1472e8f
Merge #543 load_checkpoint fix into #544
fusheng-ji dee7a78
Merge #544 load_checkpoint fix into #549
fusheng-ji 2e85d22
ci(qwen3-next): assert the full pinned runtime, retrigger on pyprojec…
fusheng-ji 74a1b1a
ci(qwen3-next): reject a malformed chain report with an explicit chec…
fusheng-ji 7fee591
Merge #543 OCR round fixes into #544
fusheng-ji 33e0ff6
Merge #544 OCR round fixes into #549
fusheng-ji 4e2c33a
fix(ci): route unreadable chain reports to exit 2; fail closed on a m…
fusheng-ji 81be081
Merge fix/543-ocr-r8 into fix/544-ocr-r8
fusheng-ji 30c0e5a
Merge fix/544-ocr-r8 into fix/549-ocr-r8
fusheng-ji b15de6e
Merge branch 'test-qwennext' into feat/ws1-qwen3-next-distributed-log…
Flink-ddd File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # Qwen3-Next chain-report adapter checks; with a report path, also validate that report. | ||
| # bash ci/scripts/run_qwen3_next_chain_gate.sh [chain-report.json] | ||
| set -euo pipefail | ||
| ROOT="$(cd "$(dirname "$0")/../.." && pwd)" | ||
| cd "$ROOT" | ||
| PY="${PY:-python3}" | ||
| export PYTHONPATH="$ROOT${PYTHONPATH:+:$PYTHONPATH}" | ||
| "$PY" -m pytest -q tests/models/qwen3_next/test_qwen3_next_chain_report.py | ||
| if [ "$#" -ge 1 ]; then | ||
| "$PY" - "$1" <<'PY' | ||
| import dataclasses, json, sys | ||
|
|
||
| from rl_engine.validation.models.chain_gate import ChainGateReport | ||
| from rl_engine.validation.models.qwen3_next.qwen3_next_chain_report import COMPATIBLE_SCHEMA, SCHEMA_VERSION | ||
|
|
||
|
|
||
|
|
||
| def invalid(message): | ||
| """Reject a malformed report with exit 2, apart from exit 1 for a failed gate.""" | ||
| print(f"invalid chain report: {message}", file=sys.stderr) | ||
| sys.exit(2) | ||
|
|
||
|
|
||
| try: | ||
| with open(sys.argv[1]) as handle: | ||
| report = json.load(handle) | ||
| except (OSError, ValueError) as error: | ||
| invalid(f"cannot read {sys.argv[1]!r} as JSON: {error}") | ||
| if not isinstance(report, dict): | ||
| invalid(f"top level is {type(report).__name__}, expected an object") | ||
| # first_drift_family is added to the payload by build_report; it is not a dataclass field. | ||
| missing = ({f.name for f in dataclasses.fields(ChainGateReport)} | {"first_drift_family"}) - set(report) | ||
| if missing: | ||
| invalid(f"lacks report fields: {sorted(missing)}") | ||
| if not isinstance(report["cells"], dict): | ||
| invalid(f"cells is {type(report['cells']).__name__}, expected an object") | ||
| for key, expected in (("schema_version", SCHEMA_VERSION), ("compatible_schema", COMPATIBLE_SCHEMA)): | ||
| if report.get(key) != expected: | ||
| invalid(f"{key} is {report.get(key)!r}, expected {expected!r}") | ||
| print(f"passed={report['passed']} first_drift={report['first_drift']} " | ||
| f"family={report['first_drift_family']} cells={len(report['cells'])}") | ||
| sys.exit(0 if report["passed"] else 1) | ||
| PY | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| #!/usr/bin/env bash | ||
| # Required operator acceptance and diagnostics use separate declared case sets. | ||
| set -euo pipefail | ||
| ROOT="$(cd "$(dirname "$0")/../.." && pwd)" | ||
| cd "$ROOT" | ||
| PY="${PY:-python3}" | ||
| OUT="${QWEN3_NEXT_EVIDENCE_DIR:?Set QWEN3_NEXT_EVIDENCE_DIR to an artifact directory}" | ||
| export PYTHONPATH="$ROOT${PYTHONPATH:+:$PYTHONPATH}" | ||
| exec "$PY" tools/validation/models/qwen3_next/qwen3_next_operator_gates.py --out "$OUT" \ | ||
| --suite "${QWEN3_NEXT_OPERATOR_SUITE:-all}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| # Qwen3-Next distributed logprob parity (TP) | ||
|
|
||
| ## Summary | ||
|
|
||
| RFC #428 row `distributed_logprob_parity` asks for a locked TP/EP reduction | ||
| order and an exactly equal selected-token logprob between VIME replay and vLLM | ||
| rollout. This page covers tensor parallelism (TP4, EP = 1, CP = 1) for | ||
| Qwen3-Next-80B-A3B-Instruct. Expert parallelism is a separate PR. | ||
|
|
||
| Every place where TP ranks combine values either moves bytes (an NCCL | ||
| all-gather, which is exact whatever NCCL algorithm runs) or sums them with | ||
| RL-Kernel's fixed-order collective, whose order does not depend on NCCL at all. | ||
| No new collective or kernel is added; the row documents and proves the order. | ||
|
|
||
| ## Entry Point | ||
|
|
||
| ```python | ||
| from rl_engine.models.qwen3_next.qwen3_next_tp import _ReduceFromTP, _CopyToTP | ||
| from rl_engine.models.qwen3_next.qwen3_next_logprob import qwen_logprob_contract | ||
| from rl_engine.models.qwen3_next.qwen3_next_forward import shared_logprob | ||
|
|
||
| contract = qwen_logprob_contract(local_logits, rank=rank, world_size=4) | ||
| selected, lse, entropy = shared_logprob(local_logits, targets, contract=contract, tp_group=tp_group) | ||
| ``` | ||
|
|
||
| vLLM rollout (`RLKernelQwen3NextForCausalLM.compute_selected_logprobs`) and VIME | ||
| replay (`calculate_log_probs_and_entropy`, installed by | ||
| `install_vime_logprob_provider`) both call `shared_logprob` on their own | ||
| 37984-column LM-head shard. | ||
|
|
||
| ## Contract | ||
|
|
||
| | Cross-rank point | Where | Operation | Order / dtype | | ||
| | --- | --- | --- | --- | | ||
| | Vocabulary-parallel embedding | `qwen3_next_embedding.shared_embedding` | each rank selects its rows, zeros elsewhere; sum | fixed-order collective, BF16; exactly one non-zero term per element | | ||
| | Embedding weight gradient | same, backward | stable sort by row, pairwise FP32 tree in token order | local, no collective | | ||
| | GDN `out_proj`, attention `o_proj`, MoE routed + shared output | `_ReduceFromTP` in `qwen3_next_tp_gdn`, `qwen3_next_tp_blocks`, vLLM adapters | row-parallel partial sums | fixed-order collective, BF16 | | ||
| | Input gradients of column-parallel projections | `_CopyToTP.backward` | sum of per-rank partial gradients | fixed-order collective | | ||
| | Selected-token logprob, LSE | `shared_logprob` (WS2 `CudaVocabParallelLogprobOp`) | per-rank FP32 max / sum-exp per tile, all-gather of tile statistics, merge in global tile order over a fixed `[tokens, 64]` grid | FP32; independent of the TP degree | | ||
| | Entropy | same | per-shard contributions, all-gather, sum in TP-rank order | FP32; depends on the TP degree (outside the selected-logprob contract) | | ||
| | Sampling logits | `compute_logits` (vLLM) | all-gather of the column shards | exact data movement | | ||
|
|
||
| RL-Kernel's fixed-order collective (`rl_engine.distributed.algorithms.collectives`, | ||
| backend `cuda_ipc_fixed_tree`) stages every rank's input in a symmetric CUDA IPC | ||
| buffer and runs one reduction kernel that reads all peers and applies the same | ||
| balanced tree on every rank; NCCL is not involved in the arithmetic, so | ||
| `NCCL_ALGO`, `NCCL_PROTO` and the channel count cannot change the result. The | ||
| logprob all-gathers move per-tile statistics whose merge order is fixed by the | ||
| tile index, so changing the TP degree does not change selected logprobs or LSE | ||
| either. | ||
|
|
||
| Fail-closed: `shared_logprob` rejects a vocabulary other than 151936, CP > 1, a | ||
| TP degree other than 1 or 4, non-finite logits and a missing contract; the | ||
| fixed-order collective checks on first use that every rank calls it with the | ||
| same shape and dtype, and supports only TP sizes 1, 2, 4 and 8. `tests/models/qwen3_next/test_qwen3_next_tp_reductions.py` fails | ||
| if a raw `torch.distributed` reduction appears in a Qwen3-Next model, engine or | ||
| VIME module (an integer MAX of per-rank flags is the one documented exception). | ||
|
|
||
| ## Results | ||
|
|
||
| Evidence: [`reports/experiments/qwen3_next/qwen3-next-distributed-logprob-b200.md`](https://github.com/fusheng-ji/RL-Kernel/blob/feat/ws1-qwen3-next-distributed-logprob-tp/reports/experiments/qwen3_next/qwen3-next-distributed-logprob-b200.md). | ||
|
|
||
| - 13 NCCL settings (defaults; Ring and all-reduce Tree × LL / LL128 / Simple × | ||
| 1 channel / default channels), real Qwen3-Next weights, 1024 tokens, 4 × B200: | ||
| every point in the contract table has one digest per rank across all | ||
| settings, and the TP4 selected logprob and LSE equal TP1. The negative | ||
| control -- real attention `o_proj` partials summed with NCCL `all_reduce` -- | ||
| gives 5 different results over the same settings. | ||
| - The full-model chain (#543) shows the end-to-end consequence: selected | ||
| logprobs of vLLM rollout and VIME recompute are bitwise equal for 16 cases at | ||
| four weight versions. | ||
|
|
||
| ## Tests | ||
|
|
||
| | File | Device | Covers | | ||
| | --- | --- | --- | | ||
| | `tests/models/qwen3_next/test_qwen3_next_tp_reductions.py` | CPU | no raw NCCL reduction in Qwen3-Next forward paths; the TP reduction uses the fixed-order collective | | ||
| | `tools/validation/models/qwen3_next/qwen3_next_logprob_parity.py` + `..._compare.py` | 4 × CUDA | digests at every cross-rank point under each NCCL setting; TP4 == TP1 | | ||
| | `tests/distributed/tp/test_vocab_parallel_logp.py` (existing) | CUDA | WS2 TP = n == TP = 1 on synthetic shapes | | ||
|
|
||
| ## Known Limitations | ||
|
|
||
| - TP4 with EP = 1 and CP = 1 only; expert parallelism is the second PR of this row. | ||
| - Entropy merges per-shard contributions in rank order and therefore changes | ||
| with the TP degree; rollout and replay run the same TP degree, so they still | ||
| agree, but entropy is not TP-degree invariant. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This 28-module qwen3_next block was appended to the "Run Mocked Hardware Discovery Tests" step, which does not build the
rl_engine._Cextension and is CPU-only. Several of the newly listed modules (e.g.test_qwen3_next_imported_binary.py,test_qwen3_next_installed_provider_identity.py,test_qwen3_next_storage.py,test_qwen3_next_training_checkpoint.py,test_qwen3_next_l2.py) exercise compiled-extension, vLLM-import or on-disk binary/provider evidence paths. Any module that imports those at module scope withoutpytest.importorskip/skip markers will fail collection here, turning the required CI job red instead of skipping. Please confirm each newly added module is import-safe on a runner without the extension/vLLM (or split these into their own step with the appropriate skip guard).