Repository navigation
[WS1][CUDA][Qwen3-Next] EP4 distributed logprob parity: expert-parallel MoE, full-model L2 bitwise on TP4 x EP4 (RFC #428 stack 10/11) #557
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
fusheng-ji
wants to merge
155
commits into
RL-Align:test-qwennext
Choose a base branch
from
fusheng-ji:feat/ws1-qwen3-next-distributed-logprob-ep
base: test-qwennext
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+29,886
−34
Open
Changes from 136 commits
Commits
Show all changes
155 commits
Select commit
Hold shift + click to select a range
cbdc00a
feat(qwen3-next): MoE route/combine contract with a batch-invariant g…
fusheng-ji fce346d
feat(qwen3-next): TP4 GDN, convolution and D=256 attention mixers
fusheng-ji 362b80b
docs(qwen3-next): MoE route/combine contract, prior-art and TP4 evide…
fusheng-ji 89b5212
Merge MoE route/combine evidence into the TP4 mixers
fusheng-ji bd13adc
test(qwen3-next): prior-art runner for the TP4 D=256 attention core
fusheng-ji c318687
docs(qwen3-next): TP4 mixer gates and D=256 attention prior-art evide…
fusheng-ji ce07836
Merge TP4 mixer evidence into the full-model chain
fusheng-ji 92e93a4
feat(qwen3-next): full-model chain with first-drift localisation
fusheng-ji d6106c2
feat(qwen3-next): batch-invariance matrix for the full model
fusheng-ji 02c576c
fix(qwen3-next): native state shapes for any prompt that fits the rol…
fusheng-ji 193805d
Merge the evidence publisher fix into the batch-invariance matrix
fusheng-ji f431bad
fix(qwen3-next): evidence publisher redacts every recorded site path
fusheng-ji dc79953
test(qwen3-next): measure Megatron-core/TE and SGLang in the MoE prio…
fusheng-ji 53dce6e
docs(qwen3-next): MoE prior art with Megatron-core/TE and SGLang meas…
fusheng-ji 3c40295
Merge the MoE prior-art update into the TP4 mixers
fusheng-ji a97a6fe
test(qwen3-next): measure FlashAttention-4, Transformer Engine and Me…
fusheng-ji 137d0fb
fix(qwen3-next): merge flat and per-size attention latency sections
fusheng-ji f01ab18
Merge test-qwennext refactor and migrate Qwen3-Next code and tests
fusheng-ji 6b74b80
Merge test-qwennext refactor and migrate Qwen3-Next code and tests
fusheng-ji 7317e5a
style(tools): mark the Qwen3-Next TP gate imports after sys.path setu…
fusheng-ji 5cacf98
style(tools): mark the Qwen3-Next TP gate imports after sys.path setu…
fusheng-ji 52b97e3
fix(qwen3-next): type annotations for mypy in the MoE prior-art runner
fusheng-ji d44704b
Merge the MoE prior-art type fix into the TP4 mixers
fusheng-ji 4d7d3d8
fix(qwen3-next): type annotations for mypy in the attention prior-art…
fusheng-ji c331746
Merge test-qwennext refactor and migrate the Qwen3-Next full-model chain
fusheng-ji 7bf6ef7
Merge test-qwennext refactor and migrate the Qwen3-Next batch-invaria…
fusheng-ji e485f0f
Merge the full-model chain evidence (c67ec14) into the migrated chain
fusheng-ji b66a8d0
build(pre-commit): leave the byte-exact Qwen3-Next chain evidence unt…
fusheng-ji 13ba74a
Merge the full-model chain evidence into the batch-invariance matrix
fusheng-ji 848ea44
build(pre-commit): leave the byte-exact batch-invariance evidence unt…
fusheng-ji 84a7bfe
test(qwen3-next): pin the case-manifest subprocess rootdir to its tem…
fusheng-ji 6684427
Merge the case-manifest test fix into the batch-invariance matrix
fusheng-ji cbb7b3d
docs(qwen3-next): figure for the full-model chain evidence
fusheng-ji e5a6987
Merge the chain evidence figure into the batch-invariance matrix
fusheng-ji 9cc2fbc
docs(qwen3-next): figure for the batch-invariance evidence
fusheng-ji bddabba
test(qwen3-next): digest every TP4 cross-rank point and compare acros…
fusheng-ji 99c244a
test(qwen3-next): require TP4 == TP1 for selected logprob and LSE only
fusheng-ji c5cea58
test(qwen3-next): prior art for the vocabulary-parallel selected-toke…
fusheng-ji da566bb
test(qwen3-next): no raw torch.distributed reduction in Qwen3-Next fo…
fusheng-ji f3a213d
docs(qwen3-next): TP distributed logprob contract and evidence figure
fusheng-ji 682c233
test(qwen3-next): publisher for the distributed logprob parity evidence
fusheng-ji 16171f5
docs(qwen3-next): record the prior-art NCCL-setting results and latency
fusheng-ji 46ea845
feat(qwen3-next): EP4 routed experts over the TP4 group
fusheng-ji 2314cc2
feat(qwen3-next): run the shared model at EP4 in vLLM rollout and VIM…
fusheng-ji 1106e46
refactor(qwen3-next): focus TP review on validation and externalize e…
fusheng-ji a9d0ba8
fix(qwen3-next): hash validation artifacts on Python 3.10
fusheng-ji 722ef1c
docs(qwen3-next): record post-review B200 validation results
fusheng-ji d8333f4
fix(validation): stream artifact SHA-256 on Python 3.10
fusheng-ji b5708e9
fix(validation): retain batch invariance with Python 3.10 hashing
fusheng-ji 7756969
fix(validation): keep standalone evidence runtime free of torch imports
fusheng-ji 6c7c49c
fix(validation): keep standalone evidence runtime free of torch imports
fusheng-ji 3c3891f
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji a7d5c88
Merge the Qwen3-Next gate-script root fix
fusheng-ji a8d2f64
Merge the Qwen3-Next gate-script root fix
fusheng-ji e9bf995
Merge the Qwen3-Next gate-script root fix
fusheng-ji 5a5aed4
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji 25ded88
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji ffa0d1d
fix(ci): resolve the repository root from ci/scripts/ in the Qwen3-Ne…
fusheng-ji caf7212
Merge the reviewed TP logprob PR into the EP branch and adopt its layout
fusheng-ji 343c8c9
Merge the batch-invariance fixes into the TP logprob PR
fusheng-ji 4746836
test(qwen3-next): drop the duplicate file_sha256 import left by the m…
fusheng-ji c78a535
Merge the TP logprob launcher fix into the EP branch
fusheng-ji 78b0db0
fix(qwen3-next): accept EP4 in the scheduler and L2 report checks
fusheng-ji 3185114
fix(qwen3-next): name the measured EP size in the chain report
fusheng-ji 6b1cb25
docs(qwen3-next): EP4 reduction contract and B200 evidence summary
fusheng-ji 4daf3c4
Merge the latest #469 branch into the MoE route branch
fusheng-ji 57216db
refactor(qwen3-next): MoE route branch to the reviewed layout
fusheng-ji f7338a7
Merge the MoE route layout change; TP4 mixers to the reviewed layout
fusheng-ji 4ceaa22
Merge the TP4 mixers layout change into the full-model chain branch
fusheng-ji 0d04347
refactor(qwen3-next): full-model chain to the reviewed layout
fusheng-ji 0f5266d
Merge the full-model chain layout change into the batch-invariance br…
fusheng-ji 299ea4c
refactor(qwen3-next): batch-invariance branch to the reviewed layout
fusheng-ji d18e0ff
Merge the batch-invariance layout change into the TP logprob branch
fusheng-ji 72ce0a9
Merge the TP logprob layout follow-up into the EP branch
fusheng-ji f05aaef
docs(qwen3-next): EP4 evidence ZIP joins the shared Qwen3-Next release
fusheng-ji d75112c
test(qwen3-next): vLLM model contract covers EP4
fusheng-ji 2cea4e9
test(qwen3-next): check the shared vLLM model with a real config and …
fusheng-ji 78b3e1a
test(qwen3-next): initialise vLLM's parallel state inside the config …
fusheng-ji e5b15c7
test(qwen3-next): real-model check uses the certified cache dtypes an…
fusheng-ji dda0c2e
test(qwen3-next): real-model check uses the BF16 KV cache and the rig…
fusheng-ji 1cb70f7
test(qwen3-next): build the real model under the loader's dtype and d…
fusheng-ji d8974d0
test(qwen3-next): real-model check builds the wrapper only
fusheng-ji 3aa220d
test(qwen3-next): preflight runs the real vLLM config and four-rank m…
fusheng-ji d84fbf3
style(qwen3-next): black-format the relocated runtime path in the dig…
fusheng-ji e7d08a8
Merge the digest-test formatting fix
fusheng-ji c86ce74
Merge the digest-test formatting fix
fusheng-ji fa42043
Merge the digest-test formatting fix
fusheng-ji 32040a7
docs(qwen3-next): record the full chain rerun on the stack top
fusheng-ji b59f628
Merge branch 'feat/ws1-qwen3-next-full-model-chain' into feat/ws1-qwe…
fusheng-ji 7b71b93
docs(qwen3-next): record the full BI suite rerun on the stack top
fusheng-ji 0165a26
Merge branch 'feat/ws1-qwen3-next-batch-invariance-ci' into feat/ws1-…
fusheng-ji 91dceda
Merge branch 'feat/ws1-qwen3-next-distributed-logprob-tp' into feat/w…
fusheng-ji c4674df
docs(qwen3-next): record the EP4 full BI suite on the stack top
fusheng-ji 4e00111
fix(qwen3-next): fail closed on a missing EP group and run the real-m…
fusheng-ji 8943c14
docs: move the measurement outputs out of the pull request
fusheng-ji 0fdbf80
docs: move the measurement outputs out of the pull request
fusheng-ji 46d5108
docs: move the measurement outputs out of the pull request
fusheng-ji 8fd657c
docs: move the measurement outputs out of the pull request
fusheng-ji 3d4fb2e
docs: move the measurement outputs out of the pull request
fusheng-ji d3d9dda
docs: move the measurement outputs out of the pull request
fusheng-ji a705d28
fix(qwen3-next): allow Hopper H100/H200 in the four-GPU preflight and…
fusheng-ji bc43113
docs: point commit hashes at the rewritten history
fusheng-ji f9b85d1
docs: point commit hashes at the rewritten history
fusheng-ji 4f61a39
docs: point commit hashes at the rewritten history
fusheng-ji 804f0a4
docs: point commit hashes at the rewritten history
fusheng-ji 78fc14a
docs: point commit hashes at the rewritten history
fusheng-ji 16c3767
docs: point commit hashes at the rewritten history
fusheng-ji 763e2f9
fix(qwen3-next): shorten qwen3_next_run docstring for flake8 E501
fusheng-ji 68240ec
Merge #469 after #467 landed in test-qwennext
fusheng-ji b36b8f6
Merge #503 after #467 landed in test-qwennext
fusheng-ji d243839
Merge #507 after #467 landed in test-qwennext
fusheng-ji eb04b15
Merge #543 after #467 landed in test-qwennext
fusheng-ji fe6ed38
Merge #544 after #467 landed in test-qwennext
fusheng-ji 8529700
Merge #549 after #467 landed in test-qwennext
fusheng-ji cfbd101
Merge #469 review fixes
fusheng-ji d53ac95
ci(qwen3-next): trigger the provider job on validation/common
fusheng-ji c9a0aed
Merge #503 review fixes
fusheng-ji b9390f3
ci(qwen3-next): trigger the provider job on the CUDA sources it builds
fusheng-ji 47b0c78
Merge #507 review fixes
fusheng-ji 73872d5
chore(qwen3-next): executable operator-gate script; say what is check…
fusheng-ji 9220f1a
Merge #543 review fixes
fusheng-ji b3bd4ea
Merge #544 review fixes
fusheng-ji 6d7d4bd
Merge #549 review fixes
fusheng-ji 6dc9d41
ci(qwen3-next): trigger the provider job on the logprob kernels and t…
fusheng-ji 9800e1a
Merge #469 second-round review fixes
fusheng-ji d849e18
Merge #503 second-round review fixes
fusheng-ji ecafcc2
Merge #507 second-round review fixes
fusheng-ji 349d6ba
Merge #543 second-round review fixes
fusheng-ji e20c29b
Merge #544 second-round review fixes
fusheng-ji 5ffd4c3
ci: run test_file_digest.py once in the CPU job
fusheng-ji 65eaae4
Merge #549 second-round review fixes
fusheng-ji d78b22b
ci(qwen3-next): trigger the provider job on the code this PR's checks…
fusheng-ji dfb0832
Merge #543 provider path-filter fix
fusheng-ji db1eb37
ci(qwen3-next): trigger the provider job on the distributed collectives
fusheng-ji 3f7cf6e
Merge #544 provider path-filter fixes
fusheng-ji 88c4f02
Merge #549 provider path-filter fixes
fusheng-ji 5294b76
Merge branch 'test-qwennext' into feat/ws1-qwen3-next-distributed-log…
Flink-ddd 62cc122
fix(workload): sync operator case ids with pytest collection
fusheng-ji fd87c0e
feat(vime): load the Megatron model without DDP or an optimizer for r…
fusheng-ji f2e9aa9
Merge #543 operator-case ids and no-DDP recompute into #544
fusheng-ji 7849c9c
Merge #544 operator-case ids and no-DDP recompute into #549
fusheng-ji 34e475f
Merge #549 operator-case ids and no-DDP recompute into #557
fusheng-ji 1c68102
fix(vime): pass skip_load_to_model_and_opt to the VIME checkpoint loader
fusheng-ji 1472e8f
Merge #543 load_checkpoint fix into #544
fusheng-ji dee7a78
Merge #544 load_checkpoint fix into #549
fusheng-ji 7261a37
Merge #549 load_checkpoint fix into #557
fusheng-ji 2e85d22
ci(qwen3-next): assert the full pinned runtime, retrigger on pyprojec…
fusheng-ji 74a1b1a
ci(qwen3-next): reject a malformed chain report with an explicit chec…
fusheng-ji 7fee591
Merge #543 OCR round fixes into #544
fusheng-ji 33e0ff6
Merge #544 OCR round fixes into #549
fusheng-ji 0a054e5
Merge #549 OCR round fixes into #557
fusheng-ji 4e2c33a
fix(ci): route unreadable chain reports to exit 2; fail closed on a m…
fusheng-ji 81be081
Merge fix/543-ocr-r8 into fix/544-ocr-r8
fusheng-ji 30c0e5a
Merge fix/544-ocr-r8 into fix/549-ocr-r8
fusheng-ji f935bf3
Merge fix/549-ocr-r8 into fix/557-ocr-r8
fusheng-ji File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # Qwen3-Next chain-report adapter checks; with a report path, also validate that report. | ||
| # bash ci/scripts/run_qwen3_next_chain_gate.sh [chain-report.json] | ||
| set -euo pipefail | ||
| ROOT="$(cd "$(dirname "$0")/../.." && pwd)" | ||
| cd "$ROOT" | ||
| PY="${PY:-python3}" | ||
| export PYTHONPATH="$ROOT${PYTHONPATH:+:$PYTHONPATH}" | ||
| "$PY" -m pytest -q tests/models/qwen3_next/test_qwen3_next_chain_report.py | ||
| if [ "$#" -ge 1 ]; then | ||
| "$PY" - "$1" <<'PY' | ||
| import dataclasses, json, sys | ||
|
|
||
| from rl_engine.validation.models.chain_gate import ChainGateReport | ||
| from rl_engine.validation.models.qwen3_next.qwen3_next_chain_report import COMPATIBLE_SCHEMA, SCHEMA_VERSION | ||
|
|
||
| report = json.load(open(sys.argv[1])) | ||
| missing = {f.name for f in dataclasses.fields(ChainGateReport)} - set(report) | ||
| assert not missing, f"chain report lacks ChainGateReport fields: {sorted(missing)}" | ||
| assert report["schema_version"] == SCHEMA_VERSION, report["schema_version"] | ||
| assert report["compatible_schema"] == COMPATIBLE_SCHEMA, report["compatible_schema"] | ||
| print(f"passed={report['passed']} first_drift={report['first_drift']} " | ||
| f"family={report['first_drift_family']} cells={len(report['cells'])}") | ||
| sys.exit(0 if report["passed"] else 1) | ||
| PY | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| #!/usr/bin/env bash | ||
| # Required operator acceptance and diagnostics use separate declared case sets. | ||
| set -euo pipefail | ||
| ROOT="$(cd "$(dirname "$0")/../.." && pwd)" | ||
| cd "$ROOT" | ||
| PY="${PY:-python3}" | ||
| OUT="${QWEN3_NEXT_EVIDENCE_DIR:?Set QWEN3_NEXT_EVIDENCE_DIR to an artifact directory}" | ||
| export PYTHONPATH="$ROOT${PYTHONPATH:+:$PYTHONPATH}" | ||
| exec "$PY" tools/validation/models/qwen3_next/qwen3_next_operator_gates.py --out "$OUT" \ | ||
| --suite "${QWEN3_NEXT_OPERATOR_SUITE:-all}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
grep -Eq '[0-9]+ passed'only proves that at least one test passed; it does not prove the intended provider modules actually ran. If a provider module errors during collection or fails to import (e.g. a missing symbol after a refactor), pytest reports that as an error, not a skip, and this check does not test forerror/failedcounts. So a run where most provider modules fail to collect but a single case in one module passes would exit 0 and be reported as green provider evidence.Suggest asserting an explicit minimum passed count (or that no
error/failedlines appear), so partial-collection regressions fail closed the way the skip check does.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not changed. The step runs under
bash -ewithset -o pipefail, so pytest's exit status decides first: a module that fails to import or collect makes pytest exit 2 with no tests run, a wrong path exits 4 and an empty selection exits 5, and each fails the step before the greps. There is no-k, noaddoptsand no--continue-on-collection-errors, so a partial collection cannot pass. The greps add the two cases in which pytest exits 0: any skip (e8daba5, r4372859876) and nothing passed.