Repository navigation
Conversation
TianHao65
force-pushed
the
qwen4-pr20-shared-gate-issue60
branch
from
November 3, 2026 04:33
61f3cb4 to
bbbada6
Compare
Fuse the BF16 gate GEMV, sigmoid, EP scale and shared-expert append for supported MI308X prefill batches with at least 256 rows. Pass ForwardBatch through the router and dual-stream paths; retain the existing fused Triton kernel for other phases and unsupported inputs. Port the native kernel from 1c019c6fb29d2de04d40ff21581846aab0688551 onto qwen4_v0.5.21 without the intermediate standalone gate implementation. Refs: apinge#60 Source: apinge#20 (native HIP follow-up) Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
TianHao65
force-pushed
the
qwen4-pr20-shared-gate-issue60
branch
from
November 3, 2026 07:34
bbbada6 to
29e5a09
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Refs #60. Port the previously validated native HIP shared-expert gate follow-up to PR20 onto
qwen4_v0.5.21. The target already contains shared-expert fusion; supported BF16/H4096 prefill batches with M>=256 now fuse gate GEMV, sigmoid, EP scale and top-k append in one HIP kernel.ForwardBatchreaches router, dual-stream and MegaMoE callers. Decode and unsupported inputs retain the existing fused Triton path.29e5a0918a, PR20 native follow-upRebased onto
apinge:qwen4_v0.5.21atc0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. The PR comparison now contains only one commit /three feature files. This remains a partial contribution to issue60; Qwen3.8 model validation is outstanding.Data
Historical October30 measurements, Qwen3.5-397B-A17B-PTPC-FP8, TP4 /4xMI308X, Torch2.12/ROCm7.14.60850. Baseline
e04b1a37-> candidate51565668; the feature files and model implementation match this new port exactly, but no fresh model run on this rebased head is claimed.Main pairs use GPUs0-3 twice/side and GPUs4-7 once/side after external GPU0 contexts interrupted startup before formal samples. Each pair uses the same GPU set; the original fixed-GPU three-start protocol was not completed. The three-pair descriptive aggregate ETE6040.355->6035.626ms (-0.08%), sample SD6.088/11.514ms, has inconsistent paired directions and does not establish stable full-model improvement. Supplemental rows have one start/side.
Benchmarking and Profiling
Six valid benchmark starts, 12 formal rounds /2368 requests /248 discarded warmups, all rounds>=60s. Non-speculative, radix disabled; actual KV262144/request64/Mamba128/graph32, effective memory fraction0.765, AITER attention/Triton GDN/NHD/page1/FP8 KV. Detailed per-start metrics and protocols are in the validation report.
Historical matched diagnostic profile: 8000 input /20 output, c1, TP4, rank0. Prefill screenshots select chronological gate call30 from each complete original trace. Rank0 gate median61.800->56.4985us and 60-call sum3708.234->3380.566us are local operator evidence, not TTFT/ETE attribution. Candidate prefill records60 HIP gates/rank; decode records60 original Triton gates/rank. Baseline decode is partial (48/38/41/49 gates/rank); decode images show first-recorded reachability examples, not matched-layer timing comparisons. Native importer diagnostics are preserved.
Prefill
Before
Open prefill before in Perfetto
After
Open prefill after in Perfetto
Decode
Before
Open decode before in Perfetto
After
Open decode after in Perfetto
Validation
Current rebase checks:21 CI CPU tests,10 evaluation shell syntax checks, clean diff and identical feature blobs. CI files come directly from the merged target; no CI patch is included. Reused same-code ROCm checks: 8 numerical/25 guard/30 phase-EP/3 changed-input graph/2 operator-profile cases passed, max absolute error7.7486e-7. Earlier lightweight real-class caller tests mocked expert/distributed arithmetic.
Historical full GSM8K1319:1288/1319 (97.6497%) ->1283/1319 (97.2707%), -5 answers /-0.3791 percentage points. Zero request errors; one cutoff at id119 retained per side. One incorrect->correct and six correct->incorrect; no no-regression claim. Taobao n1 passed the seven accepted automatic rules on both sides; semantic review remains pending, and different thinking lengths make its durations unsuitable as performance evidence. All quality runs used GPUs0-3. Task-owned October30 services were stopped and audited.
This remains Draft. Six-model GPU CI and Qwen3.8 model evaluation have not run for this source revision.
Detailed validation report · Artifact hashes · Original selections and importer diagnostics