Skip to content

[ROCm] Port PR20 native shared-expert gate follow-up to qwen4 - #83

Draft
TianHao65 wants to merge 1 commit into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr20-shared-gate-issue60
Draft

TianHao65 wants to merge 1 commit into
apinge:qwen4_v0.5.21from
TianHao65:qwen4-pr20-shared-gate-issue60

Conversation

@TianHao65

@TianHao65 TianHao65 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What

Refs #60. Port the previously validated native HIP shared-expert gate follow-up to PR20 onto qwen4_v0.5.21. The target already contains shared-expert fusion; supported BF16/H4096 prefill batches with M>=256 now fuse gate GEMV, sigmoid, EP scale and top-k append in one HIP kernel. ForwardBatch reaches router, dual-stream and MegaMoE callers. Decode and unsupported inputs retain the existing fused Triton path.

Commit / source Change
29e5a0918a, PR20 native follow-up One feature commit, three files, +216/-6

Rebased onto apinge:qwen4_v0.5.21 at c0a27e481010cd2f285fe426eed35860ae1920a5. CI PR65 is merged into this target; all separately cherry-picked CI and stacked-CI compatibility commits were removed. The PR comparison now contains only one commit /three feature files. This remains a partial contribution to issue60; Qwen3.8 model validation is outstanding.

Data

Historical October30 measurements, Qwen3.5-397B-A17B-PTPC-FP8, TP4 /4xMI308X, Torch2.12/ROCm7.14.60850. Baseline e04b1a37 -> candidate 51565668; the feature files and model implementation match this new port exactly, but no fresh model run on this rebased head is claimed.

Workload / scope c Output tok/s before → after TTFT ms before → after TPOT ms before → after ETE ms before → after
8000/500, GPUs0–3, two starts/side 1 82.722 → 82.908 (+0.22%) 595.788 → 594.443 (-0.23%) 10.915 → 10.891 (-0.22%) 6042.514 → 6028.996 (-0.22%)
8000/500, GPUs4–7, one start/side 1 82.810 → 82.633 (-0.21%) 595.656 → 595.591 (-0.01%) 10.903 → 10.928 (+0.24%) 6036.038 → 6048.888 (+0.21%)
512/256, GPUs0–3, one start/side 1 85.685 → 86.212 (+0.62%) 216.879 → 204.376 (-5.77%) 10.861 → 10.839 (-0.20%) 2986.355 → 2968.215 (-0.61%)
512/256, GPUs0–3, one start/side 8 437.179 → 439.622 (+0.56%) 486.124 → 453.107 (-6.79%) 16.451 → 16.475 (+0.14%) 4681.171 → 4654.224 (-0.58%)
512/256, GPUs0–3, one start/side 16 695.914 → 694.473 (-0.21%) 703.505 → 713.910 (+1.48%) 20.297 → 20.305 (+0.04%) 5879.348 → 5891.665 (+0.21%)

Main pairs use GPUs0-3 twice/side and GPUs4-7 once/side after external GPU0 contexts interrupted startup before formal samples. Each pair uses the same GPU set; the original fixed-GPU three-start protocol was not completed. The three-pair descriptive aggregate ETE6040.355->6035.626ms (-0.08%), sample SD6.088/11.514ms, has inconsistent paired directions and does not establish stable full-model improvement. Supplemental rows have one start/side.

Benchmarking and Profiling

Six valid benchmark starts, 12 formal rounds /2368 requests /248 discarded warmups, all rounds>=60s. Non-speculative, radix disabled; actual KV262144/request64/Mamba128/graph32, effective memory fraction0.765, AITER attention/Triton GDN/NHD/page1/FP8 KV. Detailed per-start metrics and protocols are in the validation report.

Historical matched diagnostic profile: 8000 input /20 output, c1, TP4, rank0. Prefill screenshots select chronological gate call30 from each complete original trace. Rank0 gate median61.800->56.4985us and 60-call sum3708.234->3380.566us are local operator evidence, not TTFT/ETE attribution. Candidate prefill records60 HIP gates/rank; decode records60 original Triton gates/rank. Baseline decode is partial (48/38/41/49 gates/rank); decode images show first-recorded reachability examples, not matched-layer timing comparisons. Native importer diagnostics are preserved.

Prefill

Before

Prefill before — original TP4 trace

Open prefill before in Perfetto

After

Prefill after — original TP4 trace

Open prefill after in Perfetto

Decode

Before

Decode before — original TP4 trace

Open decode before in Perfetto

After

Decode after — original TP4 trace

Open decode after in Perfetto

Validation

Current rebase checks:21 CI CPU tests,10 evaluation shell syntax checks, clean diff and identical feature blobs. CI files come directly from the merged target; no CI patch is included. Reused same-code ROCm checks: 8 numerical/25 guard/30 phase-EP/3 changed-input graph/2 operator-profile cases passed, max absolute error7.7486e-7. Earlier lightweight real-class caller tests mocked expert/distributed arithmetic.

Historical full GSM8K1319:1288/1319 (97.6497%) ->1283/1319 (97.2707%), -5 answers /-0.3791 percentage points. Zero request errors; one cutoff at id119 retained per side. One incorrect->correct and six correct->incorrect; no no-regression claim. Taobao n1 passed the seven accepted automatic rules on both sides; semantic review remains pending, and different thinking lengths make its durations unsuitable as performance evidence. All quality runs used GPUs0-3. Task-owned October30 services were stopped and audited.

This remains Draft. Six-model GPU CI and Qwen3.8 model evaluation have not run for this source revision.

Detailed validation report · Artifact hashes · Original selections and importer diagnostics

@TianHao65
TianHao65 force-pushed the qwen4-pr20-shared-gate-issue60 branch from 61f3cb4 to bbbada6 Compare November 3, 2026 04:33
Fuse the BF16 gate GEMV, sigmoid, EP scale and shared-expert append for supported MI308X prefill batches with at least 256 rows. Pass ForwardBatch through the router and dual-stream paths; retain the existing fused Triton kernel for other phases and unsupported inputs.

Port the native kernel from 1c019c6fb29d2de04d40ff21581846aab0688551 onto qwen4_v0.5.21 without the intermediate standalone gate implementation.

Refs: apinge#60
Source: apinge#20 (native HIP follow-up)
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
@TianHao65
TianHao65 force-pushed the qwen4-pr20-shared-gate-issue60 branch from bbbada6 to 29e5a09 Compare November 3, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant