Repository navigation
Conversation
Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
… entrypoints Signed-off-by: TianHao65 <283868507+TianHao65@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #88; follows #63 on
qwen4_v0.5.21(9ac4ebe). Route supported FP8/BF16 prefill through AITER tgemm, retain CK for FP8M <= 16decode, and preserve dense BF16 weights for DFlash. Include validated tuning and reproduction tools.f315e8d05cd615The two linear flags default off. Use the tuning kit to merge selected rows into complete AITER tables.
Data
Nov12 local prototype CI: PR63
8511084989b4→ same source plus tgemm and accepted tables. MI308X/PTL34 and PTL52; Torch 2.12 / ROCm 7.14, AITER514c166, pyhip8027faa, clientfc6ddbb. Con1, random inputs, configured 12000/350 for Qwen3.8 and 8000/500 for Qwen3.5; 10 warmups,128 performance requests and200 GSM8K questions per case/arm. One startup/arm; cross-start noise unestimated. Both arms use the flags above. These results predate final integration/tuple fixes; percentages use unrounded values.35B/TP4's +10.29% mean ETE regression includes a sustained late slowdown; its trigger remains unresolved.
Follow-ups: two starts/arm in ABBA order, con1. Nov10 changes only tuning tables on image
4d63dea/ PTL34: Qwen3.8 TP2, generator means 11000/350, sigma=4, prefix=0.35, MTP/radix on. Nov16 repeats the frozen851 source pair on PTL52: 35B TP4, 8000/500, seed42, radix off, decode Graph on. Both use pool262144; values average the two startup metrics.The initial untuned Qwen3.8 TP2 prototype regressed ETE by 3.52% versus transplanted PR63. The after-only large-pool 35B control measured 2620.343ms ETE; it does not establish the old slowdown's cause. Full comparisons, QPS/acceptance, gates and protocols retain these distinct baselines, all tuning candidates and unmeasured configurations. Custom 11K/350 specifies generator means, not exact token lengths or cache hit rates.
Benchmarking and Profiling
Complete original traces: frozen851 before/prototype after, Quark Qwen3.5-35B, TP1/rank0/PTL52 GPU6, identical 8000 input IDs/64 output tokens, con1, two warmups/start, decode Graph on, radix off. These operator samples are separate from model ETE and final-head CI.
Prefill shapes use CPU/GPU External-id matching; decode shapes/layer attribution are inferred from the original graph/source order. Read each GPU row, not summed overlapping scopes.
Prefill
Before:
Open prefill before in Perfetto.
After:
Open prefill after in Perfetto.
Decode
Before:
Open decode before in Perfetto.
After:
Open decode after in Perfetto.
Complete originals, sample positions and SHA256.
Validation
f315e8d, unchanged in05cd615: 8 CPU regressions,16 native caller cases/32 changed-input Graph replays passed (max relative L2 0.002740). The tuning kit's 10 CPU checks and changed-file syntax/isort/Ruff/registry checks passed. Repository-wide registry retains an unchanged base taxonomy error intest_pyhip_qsa.py.Final-head six-model GPU CI, daily pressure and full 1319-question GSM8K remain unmeasured.