Repository navigation
perf(gfx1250): drop a16w16's 4 GiB pre-check, fail on wrong results - #5074
Closed
JiaoliangYu wants to merge 1 commit into
Closed
JiaoliangYu wants to merge 1 commit into
JiaoliangYu wants to merge 1 commit into
Conversation
The bench skipped any a16w16 shape whose largest operand passed 4 GiB, on the stated grounds that "the heuristic refuses" it. That reads the guard too broadly. opus_dispatch_a16w16_gfx1250 (opus_gemm_arch_gfx1250.cuh:150-183) searches the tuned table first and returns on a hit. check_shape_4g runs only after that misses, on the way to the split-K heuristic kid, and it is that kid's launcher that builds the 32-bit gmem descriptors. A tuned 4wave_wl_co winner never reaches the check: per gen_instances_gfx1250.py:770-778 the pipeline "builds no gmem descriptor at all" and clamps every dimension through TDM descriptors instead. So 4 GiB bounds one fallback path, not a16w16, and a pre-check in Python keeps skipping shapes that tuning has already made runnable. Removed; the kernel raises if it must, and the exception is recorded as a row. The 20260827 sweep shows what the fallback costs. At N=129280 the tuned 4wave_wl_co kid does M=512 in 449us (2112 TFLOPS); M=1024 has no tuned winner, drops to split-K, and takes 3343us (568 TFLOPS) -- 7.4x slower for 2x the work. 11 of 60 shapes hit a 4wave_wl_co kid; the rest are split-K, so most of the low numbers in this table measure tuning coverage rather than the hardware. Widening that coverage is a job for csrc/gemm_a16w16/gemm_a16w16_tune.py --libtype opus, not for this file. Worse, split-K is not just slow at the top of the range: all four M=65536 shapes came back err=0.96-0.99, an unrelated result, while every other row was 0 or ~1e-5. None of them trip the 4 GiB guard (M*K*2 = 896 MB, M*N*2 <= 256 MB), and the UT neither raises nor warns -- it returns the ratio and prints a number. The sweep reported them as data. a16w16 now checks the returned ratio against _A16W16_MAX_ERR and calls _note_failure, so a silent miscompare shows up in the failed-op list. a16w16 also gets its own M list. The global sweep jumps 2048 -> 65536, so the prefill chunk sizes were never measured on the BF16 linears; _A16W16_MS adds 4096/8192/16384 and AITER_BENCH_TOKENS still overrides it. The lm_head cap stays. It is a statement about what DSv4 runs -- one row per sequence -- not about what the kernel can do, and its comment no longer leans on the 4 GiB number. Separately, put a8w8_blockscale back in --dsv4 and correct its note. The note blamed ROCm#4773's gluon tuning rows for the make_llir crash. The real cause is the UT's extra "ck strided x_scale" check (test_gemm_a8w8_blockscale.py:120), added by ROCm#4406 and gated on ck_preshuffle alone. The mxfp8_128 path declares its layout with is_x_scale_transposed=True and never reads the stride, so a strided x_scale tests nothing there and only gives triton a specialization that fails to compile. A/B with that line as the only variable, over a 162-case matrix (27 default M x six (n,k)): case 2 before it dies, case 160 after -- M=16 and M=64 included, which is what ROCm#4773 covers. Fixing it properly is upstream's call; meanwhile _A8W8_BLOCKSCALE_TOKENS starts at 1024, clear of the M that reach those rows. Verified on gfx1250-atom--20260827-ubench: 36/36 cases, err=0, 2207-7003 TFLOPS.
JiaoliangYu
force-pushed
the
gfx1250/a16w16-tune-skips
branch
from
August 28, 2026 08:06
14c6a1c to
0c836cb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
a16w16
opus_dispatch_a16w16_gfx1250searches the tunedtable first and returns on a hit;
check_shape_4gonly runs after that misses,on the way to the split-K heuristic kid whose launcher builds the 32-bit gmem
descriptors. A tuned
4wave_wl_cowinner never reaches it (that pipeline usesTDM descriptors, no gmem descriptor at all). So 4 GiB bounds one fallback path,
not a16w16 — predicting it in Python kept skipping shapes tuning had already
made runnable. The kernel now raises if it must, and that is recorded as a row.
every other row is 0 or ~1e-5. None trip the 4 GiB guard, and the UT neither
raises nor warns — it just returns the ratio. Those rows were printed as data.
Now checked against
_A16W16_MAX_ERRand reported via_note_failure.were never measured on the BF16 linears.
_A16W16_MSadds 4096/8192/16384.gfx1250_a16w16_untuned_gemm.csv(70 shapes) forgemm_a16w16_tune.py --libtype opus. Tuning coverage is what this op's numbersactually measure: at N=129280 the tuned kid does M=512 in 449us (2112 TFLOPS)
while M=1024 falls back to split-K at 3343us (568 TFLOPS).
a8w8_blockscale — back in
--dsv4The earlier note blamed #4773's gluon rows for the
make_llircrash. Wrong. Thecause is the UT's extra "ck strided x_scale" check
(
test_gemm_a8w8_blockscale.py:120), added by #4406 and gated onck_preshufflealone. The mxfp8_128 path declares its layout with
is_x_scale_transposed=Trueand never reads the stride, so a strided x_scale tests nothing there — it only
hands triton a
stride != 1specialization that fails to compile.A/B with that line as the only variable, over a 162-case matrix: dies on case 2
as written, reaches case 160 with
.view(). M=16 and M=64 included, which isexactly what #4773 covers — the gluon kernel compiles and runs fine.
The real fix belongs upstream (
ck_preshuffle and not use_flydsl_fp8_scale).Meanwhile
_A8W8_BLOCKSCALE_TOKENSstarts at 1024, clear of the M that reachthose rows. Verified on
gfx1250-atom--20260827-ubench: 36/36, err=0,2207-7003 TFLOPS.