Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions benchmark/rocm_tgemm/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Validated gfx942 linear configurations

This reproduces the completed PTPC FP8, CK decode and BF16 ViT tuning work for
Qwen3.8 TP2 on MI308X (`gfx942`, 80 CUs). It uses the existing AITER dispatcher and
tuners; it does not vendor AITER or change its GEMM algorithms.

The recorded runtime was AITER `514c166e9abad59acde74309923c2d923a89eee0`, Torch
2.12 / ROCm 7.14 / Triton 3.8. Solution indexes and CK kernel IDs belong to that
runtime. Keep the version and hardware identity with each measurement.

## Prepare complete runtime tables

`configs/ck_accepted_rows.csv` contains the one validated CK override;
`configs/tgemm_accepted_rows.csv` contains the four validated FP8/BF16 overrides.
They are **deltas**, not complete runtime tables. Preserve all other baseline
entries by merging them before setting the AITER environment variables:

```bash
python3 benchmark/rocm_tgemm/prepare_configs.py \
--ck-base /path/to/recorded/ck_base.csv \
--tgemm-base /path/to/recorded/tgemm_base.csv \
--output-dir /path/to/fresh/merged-configs

export AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE=/path/to/fresh/merged-configs/ck_accepted.csv
export AITER_CONFIG_GEMM_BF16=/path/to/fresh/merged-configs/tgemm_accepted.csv
export SGLANG_USE_AITER=1
export SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1
export SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1
```

Supply the baseline tables from the verified AITER runtime. The helper records
input/overlay/output hashes, preserves unmatched entries and rejects duplicate
shape keys or input/output overwrites. It also copies the two baseline files for
the before measurement. Complete baseline snapshots and raw results belong to
the separate PR evidence branch, rather than this source directory.

## Reproduce the native comparison

Choose a free reserved GPU and inspect utilization, VRAM and process ownership
before running. The following uses the recorded five eager and three graph
samples per shape, 10 warmups / 101 iterations, 16 rotating tensor sets, and two
changed-input graph replays. Numerical output is checked against dense FP32
arithmetic. All 16 shapes are retained, including unchanged-path controls.

```bash
export HIP_VISIBLE_DEVICES=0
export AITER_USE_SYSTEM_TRITON=1
python3 benchmark/rocm_tgemm/benchmark_gemm.py \
--configs-dir /path/to/fresh/merged-configs \
--phase base \
--output /path/to/results/base.json
python3 benchmark/rocm_tgemm/benchmark_gemm.py \
--configs-dir /path/to/fresh/merged-configs \
--phase accepted \
--output /path/to/results/accepted.json
```

These are operator measurements, not model TTFT/TPOT/ETE or GSM8K. Use the same
original AITER source and prepared kernels for both phases; do not compare one
prebuilt cache with a differently patched rebuilt runtime.

## Reproduce all six regular CI cases

Native `qwen4_ci.py` cleans inherited environment variables. Use this thin wrapper
so the two flags and prepared table paths reach the **server**; client environment,
workloads and accuracy gates remain native and unchanged. The before/accepted
phase selects tables, not a source checkout: run the matching source revision in
each comparison arm and record that revision separately.

```bash
python3 benchmark/rocm_tgemm/run_ci.py \
--configs-dir /path/to/fresh/merged-configs \
--phase accepted \
run \
--case Qwen3.8-Flash-TP2-Base,Qwen3.8-Flash-TP2-MTP,Qwen3.5-35B-TP4-Base,Qwen3.5-35B-TP1-DFlash,Qwen3.5-397B-TP8-Base,Qwen3.5-397B-TP8-DFlash \
--models-root /models \
--datasets-root /datasets \
--client-root /ci/client \
--gpus 0,1,2,3,4,5,6,7 \
--output /path/to/results/ci-accepted
```

Use `--phase base` with another output directory for the baseline tables. This
runs the existing 128-request / 200-example regular CI protocol; it does not alter
the native case selection, GPU mask, benchmark specifications or accuracy gates.

## Tuner entry points and accepted scope

The input files contain nine CK decode shapes (`M=1/4/16`) and seven tgemm shapes
(three FP8 prefill shapes plus four BF16 ViT shapes). Existing AITER commands were:

```bash
python3 /path/to/aiter/csrc/ck_gemm_a8w8_bpreshuffle/gemm_a8w8_bpreshuffle_tune.py \
-i benchmark/rocm_tgemm/configs/ck_decode_shapes.csv \
-o /path/to/results/ck_decode_tuned.csv \
--profile_file /path/to/results/ck_candidates.csv \
--libtype ck \
--splitK \
--mp 1 \
--warmup 10 \
--iters 101 \
--errRatio 0.01
python3 /path/to/aiter/csrc/gemm_a16w16/gemm_a16w16_tune.py \
-i benchmark/rocm_tgemm/configs/tgemm_shapes.csv \
-o /path/to/results/tgemm_tuned.csv \
--profile_file /path/to/results/tgemm_candidates.csv \
--libtype hipblaslt,asm,torch \
--with-hipblaslt \
--mp 1 \
--errRatio 0.01
```

The historical AITER search needed two temporary tuning-only fixes: BF16
`scaleAB=False` must supply no scale tensors, and unsupported hipBLASLt solutions
must raise instead of aborting the worker. Those patches/checkpoint recovery are
retained as external tuning evidence. Apply them only in an isolated AITER tuning
checkout if reproducing that exact search, and restore the original source before
production-interface verification. This source kit applies already validated
rows; it never patches the installed library or automatically accepts tuner output.

The selected changes are one CK row `(4,2560,3072)`; two FP8 rows
`(8192,8192,2560)` / `(8192,6656,2560)`; and two BF16 rows
`(256,1152,2152)` / `(256,1152,576)`. The other candidates were not accepted.
Selection required at least 3% lower median latency in both eager and graph
comparison, followed by numerical and changed-input replay validation. Re-running
a tuner produces new experimental data, not automatically validated replacements.
203 changes: 203 additions & 0 deletions benchmark/rocm_tgemm/benchmark_gemm.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
"""Reproduce the recorded 16-shape AITER comparison; use a free, reserved gfx942 GPU."""

import argparse
import csv
import json
import os
from pathlib import Path


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--configs-dir", type=Path, required=True)
parser.add_argument("--phase", choices=("base", "accepted"), required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
if args.output.exists():
raise ValueError("Output already exists; preserve the previous measurement")
os.environ["AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE"] = str(
(args.configs_dir / ("ck_" + args.phase + ".csv")).resolve()
)
os.environ["AITER_CONFIG_GEMM_BF16"] = str(
(args.configs_dir / ("tgemm_" + args.phase + ".csv")).resolve()
)
for name in ("AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE", "AITER_CONFIG_GEMM_BF16"):
if not Path(os.environ[name]).is_file():
raise FileNotFoundError(
"Prepare the complete AITER configuration tables first"
)

import aiter
import torch
from aiter.jit.core import AITER_CONFIGS
from aiter.jit.utils.chip_info import get_cu_num, get_gfx
from aiter.ops.shuffle import shuffle_weight
from aiter.test_common import run_perftest
from aiter.tuned_gemm import get_GEMM_A16W16_config, tgemm

ROOT = Path(__file__).resolve().parent
if get_gfx() != "gfx942" or get_cu_num() != 80:
raise ValueError("Recorded configurations require gfx942 / 80 CUs")
args.output.parent.mkdir(parents=True, exist_ok=True)
torch.manual_seed(42)
torch.set_grad_enabled(False)
records = []

def check(out, ref):
assert out.shape == ref.shape and out.dtype == torch.bfloat16
assert torch.isfinite(out).all()
rel = float(
(out.float() - ref.float()).norm() / ref.float().norm().clamp_min(1e-8)
)
assert rel < 0.008, rel
return rel

shapes = []
for row in csv.DictReader((ROOT / "configs/ck_decode_shapes.csv").open()):
shapes.append(
(
"ck_decode",
int(row["M"]),
int(row["N"]),
int(row["K"]),
True,
False,
True,
)
)
for row in csv.DictReader((ROOT / "configs/tgemm_shapes.csv").open()):
shapes.append(
(
"tgemm",
int(row["M"]),
int(row["N"]),
int(row["K"]),
row["scaleAB"] == "True",
row["bias"] == "True",
row["bpreshuffle"] == "True",
)
)

for family, m, n, k, fp8, has_bias, packed in shapes:
x = torch.randn(m, k, device="cuda", dtype=torch.bfloat16)
w = torch.randn(n, k, device="cuda", dtype=torch.bfloat16)
bias = torch.randn(n, device="cuda", dtype=torch.bfloat16) if has_bias else None
if fp8:
x, sx = aiter.per_token_quant_hip(x, quant_dtype=aiter.dtypes.fp8)
w, sw = aiter.per_token_quant_hip(w, quant_dtype=aiter.dtypes.fp8)
canonical_w = w.float() * sw
ref = (x.float() * sx) @ canonical_w.t()
else:
sx = sw = None
canonical_w = w.float()
ref = x.float() @ canonical_w.t()
if bias is not None:
ref += bias.float()
wp = shuffle_weight(w, (16, 16)) if packed else w
if packed:
wp.is_shuffled = True

def fn(inp, weight, scale_a, scale_b, bias_tensor):
if family == "ck_decode":
return aiter.gemm_a8w8_bpreshuffle(
inp, weight, scale_a, scale_b, None, torch.bfloat16
)
return tgemm.mm(
inp,
weight,
bias_tensor,
otype=torch.bfloat16,
scale_a=scale_a,
scale_b=scale_b.t() if scale_b is not None else None,
)

kernel_args = (x, wp, sx, sw, bias)
out = fn(*kernel_args)
relative_l2 = check(out, ref)
samples = []
for _ in range(5):
out, us = run_perftest(
fn, *kernel_args, num_warmup=10, num_iters=101, num_rotate_args=16
)
check(out, ref)
samples.append(float(us))
graph_samples = []
for _ in range(3):
out, us = run_perftest(
fn,
*kernel_args,
num_warmup=10,
num_iters=101,
num_rotate_args=16,
testGraph=True,
)
check(out, ref)
graph_samples.append(float(us))

graph = torch.cuda.CUDAGraph()
torch.cuda.synchronize()
with torch.cuda.graph(graph):
graph_out = fn(*kernel_args)
graph_errors = []
for seed in [43, 44]:
torch.manual_seed(seed)
changed = torch.randn(m, k, device="cuda", dtype=torch.bfloat16)
if fp8:
q, qs = aiter.per_token_quant_hip(changed, quant_dtype=aiter.dtypes.fp8)
x.copy_(q)
sx.copy_(qs)
reference = (x.float() * sx) @ canonical_w.t()
else:
x.copy_(changed)
reference = x.float() @ canonical_w.t()
if bias is not None:
reference += bias.float()
graph.replay()
torch.cuda.synchronize()
graph_errors.append(check(graph_out, reference))
if family == "tgemm":
config = get_GEMM_A16W16_config(
m, n, k, has_bias, str(x.dtype), str(torch.bfloat16), fp8, packed
)
else:
from aiter.ops.gemm_op_a8w8 import get_GEMM_config_with_quant_type

config = get_GEMM_config_with_quant_type(
m,
n,
k,
str(w.dtype),
AITER_CONFIGS.AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE_FILE,
)
record = dict(
family=family,
M=m,
N=n,
K=k,
dtype=str(x.dtype),
bias=has_bias,
packed=packed,
us=samples,
graph_us=graph_samples,
relative_l2=relative_l2,
changed_input_graph_relative_l2=graph_errors,
config=config,
)
records.append(record)
print(json.dumps(record, default=str), flush=True)
args.output.write_text(json.dumps(records, indent=2, default=str))
del graph, graph_out, out, ref, wp, w, x, canonical_w
torch.cuda.empty_cache()

print(
"PASS",
len(records),
"GEMMs;",
len(records) * 2,
"changed-input graph replays",
flush=True,
)


if __name__ == "__main__":
main()
2 changes: 2 additions & 0 deletions benchmark/rocm_tgemm/configs/ck_accepted_rows.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
gfx,cu_num,M,N,K,q_dtype_w,libtype,kernelId,splitK,us,kernelName,tflops,bw,errRatio
gfx942,80,4,2560,3072,torch.float8_e4m3fnuz,ck,10,0,7.1939,a8w8_bpreshuffle_128x16x32x512_16x16_16x16_32x4x1_32x4x1_1x16x1x8_4x4x1_1x1_intrawave_v1,8.75,1097.75,0.0
10 changes: 10 additions & 0 deletions benchmark/rocm_tgemm/configs/ck_decode_shapes.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
M,N,K,q_dtype_w
1,8192,2560,torch.float8_e4m3fnuz
4,8192,2560,torch.float8_e4m3fnuz
16,8192,2560,torch.float8_e4m3fnuz
1,2560,3072,torch.float8_e4m3fnuz
4,2560,3072,torch.float8_e4m3fnuz
16,2560,3072,torch.float8_e4m3fnuz
1,6656,2560,torch.float8_e4m3fnuz
4,6656,2560,torch.float8_e4m3fnuz
16,6656,2560,torch.float8_e4m3fnuz
5 changes: 5 additions & 0 deletions benchmark/rocm_tgemm/configs/tgemm_accepted_rows.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
gfx,cu_num,M,N,K,bias,dtype,outdtype,scaleAB,bpreshuffle,libtype,solidx,splitK,us,kernelName,err_ratio,tflops,bw
gfx942,80,8192,8192,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True,hipblaslt,169251,0,701.2096,Cijk_Alik_Bljk_F8NBS_STA_BH_SABV_UserArgs_MT256x96x128_MI16x16x1_SN_LDSB0_AFC0_AG0_AGGSUA0_AGNTAB0_AFEM1_AFEM1_ASEM1_CD1_1_CLR1_CLS0_CADS0_DTLA0_DTLB0_DTLM0_DTVA1_DTVB0_DTVMXSA0_DTVMXSB0_DTVSM0_DPLB0_EPS0_ELFLR0_EMLLn1_FDSI0_GRPM1_GRVWA16_GRVWB16_GSUAMBSK_GLS0_HPLR0_ISA942_ICIW0_IU1_K1_LDSTI0_LBSPPA0_LBSPPB256_LBSPPMXSA0_LBSPPMXSB0_LBSPPM0_LPA0_LPB32_LPMXSA0_LPMXSB0_LPM0_LRVW16_LWPMn1_MIAV0_MIWT4_6_MXLIBL_MXSFNS_MO40_MGRIPM1_NTn1_NTA0_NTB0_NTC0_NTD4_NTE0_NTMXSA0_NTMXSB0_NTM0_NTWS0_NVn1_NVA0_NVB0_NVC0_NVD0_NVE0_NVMXSA0_NVMXSB0_NVM0_NVWS0_NEPBS16_NLCA2_NLCB1_ONLL0_PAP0_PGL0_PGR2_PLR1_PKA1_SGROB0_SIA3_SS1_SPO0_SRVW0_SSO0_SVW1_SK0_SKFTR0_SKFDPO0_SKXCCM0_SNLL0_SIP1_SGRO0_TDMI0_TDMIM0_TDMS0_TIN0_THn1_THA0_THB0_THC0_THD0_THE0_THMXSA0_THMXSB0_THM0_THWS0_TLDS1_TLDSM1_ULSGRO0_USL1_USLMX0_UIOFGRO0_UPLRP0_USFGROn1_USI0_VSn1_VWA1_VWB2_WSGRA0_WSGRB0_WS64_WG64_4_1,0.0,490.01,311.04
gfx942,80,8192,6656,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True,hipblaslt,169251,0,575.4129,Cijk_Alik_Bljk_F8NBS_STA_BH_SABV_UserArgs_MT256x96x128_MI16x16x1_SN_LDSB0_AFC0_AG0_AGGSUA0_AGNTAB0_AFEM1_AFEM1_ASEM1_CD1_1_CLR1_CLS0_CADS0_DTLA0_DTLB0_DTLM0_DTVA1_DTVB0_DTVMXSA0_DTVMXSB0_DTVSM0_DPLB0_EPS0_ELFLR0_EMLLn1_FDSI0_GRPM1_GRVWA16_GRVWB16_GSUAMBSK_GLS0_HPLR0_ISA942_ICIW0_IU1_K1_LDSTI0_LBSPPA0_LBSPPB256_LBSPPMXSA0_LBSPPMXSB0_LBSPPM0_LPA0_LPB32_LPMXSA0_LPMXSB0_LPM0_LRVW16_LWPMn1_MIAV0_MIWT4_6_MXLIBL_MXSFNS_MO40_MGRIPM1_NTn1_NTA0_NTB0_NTC0_NTD4_NTE0_NTMXSA0_NTMXSB0_NTM0_NTWS0_NVn1_NVA0_NVB0_NVC0_NVD0_NVE0_NVMXSA0_NVMXSB0_NVM0_NVWS0_NEPBS16_NLCA2_NLCB1_ONLL0_PAP0_PGL0_PGR2_PLR1_PKA1_SGROB0_SIA3_SS1_SPO0_SRVW0_SSO0_SVW1_SK0_SKFTR0_SKFDPO0_SKXCCM0_SNLL0_SIP1_SGRO0_TDMI0_TDMIM0_TDMS0_TIN0_THn1_THA0_THB0_THC0_THD0_THE0_THMXSA0_THMXSB0_THM0_THWS0_TLDS1_TLDSM1_ULSGRO0_USL1_USLMX0_UIOFGRO0_UPLRP0_USFGROn1_USI0_VSn1_VWA1_VWB2_WSGRA0_WSGRB0_WS64_WG64_4_1,0.0,485.17,321.64
gfx942,80,256,1152,2152,True,torch.bfloat16,torch.bfloat16,False,False,hipblaslt,82636,0,17.0225,Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT64x64x128_MI16x16x1_SN_LDSB1_AFC0_AG0_AGGSUA0_AGNTAB0_AFEM1_AFEM1_ASEM1_CD1_1_CLR1_CLS0_CADS0_DTLA0_DTLB0_DTLM0_DTVA0_DTVB0_DTVMXSA0_DTVMXSB0_DTVSM0_DPLB0_EPS0_ELFLR0_EMLLn1_FDSI0_GRPM1_GRVWA8_GRVWB8_GSUAMB_GLS0_HPLR0_ISA942_ICIW0_IU1_K1_LDSTI0_LBSPPA1024_LBSPPB512_LBSPPMXSA0_LBSPPMXSB0_LBSPPM0_LPA16_LPB16_LPMXSA0_LPMXSB0_LPM0_LRVW8_LWPMn1_MIAV0_MIWT4_2_MXLIBL_MXSFNS_MO40_MGRIPM1_NTn1_NTA0_NTB0_NTC0_NTD7_NTE0_NTMXSA0_NTMXSB0_NTM0_NTWS0_NVn1_NVA0_NVB0_NVC0_NVD0_NVE0_NVMXSA0_NVMXSB0_NVM0_NVWS0_NEPBS8_NLCA1_NLCB1_ONLL1_PAP0_PGL0_PGR2_PLR1_PKA1_SGROB0_SIA3_SS1_SPO0_SRVW0_SSO0_SVW4_SK0_SKFTR0_SKFDPO0_SKXCCM0_SNLL0_SIP1_SGRO0_TDMI0_TDMIM0_TDMS0_TIN0_THn1_THA0_THB0_THC0_THD0_THE0_THMXSA0_THMXSB0_THM0_THWS0_TLDS1_TLDSM1_ULSGRO1_USL1_USLMX0_UIOFGRO0_UPLRP0_USFGROn1_USI0_VSn1_VWA4_VWB2_WSGRA0_WSGRB0_WS64_WG16_8_2,0.0,74.57,390.65
gfx942,80,256,1152,576,True,torch.bfloat16,torch.bfloat16,False,True,asm,5,1,5.9612,_ZN5aiter42bf16gemm_fp32bf16_tn_64x64_bshuffle_splitkE,0.0,56.99,371.04
8 changes: 8 additions & 0 deletions benchmark/rocm_tgemm/configs/tgemm_shapes.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
M,N,K,bias,dtype,outdtype,scaleAB,bpreshuffle
8192,8192,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True
8192,2560,3072,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True
8192,6656,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True
256,1728,1152,True,torch.bfloat16,torch.bfloat16,False,True
256,2152,1152,True,torch.bfloat16,torch.bfloat16,False,False
256,1152,2152,True,torch.bfloat16,torch.bfloat16,False,False
256,1152,576,True,torch.bfloat16,torch.bfloat16,False,True
Loading