diff --git a/benchmark/rocm_tgemm/README.md b/benchmark/rocm_tgemm/README.md new file mode 100644 index 000000000000..f47978fd06dc --- /dev/null +++ b/benchmark/rocm_tgemm/README.md @@ -0,0 +1,126 @@ +# Validated gfx942 linear configurations + +This reproduces the completed PTPC FP8, CK decode and BF16 ViT tuning work for +Qwen3.8 TP2 on MI308X (`gfx942`, 80 CUs). It uses the existing AITER dispatcher and +tuners; it does not vendor AITER or change its GEMM algorithms. + +The recorded runtime was AITER `514c166e9abad59acde74309923c2d923a89eee0`, Torch +2.12 / ROCm 7.14 / Triton 3.8. Solution indexes and CK kernel IDs belong to that +runtime. Keep the version and hardware identity with each measurement. + +## Prepare complete runtime tables + +`configs/ck_accepted_rows.csv` contains the one validated CK override; +`configs/tgemm_accepted_rows.csv` contains the four validated FP8/BF16 overrides. +They are **deltas**, not complete runtime tables. Preserve all other baseline +entries by merging them before setting the AITER environment variables: + +```bash +python3 benchmark/rocm_tgemm/prepare_configs.py \ + --ck-base /path/to/recorded/ck_base.csv \ + --tgemm-base /path/to/recorded/tgemm_base.csv \ + --output-dir /path/to/fresh/merged-configs + +export AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE=/path/to/fresh/merged-configs/ck_accepted.csv +export AITER_CONFIG_GEMM_BF16=/path/to/fresh/merged-configs/tgemm_accepted.csv +export SGLANG_USE_AITER=1 +export SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=1 +export SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE=1 +``` + +Supply the baseline tables from the verified AITER runtime. The helper records +input/overlay/output hashes, preserves unmatched entries and rejects duplicate +shape keys or input/output overwrites. It also copies the two baseline files for +the before measurement. Complete baseline snapshots and raw results belong to +the separate PR evidence branch, rather than this source directory. + +## Reproduce the native comparison + +Choose a free reserved GPU and inspect utilization, VRAM and process ownership +before running. The following uses the recorded five eager and three graph +samples per shape, 10 warmups / 101 iterations, 16 rotating tensor sets, and two +changed-input graph replays. Numerical output is checked against dense FP32 +arithmetic. All 16 shapes are retained, including unchanged-path controls. + +```bash +export HIP_VISIBLE_DEVICES=0 +export AITER_USE_SYSTEM_TRITON=1 +python3 benchmark/rocm_tgemm/benchmark_gemm.py \ + --configs-dir /path/to/fresh/merged-configs \ + --phase base \ + --output /path/to/results/base.json +python3 benchmark/rocm_tgemm/benchmark_gemm.py \ + --configs-dir /path/to/fresh/merged-configs \ + --phase accepted \ + --output /path/to/results/accepted.json +``` + +These are operator measurements, not model TTFT/TPOT/ETE or GSM8K. Use the same +original AITER source and prepared kernels for both phases; do not compare one +prebuilt cache with a differently patched rebuilt runtime. + +## Reproduce all six regular CI cases + +Native `qwen4_ci.py` cleans inherited environment variables. Use this thin wrapper +so the two flags and prepared table paths reach the **server**; client environment, +workloads and accuracy gates remain native and unchanged. The before/accepted +phase selects tables, not a source checkout: run the matching source revision in +each comparison arm and record that revision separately. + +```bash +python3 benchmark/rocm_tgemm/run_ci.py \ + --configs-dir /path/to/fresh/merged-configs \ + --phase accepted \ + run \ + --case Qwen3.8-Flash-TP2-Base,Qwen3.8-Flash-TP2-MTP,Qwen3.5-35B-TP4-Base,Qwen3.5-35B-TP1-DFlash,Qwen3.5-397B-TP8-Base,Qwen3.5-397B-TP8-DFlash \ + --models-root /models \ + --datasets-root /datasets \ + --client-root /ci/client \ + --gpus 0,1,2,3,4,5,6,7 \ + --output /path/to/results/ci-accepted +``` + +Use `--phase base` with another output directory for the baseline tables. This +runs the existing 128-request / 200-example regular CI protocol; it does not alter +the native case selection, GPU mask, benchmark specifications or accuracy gates. + +## Tuner entry points and accepted scope + +The input files contain nine CK decode shapes (`M=1/4/16`) and seven tgemm shapes +(three FP8 prefill shapes plus four BF16 ViT shapes). Existing AITER commands were: + +```bash +python3 /path/to/aiter/csrc/ck_gemm_a8w8_bpreshuffle/gemm_a8w8_bpreshuffle_tune.py \ + -i benchmark/rocm_tgemm/configs/ck_decode_shapes.csv \ + -o /path/to/results/ck_decode_tuned.csv \ + --profile_file /path/to/results/ck_candidates.csv \ + --libtype ck \ + --splitK \ + --mp 1 \ + --warmup 10 \ + --iters 101 \ + --errRatio 0.01 +python3 /path/to/aiter/csrc/gemm_a16w16/gemm_a16w16_tune.py \ + -i benchmark/rocm_tgemm/configs/tgemm_shapes.csv \ + -o /path/to/results/tgemm_tuned.csv \ + --profile_file /path/to/results/tgemm_candidates.csv \ + --libtype hipblaslt,asm,torch \ + --with-hipblaslt \ + --mp 1 \ + --errRatio 0.01 +``` + +The historical AITER search needed two temporary tuning-only fixes: BF16 +`scaleAB=False` must supply no scale tensors, and unsupported hipBLASLt solutions +must raise instead of aborting the worker. Those patches/checkpoint recovery are +retained as external tuning evidence. Apply them only in an isolated AITER tuning +checkout if reproducing that exact search, and restore the original source before +production-interface verification. This source kit applies already validated +rows; it never patches the installed library or automatically accepts tuner output. + +The selected changes are one CK row `(4,2560,3072)`; two FP8 rows +`(8192,8192,2560)` / `(8192,6656,2560)`; and two BF16 rows +`(256,1152,2152)` / `(256,1152,576)`. The other candidates were not accepted. +Selection required at least 3% lower median latency in both eager and graph +comparison, followed by numerical and changed-input replay validation. Re-running +a tuner produces new experimental data, not automatically validated replacements. diff --git a/benchmark/rocm_tgemm/benchmark_gemm.py b/benchmark/rocm_tgemm/benchmark_gemm.py new file mode 100644 index 000000000000..70f80c39149b --- /dev/null +++ b/benchmark/rocm_tgemm/benchmark_gemm.py @@ -0,0 +1,203 @@ +"""Reproduce the recorded 16-shape AITER comparison; use a free, reserved gfx942 GPU.""" + +import argparse +import csv +import json +import os +from pathlib import Path + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--configs-dir", type=Path, required=True) + parser.add_argument("--phase", choices=("base", "accepted"), required=True) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + if args.output.exists(): + raise ValueError("Output already exists; preserve the previous measurement") + os.environ["AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE"] = str( + (args.configs_dir / ("ck_" + args.phase + ".csv")).resolve() + ) + os.environ["AITER_CONFIG_GEMM_BF16"] = str( + (args.configs_dir / ("tgemm_" + args.phase + ".csv")).resolve() + ) + for name in ("AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE", "AITER_CONFIG_GEMM_BF16"): + if not Path(os.environ[name]).is_file(): + raise FileNotFoundError( + "Prepare the complete AITER configuration tables first" + ) + + import aiter + import torch + from aiter.jit.core import AITER_CONFIGS + from aiter.jit.utils.chip_info import get_cu_num, get_gfx + from aiter.ops.shuffle import shuffle_weight + from aiter.test_common import run_perftest + from aiter.tuned_gemm import get_GEMM_A16W16_config, tgemm + + ROOT = Path(__file__).resolve().parent + if get_gfx() != "gfx942" or get_cu_num() != 80: + raise ValueError("Recorded configurations require gfx942 / 80 CUs") + args.output.parent.mkdir(parents=True, exist_ok=True) + torch.manual_seed(42) + torch.set_grad_enabled(False) + records = [] + + def check(out, ref): + assert out.shape == ref.shape and out.dtype == torch.bfloat16 + assert torch.isfinite(out).all() + rel = float( + (out.float() - ref.float()).norm() / ref.float().norm().clamp_min(1e-8) + ) + assert rel < 0.008, rel + return rel + + shapes = [] + for row in csv.DictReader((ROOT / "configs/ck_decode_shapes.csv").open()): + shapes.append( + ( + "ck_decode", + int(row["M"]), + int(row["N"]), + int(row["K"]), + True, + False, + True, + ) + ) + for row in csv.DictReader((ROOT / "configs/tgemm_shapes.csv").open()): + shapes.append( + ( + "tgemm", + int(row["M"]), + int(row["N"]), + int(row["K"]), + row["scaleAB"] == "True", + row["bias"] == "True", + row["bpreshuffle"] == "True", + ) + ) + + for family, m, n, k, fp8, has_bias, packed in shapes: + x = torch.randn(m, k, device="cuda", dtype=torch.bfloat16) + w = torch.randn(n, k, device="cuda", dtype=torch.bfloat16) + bias = torch.randn(n, device="cuda", dtype=torch.bfloat16) if has_bias else None + if fp8: + x, sx = aiter.per_token_quant_hip(x, quant_dtype=aiter.dtypes.fp8) + w, sw = aiter.per_token_quant_hip(w, quant_dtype=aiter.dtypes.fp8) + canonical_w = w.float() * sw + ref = (x.float() * sx) @ canonical_w.t() + else: + sx = sw = None + canonical_w = w.float() + ref = x.float() @ canonical_w.t() + if bias is not None: + ref += bias.float() + wp = shuffle_weight(w, (16, 16)) if packed else w + if packed: + wp.is_shuffled = True + + def fn(inp, weight, scale_a, scale_b, bias_tensor): + if family == "ck_decode": + return aiter.gemm_a8w8_bpreshuffle( + inp, weight, scale_a, scale_b, None, torch.bfloat16 + ) + return tgemm.mm( + inp, + weight, + bias_tensor, + otype=torch.bfloat16, + scale_a=scale_a, + scale_b=scale_b.t() if scale_b is not None else None, + ) + + kernel_args = (x, wp, sx, sw, bias) + out = fn(*kernel_args) + relative_l2 = check(out, ref) + samples = [] + for _ in range(5): + out, us = run_perftest( + fn, *kernel_args, num_warmup=10, num_iters=101, num_rotate_args=16 + ) + check(out, ref) + samples.append(float(us)) + graph_samples = [] + for _ in range(3): + out, us = run_perftest( + fn, + *kernel_args, + num_warmup=10, + num_iters=101, + num_rotate_args=16, + testGraph=True, + ) + check(out, ref) + graph_samples.append(float(us)) + + graph = torch.cuda.CUDAGraph() + torch.cuda.synchronize() + with torch.cuda.graph(graph): + graph_out = fn(*kernel_args) + graph_errors = [] + for seed in [43, 44]: + torch.manual_seed(seed) + changed = torch.randn(m, k, device="cuda", dtype=torch.bfloat16) + if fp8: + q, qs = aiter.per_token_quant_hip(changed, quant_dtype=aiter.dtypes.fp8) + x.copy_(q) + sx.copy_(qs) + reference = (x.float() * sx) @ canonical_w.t() + else: + x.copy_(changed) + reference = x.float() @ canonical_w.t() + if bias is not None: + reference += bias.float() + graph.replay() + torch.cuda.synchronize() + graph_errors.append(check(graph_out, reference)) + if family == "tgemm": + config = get_GEMM_A16W16_config( + m, n, k, has_bias, str(x.dtype), str(torch.bfloat16), fp8, packed + ) + else: + from aiter.ops.gemm_op_a8w8 import get_GEMM_config_with_quant_type + + config = get_GEMM_config_with_quant_type( + m, + n, + k, + str(w.dtype), + AITER_CONFIGS.AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE_FILE, + ) + record = dict( + family=family, + M=m, + N=n, + K=k, + dtype=str(x.dtype), + bias=has_bias, + packed=packed, + us=samples, + graph_us=graph_samples, + relative_l2=relative_l2, + changed_input_graph_relative_l2=graph_errors, + config=config, + ) + records.append(record) + print(json.dumps(record, default=str), flush=True) + args.output.write_text(json.dumps(records, indent=2, default=str)) + del graph, graph_out, out, ref, wp, w, x, canonical_w + torch.cuda.empty_cache() + + print( + "PASS", + len(records), + "GEMMs;", + len(records) * 2, + "changed-input graph replays", + flush=True, + ) + + +if __name__ == "__main__": + main() diff --git a/benchmark/rocm_tgemm/configs/ck_accepted_rows.csv b/benchmark/rocm_tgemm/configs/ck_accepted_rows.csv new file mode 100644 index 000000000000..6d49ff561757 --- /dev/null +++ b/benchmark/rocm_tgemm/configs/ck_accepted_rows.csv @@ -0,0 +1,2 @@ +gfx,cu_num,M,N,K,q_dtype_w,libtype,kernelId,splitK,us,kernelName,tflops,bw,errRatio +gfx942,80,4,2560,3072,torch.float8_e4m3fnuz,ck,10,0,7.1939,a8w8_bpreshuffle_128x16x32x512_16x16_16x16_32x4x1_32x4x1_1x16x1x8_4x4x1_1x1_intrawave_v1,8.75,1097.75,0.0 diff --git a/benchmark/rocm_tgemm/configs/ck_decode_shapes.csv b/benchmark/rocm_tgemm/configs/ck_decode_shapes.csv new file mode 100644 index 000000000000..0ee4fd7f675c --- /dev/null +++ b/benchmark/rocm_tgemm/configs/ck_decode_shapes.csv @@ -0,0 +1,10 @@ +M,N,K,q_dtype_w +1,8192,2560,torch.float8_e4m3fnuz +4,8192,2560,torch.float8_e4m3fnuz +16,8192,2560,torch.float8_e4m3fnuz +1,2560,3072,torch.float8_e4m3fnuz +4,2560,3072,torch.float8_e4m3fnuz +16,2560,3072,torch.float8_e4m3fnuz +1,6656,2560,torch.float8_e4m3fnuz +4,6656,2560,torch.float8_e4m3fnuz +16,6656,2560,torch.float8_e4m3fnuz diff --git a/benchmark/rocm_tgemm/configs/tgemm_accepted_rows.csv b/benchmark/rocm_tgemm/configs/tgemm_accepted_rows.csv new file mode 100644 index 000000000000..fcca4c241326 --- /dev/null +++ b/benchmark/rocm_tgemm/configs/tgemm_accepted_rows.csv @@ -0,0 +1,5 @@ +gfx,cu_num,M,N,K,bias,dtype,outdtype,scaleAB,bpreshuffle,libtype,solidx,splitK,us,kernelName,err_ratio,tflops,bw +gfx942,80,8192,8192,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True,hipblaslt,169251,0,701.2096,Cijk_Alik_Bljk_F8NBS_STA_BH_SABV_UserArgs_MT256x96x128_MI16x16x1_SN_LDSB0_AFC0_AG0_AGGSUA0_AGNTAB0_AFEM1_AFEM1_ASEM1_CD1_1_CLR1_CLS0_CADS0_DTLA0_DTLB0_DTLM0_DTVA1_DTVB0_DTVMXSA0_DTVMXSB0_DTVSM0_DPLB0_EPS0_ELFLR0_EMLLn1_FDSI0_GRPM1_GRVWA16_GRVWB16_GSUAMBSK_GLS0_HPLR0_ISA942_ICIW0_IU1_K1_LDSTI0_LBSPPA0_LBSPPB256_LBSPPMXSA0_LBSPPMXSB0_LBSPPM0_LPA0_LPB32_LPMXSA0_LPMXSB0_LPM0_LRVW16_LWPMn1_MIAV0_MIWT4_6_MXLIBL_MXSFNS_MO40_MGRIPM1_NTn1_NTA0_NTB0_NTC0_NTD4_NTE0_NTMXSA0_NTMXSB0_NTM0_NTWS0_NVn1_NVA0_NVB0_NVC0_NVD0_NVE0_NVMXSA0_NVMXSB0_NVM0_NVWS0_NEPBS16_NLCA2_NLCB1_ONLL0_PAP0_PGL0_PGR2_PLR1_PKA1_SGROB0_SIA3_SS1_SPO0_SRVW0_SSO0_SVW1_SK0_SKFTR0_SKFDPO0_SKXCCM0_SNLL0_SIP1_SGRO0_TDMI0_TDMIM0_TDMS0_TIN0_THn1_THA0_THB0_THC0_THD0_THE0_THMXSA0_THMXSB0_THM0_THWS0_TLDS1_TLDSM1_ULSGRO0_USL1_USLMX0_UIOFGRO0_UPLRP0_USFGROn1_USI0_VSn1_VWA1_VWB2_WSGRA0_WSGRB0_WS64_WG64_4_1,0.0,490.01,311.04 +gfx942,80,8192,6656,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True,hipblaslt,169251,0,575.4129,Cijk_Alik_Bljk_F8NBS_STA_BH_SABV_UserArgs_MT256x96x128_MI16x16x1_SN_LDSB0_AFC0_AG0_AGGSUA0_AGNTAB0_AFEM1_AFEM1_ASEM1_CD1_1_CLR1_CLS0_CADS0_DTLA0_DTLB0_DTLM0_DTVA1_DTVB0_DTVMXSA0_DTVMXSB0_DTVSM0_DPLB0_EPS0_ELFLR0_EMLLn1_FDSI0_GRPM1_GRVWA16_GRVWB16_GSUAMBSK_GLS0_HPLR0_ISA942_ICIW0_IU1_K1_LDSTI0_LBSPPA0_LBSPPB256_LBSPPMXSA0_LBSPPMXSB0_LBSPPM0_LPA0_LPB32_LPMXSA0_LPMXSB0_LPM0_LRVW16_LWPMn1_MIAV0_MIWT4_6_MXLIBL_MXSFNS_MO40_MGRIPM1_NTn1_NTA0_NTB0_NTC0_NTD4_NTE0_NTMXSA0_NTMXSB0_NTM0_NTWS0_NVn1_NVA0_NVB0_NVC0_NVD0_NVE0_NVMXSA0_NVMXSB0_NVM0_NVWS0_NEPBS16_NLCA2_NLCB1_ONLL0_PAP0_PGL0_PGR2_PLR1_PKA1_SGROB0_SIA3_SS1_SPO0_SRVW0_SSO0_SVW1_SK0_SKFTR0_SKFDPO0_SKXCCM0_SNLL0_SIP1_SGRO0_TDMI0_TDMIM0_TDMS0_TIN0_THn1_THA0_THB0_THC0_THD0_THE0_THMXSA0_THMXSB0_THM0_THWS0_TLDS1_TLDSM1_ULSGRO0_USL1_USLMX0_UIOFGRO0_UPLRP0_USFGROn1_USI0_VSn1_VWA1_VWB2_WSGRA0_WSGRB0_WS64_WG64_4_1,0.0,485.17,321.64 +gfx942,80,256,1152,2152,True,torch.bfloat16,torch.bfloat16,False,False,hipblaslt,82636,0,17.0225,Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT64x64x128_MI16x16x1_SN_LDSB1_AFC0_AG0_AGGSUA0_AGNTAB0_AFEM1_AFEM1_ASEM1_CD1_1_CLR1_CLS0_CADS0_DTLA0_DTLB0_DTLM0_DTVA0_DTVB0_DTVMXSA0_DTVMXSB0_DTVSM0_DPLB0_EPS0_ELFLR0_EMLLn1_FDSI0_GRPM1_GRVWA8_GRVWB8_GSUAMB_GLS0_HPLR0_ISA942_ICIW0_IU1_K1_LDSTI0_LBSPPA1024_LBSPPB512_LBSPPMXSA0_LBSPPMXSB0_LBSPPM0_LPA16_LPB16_LPMXSA0_LPMXSB0_LPM0_LRVW8_LWPMn1_MIAV0_MIWT4_2_MXLIBL_MXSFNS_MO40_MGRIPM1_NTn1_NTA0_NTB0_NTC0_NTD7_NTE0_NTMXSA0_NTMXSB0_NTM0_NTWS0_NVn1_NVA0_NVB0_NVC0_NVD0_NVE0_NVMXSA0_NVMXSB0_NVM0_NVWS0_NEPBS8_NLCA1_NLCB1_ONLL1_PAP0_PGL0_PGR2_PLR1_PKA1_SGROB0_SIA3_SS1_SPO0_SRVW0_SSO0_SVW4_SK0_SKFTR0_SKFDPO0_SKXCCM0_SNLL0_SIP1_SGRO0_TDMI0_TDMIM0_TDMS0_TIN0_THn1_THA0_THB0_THC0_THD0_THE0_THMXSA0_THMXSB0_THM0_THWS0_TLDS1_TLDSM1_ULSGRO1_USL1_USLMX0_UIOFGRO0_UPLRP0_USFGROn1_USI0_VSn1_VWA4_VWB2_WSGRA0_WSGRB0_WS64_WG16_8_2,0.0,74.57,390.65 +gfx942,80,256,1152,576,True,torch.bfloat16,torch.bfloat16,False,True,asm,5,1,5.9612,_ZN5aiter42bf16gemm_fp32bf16_tn_64x64_bshuffle_splitkE,0.0,56.99,371.04 diff --git a/benchmark/rocm_tgemm/configs/tgemm_shapes.csv b/benchmark/rocm_tgemm/configs/tgemm_shapes.csv new file mode 100644 index 000000000000..377fcbafd7fb --- /dev/null +++ b/benchmark/rocm_tgemm/configs/tgemm_shapes.csv @@ -0,0 +1,8 @@ +M,N,K,bias,dtype,outdtype,scaleAB,bpreshuffle +8192,8192,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True +8192,2560,3072,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True +8192,6656,2560,False,torch.float8_e4m3fnuz,torch.bfloat16,True,True +256,1728,1152,True,torch.bfloat16,torch.bfloat16,False,True +256,2152,1152,True,torch.bfloat16,torch.bfloat16,False,False +256,1152,2152,True,torch.bfloat16,torch.bfloat16,False,False +256,1152,576,True,torch.bfloat16,torch.bfloat16,False,True diff --git a/benchmark/rocm_tgemm/prepare_configs.py b/benchmark/rocm_tgemm/prepare_configs.py new file mode 100644 index 000000000000..6894d81830af --- /dev/null +++ b/benchmark/rocm_tgemm/prepare_configs.py @@ -0,0 +1,159 @@ +"""Merge the validated gfx942 overrides into supplied AITER tables without replacing inputs.""" + +import argparse +import csv +import hashlib +import io +import json +from decimal import Decimal, InvalidOperation +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +AITER_REVISION = "514c166e9abad59acde74309923c2d923a89eee0" +KEYS = { + "ck": ("gfx", "cu_num", "M", "N", "K", "q_dtype_w"), + "tgemm": ( + "gfx", + "cu_num", + "M", + "N", + "K", + "bias", + "dtype", + "outdtype", + "scaleAB", + "bpreshuffle", + ), +} +REQUIRED = { + "ck": set(KEYS["ck"]) | {"libtype", "kernelId", "splitK", "kernelName"}, + "tgemm": set(KEYS["tgemm"]) | {"libtype", "solidx", "splitK", "kernelName"}, +} +OVERLAYS = { + "ck": ROOT / "configs/ck_accepted_rows.csv", + "tgemm": ROOT / "configs/tgemm_accepted_rows.csv", +} + + +def shape_key(row, family): + """The original pandas merge used numeric coordinates and boolean layout keys.""" + result = [] + for name in KEYS[family]: + value = str(row[name]).strip() + if name in ("cu_num", "M", "N", "K"): + try: + number = Decimal(value) + except InvalidOperation as exc: + raise ValueError(f"Invalid integer coordinate {name}={value}") from exc + if not number.is_finite() or number != number.to_integral_value(): + raise ValueError(f"Invalid integer coordinate {name}={value}") + value = int(number) + elif name in ("bias", "scaleAB", "bpreshuffle"): + if value.lower() not in ("true", "false"): + raise ValueError(f"Invalid boolean shape key {name}={value}") + value = value.lower() == "true" + result.append(value) + return tuple(result) + + +def read_table(path, family): + with Path(path).open(newline="") as stream: + reader = csv.DictReader(stream) + names = reader.fieldnames or [] + if len(set(names)) != len(names) or not REQUIRED[family].issubset(names): + raise ValueError(f"Invalid {family} CSV schema: {path}") + rows = list(reader) + if not rows or any( + None in row or any(value is None for value in row.values()) for row in rows + ): + raise ValueError(f"Empty or malformed {family} table: {path}") + seen = set() + for row in rows: + key = shape_key(row, family) + if key in seen: + raise ValueError(f"Duplicate {family} shape key: {key}") + seen.add(key) + return names, rows + + +def merge_tables(base_names, base_rows, overlay_names, overlay_rows, family): + replacements = {shape_key(row, family) for row in overlay_rows} + names = base_names + [name for name in overlay_names if name not in base_names] + # Same keep-last operation as concat(base, validated_rows).drop_duplicates(keys). + rows = [row for row in base_rows if shape_key(row, family) not in replacements] + rows += overlay_rows + return names, rows + + +def encode_table(names, rows): + stream = io.StringIO(newline="") + writer = csv.DictWriter(stream, fieldnames=names, lineterminator="\n") + writer.writeheader() + writer.writerows(rows) + return stream.getvalue().encode() + + +def sha256(data): + return hashlib.sha256(data).hexdigest() + + +def prepare_configs(ck_base, tgemm_base, output_dir): + bases = {"ck": Path(ck_base), "tgemm": Path(tgemm_base)} + output_dir = Path(output_dir) + input_paths = {path.resolve() for path in [*bases.values(), *OVERLAYS.values()]} + outputs = { + family + "_" + variant + ".csv": output_dir / (family + "_" + variant + ".csv") + for family in bases + for variant in ("base", "accepted") + } + outputs["MANIFEST.json"] = output_dir / "MANIFEST.json" + if any(path.resolve() in input_paths for path in outputs.values()): + raise ValueError("Output would overwrite an input table") + if any(path.exists() for path in outputs.values()): + raise ValueError("Output files already exist; use a fresh output directory") + data, manifest = {}, {"validated_aiter_revision": AITER_REVISION, "tables": {}} + for family, base_path in bases.items(): + base_names, base_rows = read_table(base_path, family) + overlay_path = OVERLAYS[family] + overlay_names, overlay_rows = read_table(overlay_path, family) + if len(overlay_rows) != (1 if family == "ck" else 4) or any( + row["gfx"] != "gfx942" or Decimal(row["cu_num"]) != 80 + for row in overlay_rows + ): + raise ValueError( + "Validated overlay must retain the recorded gfx942/80-CU rows" + ) + names, merged = merge_tables( + base_names, base_rows, overlay_names, overlay_rows, family + ) + base_bytes = base_path.read_bytes() + data[family + "_base.csv"] = base_bytes + data[family + "_accepted.csv"] = encode_table(names, merged) + manifest["tables"][family] = { + "base_path": str(base_path.resolve()), + "base_sha256": sha256(base_bytes), + "overlay_sha256": sha256(overlay_path.read_bytes()), + "merged_sha256": sha256(data[family + "_accepted.csv"]), + "base_rows": len(base_rows), + "overlay_rows": len(overlay_rows), + "merged_rows": len(merged), + } + output_dir.mkdir(parents=True, exist_ok=True) + for name, value in data.items(): + outputs[name].write_bytes(value) + outputs["MANIFEST.json"].write_text(json.dumps(manifest, indent=2) + "\n") + return manifest + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--ck-base", type=Path, required=True) + parser.add_argument("--tgemm-base", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + manifest = prepare_configs(args.ck_base, args.tgemm_base, args.output_dir) + print(json.dumps(manifest, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/benchmark/rocm_tgemm/run_ci.py b/benchmark/rocm_tgemm/run_ci.py new file mode 100644 index 000000000000..e7b5a55d772b --- /dev/null +++ b/benchmark/rocm_tgemm/run_ci.py @@ -0,0 +1,48 @@ +"""Run native Qwen4 CI with the selected prepared AITER tables in server environments.""" + +import argparse +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] + + +def server_environment(original, configs_dir, phase): + configs_dir = Path(configs_dir).resolve() + + def configured(**extra): + env = original(**extra) + if "MODEL_PATH" in extra: + env.update( + SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB="1", + SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE="1", + AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE=str( + configs_dir / ("ck_" + phase + ".csv") + ), + AITER_CONFIG_GEMM_BF16=str(configs_dir / ("tgemm_" + phase + ".csv")), + ) + return env + + return configured + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, allow_abbrev=False) + parser.add_argument("--configs-dir", type=Path, required=True) + parser.add_argument("--phase", choices=("base", "accepted"), required=True) + args, native_args = parser.parse_known_args() + for family in ("ck", "tgemm"): + if not (args.configs_dir / (family + "_" + args.phase + ".csv")).is_file(): + raise FileNotFoundError("Prepare complete configuration tables before CI") + sys.path.insert(0, str(ROOT / "scripts/ci")) + import qwen4_ci as ci + + ci.clean_environment = server_environment( + ci.clean_environment, args.configs_dir, args.phase + ) + sys.argv = [sys.argv[0], *native_args] + return ci.main() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/python/sglang/srt/environ.py b/python/sglang/srt/environ.py index d67ed5797fb4..13bcf17923dd 100644 --- a/python/sglang/srt/environ.py +++ b/python/sglang/srt/environ.py @@ -955,6 +955,8 @@ class Envs: # unset, the self-check decides. SGLANG_ROCM_JPEG_COLOR_CORRECTION = EnvBool(None) USE_ROCM_AITER_ROPE_BACKEND = EnvStr("0") + SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE = EnvBool(False) + SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB = EnvBool(False) # Enable dual-stream MoE (shared experts vs routed experts) on the # ROCm/AITER path. Requires GPU_MAX_HW_QUEUES>=5 to avoid HW-queue serialization. SGLANG_ROCM_USE_MULTI_STREAM = EnvBool(False) diff --git a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py b/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py index a829de606a93..11fe723d8693 100644 --- a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py +++ b/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py @@ -178,11 +178,15 @@ def process_weights_after_loading(self, layer) -> None: weight_scale = layer.weight_scale.data if _use_aiter: + layout = (16, 16) # keep the weight as (N, K) layer.weight = Parameter( - shuffle_weight(weight, (16, 16)), requires_grad=False + shuffle_weight(weight, layout), requires_grad=False ) + # Parameter construction drops the shuffle tensor's metadata. + layer.weight.is_shuffled = True else: + # keep the weight as (K, N) layer.weight = Parameter(weight.t(), requires_grad=False) # required by torch.compile to be torch.nn.Parameter @@ -234,8 +238,21 @@ def apply_weights( if isinstance(x, tuple): # Pre-quantized activation from a fused RMSNorm+FP8 quant kernel: # x = (fp8_input, per_tensor_input_scale[, orig_dtype]). - # apply_fp8_linear detects the fp8 dtype and skips re-quantizing; - # orig_dtype (when present) sets the GEMM output dtype. + # Packed channel weights require layout-aware PTPC dispatch even + # when tgemm is disabled or the producer requests FP16 output. + # The generic path keeps the caller's original output dtype. + if ( + _use_aiter + and getattr(layer.weight, "is_shuffled", False) + and self.strategy == QuantizationStrategy.CHANNEL + ): + return apply_fp8_ptpc_linear( + input=x, + weight=layer.weight, + weight_scale=layer.weight_scale, + input_scale=layer.input_scale, + bias=bias, + ) qx, x_scale = x[0], x[1] out_dtype = x[2] if len(x) > 2 else None return apply_fp8_linear( diff --git a/python/sglang/srt/layers/quantization/fp8_utils.py b/python/sglang/srt/layers/quantization/fp8_utils.py index fd86e97283fd..79fa21c489d7 100755 --- a/python/sglang/srt/layers/quantization/fp8_utils.py +++ b/python/sglang/srt/layers/quantization/fp8_utils.py @@ -45,6 +45,7 @@ is_cuda, is_flashinfer_available, is_gfx95_supported, + is_gfx942_supported, is_gfx1250_supported, is_hip, is_musa, @@ -2265,13 +2266,34 @@ def apply_fp8_linear( # x_scale -> input scale tensor, shape = (m, 1) # w_scale -> weight scale tensor, shape = (n ,1) # dtype -> output dtype - output = gemm_a8w8_bpreshuffle( - XQ=qinput, - WQ=weight.T, - x_scale=x_scale, - w_scale=weight_scale, - dtype=output_dtype, - ) + if ( + envs.SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB.get() + and is_gfx942_supported() + and output_dtype == torch.bfloat16 + and qinput.shape[0] > 16 + and x_scale.numel() == qinput.shape[0] + and weight_scale.numel() == weight.shape[1] + ): + from aiter.tuned_gemm import tgemm + + # This branch shares Quark's load-time bpreshuffle predicate. + packed_weight = weight.T + packed_weight.is_shuffled = True + output = tgemm.mm( + qinput, + packed_weight, + otype=output_dtype, + scale_a=x_scale.view(-1, 1), + scale_b=weight_scale.view(1, -1), + ) + else: + output = gemm_a8w8_bpreshuffle( + XQ=qinput, + WQ=weight.T, + x_scale=x_scale, + w_scale=weight_scale, + dtype=output_dtype, + ) if bias is not None: output += bias return _process_scaled_mm_output(output, input_2d.shape, output_shape) @@ -2343,7 +2365,11 @@ def can_auto_enable_marlin_fp8() -> bool: def apply_fp8_ptpc_linear( - input: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]], + input: Union[ + torch.Tensor, + Tuple[torch.Tensor, torch.Tensor], + Tuple[torch.Tensor, torch.Tensor, torch.dtype], + ], weight: torch.Tensor, weight_scale: torch.Tensor, input_scale: Optional[torch.Tensor] = None, @@ -2355,36 +2381,80 @@ def apply_fp8_ptpc_linear( compressed_tensor_quant: bool = False, ) -> torch.Tensor: """FP8 per-token per-channel linear. Only used with the aiter (ROCm) backend.""" - # Handle pre-quantized (fp8_tensor, scale) tuple from fused RMSNorm+Quant + # Reuse fused quantization and preserve the original leading dimensions. if isinstance(input, tuple): - q_input, x_scale = input - q_input = q_input.view(-1, q_input.shape[-1]) - output_shape = [*q_input.shape[:-1], weight.shape[0]] - output = aiter.gemm_a8w8_bpreshuffle( - q_input, weight, x_scale, weight_scale, None, torch.bfloat16 + if len(input) not in (2, 3): + raise ValueError("FP8 PTPC input must be a two- or three-element tuple") + q_input, x_scale = input[:2] + if not isinstance(q_input, torch.Tensor) or q_input.ndim == 0: + raise ValueError("FP8 PTPC input must have a hidden dimension") + if weight.ndim != 2 or q_input.shape[-1] != weight.shape[1]: + raise ValueError("FP8 PTPC packed weight must have shape [N, K]") + leading_shape = q_input.shape[:-1] + out_dtype = input[2] if len(input) > 2 else torch.bfloat16 + q_input = q_input.reshape(-1, q_input.shape[-1]).contiguous() + num_tokens = q_input.shape[0] + if not isinstance(x_scale, torch.Tensor): + raise ValueError("FP8 PTPC activation scale must be a tensor") + # Remember the producer's scale layout before broadcasting a scalar. + per_token_scales = x_scale.numel() == num_tokens + if x_scale.numel() == 1 and num_tokens > 1: + x_scale = x_scale.reshape(1, 1).expand(num_tokens, 1).contiguous() + elif not per_token_scales: + raise ValueError("FP8 PTPC activation scale must be scalar or per-token") + elif x_scale.shape != (num_tokens, 1) or not x_scale.is_contiguous(): + x_scale = x_scale.reshape(num_tokens, 1).contiguous() + if ( + not isinstance(weight_scale, torch.Tensor) + or weight_scale.numel() != weight.shape[0] + ): + raise ValueError("FP8 PTPC weight scale must have one value per channel") + if ( + weight_scale.shape != (weight.shape[0], 1) + or not weight_scale.is_contiguous() + ): + weight_scale = weight_scale.reshape(-1, 1).contiguous() + output_shape = [*leading_shape, weight.shape[0]] + else: + input_2d = input.view(-1, input.shape[-1]) + out_dtype = torch.bfloat16 if input_scale is not None else input.dtype + q_input, x_scale = aiter.per_token_quant_hip( + input_2d, quant_dtype=aiter.dtypes.fp8 ) - if bias is not None: - output = output + bias - return output.view(*output_shape) - - # View input as 2D matrix for fp8 methods - input_2d = input.view(-1, input.shape[-1]) - - # weight is transposed (K, N) - output_shape = [*input.shape[:-1], weight.shape[1]] - - q_input, x_scale = aiter.per_token_quant_hip(input_2d, quant_dtype=aiter.dtypes.fp8) - - per_tensor_weights = (weight_scale.numel() == 1) and weight_scale.dim() < 2 - per_tensor_activations = (x_scale.numel() == 1) and x_scale.dim() < 2 + per_token_scales = x_scale.numel() == q_input.shape[0] + per_tensor_weights = (weight_scale.numel() == 1) and weight_scale.dim() < 2 + per_tensor_activations = (x_scale.numel() == 1) and x_scale.dim() < 2 + output_shape = [ + *input.shape[:-1], + weight.shape[1] + if per_tensor_weights and per_tensor_activations + else weight.shape[0], + ] - if not (per_tensor_weights and per_tensor_activations): - # weight is in (N, K) - output_shape = [*input.shape[:-1], weight.shape[0]] + if ( + envs.SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB.get() + and is_gfx942_supported() + and getattr(weight, "is_shuffled", False) + and out_dtype == torch.bfloat16 + and per_token_scales + and q_input.shape[0] > 16 + and x_scale.numel() == q_input.shape[0] + and weight_scale.numel() == weight.shape[0] + ): + from aiter.tuned_gemm import tgemm - output = aiter.gemm_a8w8_bpreshuffle( - q_input, weight, x_scale, weight_scale, None, input.dtype - ) + # tgemm takes packed [N, K] weights and transposes them internally. + output = tgemm.mm( + q_input, + weight, + otype=out_dtype, + scale_a=x_scale.view(-1, 1), + scale_b=weight_scale.view(1, -1), + ) + else: + output = aiter.gemm_a8w8_bpreshuffle( + q_input, weight, x_scale, weight_scale, None, out_dtype + ) if bias is not None: output = output + bias return output.view(*output_shape) diff --git a/python/sglang/srt/layers/quantization/unquant.py b/python/sglang/srt/layers/quantization/unquant.py index 689319b98d24..9999113f6b0b 100644 --- a/python/sglang/srt/layers/quantization/unquant.py +++ b/python/sglang/srt/layers/quantization/unquant.py @@ -47,6 +47,7 @@ is_cpu, is_cuda, is_gfx95_supported, + is_gfx942_supported, is_hip, is_npu, is_xpu, @@ -74,6 +75,7 @@ _use_aiter = get_bool_env_var("SGLANG_USE_AITER") and _is_hip if _use_aiter: + from aiter.ops.shuffle import shuffle_weight from aiter.tuned_gemm import tgemm @@ -468,6 +470,21 @@ def process_weights_after_loading(self, layer: torch.nn.Module) -> None: if _is_cpu and _is_cpu_amx_available: _amx_process_weight_after_loading(layer, ["weight"]) + weight = layer.weight + if ( + _use_aiter + and is_gfx942_supported() + and get_bool_env_var("SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE") + and weight.dtype == torch.bfloat16 + and weight.dim() == 2 + and weight.shape[0] % 16 == 0 + and weight.shape[1] % 32 == 0 + ): + packed = Parameter(shuffle_weight(weight, (16, 16)), requires_grad=False) + packed.is_shuffled = True + # Keep dense weights available to decode and DFlash KV slicing. + layer.register_parameter("aiter_prefill_weight", packed) + def apply( self, layer: torch.nn.Module, @@ -489,6 +506,14 @@ def apply( return output elif _use_aiter and type(layer.weight.data) is torch.Tensor: + if x.numel() // x.shape[-1] > 16 and x.dtype == torch.bfloat16: + packed = getattr(layer, "aiter_prefill_weight", None) + if packed is not None: + # tgemm's non-viewable input fallback expects dense weights. + output = tgemm.mm( + x.reshape(-1, x.shape[-1]), packed, bias, otype=x.dtype + ) + return output.view(*x.shape[:-1], packed.shape[0]) return tgemm.mm(x, layer.weight, bias, otype=x.dtype) elif ( @@ -537,6 +562,9 @@ def apply_into( bias: Optional[torch.Tensor] = None, ) -> torch.Tensor: """Run an inference-only BF16 linear into caller-owned storage.""" + if getattr(layer.weight, "aiter_trans_weight", False): + output.copy_(self.apply(layer, x, bias)) + return output if ( _enable_bf16_splitk_gemm and bias is None diff --git a/python/sglang/srt/speculative/dflash_utils.py b/python/sglang/srt/speculative/dflash_utils.py index 44c3673cf50d..c9c155bb25cf 100644 --- a/python/sglang/srt/speculative/dflash_utils.py +++ b/python/sglang/srt/speculative/dflash_utils.py @@ -852,8 +852,16 @@ def can_dflash_slice_qkv_weight(qkv_proj: Any) -> Tuple[bool, str]: "quantized qkv_proj is not supported for this path " f"(quant_method={type(quant_method).__name__})", ) - if not hasattr(qkv_proj, "weight"): + weight = getattr(qkv_proj, "weight", None) + if weight is None: return False, "qkv weight tensor is missing" + # UnquantizedLinearMethod may store an AITER-swizzled [K, N] matrix. + # Both KV-only paths read rows directly, so such weights must instead + # go through the linear layer's layout-aware dispatch. + if getattr(weight, "aiter_trans_weight", False): + return False, "AITER-swizzled qkv weight requires the linear dispatch" + if weight.ndim != 2 or weight.dtype not in _DENSE_HEAD_DTYPES: + return False, "qkv weight is not a dense floating-point matrix" return True, "" diff --git a/test/registered/unit/layers/quantization/test_rocm_tgemm_configs.py b/test/registered/unit/layers/quantization/test_rocm_tgemm_configs.py new file mode 100644 index 000000000000..cf3f344c6ad6 --- /dev/null +++ b/test/registered/unit/layers/quantization/test_rocm_tgemm_configs.py @@ -0,0 +1,200 @@ +"""Standard-library tests for validated-overlay merges; no ROCm runtime imports.""" + +import csv +import hashlib +import importlib.util +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[5] +KIT = ROOT / "benchmark/rocm_tgemm" + + +def load_file(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +# The registry itself is stdlib-only; avoid importing the runtime-heavy public API. +registry = load_file( + "_rocm_tgemm_config_registry", ROOT / "python/sglang/test/ci/ci_register.py" +) +register_cpu_ci = registry.register_cpu_ci +register_cpu_ci(est_time=2, suite="base-a-test-cpu") +prepare = load_file("_rocm_tgemm_prepare_configs", KIT / "prepare_configs.py") +ci_wrapper = load_file("_rocm_tgemm_run_ci", KIT / "run_ci.py") + + +class TestRocmTgemmConfigs(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.directory = Path(self.temporary.name) + + def write_table(self, name, names, rows): + path = self.directory / name + with path.open("w", newline="") as stream: + writer = csv.DictWriter(stream, fieldnames=names) + writer.writeheader() + writer.writerows(rows) + return path + + def base_tables(self): + paths = {} + for family, path in prepare.OVERLAYS.items(): + names, rows = prepare.read_table(path, family) + matched = dict(rows[0], libtype="old_backend") + control = dict(rows[0], M="1", N="4096", K="2048") + paths[family] = self.write_table( + family + "_base.csv", names, [matched, control] + ) + return paths + + def test_validated_rows_have_recorded_scope(self): + self.assertEqual(len(prepare.read_table(prepare.OVERLAYS["ck"], "ck")[1]), 1) + self.assertEqual( + len(prepare.read_table(prepare.OVERLAYS["tgemm"], "tgemm")[1]), 4 + ) + for family, path in prepare.OVERLAYS.items(): + for row in prepare.read_table(path, family)[1]: + self.assertEqual(row["gfx"], "gfx942") + self.assertEqual(int(row["cu_num"]), 80) + + def test_merge_preserves_unmatched_rows_and_replaces_exact_key(self): + for family, base_path in self.base_tables().items(): + with self.subTest(family=family): + names, base = prepare.read_table(base_path, family) + overlay_names, overlay = prepare.read_table( + prepare.OVERLAYS[family], family + ) + output_names, output = prepare.merge_tables( + names, base, overlay_names, overlay, family + ) + self.assertEqual(output_names, names) + self.assertEqual(output[0], base[1]) + self.assertEqual(output[1:], overlay) + self.assertEqual(len(output), len(overlay) + 1) + + def test_tgemm_key_retains_dtype_bias_scale_and_packing(self): + names, overlay = prepare.read_table(prepare.OVERLAYS["tgemm"], "tgemm") + row = overlay[0] + controls = [ + dict(row, **{field: value}) + for field, value in [ + ("dtype", "torch.bfloat16"), + ("outdtype", "torch.float16"), + ("bias", "True"), + ("scaleAB", "False"), + ("bpreshuffle", "False"), + ] + ] + _, merged = prepare.merge_tables(names, [row, *controls], names, [row], "tgemm") + self.assertEqual(merged, [*controls, row]) + + def test_invalid_schema_and_row_width_fail(self): + path = self.directory / "invalid.csv" + path.write_text("M,N,K\n1,2,3\n") + with self.assertRaisesRegex(ValueError, "schema"): + prepare.read_table(path, "ck") + names, rows = prepare.read_table(prepare.OVERLAYS["ck"], "ck") + path = self.write_table("bad_width.csv", names, rows) + with path.open("a") as stream: + stream.write("1,2\n") + with self.assertRaisesRegex(ValueError, "malformed"): + prepare.read_table(path, "ck") + + def test_duplicate_numeric_shape_is_rejected(self): + for family, overlay_path in prepare.OVERLAYS.items(): + with self.subTest(family=family): + names, rows = prepare.read_table(overlay_path, family) + duplicate = dict(rows[0], cu_num="80.0", libtype="different_backend") + path = self.write_table( + family + "_duplicate.csv", names, [rows[0], duplicate] + ) + with self.assertRaisesRegex(ValueError, "Duplicate"): + prepare.read_table(path, family) + + def test_prepare_preserves_inputs_and_complete_base_snapshots(self): + bases = self.base_tables() + original = {family: path.read_bytes() for family, path in bases.items()} + output = self.directory / "merged" + manifest = prepare.prepare_configs(bases["ck"], bases["tgemm"], output) + for family, path in bases.items(): + self.assertEqual(path.read_bytes(), original[family]) + self.assertEqual( + (output / (family + "_base.csv")).read_bytes(), original[family] + ) + self.assertEqual( + manifest["tables"][family]["base_sha256"], + hashlib.sha256(original[family]).hexdigest(), + ) + _, merged = prepare.read_table(output / (family + "_accepted.csv"), family) + count = 1 if family == "ck" else 4 + self.assertEqual(len(merged), count + 1) + self.assertEqual(manifest["tables"][family]["overlay_rows"], count) + self.assertTrue((output / "MANIFEST.json").is_file()) + + def test_input_overwrite_is_rejected(self): + bases = self.base_tables() + original = bases["ck"].read_bytes() + with self.assertRaisesRegex(ValueError, "overwrite an input"): + prepare.prepare_configs(bases["ck"], bases["tgemm"], self.directory) + self.assertEqual(bases["ck"].read_bytes(), original) + + def test_ci_server_environment_injects_without_changing_native_extra(self): + calls = [] + + def original(**extra): + calls.append(dict(extra)) + return {"PATH": "/runtime/bin", **extra} + + configure = ci_wrapper.server_environment(original, self.directory, "accepted") + extra = { + "MODEL_PATH": "/models/model", + "HIP_VISIBLE_DEVICES": "3,1", + "PORT": "9000", + } + env = configure(**extra) + self.assertEqual(calls, [extra]) + self.assertEqual(env["HIP_VISIBLE_DEVICES"], "3,1") + self.assertEqual(env["PORT"], "9000") + self.assertEqual(env["MODEL_PATH"], "/models/model") + self.assertEqual(env["SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB"], "1") + self.assertEqual(env["SGLANG_ROCM_USE_AITER_LINEAR_SHUFFLE"], "1") + self.assertEqual( + env["AITER_CONFIG_GEMM_BF16"], str(self.directory / "tgemm_accepted.csv") + ) + self.assertEqual( + env["AITER_CONFIG_GEMM_A8W8_BPRESHUFFLE"], + str(self.directory / "ck_accepted.csv"), + ) + + def test_ci_client_environment_is_unchanged(self): + def original(**extra): + return {"PATH": "/runtime/bin", **extra} + + configure = ci_wrapper.server_environment(original, self.directory, "base") + extra = { + "TOKENIZER_PATH": "/models/model", + "HIP_VISIBLE_DEVICES": "6", + "CUSTOM": "kept", + } + self.assertEqual(configure(**extra), original(**extra)) + + def test_existing_measurement_configs_are_not_overwritten(self): + bases = self.base_tables() + output = self.directory / "merged" + prepare.prepare_configs(bases["ck"], bases["tgemm"], output) + original = (output / "ck_accepted.csv").read_bytes() + with self.assertRaisesRegex(ValueError, "already exist"): + prepare.prepare_configs(bases["ck"], bases["tgemm"], output) + self.assertEqual((output / "ck_accepted.csv").read_bytes(), original) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/registered/unit/layers/quantization/test_rocm_tgemm_linear.py b/test/registered/unit/layers/quantization/test_rocm_tgemm_linear.py new file mode 100644 index 000000000000..a4c4dd0eb784 --- /dev/null +++ b/test/registered/unit/layers/quantization/test_rocm_tgemm_linear.py @@ -0,0 +1,307 @@ +"""CPU regression checks for AITER linear dispatch and packed-weight ownership. + +Load the production methods without importing ROCm extensions. Numerical backend +stand-ins deliberately scramble packed weights, so a dense fallback cannot +silently consume packed storage. +""" + +import ast +import sys +import unittest +from pathlib import Path +from types import ModuleType, SimpleNamespace +from unittest.mock import patch + +import torch +import torch.nn.functional as F +from torch.nn import Parameter + +from sglang.test.ci.ci_register import register_cpu_ci + +register_cpu_ci(est_time=5, suite="base-a-test-cpu") + +ROOT = Path(__file__).resolve().parents[5] +QUANT = ROOT / "python/sglang/srt/layers/quantization" + + +def load_definition(path, name, namespace, methods=None): + tree = ast.parse(path.read_text()) + node = next(item for item in tree.body if getattr(item, "name", None) == name) + if methods is not None: + node.bases = [] + node.body = [ + item for item in node.body if getattr(item, "name", None) in methods + ] + module = ast.Module( + body=[ast.ImportFrom("__future__", [ast.alias("annotations")], 0), node], + type_ignores=[], + ) + exec(compile(ast.fix_missing_locations(module), str(path), "exec"), namespace) + return namespace[name] + + +class Flag: + def __init__(self, value): + self.value = value + + def get(self): + return self.value + + +def pack(weight, layout=None): + return weight.float().flip(1).to(weight.dtype) + + +class Backends: + def __init__(self): + self.calls = [] + + def mm(self, x, weight, bias=None, *, otype, scale_a=None, scale_b=None): + packed = getattr(weight, "is_shuffled", False) + self.calls.append(("tgemm", weight, x.shape, packed)) + dense = weight.float().flip(1) if packed else weight.float() + if scale_a is not None: + return ((x.float() * scale_a) @ (dense * scale_b.t()).t()).to(otype) + if packed: + # The native non-viewable-input fallback requires dense weights. + assert x.ndim == 2 and x.is_contiguous() + result = F.linear(x.float(), dense, None if bias is None else bias.float()) + return result.to(otype) + + def ck(self, x, weight, x_scale, weight_scale, unused_bias, out_dtype): + self.calls.append(("ck", weight, x.shape, True)) + assert x.is_contiguous() + assert x_scale.shape == (x.shape[0], 1) and x_scale.is_contiguous() + dense = weight.float().flip(1) + return ((x.float() * x_scale) @ (dense * weight_scale.view(-1, 1)).t()).to( + out_dtype + ) + + def quant(self, *args, **kwargs): + raise AssertionError("pre-quantized tuples must not be quantized again") + + +class TestBF16TgemmWeightOwnership(unittest.TestCase): + def setUp(self): + torch.manual_seed(42) + self.backends = Backends() + self.shuffle = Flag(True) + self.namespace = { + "torch": torch, + "F": F, + "Parameter": Parameter, + "_is_cpu": False, + "_is_cpu_amx_available": False, + "_use_aiter": True, + "is_gfx942_supported": lambda: True, + "get_bool_env_var": lambda name: self.shuffle.get(), + "shuffle_weight": pack, + "use_intel_amx_backend": lambda layer: False, + "tgemm": self.backends, + "get_bf16_gemm_backend": lambda: SimpleNamespace(is_cutedsl=lambda: False), + } + self.method = load_definition( + QUANT / "unquant.py", + "UnquantizedLinearMethod", + self.namespace, + {"process_weights_after_loading", "apply"}, + )() + + def layer(self, n=16, k=64, dtype=torch.bfloat16): + layer = torch.nn.Module() + layer.weight = Parameter(torch.randn(n, k, dtype=dtype) / 8, False) + return layer + + def test_dense_parameter_is_preserved_for_decode_and_kv_slicing(self): + layer = self.layer() + original = layer.weight + dense = original.detach().clone() + self.method.process_weights_after_loading(layer) + self.assertIs(layer.weight, original) + torch.testing.assert_close(layer.weight, dense, rtol=0, atol=0) + self.assertIn("aiter_prefill_weight", dict(layer.named_parameters())) + self.assertTrue(layer.aiter_prefill_weight.is_shuffled) + hidden = torch.randn(3, 64) + torch.testing.assert_close( + F.linear(hidden, layer.weight[8:].float()), + F.linear(hidden, dense.float())[..., 8:], + ) + + def test_small_m_dense_and_large_m_packed(self): + layer = self.layer() + dense = layer.weight.detach().clone() + self.method.process_weights_after_loading(layer) + for m in (1, 16, 17, 256): + with self.subTest(m=m): + x = torch.randn(1, m, 64, dtype=torch.bfloat16) + bias = torch.randn(16, dtype=torch.bfloat16) + output = self.method.apply(layer, x, bias) + expected = F.linear(x.float(), dense.float(), bias.float()).to(x.dtype) + torch.testing.assert_close(output, expected, rtol=0, atol=0) + name, used_weight, _, packed = self.backends.calls[-1] + self.assertEqual(name, "tgemm") + self.assertEqual(packed, m > 16) + self.assertIs( + used_weight, layer.aiter_prefill_weight if m > 16 else layer.weight + ) + + def test_shared_method_does_not_mix_layer_layouts(self): + aligned, unaligned = self.layer(), self.layer(n=15, k=63) + for layer in (aligned, unaligned): + self.method.process_weights_after_loading(layer) + self.assertFalse(hasattr(unaligned, "aiter_prefill_weight")) + for layer in (aligned, unaligned, aligned): + x = torch.randn(17, layer.weight.shape[1], dtype=torch.bfloat16) + output = self.method.apply(layer, x) + torch.testing.assert_close( + output, F.linear(x.float(), layer.weight.float()).to(x.dtype) + ) + self.assertEqual(self.backends.calls[-1][3], layer is aligned) + + def test_shuffle_off_keeps_existing_dense_tgemm_path(self): + self.shuffle.value = False + layer = self.layer() + self.method.process_weights_after_loading(layer) + self.assertFalse(hasattr(layer, "aiter_prefill_weight")) + x = torch.randn(17, 64, dtype=torch.bfloat16) + output = self.method.apply(layer, x) + self.assertIs(self.backends.calls[-1][1], layer.weight) + self.assertFalse(self.backends.calls[-1][3]) + torch.testing.assert_close( + output, F.linear(x.float(), layer.weight.float()).to(x.dtype) + ) + + def test_non_viewable_prefill_is_flattened_before_packed_dispatch(self): + layer = self.layer() + self.method.process_weights_after_loading(layer) + x = torch.randn(2, 12, 64, dtype=torch.bfloat16).transpose(0, 1) + self.assertFalse(x.is_contiguous()) + output = self.method.apply(layer, x) + self.assertEqual(self.backends.calls[-1][2], (24, 64)) + self.assertEqual(output.shape, (12, 2, 16)) + torch.testing.assert_close( + output, F.linear(x.float(), layer.weight.float()).to(x.dtype) + ) + + +class TestPackedFP8Tuples(unittest.TestCase): + def setUp(self): + self.backends = Backends() + self.enabled = Flag(True) + self.namespace = { + "torch": torch, + "_use_aiter": True, + "envs": SimpleNamespace(SGLANG_ROCM_USE_AITER_LINEAR_FP8HIPB=self.enabled), + "is_gfx942_supported": lambda: True, + "cutlass_fp8_supported": lambda: False, + "aiter": SimpleNamespace( + gemm_a8w8_bpreshuffle=self.backends.ck, + per_token_quant_hip=self.backends.quant, + ), + } + ptpc = load_definition( + QUANT / "fp8_utils.py", "apply_fp8_ptpc_linear", self.namespace + ) + self.namespace["apply_fp8_ptpc_linear"] = ptpc + self.namespace["apply_fp8_linear"] = lambda **kwargs: (_ for _ in ()).throw( + AssertionError("packed CHANNEL weights reached the dense [K,N] path") + ) + self.channel = object() + self.namespace["QuantizationStrategy"] = SimpleNamespace(CHANNEL=self.channel) + cls = load_definition( + QUANT / "compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py", + "CompressedTensorsW8A8Fp8", + self.namespace, + {"apply_weights"}, + ) + self.method = cls() + self.method.strategy = self.channel + self.method.weight_block_size = None + self.dense = ( + torch.arange(32 * 64, dtype=torch.float32).remainder(13).view(32, 64) / 8 + ) + self.dense = self.dense.to(torch.float8_e4m3fnuz).float() + self.layer = torch.nn.Module() + self.layer.weight = Parameter(pack(self.dense).to(torch.float8_e4m3fnuz), False) + self.layer.weight.is_shuffled = True + self.layer.weight_scale = Parameter( + torch.linspace(0.01, 0.02, 32).view(32, 1), False + ) + self.layer.input_scale = None + module = ModuleType("aiter.tuned_gemm") + module.tgemm = self.backends + parent = ModuleType("aiter") + parent.__path__ = [] + parent.tuned_gemm = module + self.modules = patch.dict( + sys.modules, {"aiter": parent, "aiter.tuned_gemm": module} + ) + self.modules.start() + self.addCleanup(self.modules.stop) + + def activation(self, m): + return ( + torch.arange(m * 64, dtype=torch.float32).remainder(11).view(1, m, 64) / 8 + ).to(torch.float8_e4m3fnuz) + + def reference(self, qx, scale, dtype, bias): + x = qx.float().reshape(-1, 64) + scale = scale.reshape(-1, 1) if scale.numel() > 1 else scale + output = ((x * scale) @ (self.dense * self.layer.weight_scale).t()).to(dtype) + return (output + bias).view(*qx.shape[:-1], 32) + + def test_tuple_dispatch_preserves_layout_scale_and_dtype(self): + for m in (1, 16, 17): + qx = self.activation(m) + for enabled in (False, True): + self.enabled.value = enabled + for per_token in (False, True): + scale = ( + torch.linspace(0.1, 0.2, m).view(m, 1) + if per_token + else torch.tensor(0.125) + ) + for dtype in (None, torch.bfloat16, torch.float16): + with self.subTest( + m=m, enabled=enabled, per_token=per_token, dtype=dtype + ): + output_dtype = dtype or torch.bfloat16 + bias = torch.linspace(-0.1, 0.1, 32).to(output_dtype) + value = (qx, scale) if dtype is None else (qx, scale, dtype) + output = self.method.apply_weights(self.layer, value, bias) + expected = self.reference(qx, scale, output_dtype, bias) + self.assertEqual(output.dtype, output_dtype) + torch.testing.assert_close(output, expected, rtol=0, atol=0) + route = ( + "tgemm" + if enabled + and m > 16 + and per_token + and output_dtype == torch.bfloat16 + else "ck" + ) + self.assertEqual(self.backends.calls[-1][0], route) + + def test_invalid_scale_counts_are_rejected_before_kernel(self): + qx = self.activation(17) + for scale in (torch.ones(2), torch.ones(17, 2)): + with self.subTest(shape=scale.shape): + with self.assertRaisesRegex(ValueError, "scale"): + self.method.apply_weights(self.layer, (qx, scale)) + self.layer.weight_scale = Parameter(torch.ones(31, 1), False) + with self.assertRaisesRegex(ValueError, "scale"): + self.method.apply_weights(self.layer, (qx, torch.ones(17, 1))) + + def test_non_viewable_tuple_preserves_leading_dimensions(self): + qx = self.activation(34).view(2, 17, 64).transpose(0, 1) + scale = torch.ones(34, 1) / 8 + bias = torch.zeros(32, dtype=torch.float16) + output = self.method.apply_weights(self.layer, (qx, scale, torch.float16), bias) + self.assertEqual(output.shape, (17, 2, 32)) + torch.testing.assert_close( + output, self.reference(qx, scale, torch.float16, bias), rtol=0, atol=0 + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/registered/unit/spec/test_dflash_qkv_weight_layout.py b/test/registered/unit/spec/test_dflash_qkv_weight_layout.py new file mode 100644 index 000000000000..b0420422e077 --- /dev/null +++ b/test/registered/unit/spec/test_dflash_qkv_weight_layout.py @@ -0,0 +1,126 @@ +"""DFlash KV materialization must respect post-load linear weight layouts.""" + +import unittest +from types import SimpleNamespace + +import torch +from torch import nn + +from sglang.srt.layers.quantization.unquant import UnquantizedLinearMethod +from sglang.srt.models.dflash import DFlashAttention +from sglang.srt.speculative.dflash_utils import ( + can_dflash_slice_qkv_weight, + can_dflash_use_fused_qkv_proj, +) +from sglang.test.ci.ci_register import register_cpu_ci + +register_cpu_ci(est_time=5, suite="base-a-test-cpu") + + +class LayoutAwareQKV(nn.Module): + """CPU stand-in: expose packed storage and consume canonical values in forward.""" + + def __init__(self, canonical, *, packed, bias=None): + super().__init__() + self.canonical = canonical + physical = canonical.t().flip(1) if packed else canonical + self.weight = nn.Parameter(physical, requires_grad=False) + self.weight.aiter_trans_weight = packed + self.bias = bias + self.quant_method = UnquantizedLinearMethod() + self.calls = 0 + + def forward(self, hidden): + self.calls += 1 + return torch.nn.functional.linear(hidden, self.canonical, self.bias), None + + +def attention(proj, q_size, kv_size, v_scale=None): + result = DFlashAttention.__new__(DFlashAttention) + nn.Module.__init__(result) + result.qkv_proj = proj + result.q_size = q_size + result.kv_size = kv_size + result.v_scale = v_scale + return result + + +class TestDFlashQKVWeightLayout(unittest.TestCase): + def test_dense_weights_keep_kv_only_path(self): + for dtype in (torch.float16, torch.bfloat16, torch.float32): + with self.subTest(dtype=dtype): + proj = LayoutAwareQKV( + torch.arange(48).reshape(12, 4).to(dtype), packed=False + ) + attn = attention(proj, 8, 2) + hidden = torch.ones(3, 4, dtype=dtype) + expected = torch.nn.functional.linear(hidden, proj.canonical)[..., 8:] + k, v = attn.kv_proj_only(hidden) + torch.testing.assert_close(torch.cat((k, v), dim=-1), expected) + self.assertEqual(proj.calls, 0) + self.assertTrue(can_dflash_slice_qkv_weight(proj)[0]) + self.assertTrue(can_dflash_use_fused_qkv_proj(proj)[0]) + + def test_packed_weights_use_layout_aware_fallback(self): + for hidden_size in (4, 12): + with self.subTest(hidden_size=hidden_size): + canonical = ( + torch.arange(12 * hidden_size, dtype=torch.float32).reshape( + 12, hidden_size + ) + / 16 + ) + proj = LayoutAwareQKV(canonical, packed=True) + attn = attention(proj, 8, 2, v_scale=torch.tensor(0.5)) + hidden = ( + torch.arange(3 * hidden_size, dtype=torch.float32).reshape( + 3, hidden_size + ) + / 8 + ) + qkv = torch.nn.functional.linear(hidden, canonical) + k, v = attn.kv_proj_only(hidden) + torch.testing.assert_close(k, qkv[..., 8:10]) + torch.testing.assert_close(v, qkv[..., 10:12] * 0.5) + self.assertEqual(proj.calls, 1) + eligible, reason = can_dflash_use_fused_qkv_proj(proj) + self.assertFalse(eligible) + self.assertIn("AITER-swizzled", reason) + + def test_ci_qkv_shape_is_rejected_before_empty_slice(self): + weight = nn.Parameter( + torch.empty(2048, 6144, dtype=torch.bfloat16), requires_grad=False + ) + weight.aiter_trans_weight = True + proj = SimpleNamespace( + quant_method=UnquantizedLinearMethod(), weight=weight, bias=None + ) + self.assertEqual(tuple(weight[4096:6144].shape), (0, 6144)) + self.assertFalse(can_dflash_slice_qkv_weight(proj)[0]) + self.assertFalse(can_dflash_use_fused_qkv_proj(proj)[0]) + + def test_invalid_dense_storage_is_rejected(self): + for weight in ( + None, + torch.empty(2, 3, 4), + torch.empty(3, 4, dtype=torch.uint8), + ): + with self.subTest(shape=None if weight is None else tuple(weight.shape)): + proj = SimpleNamespace( + quant_method=UnquantizedLinearMethod(), weight=weight, bias=None + ) + self.assertFalse(can_dflash_slice_qkv_weight(proj)[0]) + + def test_quantized_and_biased_fused_guards_remain(self): + proj = SimpleNamespace( + quant_method=object(), weight=torch.empty(12, 4), bias=None + ) + self.assertFalse(can_dflash_slice_qkv_weight(proj)[0]) + proj.quant_method = UnquantizedLinearMethod() + proj.bias = torch.ones(12) + self.assertTrue(can_dflash_slice_qkv_weight(proj)[0]) + self.assertFalse(can_dflash_use_fused_qkv_proj(proj)[0]) + + +if __name__ == "__main__": + unittest.main()