Repository navigation
Conversation
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
One backend per PR: PR title tags & labels: |
github-actions
Bot
changed the title
|
+Zoe for tune flow review |
0e7e7bc to
6444612
Compare
|
This is not merge-ready yet. The new heuristic routes Please exclude every known-incorrect FlyDSL shape and route it to a verified-correct fallback, then add dispatch regression coverage around the |
On gfx1250 a BF16 GEMM without a tuned row runs on Triton. Where gluon has no config for the (N, K), it runs DEFAULT.json, and the FlyDSL gfx1250 kernel with a fixed config takes 0.35 to 0.68 of that time for M >= 512, N >= 4096 and K >= 512. Pick FlyDSL in that case (BF16 input, BF16 or FP32 output): 128x128x64 tiles up to M = 768, 256x128x64 above, 4 stages. Smaller grids, short K, fp16 and (N, K) pairs with their own gluon config stay on Triton. Signed-off-by: Jin Pan <jin.pan@amd.com>
The gfx1250 FlyDSL kernel writes C with 32-bit byte offsets and drops the rows of its last M tile past M only through C's record count. Once those rows reach byte 2^32 they wrap onto the first rows of C, so the default now picks FlyDSL only when the padded last tile ends below 2^32. Larger outputs go to torch: Triton's default config leaves the rows past byte 2^31 unwritten. FP32 outputs with a bias stay on Triton, because flydsl_gemm adds an unfused bias in the bias dtype and a BF16 bias rounds the FP32 result.
The gfx1250 gluon DEFAULT.json was retuned (#6549). Against it, the two FlyDSL default configs are slower than Triton in every repeat on 107 of the 368 untuned shapes with 1024 < M <= 4096 (up to 1.28x), the M_LEQ_2048 and M_LEQ_4096 buckets of that file, and stay faster for M from 512 to 1024 or above 4096 (0.50 to 1.10, geomean 0.78). The FlyDSL default now skips that range.
… on gfx1250 Triton's default gfx1250 config leaves the rows of C past byte 2^31 unwritten. An untuned shape without its own gluon config now runs on it only when C fits in 2^31 bytes, and on torch otherwise. The FlyDSL default keeps 1024 < M <= 4096 when C is larger, because FlyDSL writes C up to 2^32 bytes and runs about 10x faster than torch on such outputs.
The docstring now covers the validation run at M values the range was not chosen from (896, 1100, 1800, 2600, 3500, 4500, 5000, 7000, 10000) as well as the two sets the range came from.
6444612 to
27670ca
Compare
|
@zufayu thanks, fixed in 27670ca (rebased onto main c118faf). The description now has the gfx1250 results at that head.
The retest changed two more things. #6549 retuned the gluon @yzhou103 the gfx1250 tests are in the description. |
|
The updated PR still has correctness and dispatch blockers. For large outputs, the Torch fallback converts through the input dtype before producing the requested output dtype. For BF16-to-FP16 and FP16-to-BF16 cases, that introduces an extra rounding step compared with the original Gluon path, which converts the FP32 accumulator directly to the output type. Values around rounding boundaries can therefore change when the dispatch crosses the size threshold. Also, current main already contains #6629, which removes the Gluon 2 GiB limitation that motivated this fallback. Keeping the old size-based switch after that fix can route supported shapes to a slower path. The newly selected FlyDSL Please rebase on current main including #6629, redesign the dispatch around the remaining real limitations, and verify every newly selected FlyDSL path. The validation should cover cross-dtype output, values around the dispatch threshold, both bias modes, and same-machine performance against the current default path. |
READY FOR HUMAN REVIEWPR Self-ReviewRevision Author self-review accepted. Readiness report: Author answers
Show the author's self-review answersBehavior and intentValidation and unverified scopePR-specific questionChanged op testsPR state
Copilot findings
Code and op-test hygiene
Required CI
Shadow mode reports what would block but keeps the status successful. Set |
|
Hi @jhinpan please resolve the conflicts |
Refs #6345, #6402.
On gfx1250, a BF16 GEMM without a tuned row runs on Triton (gluon). Where gluon has no config for the shape's (N, K), it runs its
DEFAULT.jsonconfig.get_GEMM_A16W16_confignow picks AITER's FlyDSL gfx1250 kernel with a fixed config for such a shape when all of these hold:GEMM-A16W16-N={N}-K={K}.jsonamong the gfx1250 gluon configs;M ≤ 768 gets 128x128x64 tiles with 4 stages and 2x2 waves, larger M gets 256x128x64 with 4 stages and 4x2 waves. Other untuned shapes without their own gluon config run on Triton's default config when C fits in 2^31 bytes, and on torch otherwise. Tuned rows still win because they are matched first. Only gfx1250 changes.
Why the limits
M_LEQ_2048andM_LEQ_4096buckets of the retunedDEFAULT.json([Triton/Gluon] [Config] Tune gfx1250 Gluon a16w16 GEMMs for GPT-OSS 120B #6549) are as fast as these FlyDSL configs.flydsl_gemmadds a bias it cannot fuse in the bias dtype, so a BF16 bias rounded an FP32 output (max row rel-L2 0.0023, against 0.0000 on Triton).get_gemm_config'sis_tuned: these files cover most M through theiranyentry, for whichis_tunedis false.Measurements
gfx1250 B0 (
heliosr-1b114-d04-1, ASIC revision 1),lmsysorg/sglang-rocm:v0.5.20-rocm10-mi45x-20260921(ROCm 10.0.0, torch 2.11.0, Triton 3.8.0), FlyDSL 0.4.0, main c118faf. Each kernel called the waytuned_gemmcalls it, same inputs in one process, median of 7 interleaved rounds, 3 repeats in separate runs. A shape counts as slower only if every repeat is above 1.0081, the noise of this GEMM on this host. Every output was checked row by row against FP32. The shapes are the 98 (N, K, bias, output dtype) combinations of the BF16 untuned CSVs with N ≥ 4096, K ≥ 512 and no gluon config, without the ones that have a tuned row.FlyDSL / Triton's default config:
The M range comes from this table, so I checked it again at M values it was not chosen from:
Between M = 4096 and 8192, up to 8% of the shapes stay slower on FlyDSL, by at most 1.10x. They have N ≤ 9216 and K ≥ 6144, for example 5120 x 17408 and 5120 x 25600. At M = 640, the 128x128 config took 0.84 of the time of 256x128; at M = 1000, 1.00.
Large outputs
Wrong rows against FP32, output memory filled with NaN first. Kernels on main c118faf; the last column is this PR's dispatch at 27670ca:
The MI450 A0 benchtop gives the same rows. torch takes 141 ms on 16384 x 151936 x 4096; moving C's buffer base per tile in both kernels (#6402) would let them write such outputs.
DeepSeek shapes on gfx1250
I replayed the dispatch on every (N, K) of the DeepSeek CSVs in
aiter/configs/model_configs/(dsv4, dsv41, dsr1, dsv3) with the gfx1250 tuned rows (cu_num 256) and the gluon configs. Four (N, K) reach the FlyDSL default; the others have a tuned row, a gluon config, or N < 4096. DeepSeek-V4-Flash on one GPU does not change: its lm_head (129280 x 4096) has a gluon config. Every row of these shapes is right through the new dispatch, and FlyDSL / Triton is:At M = 2048 these shapes stay on Triton (FlyDSL took 0.95 to 0.98 there). 32768 x 512 reaches the default above M = 4096 because its gfx1250 tuned rows stop at M = 6144 and come in steps of 512 above 4096.
Tests
TestGfx1250DefaultDispatchmocks gfx1250 and checks each branch: both FlyDSL tiles, the M range, both sides of FlyDSL's 2^32 limit (14080 / 14081 x 151936 x 4096) and of the 2^31 limit of Triton's default config (7067 / 7068 x 151936), and bias with FP32 output. It runs in the nightly Tuning Tests workflow, not in the PR checks. It passes at 27670ca on the B0 host above, and its new cases fail on the previous heads.PR Self-Review
Reviewed revision
6b05290
Behavior and intent
On gfx1250, a BF16 GEMM without a tuned row and without a gluon config for its (N, K) ran Triton's
DEFAULT.jsonconfig.get_GEMM_A16W16_confignow gives such a shape AITER's FlyDSL gfx1250 kernel with a fixed 128x128x64 or 256x128x64 config inside the M, N and K limits above, when its padded last M tile of C ends below byte 2^32. A shape that FlyDSL cannot take and whose C passes byte 2^31 goes to torch, because Triton's default config leaves those rows unwritten. Tuned rows and gluon (N, K) configs still win. Other architectures do not change.Validation and unverified scope
heliosr-1b114-d04-1, FlyDSL 0.4.0, main c118faf).python3 -m pytest op_tests/tuning_tests/test_a16w16_tuned_config_lookup.py: the 5TestGfx1250DefaultDispatchtests and the other test methods passed. 8 subtests of main'sTestLegacyFlydslHgemmConfigs::test_gfx1250_legacy_wrapper_routingfailed the same way on main, because they launch the TDM kernel and need a gfx1250 GPU.flydsl_hgemm_kernel_namestill builds..._pft_gfx1250from this config, andget_flydsl_hgemm_kernel_paramsparses it. Since c118faf, the gfx1250 gluonDEFAULT.jsononly gainedREGISTER_STORE: 0, which is the kernel's default, andgemm_a16w16_gfx1250.pyonly changed its split-K counter cache.PR-specific question
No CSV or JSON file changes. The new default applies only when
get_GEMM_A16W16_tuned_configmisses on gfx1250 with BF16 input, BF16 or FP32 output and no scales, the (N, K) has noGEMM-A16W16-N={N}-K={K}.jsonamong the gfx1250 gluon configs, and the shape is inside the limits above. Every other key keeps its row or its previous default: the branch sits inside the gfx1250 default path, and tuned rows are matched before it.test_other_shapes_stay_on_tritonandTestA16W16TunedConfigLookupcover the unchanged selections (M = 256, 1025 to 4096, N < 4096, K < 512, an (N, K) with a gluon config, FP16, tuned rows).Changed op tests
op_tests/tuning_tests/test_a16w16_tuned_config_lookup.py: addsTestGfx1250DefaultDispatch. It mocks gfx1250 and checks each branch of the new default: both FlyDSL tiles, the M range, both sides of FlyDSL's 2^32 limit and of the 2^31 limit of Triton's default config, and bias with FP32 output. No existing test covers the gfx1250 default dispatch, and the new cases fail on the previous heads.