[Qubitium]

Summary

This PR makes gfx950 MXFP4 FlyDSL tuning practical for GLM-5.3-Flash TP=1. It moves cold kernel compilation out of the serial GPU candidate loop, compiles deduplicated variants in parallel, reuses one immutable shape fixture and torch reference, and applies an accuracy-aware two-stage scan.

This PR also addresses an important accuracy-selection issue: main can select a less accurate kernel for a negligible speed advantage. In the new --fast-scan path, final selection prefers the lowest worst observed numerical error among validated, retimed finalists within 1% of the fastest finalist. This preserves better numerical accuracy in cases where the latency difference is minor.

The timing score is summed active GPU-kernel execution time for the coupled sorting + G1 + G2 operation. Python dispatch, HIP launch overhead, queue gaps, and GPU-idle gaps are excluded. The coupled port produces one measurement rather than independent stage measurements, so the CSV stores it in us/us1; us2=0 explicitly means no standalone G2 split was measured.

Accuracy-selection fix: preserve accuracy when latency is nearly equal

Main uses numerical accuracy as a pass/fail gate, then chooses the lowest measured latency. Once candidates pass --errRatio, lower error does not influence the winner. Consequently, even a tiny measured speed advantage can displace a substantially more accurate candidate; exact latency ties retain the earlier candidate regardless of accuracy.

The new --fast-scan final selection changes that behavior:

1. Find the fastest valid, retimed finalist.
2. Admit finalists whose latency is at most 1% higher.
3. Select the lowest worst observed numerical error in that set, using latency to break an accuracy tie.

The raw, unrounded error is retained across coarse warm-up, coarse measured-output, and final measured-output checks. A final retime cannot overwrite a larger previously observed error with a smaller one. Numerical rejection thresholds remain in force.

Illustrative example, assuming both candidates pass validation and reach final retiming:

Candidate	Measured latency	Observed error	Main selection	New fast-scan final selection
A	100.0 µs	0.020	Selected for lowest latency	Not selected
B	100.5 µs	0.005	Discarded despite lower error	Selected: 75% lower error for 0.5% higher latency

These are illustrative values demonstrating the selection rule, not new GPU benchmark results or a claim about downstream model accuracy. The reported token-1 benchmark below selected the same kernel pair on main and the PR.

Scope: this corrects cases of unnecessary accuracy loss within the retimed finalist set. It does not guarantee the most accurate eligible candidate across the entire coarse pool, because the funnel may exclude a candidate before final retiming. The ordinary scan path without --fast-scan remains latency-first.

Sources: main's accuracy gate, main's latency-only winner selection, and PR's 1% band and accuracy-based final selection.

Main behavior addressed

Main behavior	Final PR behavior	Source
Cold variants JIT-compile serially inside the GPU candidate loop	Convert legal pairs into deduplicated jobs and precompile them in parallel before GPU scoring	gemm_moe_tune.py#L6603-L6650, common.py#L345-L379
Every coupled pair rebuilds and reshuffles the large expert fixture/reference	Build one immutable fixture and one torch reference for the complete shape sweep	gemm_moe_tune.py#L6689-L6720
GLM-5.3-Flash TP=1 had not been swept	Add (NE=288, H=4096, INTER=2048, TOPK=8) auxiliary generation and 16 tuned token tiers	gen_instances.py#L17-L44, glm53_fp4_tuned_fmoe.csv

Correctness and worker-policy changes

Issue	Final behavior	Source
Shared use_cuda_event=True semantics could be changed by the scan work	Restored base per-invocation event timing: each iteration is independently recorded, synchronized, and averaged. MXFP4 fast-scan does not use this shared path; it uses profiler-derived active-kernel timing.	test_common.py#L55-L110, mp_tuner.py#L57-L118, gemm_moe_tune.py#L6517-L6582
Accuracy-first coarse bucket ordering could drop the true fastest candidate	Performance finalists are the lowest-latency coarse survivors, so the coarse global fastest is always retimed. Accuracy promotion remains a separate distinct path.	gemm_moe_tune.py#L300-L374, test_fast_tuning.py#L75-L94
Final-validation failures could exhaust only the initial six finalists	Failed finalists are skipped and the next-lowest-latency unselected coarse survivors are retimed until the requested successful-retime count is reached or the pool is exhausted.	gemm_moe_tune.py#L6757-L6860, test_fast_tuning.py#L282-L416
Shape-grouped failures could be reported against the wrong kernel after regrouping	Failure diagnostics now use the info returned by mp_tuner, which remains paired with the actual task after shape grouping.	gemm_moe_tune.py#L5171-L5188
Worker fanout needed one source of truth and a bounded default	AITER_MAX_JOBS is the only explicit worker override. Without it, automatic sizing is min(80% of process-available CPUs, available-memory budget) using the fixed observed 1.5 GB/worker estimate. Descendants force AITER_MAX_JOBS=1.	aiter_worker_limits.py#L181-L241, common.py#L209-L210, flydsl/README.md#L60-L83
Profiler controls accepted one iteration even though the profiler requires more than one	--fast-scan-iters and --fast-scan-final-iters reject values below 2 at parse time and runtime.	gemm_moe_tune.py#L283-L291, test_fast_tuning.py#L44-L73
Candidate failures had inconsistent shape-level documentation	A candidate-local accuracy or launch failure is rejected individually. A shape is marked failed only when no candidate survives. The final documentation calls these the lowest-latency performance finalists and documents latency-ordered backfill.	README.md#L141-L174

Final measured errors preserve the maximum observed error across coarse warm-up, coarse timing output, and final timing output: gemm_moe_tune.py#L6502-L6600.

Final scan policy

Phase	Policy
Inventory	Enumerate all 174 legal coupled G1/G2 candidates
Compile	Parallel-precompile 143 deduplicated FlyDSL jobs
Shared data	Build one GPU fixture and one torch reference per model shape
Coarse funnel	Per candidate: one untimed warm-up-and-accuracy launch, then 10 active-kernel profiler samples
Coarse ranking	Accuracy-aware anchored 1% timing buckets for diagnostic ranking
Performance finalists	Five lowest-latency coarse survivors, always including the coarse global fastest
Accuracy promotion	Most accurate distinct unselected coarse survivor
Final retime	100 profiler samples per successful finalist target, no repeated warm-up, with backfill after failures
Final selection	Find the global fastest finalist, admit candidates within 1%, choose lowest worst-case cosine error, then latency as tie-breaker
Shape failure	Only when the coarse/final candidate pool has no valid survivor

Fast-scan arguments and usage

The following controls are implemented by Mxfp4FlydslTuner. Enable them with --mxfp4-flydsl --fast-scan; --fast-scan is rejected for unsupported tuner modes. An iteration here means one complete coupled MoE invocation (sorting, applicable quantization, G1, G2, and associated reduction), not one individual GPU kernel.

Argument	Type	Default	Behavior and limits
--fast-scan	Boolean flag	Off	Enable parallel candidate precompilation, one shared fixture/reference per shape, and the coarse/final measurement funnel. Omit the flag to retain the ordinary scan path.
--fast-scan-iters N	Integer	10	Measured invocations per candidate in the initial round. Must be at least 2; smaller values are rejected at argument parsing and runtime.
--fast-scan-final-iters N	Integer	100	Measured invocations per finalist in the final round. Must be at least 2. This is a configurable measurement budget, not a kernel correctness requirement.
--fast-scan-finalists N	Integer	5	Number of lowest-latency coarse survivors selected for final retiming. Runtime clamps the requested count to at least 1; the available survivor pool limits the actual count. One additional distinct accuracy candidate is promoted when available, so the default target is up to 6 successful retimes.
--fast-scan-warmup-accuracy-checks N	Integer	1	Untimed invocations whose outputs are validated before each candidate's coarse measurement. Runtime clamps negative values to 0. Setting 0 skips these premeasurement checks; the final output of the measured block is still checked.

In fast-scan mode, the dedicated controls above supply the measurement and warm-up counts instead of the ordinary --iters/--warmup settings. The existing --errRatio accuracy gate still applies. Candidate enumeration is separate: the existing --mxfp4-search-mode defaults to prune; use --mxfp4-search-mode full to enumerate all statically supported GEMM1 variants. The reported 174 candidate pairs describe the benchmark's configured search space, not every possible shape/search mode.

Default-budget example

Run from the repository root:

python3 csrc/ck_gemm_moe_2stages_codegen/gemm_moe_tune.py \
  -i aiter/configs/model_configs/glm53_fp4_untuned_fmoe.csv \
  -o /tmp/glm53_fp4_fast_scan_tuned.csv \
  --mxfp4-flydsl --fast-scan --mp 1 --batch 1 \
  --fast-scan-iters 10 \
  --fast-scan-final-iters 100 \
  --fast-scan-finalists 5 \
  --fast-scan-warmup-accuracy-checks 1

The four numeric settings are shown explicitly for discoverability; omitting them uses these same defaults. For the reported 174-candidate case with no failures, this means 174 untimed warm-up/check invocations, 1,740 coarse measured invocations, and 600 final measured invocations.

Validate the generated rows through production dispatch:

python3 csrc/ck_gemm_moe_2stages_codegen/gemm_moe_tune.py \
  --mxfp4-flydsl \
  --run_config /tmp/glm53_fp4_fast_scan_tuned.csv

Adjusting the measurement budget

For an experimental shorter final round, use the same tuning command with --fast-scan-final-iters 32 and a separate output path. Keep --fast-scan-iters 10 to leave initial screening unchanged. This is an example of the configurable budget, not a claim that 32 iterations has been validated to retain every winner or reproduce the published benchmark. Increase the initial-round count when coarse rankings are unstable; adding final samples cannot recover a candidate already excluded from the finalist set.

Finalists do not receive another untimed warm-up: they have already been compiled and exercised during the initial round (normally 1 untimed + 10 measured invocations). Repeating initialization warm-up is intentionally avoided. This does not assert that cache contents or GPU clock conditions remain identical between rounds. There is currently no separate final-round warm-up argument.

Timing, filtering, and selection semantics

• Both rounds use the profiler-derived mean of summed active GPU execution durations, excluding host dispatch and idle gaps. The fast path uses profile_warm_iters=0, so it does not automatically discard the first profiled invocation.
• Outlier filtering is enabled independently for a measured block only when its requested iteration count is greater than 30. It removes whole-iteration totals outside the Q1 - 1.5*IQR / Q3 + 1.5*IQR fences and averages the retained samples. Therefore the default 10-iteration coarse round is unfiltered, whereas the 100-iteration final round is filtered. Changing 100 to 30 disables this filtering; changing it to 32 retains it. The threshold is currently hardcoded.
• Warm-up outputs and the final output of each measured block are accuracy-checked outside timing; every measured invocation's output is not independently checked. Worst observed error is retained across coarse and final checks. Non-finite errors, excessive error, or launch failures reject that candidate; failed finalists are backfilled in coarse-latency order.
• Final selection admits retimed candidates within 1% of the fastest retimed finalist, then chooses the lowest worst-case error, using latency to break an accuracy tie. This 1% band is currently hardcoded and is relative to measured finalists, not a guarantee against an unmeasured global optimum.
• A fixed 100-sample final round is not adaptive: it neither stops early on a clear winner nor establishes a confidence interval. Fewer iterations reduce measurement work; winner stability should be checked on the target GPU and shapes before adopting a reduced default.

Implementation references at the reviewed head: arguments, candidate execution and two-stage scan, and timing/filtering.

Full main-vs-PR benchmark

This is a complete token-1 measurement, not a sampled or extrapolated estimate. The performance benchmark used PR head e438a359; the final corrective changes preserve the valid-candidate compile/timing path while fixing API, finalist, worker-policy, and validation behavior.

Item	Value
GPU	AMD Instinct MI355X VF / gfx950
Shape	token=1, hidden=4096, inter=2048, experts=288, topk=8
Main base	dbd8bf5bd624120197a7a26780a8c72201824f0f
Benchmark PR head	e438a3593762cbd96e76a554e1ff4f5bffdd240d
Final PR head	7440ef72
Fixed seed	20260901
Legal G1/G2 pairs	174
Deduplicated compile jobs	143
Main scan	All 174 pairs with main's exhaustive policy
PR scan	All 174 coarse pairs + 6 finalists retimed for 100 samples each
Failures	0 candidate failures; 0 compile failures

End-to-end and compile timing

Measurement	Main	PR	Speedup	Wall-time reduction
Cold full scan	223.418 s	25.417 s	8.790x	88.62%
Warm full scan	99.151 s	17.067 s	5.809x	82.79%
Post-compile GPU scan/funnel	99.151 s	11.853 s	8.365x	88.05%
Compile-only, empty cache	122.814 s	12.911 s	9.512x	89.49%

Both compile-only paths covered the same 143 deduplicated jobs with zero failures. The independently observed main cold-minus-warm delta was 124.267 s, cross-checking the direct 122.814 s serial compile result.

Selected kernel and accuracy

The complete main and PR scans selected the same pair:

Stage	Kernel
G1	flydsl_mxmoe_g1_a4w4_16x256x256_f16in_nt
G2	flydsl_moe2_layout_afp4_wfp4_bf16_t16x128x256_atomic_nt_sbm16

Run	Active-kernel time	Cosine error	Tuner similarity
Main cold	27.2696 us	0.011398	98.8602%
Main warm	28.5856 us	0.011405	98.8595%
PR cold	27.7506 us	0.011405	98.8595%
PR warm	28.0113 us	0.011406	98.8594%
Corrected full regeneration, token 1	27.7777 us	0.011412	98.8588%

A separate common-fixture validation retimed the selected pair for 500 profiler samples: median 26.6825 us, range 26.6289-26.7627 us, worst cosine error 0.01141738, and worst tuner similarity 98.858262%.

For this shape, main and the PR select the same implementation. The PR reduces scan cost and makes selection robust to timing noise while applying accuracy only inside the global 1% performance envelope.

Corrected artifact regeneration

The corrected implementation completed a warm-cache regeneration of every GLM-5.3 token tier in 182.4143 seconds.

Token	G1	G2	us	Cosine error
1	16x256x256_f16in_nt	t16x128x256_atomic_nt_sbm16	27.7777	0.011412
2	16x256x256_f16in_nt	t16x256x128_atomic_sbm16	43.5156	0.011396
4	16x256x256_f16in_nt	t16x128x256_atomic_nt_sbm16	73.0846	0.011279
8	16x256x256_f16in_nt	t16x128x256_atomic_persist_sbm16	126.6584	0.011345
16	16x256x256_f16in_nt	t16x128x256_atomic_persist_nt_sbm16	232.2952	0.011478
32	16x256x256_f16in_nt	t16x128x256_reduce_persist_nt_sbm16	358.6395	0.011435
64	16x256x256_f16in_nt	t16x128x256_atomic_persist_nt_sbm16	520.0278	0.011425
128	16x256x256_f16in_nt	t16x128x256_reduce_persist_nt_sbm16	610.9232	0.011429
256	16x256x256_f16in_nt	t16x128x128_atomic_nt_sbm16	619.0457	0.011450
512	32x256x256_nt	t32x256x128_reduce_nt_sbm32	647.7988	0.011462
1024	64x256x256	t64x256x128_atomic_nt_sbm64	724.9569	0.011463
2048	64x256x256	t64x256x128_atomic_nt_sbm64	793.8585	0.011421
4096	128x256x256	t128x128x128_atomic_nt_sbm128	929.2160	0.011441
8192	128x256x256	t128x128x256_atomic_persist_sbm128	1453.7014	0.011440
16384	64x256x256	t64x256x128_atomic_sbm64	2648.9033	0.011441
32768	64x256x256	t64x256x128_atomic_sbm64	4957.2630	0.011442

Scoped validation

Only tests directly covering the changed tuner, worker policy, and artifact were run. No broad AITER/operator test suite was used for this PR validation.

Validation	Result
python3.14 -m pytest -q op_tests/tuning_tests/test_fast_tuning.py tests/test_worker_awareness.py	43 passed, 6 subtests passed in 4.26 s
Targeted python3.14 -m py_compile of changed Python modules	passed
Full 16-tier GLM-5.3 MXFP4 regeneration	passed: 16/16 tiers, 174 candidates/tier, 143/143 compile jobs
Production --mxfp4-flydsl --run_config aiter/configs/model_configs/glm53_fp4_tuned_fmoe.csv	16/16 dispatched and cosine numerical gates accepted; exit 0. Elementwise checkAllclose warnings are expected for lossy FP4 and are not the acceptance gate.
CSV tier/timing/accuracy integrity check	passed
git diff --check	passed

[Qubitium]

Added a Fast-scan arguments and usage section to the PR description, with the complete argument table, explicit tuning and production-validation commands, and measurement semantics checked against head 53f99ce.

Control	Default	Meaning
--fast-scan	Off	Enable the two-stage path; requires --mxfp4-flydsl.
--fast-scan-iters	10	Initial measured coupled MoE invocations per candidate; minimum 2.
--fast-scan-final-iters	100	Final measured invocations per finalist; minimum 2.
--fast-scan-finalists	5	Performance finalists, plus one distinct accuracy candidate when available. Values below 1 are clamped to 1.
--fast-scan-warmup-accuracy-checks	1	Initial untimed, accuracy-checked invocations. Values below 0 are clamped to 0.

Warm-up clarification: the first round already compiles and exercises every surviving candidate. A finalist normally reaches retiming after 1 untimed accuracy-check invocation and 10 measured invocations. The final round intentionally adds no redundant initialization warm-up. Cache/clock conditions may evolve between rounds, but that is separate from needing to compile or initialize the candidate again.

100 final iterations is adjustable, not a correctness requirement. The description includes a 32-iteration experimental alternative without claiming it has the same validated winner retention. The current collector enables 1.5×IQR outlier filtering only above 30 measured iterations: 10 and 30 are unfiltered; 32 and 100 are filtered. The fast path does not automatically discard the first profiled sample. These settings replace the ordinary iteration/warm-up counts in fast-scan mode.

The new documentation also explains accuracy checks, finalist backfill, the fixed 1% final selection band, and the distinction between candidate search-space selection and measurement budgets. Published benchmark results remain attributed to their original configuration. This update changes documentation only; no iteration defaults or kernel behavior were changed.

[Qubitium]

In my private AITER fork, I already run AOT compilation and tuning automatically during model loading when a model encounters a previously unseen configuration within a supported module/kernel path. The kernel implementations are prevalidated; new configurations are compiled, tuned, and numerically checked before use.

Once this PR is validated, I believe we can integrate the faster tuning flow into model loading upstream. That would make selecting and preparing kernels for new models largely automatic wherever existing kernel support applies, without requiring a separate offline tuning sweep.