[0z5a]

Summary

examples/python/CuTeDSL/cute/hopper has seven examples, but test/examples/CuTeDSL/hopper only tests three of them (FP8 2xacc, FP8 GELU, grouped GEMM). This PR adds a test file for each of the other four, written like test_dense_gemm_fp8_gelu.py (L0 compiles and launches once; L1 checks against the PyTorch reference):

File	Coverage
test_dense_gemm.py	all tile shapes x all cluster shapes; f16 (f16/f32 out, f32/f16 acc), e4m3, e5m2, s8, u8; five A/B/C major combinations; batched, partial-tile and single-tile problems
test_dense_gemm_persistent.py	the same, plus the tile scheduler's swizzle size (1/2/4) and raster order
test_fmha.py	head dims 32/64/128/256, persistent and non-persistent (d256 with the N tiles can_implement accepts); no mask, causal and two sliding windows; partial tiles incl. S_q = 1 and bottom-right causal; GQA (8:2) and MQA; bf16; fp8
test_cta_norm.py	LayerNorm and RMSNorm in f16/bf16/f32; 32 to 1024 threads; single row, N not a multiple of a warp tile, many rows, large N

Writing the tests turned up three small problems in the examples. The cta_norm.py fix is in the same commit as its test; the two fmha.py changes are a separate commit:

1. cta_norm.py: the reference check used atol = rtol = 1e-3 for every dtype. bf16 keeps 8 mantissa bits, so a one-ulp rounding difference from PyTorch is already about 4e-3 relative, and 6 of the 24 bf16 cases failed (1–7 mismatched elements out of 0.5–2 M, max abs diff 0.0039–0.0078). It now uses 1.6e-2 for bf16 (the PyTorch default rtol for bf16). The command line also ran Float16 regardless of --dtype.
2. fmha.py: scale_softmax = 0 is documented as 1/sqrt(D) but computed 1 / sqrt(q_shape[1]), i.e. 1/sqrt(S_q). It now uses the head dimension.
3. fmha.py: the module docstring says Q and K must have the same number of heads, but run() already broadcasts K/V over h // h_k query heads, and the new GQA/MQA cases pass the reference check. The constraint now reads "a multiple of".

Related: #3399 (max reduction in the FMHA softmax) is performance-only; the new FMHA tests pass with and without it (85/85 each). I posted GH200 timings there.

No overlap with #3648 (bf16 inputs for the dense GEMM examples): the dense GEMM tests use the dtypes is_valid_dtypes accepts today; bf16 cases can be added to the DTYPES list once #3648 lands.

Results (GH200)

Full files at the default level (L0 and L1 cases), pytest 9.0.2 with the cutlass_cicd plugins:

File	Result	Time
test_dense_gemm.py	62 passed	35 s
test_dense_gemm_persistent.py	68 passed	79 s
test_fmha.py	85 passed	135 s
test_cta_norm.py	40 passed (6 bf16 failures before fix 1)	11 s

The runs used torch 2.14.1 rather than the 2.11.0 pinned in cutlass_cicd requirements.

GPU: GH200 (cc 9.0) | driver 595.58 | CUDA 13.2 | nvidia-cutlass-dsl 4.8.1 | torch 2.14.1+cu130 | CUTLASS 91843b71

[arvin-chou]

Thanks for filling this coverage gap — and for the bundled example fixes.

Minor suggestions — none blocking

1. fmha.py:72 docstring still says d256 non-persistent "shoule be (64, 256)" (typo included), but can_implement rejects (64,256) at d256 for every dtype. Since you're already editing this docstring, fix the stale line in the same commit.
2. test_fmha.py's docstring says d256 non-persistent needs "an N tile below 128" — that's the 16-bit rule; fp8 permits up to 128 (your own test_l1_fp8 uses (64,128)). Worth scoping the claim: "(16-bit types; fp8 permits up to 128)".
3. test_cta_norm.py doesn't follow the L0-smoke / L1-verify split the description claims: every test is bare @pytest.mark.L0 yet always verifies (skip_ref_check=False), with no L1 cross-marks. Depending on how the cicd plugin selects on bare L0 vs L0(0)/L1(0), these 40
   verifying cases may land in a different suite than the other files'. Either split it like the GEMM files, or note the intent in a comment / adjust the description.

cc @Junkai-Wu.