[OneDNN 3.14.2][BMG] E2E Performance drops compared with OneDNN 3.13.3 #6145

Activity

1. kaileiyx

   changed the title [-][OneDNN 3.14.2][BMG] E2E Performance drops compared with OneDNN 3.14.2[/-] [+][OneDNN 3.14.2][BMG] E2E Performance drops compared with OneDNN 3.13.3[/+] on Nov 11, 2026
2. daisyden

   added

   module: gemmMatmul family (matmul/addmm/bmm/gemm)Matmul family (matmul/addmm/bmm/gemm)

   dtype: amp_fp16Autocast/AMP fp16Autocast/AMP fp16

   dependency component: oneDNNoneDNN primitive: matmul/conv/linear/quantizedoneDNN primitive: matmul/conv/linear/quantized

   performancePerformance-related failurePerformance-related failure

   regressionRegression with before/after version or commit signalRegression with before/after version or commit signal

   inferenceInference workload phaseInference workload phase

   test: e2eEnd-to-end / benchmark failure surfaceEnd-to-end / benchmark failure surface

   hw: BMGBattlemageBattlemage

   on Nov 11, 2026
3. daisyden

   added the

   Bug

   issue type on Nov 11, 2026
4. daisyden commented on Nov 11, 2026

   

   daisyden

   on Nov 11, 2026

   Contributor

   More actions

   label-issue: #6145

   Root cause: oneDNN 3.14.2 vs 3.13.3 causes E2E inference latency regressions on BMG across 4 models; both inductor and eager ratios drop (whisper-tiny eager 0.5327), so the defect is in oneDNN primitives, not inductor codegen. One group (shared cause: oneDNN version bump).

   Analyzed case: huggingface openai/whisper-tiny inference amp_fp16 (case 1 of 4; the other 3 not analyzed)

   axis	value	reason
   type	Bug	Type unset; measured performance regression between oneDNN releases.
   test	test: e2e	benchmarks/dynamo/{huggingface,timm_models,torchbench}.py --performance runs.
   module	module: gemm	oneDNN matmul primitives dominate the regressed transformer model (representative whisper-tiny).
   priority	Urgent	Measured perf regression >7% between releases: whisper-tiny ind 0.6224 / eager 0.5327 (~38-47% slower).
   hw	hw: BMG	Title [BMG]; BMG Linux oneDNN comparison.
   dtype	dtype: amp_fp16	Representative case runs --amp --amp-dtype float16.
   dependency component	dependency component: oneDNN	Regression bracketed by oneDNN 3.14.2 vs 3.13.3; originates in oneDNN primitives.
   symptom	performance	Latency ratios < 0.9 across 4 models.
   symptom	regression	OneDNN 3.14.2 (bad) vs 3.13.3 (good) before/after version signal.
   symptom	inference	All four cases are inference-mode benchmarks.

   Test cases (4):

   • huggingface openai/whisper-tiny inference amp_fp16 (ind 0.6224 / eager 0.5327)
   • timm gmlp_s16_224 inference bfloat16 (0.8735)
   • timm gmlp_s16_224 inference float16 (0.869)
   • torchbench detectron2_fcos_r_50_fpn inference float16 (0.8558)

   ---

   Applied to this issue. Generated by a local label-issue run.
5. mengfei25 commented on Nov 12, 2026

   

   mengfei25

   on Nov 12, 2026

   Contributor

   More actions

   Root-cause analysis (oneDNN 3.14.x vs 3.13.3, BMG B60)

   Env: host 818733, same image intelgpu/ubuntu-26.04-rolling:26.18, target torch 2.15.0.dev20261105+xpu (oneDNN 4db284662a), baseline torch 2.15.0+xpu (oneDNN v3.13.3). Each case: 3 perf runs on the same card, plus ONEDNN_VERBOSE=1 and TORCHINDUCTOR_SHAPE_PADDING=0 runs; all oneDNN shapes re-timed with benchdnn built from the same oneDNN commits. All oneDNN primitives run on XPU (0 CPU exec).

   Case	Inductor (tgt/base latency)	Eager	Shape padding off	Conclusion
   gmlp_s16_224 fp16	54.37 → 62.16 ms (+14.3%)	0.997	54.34 → 54.11 (0.996)	Reproduced. Inductor pad_mm mis-decision
   gmlp_s16_224 bf16	55.25 → 63.07 ms (+14.1%)	0.995	55.19 → 54.70 (0.991)	Same as above
   whisper-tiny amp_fp16	38.26 → 38.31 ms (1.001)	1.012	–	Not reproduced
   detectron2_fcos_r_50_fpn fp16	18.71 → 18.62 ms (0.995)	31.17 → 28.22 (faster)	–	Not reproduced

   For whisper-tiny / detectron2 the oneDNN verbose shapes and implementations are identical between versions, and benchdnn on all 134 XPU shapes of the two models shows no gap (oneDNN XPU time weighted by call count: whisper-tiny 500.0 → 500.1 ms, detectron2 537.3 → 536.7 ms; only 7 tiny µs-level matmuls are >5% slower). These two look like run-to-run noise in the original test.

   gmlp_s16_224: root cause = inductor pad_mm, triggered by a oneDNN kernel change

   1. Inductor benchmarks mm(98304x196, 196x196) vs padded mm(98304x200, 200x200) and pads if ori > pad * 1.1. Traced decision (same pad_mm.py in both envs):

      	ori	pad (bench)	ratio	decision
      3.13	0.415 ms	0.420 ms	0.99	no pad
      3.14	0.416 ms	0.334 ms	1.25	pad

   2. The padded benchmark is not representative of the compiled graph:

      • the bench pads the weight into a contiguous ab layout; oneDNN 3.14 made 98304x200:200x200 with wei:ab ~30% faster (0.304 → 0.215 ms), but the compiled graph runs wei:ba (verbose), which is unchanged (0.403 → 0.400 ms);
      • mat1 pad time is excluded (exclude_pad:True, assumed to be memory-planned away), but at runtime it is a real copy: 30 pad + slice per forward ≈ +8.8 ms.

   3. Net: oneDNN time per forward actually drops (25.17 → 24.20 ms); the whole regression is the extra pad/copy kernels.

   Minor oneDNN-side regression (not the cause, FYI): matmul 98304x196:196x196 is +3.4% (wei:ba) / +12% (wei:ab) slower on 3.14.

   Small reproducer

   import time, torch
   import torch._inductor.fx_passes.pad_mm as pm
   
   orig = pm.is_padded_faster
   def traced(key, ori, pad):
       r = orig(key, ori, pad)
       print(f"pad_mm bench: ori={ori:.4f} pad={pad:.4f} -> pad={r}")
       return r
   pm.is_padded_faster = traced
   
   x = torch.randn(98304, 196, device="xpu", dtype=torch.float16)
   w = torch.randn(196, 196, device="xpu", dtype=torch.float16)
   fn = torch.compile(lambda x, w: torch.nn.functional.linear(torch.nn.functional.gelu(x), w))
   for _ in range(10): fn(x, w)
   torch.xpu.synchronize(); t = time.perf_counter()
   for _ in range(100): fn(x, w)
   torch.xpu.synchronize(); print(f"{(time.perf_counter() - t) * 10:.3f} ms/iter")

   	pad_mm decision	ms/iter
   oneDNN 3.13.3	no pad	0.619
   oneDNN 3.14	pad	0.802 (+30%)
   oneDNN 3.14 + TORCHINDUCTOR_SHAPE_PADDING=0	–	0.629

   benchdnn (layout effect):

   benchdnn --mode=F --matmul --dt=f16:f16:f16 --stag=ab --wtag=ab --dtag=ab 98304x200:200x200  # 3.13: 0.304 ms, 3.14: 0.215 ms
   benchdnn --mode=F --matmul --dt=f16:f16:f16 --stag=ab --wtag=ba --dtag=ab 98304x200:200x200  # 3.13: 0.403 ms, 3.14: 0.400 ms (what the model runs)

   Next steps

   • PyTorch inductor: make pad_mm benchmark the layout the graph actually lowers to, and not exclude mat1 pad time when it can't be planned away → file upstream.
   • Workaround: TORCHINDUCTOR_SHAPE_PADDING=0 (verified on gmlp only).
   • whisper-tiny / detectron2: re-run in CI to confirm; not reproducible locally.
6. mengfei25

   assigned

   LuFinch

   on Nov 12, 2026