Skip to content

[BMG] PyTorch 2.12 performance cases where inductor is slower than eager (speedup < 1) #3761

New issue
New issue
Open
Bug
Open
[BMG] PyTorch 2.12 performance cases where inductor is slower than eager (speedup < 1)#3761
Bug
Assignees
msnliu
Labels
dtype: bfloat16bf16 dtypehw: BMGBattlemageinferenceInference workload phasemodule: inductortorch.compile backend: Triton codegen, lowering, AOTAutogradperformancePerformance-related failuretest: e2eEnd-to-end / benchmark failure surface
Milestone
PT2.16

Description

@mengfei25
mengfei25
opened on May 27, 2026

Summary

BMG PyTorch 2.12 release performance results contain multiple cases where inductor is slower than eager (speedup < 1).

These are PyTorch 2.12 release performance results collected from the raw performance artifacts:

  • Platform: BMG (Battlemage)
  • PyTorch: torch 2.12.0+xpu
  • Environment: Ubuntu 24.04 LTS, intel-opencl-icd 26.05.37020.3, libze-dev 1.27.0, intel-opencl-rt 2025.3.2, oneMKL 2025.3.1

Representative Cases

The rows below come directly from the raw PyTorch 2.12 performance CSVs and show speedup < 1, which means inductor is slower than eager on the same workload.

suite dtype mode model batch_size speedup source csv
huggingface bfloat16 inference ElectraForQuestionAnswering 128 0.662224 huggingface/bfloat16/inference/performance/inductor-results-huggingface-bfloat16-inference-xpu-performance.csv
huggingface bfloat16 inference BertForQuestionAnswering 32 0.732601 huggingface/bfloat16/inference/performance/inductor-results-huggingface-bfloat16-inference-xpu-performance.csv
huggingface bfloat16 inference BartForConditionalGeneration 4 0.741895 huggingface/bfloat16/inference/performance/inductor-results-huggingface-bfloat16-inference-xpu-performance.csv
huggingface bfloat16 inference MBartForConditionalGeneration 4 0.723328 huggingface/bfloat16/inference/performance/inductor-results-huggingface-bfloat16-inference-xpu-performance.csv
huggingface bfloat16 inference PLBartForConditionalGeneration 8 0.829213 huggingface/bfloat16/inference/performance/inductor-results-huggingface-bfloat16-inference-xpu-performance.csv
huggingface amp_bf16 inference BertForQuestionAnswering 32 0.893284 huggingface/amp_bf16/inference/performance/inductor-results-huggingface-amp_bf16-inference-xpu-performance.csv
huggingface float16 inference ElectraForQuestionAnswering 128 0.646220 huggingface/float16/inference/performance/inductor-results-huggingface-float16-inference-xpu-performance.csv
huggingface bfloat16 training OPTForCausalLM 2 0.876858 huggingface/bfloat16/training/performance/inductor-results-huggingface-bfloat16-training-xpu-performance.csv
torchbench bfloat16 inference lennard_jones 1000 0.671808 torchbench/bfloat16/inference/performance/inductor-results-torchbench-bfloat16-inference-xpu-performance.csv
torchbench bfloat16 inference maml 1 0.664822 torchbench/bfloat16/inference/performance/inductor-results-torchbench-bfloat16-inference-xpu-performance.csv
torchbench bfloat16 inference soft_actor_critic 256 0.865522 torchbench/bfloat16/inference/performance/inductor-results-torchbench-bfloat16-inference-xpu-performance.csv

There are additional PyTorch 2.12 rows with the same pattern outside this representative subset.

Notes

  • The pattern is concentrated in inference workloads, especially HuggingFace QA/seq2seq style models and several TorchBench models.
  • These rows were taken from the raw perf artifacts rather than the generated comparison CSV, because the goal here is to flag inductor-vs-eager gaps directly in the PyTorch 2.12 release results.

Ask

Please review these BMG PyTorch 2.12 cases and track the inductor performance gaps where eager is still faster than inductor.

Activity

  1. mengfei25
    changed the title [-][BMG] Baseline performance cases where inductor is slower than eager (speedup < 1)[/-] [+][BMG] PyTorch 2.12 performance cases where inductor is slower than eager (speedup < 1)[/+] on May 27, 2026
  2. mengfei25
    added
    performancePerformance-related failure
    on May 27, 2026
  3. chuanqi129
    assigned
    jianyizh
    on May 27, 2026
  4. chuanqi129
    added this to the PT2.14 milestone on May 27, 2026
  5. msnliu
    self-assigned this
    on Jul 22, 2026

  6. msnliu commented on Jul 22, 2026

    @msnliu
    msnliu
    on Jul 22, 2026
    Contributor

    speedup reproduction:
    Image

    device ratio:
    Image

    Env:

    Component torch-2.12-xpu torch-2.14-xpu Notes
    Python 3.13.14 3.13.15 Aligned
    PyTorch 2.12.0+xpu 2.14.0+xpu Official release wheels
    Triton 3.7.1 3.8.0 Default bundled versions
    Transformers 5.14.1 5.14.1 Aligned
    TorchBench Installed Installed Aligned
  7. chuanqi129
    unassigned
    jianyizh
    on Jul 22, 2026
  8. chuanqi129
    modified the milestones: PT2.14, PT2.15 on Jul 29, 2026

  9. msnliu commented on Sep 15, 2026

    @msnliu
    msnliu
    on Sep 15, 2026
    Contributor

    In PyTorch 2.12, models with 4D broadcasted attention masks (e.g. HuggingFace BERT/BART/Electra with shape [B, 1, 1, S]) fail to dispatch to the oneDNN SDPA backend (overrideable_attention) on XPU, falling back to the un-fused math backend.

    Profiling traces show that this fallback introduces extra general GEMMs and unfused softmax kernels per attention block, causing substantial performance regressions (speedup < 1.0).

    after the sdpa path fix (pytorch/pytorch#197505):
    Image
    we could see that model 05_plbart_bf16_inf and 06_bert_amp_inf did not show regression again:

    • 05_plbart_bf16_inf: 121 gemm + 12 triton safe_softmax_add + 6 micro_sdpa -> 97 gemm + 18 micro_sdpa
    • 06_bert_amp_inf: 97 gemm + 12 triton safe_softmax_add -> 73 gemm + 12 micro_sdpa
    • and all other models that have attention blocks improved (models that don't have attention: 09_lennard_bf16_inf, 10_maml_bf16_inf, 11_sac_bf16_inf)
  10. chuanqi129
    modified the milestones: PT2.15, PT2.16 on Sep 16, 2026

  11. msnliu commented on Sep 20, 2026

    @msnliu
    msnliu
    on Sep 20, 2026
    Contributor

    Transformers issues:
    huggingface/transformers#48924
    huggingface/transformers#48925
    After applying the Transformers masking fix (huggingface/transformers#48975 & huggingface/transformers#49002), all 8 validation cases pass successfully: Electra, BERT, BART, mBART, PLBART, BERT AMP, Electra FP16, and OPT BF16 training. The fix correctly enables mask skipping under torch.compile.

    Image

  12. msnliu commented on Sep 23, 2026

    @msnliu
    msnliu
    on Sep 23, 2026
    Contributor

    MAML inference performs task-specific finetuning and calls torch.autograd.grad(). With PyTorch 2.14's default Dynamo settings, autograd operations are not traced, which causes graph breaks.

    This change:

    • enables trace_autograd_ops for MAML;
    • passes parameters and BatchNorm state explicitly;
    • removes deepcopy() and Tensor.item() from the compiled path.

    MAML inference now compiles without the observed graph breaks or recompilations. Validated on Intel XPU with numerical correctness checks and approximately 2.7x Inductor speedup over eager.

    pytorch/benchmark#2709

    lennard_jones and soft_actor_critic models are host bounded

  13. daisyden
    added
    module: inductortorch.compile backend: Triton codegen, lowering, AOTAutograd
    dtype: bfloat16bf16 dtype
    inferenceInference workload phase
    test: e2eEnd-to-end / benchmark failure surface
    hw: BMGBattlemage
    on Nov 11, 2026
  14. daisyden
    added the
    Bug
    issue type on Nov 11, 2026

  15. daisyden commented on Nov 11, 2026

    @daisyden
    daisyden
    on Nov 11, 2026
    Contributor
    label-issue: #3761

    Root cause: tracking issue for BMG PyTorch 2.12 inductor-vs-eager perf gaps (speedup<1); no single defect traced, representative case is inductor slower than eager on an SDPA-heavy HF QA model.

    Analyzed case: huggingface/ElectraForQuestionAnswering/inference/bfloat16 (case 1 of 11; the other 10 not analyzed)

    axis value reason
    type Bug No native Type; reported performance defect (inductor slower than eager).
    test test: e2e huggingface/torchbench dynamo benchmark perf results.
    module module: inductor inductor torch.compile backend is slower than eager; codegen perf gap.
    priority High Native Priority field set to High; preserved verbatim.
    hw hw: BMG Title [BMG]; Battlemage platform.
    dtype dtype: bfloat16 Representative case dtype bfloat16.
    symptom performance speedup<1: inductor slower than eager throughput.
    symptom inference Representative case is an inference-phase run.

    Test cases (11):

    • huggingface ElectraForQuestionAnswering inference bfloat16 (speedup 0.662)
    • huggingface BertForQuestionAnswering inference bfloat16 (0.733)
    • huggingface BartForConditionalGeneration inference bfloat16 (0.742)
    • huggingface MBartForConditionalGeneration inference bfloat16 (0.723)
    • huggingface PLBartForConditionalGeneration inference bfloat16 (0.829)
    • huggingface BertForQuestionAnswering inference amp_bf16 (0.893)
    • huggingface ElectraForQuestionAnswering inference float16 (0.646)
    • huggingface OPTForCausalLM training bfloat16 (0.877)
    • torchbench lennard_jones inference bfloat16 (0.672)
    • torchbench maml inference bfloat16 (0.665)
    • torchbench soft_actor_critic inference bfloat16 (0.866)

    Applied to this issue. Generated by a local label-issue run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

dtype: bfloat16bf16 dtypehw: BMGBattlemageinferenceInference workload phasemodule: inductortorch.compile backend: Triton codegen, lowering, AOTAutogradperformancePerformance-related failuretest: e2eEnd-to-end / benchmark failure surface

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions