Skip to content

[release/2.12] performance regressions compared with 2.11 #3432

New issue
New issue
Open
Bug
Open
[release/2.12] performance regressions compared with 2.11#3432
Bug
Assignees
weishi-deng
Labels
dependency component: communityMatching open pytorch/pytorch issuedtype: amp_fp16Autocast/AMP fp16hw: PVCPonte Vecchio / Data Center GPU MaxinferenceInference workload phasemodule: inductortorch.compile backend: Triton codegen, lowering, AOTAutogradperformancePerformance-related failureregressionRegression with before/after version or commit signaltest: e2eEnd-to-end / benchmark failure surface
Milestone
PT2.16

Description

@mengfei25
mengfei25
opened on Apr 23, 2026

🐛 Describe the bug

Observed with 2.12 rc wheel on PVC, some models got performance dropped compared with 2.11

Higher is better

suite data_type mode model inductor_ratio eager_ratio
timm_models amp_fp16 inference tnt_s_patch16_224 0.619 1.003
timm_models float16 inference tnt_s_patch16_224 0.622 1.02
timm_models bfloat16 inference tnt_s_patch16_224 0.636 1.016
timm_models amp_bf16 inference tnt_s_patch16_224 0.651 1.007
torchbench amp_fp16 training vgg16 0.736 0.998
torchbench float16 training vgg16 0.736 0.998
torchbench amp_bf16 training vgg16 0.764 1.004
torchbench bfloat16 training vgg16 0.776 0.994
torchbench float16 inference hf_Roberta_base 0.817 0.688
torchbench float16 inference phlippe_densenet 0.823 1.204
torchbench amp_fp16 inference phlippe_densenet 0.824 1.101
torchbench amp_fp16 inference hf_Roberta_base 0.827 1.031
torchbench float32 inference phlippe_densenet 0.836 1.172
timm_models float16 training gluon_inception_v3 0.847 0.997
timm_models amp_fp16 training gluon_inception_v3 0.85 0.995
timm_models amp_fp16 training inception_v3 0.851 0.998
timm_models float16 training adv_inception_v3 0.857 0.998
timm_models bfloat16 training gluon_inception_v3 0.858 1
timm_models float16 inference gmlp_s16_224 0.869 1.008
timm_models bfloat16 training adv_inception_v3 0.871 0.997
timm_models bfloat16 training inception_v3 0.873 1
timm_models float16 training inception_v3 0.878 1.002
timm_models amp_fp16 training ese_vovnet19b_dw 0.88 1.007
timm_models bfloat16 inference gmlp_s16_224 0.887 0.94
timm_models float16 training dla102 0.892 0.993
timm_models bfloat16 training ese_vovnet19b_dw 0.897 1.001
timm_models bfloat16 inference xcit_large_24_p8_224 0.938 0.546
timm_models float16 training xcit_large_24_p8_224 0.968 0.643
timm_models float16 inference xcit_large_24_p8_224 0.97 0.541
timm_models bfloat16 training xcit_large_24_p8_224 0.998 0.695

Reproducer

docker run -it -e TZ=Aisa/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged --shm-size=8g intelgpu/ubuntu-24.04-lts2:2523.40 bash

curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR="/usr/local/bin" sh
uv venv myvenv --python 3.12 --clear
source myvenv/bin/activate
uv pip install pip numpy 'setuptools<81' wheel

uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/test/xpu  # 2.12
# pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu  # 2.11

pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}
# modify model list for all models
rm -rf torch torch-xpu-ops
git clone https://github.com/intel/torch-xpu-ops
rsync -avz ./torch-xpu-ops/.ci/benchmarks/ ./benchmarks/dynamo/

# install requirements
uv pip install pip pandas psutil scipy requests
uv pip install -U numpy==1.26.4
uv pip install transformers==4.55.2 accelerate
uv pip install timm==1.0.19
# optional for torchbench
git clone https://github.com/pytorch/benchmark tmp-benchmark
cd tmp-benchmark
python install <model name>

python ./benchmarks/dynamo/timm_models.py --performance --bfloat16 --inference -d xpu -n 10 --only tnt_s_patch16_224 --backend=inductor --cold-start-latency --timeout 10800 --disable-cudagraphs --output output.csv

Versions

pytorch 2.12 rc
pytorch 2.11

Activity

  1. mengfei25
    added
    performancePerformance-related failure
    hw: PVCPonte Vecchio / Data Center GPU Max
    regressionRegression with before/after version or commit signal
    on Apr 23, 2026
  2. chuanqi129
    added this to the PT2.12 milestone on Apr 23, 2026
  3. chuanqi129
    assigned
    jianyizh
    on Apr 23, 2026

  4. weishi-deng commented on Apr 27, 2026

    @weishi-deng
    weishi-deng
    on Apr 27, 2026
    Contributor

    regression of tnt_s_patch16_224: raised by the inductor SDPA kernels(bmm+softmax -> micro_sdpa). OP benchmark has been assigned to @LuFinch.

  5. LuFinch commented on Apr 27, 2026

    @LuFinch
    LuFinch
    on Apr 27, 2026
    Contributor

    MFDNN-14987

  6. jianyizh commented on Apr 27, 2026

    @jianyizh
    jianyizh
    on Apr 27, 2026
    Contributor

    vgg16: pytorch pr 167318 changes max_pool2d_with_indices_backward from fallback to atomic add triton op. We may not atomic

  7. jianyizh
    assigned
    weishi-deng
    on Apr 28, 2026

  8. jianyizh commented on Apr 28, 2026

    @jianyizh
    jianyizh
    on Apr 28, 2026
    Contributor

    hf_Roberta_base: 1. gemm MFDNN-14993，2. jira HUGGINGFAD-941 attention mask is broadcasted on last dim (kv seq_len), sdpa need stride of last dim=1. need change transformers https://github.com/huggingface/transformers/blob/main/src/transformers/masking_utils.py#L520 attention_mask = attention_mask.expand(batch_size, -1, q_length, kv_length) 3. jira MFDNN-14997 broadcasted mask will not trigger microsdpa kernel in onednn onednn_verbose,v1,graph,exec,gpu,100079,sdp,matmul_qk;scale_mul;mask_add;softmax;matmul_v,,in0_f16:0:strided:undef:8x12x512x64:393216s64s768s1 in1_f16:1:strided:undef:8x12x512x64:393216s64s768s1 in2_f32:2:strided:host_scalar:0:2 in3_f16:4:strided:undef:1x1x512x1:512s512s1s1 in4_f16:5:strided:undef:8x12x512x64:393216s64s768s1 out0_f16:6:strided:undef:8x12x512x64:393216s64s768s1,fpm:strict,larger_partition_kernel_t,dnnl_backend,0.39502

  9. chuanqi129
    modified the milestones: PT2.12, PT2.13 on Apr 29, 2026

  10. chuanqi129 commented on Apr 29, 2026

    @chuanqi129
    chuanqi129
    on Apr 29, 2026
    Contributor

    Need consider acceptance methodology

  11. weishi-deng commented on May 6, 2026

    @weishi-deng
    weishi-deng
    on May 6, 2026
    Contributor

    adv_inception_v3, gluon_inception_v3,inception_v3: similar with vgg16, suffer from atomic in max_pool2d_with_indices_backward, fix by Jianyi's PR

  12. weishi-deng commented on May 6, 2026

    @weishi-deng
    weishi-deng
    on May 6, 2026
    Contributor

    phlippe_densenet: cannot reproduce regression on both 1100 and 1550.

  13. 5 remaining items

  14. orrangetabby17 commented on May 27, 2026

    @orrangetabby17
    orrangetabby17
    on May 27, 2026
    Contributor

    phlippe_densenet : invoke an issue in pytorch repo issues#185346

  15. chuanqi129
    added
    dependency component: communityMatching open pytorch/pytorch issue
    on Jun 3, 2026
  16. chuanqi129
    unassigned
    jianyizh
    on Jun 3, 2026
  17. chuanqi129
    modified the milestones: PT2.13, PT2.14 on Jun 17, 2026
  18. chuanqi129
    modified the milestones: PT2.14, PT2.15 on Jul 29, 2026
  19. chuanqi129
    modified the milestones: PT2.15, PT2.16 on Sep 23, 2026
  20. daisyden
    added
    module: inductortorch.compile backend: Triton codegen, lowering, AOTAutograd
    dtype: amp_fp16Autocast/AMP fp16
    inferenceInference workload phase
    test: e2eEnd-to-end / benchmark failure surface
    on Nov 11, 2026
  21. daisyden
    added the
    Bug
    issue type on Nov 11, 2026

  22. daisyden commented on Nov 11, 2026

    @daisyden
    daisyden
    on Nov 11, 2026
    Contributor
    label-issue: #3432

    Root cause: inductor-compiled performance of several models regressed from PyTorch 2.11 to 2.12 RC on PVC (worst inductor_ratio 0.619 for tnt_s_patch16_224, ~38% slowdown); no per-model traceback, a single release-regression concern. Owner: inductor/torch.compile XPU codegen; not pinpointable to one file:line.

    axis value reason
    type Bug Native Type unset; measured performance regression between releases.
    test test: e2e timm_models / torchbench perf runs via benchmarks/dynamo.
    module module: inductor inductor_ratio drops under --backend=inductor; eager_ratio mostly ~1.0.
    priority Urgent Measured perf regression >7% between releases (inductor_ratio 0.619 = ~38%).
    hw hw: PVC hw: PVC label; regression observed on PVC.
    dtype dtype: amp_fp16 Representative case tnt_s_patch16_224 amp_fp16 inference.
    dependency component dependency component: community Existing human-set label preserved.
    symptom performance Benchmark throughput/latency regression.
    symptom regression Explicit before/after: passes on 2.11, regresses on 2.12 RC.
    symptom inference Representative case is an inference run.

    Applied to this issue. Generated by a local label-issue run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

dependency component: communityMatching open pytorch/pytorch issuedtype: amp_fp16Autocast/AMP fp16hw: PVCPonte Vecchio / Data Center GPU MaxinferenceInference workload phasemodule: inductortorch.compile backend: Triton codegen, lowering, AOTAutogradperformancePerformance-related failureregressionRegression with before/after version or commit signaltest: e2eEnd-to-end / benchmark failure surface

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions