Using oneDNN v3.6 with the maximum ISA set to avx, and the attribute post-op configured as eltwise_relu, the actual execution runs on jit:sse41.

[Li-Theone]

Summary

Provide a short summary of the issue. Sections below provide guidance on what
factors are considered important to reproduce an issue.

Using oneDNN v3.6 with the maximum ISA set to AVX, and the attribute post-op configured as eltwise_relu, the actual execution runs on jit:sse41. What could be the reason for this?
Below is the specific print/log output:

onednn_verbose,v1,info,oneDNN v3.6.0 (commit N/A)
onednn_verbose,v1,info,cpu,runtime:OpenMP,nthr:1
onednn_verbose,v1,info,cpu,isa:Intel AVX
onednn_verbose,v1,info,gpu,runtime:none
onednn_verbose,v1,info,graph,backend,0:dnnl_backend
onednn_verbose,v1,primitive,info,template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
onednn_verbose,v1,graph,info,template:operation,engine,partition_id,partition_kind,op_names,data_formats,logical_tensors,fpmath_mode,implementation,backend,exec_time
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:Acdb8a::f0,,,32x3x7x7,0.0209961
onednn_verbose,v1,primitive,exec,cpu,convolution,jit:sse41,forward_inference,src:f32:a:blocked:abcd::f0 wei:f32:a:blocked:Acdb8a::f0 bia:f32::blocked:a::f0 dst:f32:a:blocked:aBcd8b::f0,attr-post-ops:eltwise_relu,alg:convolution_direct,mb1_ic3oc32_ih540oh135kh7sh4dh0ph3_iw960ow240kw7sw4dw0pw3,9.44189
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:aBcd8b::f0 dst:f32::blocked:acdb::f0,,,1x32x135x240,0.843994
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:ABcd8b8a::f0,,,64x32x3x3,0.0200195
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:acdb::f0 dst:f32::blocked:aBcd8b::f0,,,1x32x135x240,0.38501
onednn_verbose,v1,primitive,exec,cpu,convolution,jit:sse41,forward_inference,src:f32:a:blocked:aBcd8b::f0 wei:f32:a:blocked:ABcd8b8a::f0 bia:f32::blocked:a::f0

Version

Report oneDNN version and githash. Version information is printed to stdout
in verbose mode.
onednn V3.6.0

Environment

oneDNN includes hardware-specific optimizations and may behave
differently on depending on the compiler and build environment. Include
the following information to help reproduce the issue:

• CPU make and model (try lscpu; if your lscpu does not list CPU flags,
  try running cat /proc/cpuinfo | grep flags | sort -u)
• OS version (uname -a)
• Compiler version (gcc --version)
• CMake version (cmake --version)
• CMake output log
• git hash (git log -1 --format=%H)

xxx# uname -a
Linux ubuntu 4.4.0-210-generic #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

xxx# gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

xxx# cmake --version
cmake version 3.16.3
CMake suite maintained and supported by Kitware (kitware.com/cmake).

Steps to reproduce

Please check that the issue is reproducible with the latest revision on
main. Include all the steps to reproduce the issue.

You can use verbose mode
and benchdnn
to validate correctness of all primitives the library supports. If this does not
work a short C/C++ program or modified unit tests demonstrating the issue
will greatly help with the investigation.

Observed behavior

Document behavior you observe. For performance defects, like performance
regressions or a function being slow, provide a log including output generated
by your application in
verbose mode.

Expected behavior

Document behavior you expect.

when maximum ISA set to AVX, and the attribute post-op configured as eltwise_relu, the actual execution runs on avx

[Li-Theone]

add lscpu info:
xxx# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i7-9700KF CPU @ 3.60GHz
Stepping: 13
CPU MHz: 800.015
CPU max MHz: 4900.0000
CPU min MHz: 800.0000
BogoMIPS: 7199.71
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch invpcid_single intel_pt ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities

[yehudaorel]

Hi @Li-Theone,

What you are seeing is the expected behavior as there are no AVX specific code paths for your problem configuration.
By capping max isa to AVX you are enforcing this behavior which will skip over AVX2 implementations and dispatch the next best case.

src/cpu/cpu_convolution_list.cpp:

    CPU_INSTANCE_AVX2(jit_avx2_convolution_fwd_t)
    CPU_INSTANCE_SSE41(jit_sse41_convolution_fwd_t)

You can also validate this dispatching behaviour by running your workload with ONEDNN_VERBOSE=dispatch

Given your cpu supports AVX2, is there a reason you are limiting the max isa in oneDNN to AVX?

[Li-Theone]

Hi @yehudaorel

1. The reason I set the maximum to AVX is that I don’t have any devices on my side that support only up to AVX; I did this merely to test the performance of the AVX branch.

2. What confuses me is that I also have another model A, for which the maximum ISA is set to AVX and which actually runs on the jit:avx kernel. However, for model B, even though the maximum ISA is also set to AVX, it runs on jit:sse41 instead. I’m puzzled about what causes this difference in behavior between the two models.
   model B:
   onednn_verbose,v1,info,oneDNN v3.6.0 (commit N/A)
   onednn_verbose,v1,info,cpu,runtime:OpenMP,nthr:1
   onednn_verbose,v1,info,cpu,isa:Intel AVX
   onednn_verbose,v1,info,gpu,runtime:none
   onednn_verbose,v1,info,graph,backend,0:dnnl_backend
   onednn_verbose,v1,primitive,info,template:operation,engine,primitive,implementation,prop_kind,memory_descriptors,attributes,auxiliary,problem_desc,exec_time
   onednn_verbose,v1,graph,info,template:operation,engine,partition_id,partition_kind,op_names,data_formats,logical_tensors,fpmath_mode,implementation,backend,exec_time
   onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:Acdb8a::f0,,,16x3x3x3,0.00292969
   onednn_verbose,v1,primitive,exec,cpu,convolution,jit:avx,forward_inference,src:f32:a:blocked:abcd::f0 wei:f32:a:blocked:Acdb8a::f0 bia:f32::blocked:a::f0 dst:f32:a:blocked:aBcd8b::f0,,alg:convolution_direct,mb1_ic3oc16_ih976oh488kh3sh2dh0ph1_iw1664ow832kw3sw2dw0pw1,9.54395
   onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:aBcd8b::f0 dst:f32::blocked:acdb::f0,,,1x16x488x832,4.75391

3. Another observation is that with the version of oneDNN (v2.5), for the same model B with the maximum ISA set to AVX, the actual execution used the jit:avx kernel.

onednn_verbose,info,oneDNN v2.5.0 (commit N/A)
onednn_verbose,info,cpu,runtime:OpenMP
onednn_verbose,info,cpu,isa:Intel AVX
onednn_verbose,info,gpu,runtime:none
onednn_verbose,exec,cpu,convolution,jit:avx,forward_inference,src_f32::blocked:abcd:f0 wei_f32::blocked:Acdb8a:f0 bia_f32::blocked:a:f0 dst_f32::blocked:aBcd8b:f0,attr-post-ops:eltwise_relu ,alg:convolution_direct,mb1_ic3oc32_ih540oh135kh7sh4dh0ph3_iw960ow240kw7sw4dw0pw3,5
onednn_verbose,exec,cpu,convolution,jit:avx,forward_inference,src_f32::blocked:aBcd8b:f0 wei_f32::blocked:ABcd8b8a:f0 bia_f32::blocked:a:f0 dst_f32::blocked:aBcd8b:f0,attr-post-ops:eltwise_relu ,alg:convolution_direct,mb1_ic32oc64_ih135oh68kh3sh2dh0ph1_iw240ow120kw3sw2dw0pw1,4.39307

All of the above tests were conducted on the same device.