Suboptimal `libxsmm` ?

[mboisson]

@bartoldeman, our version of libxsmm uses MKL toolchains and specifies AVX=2, AVX=3 MIC=0.
https://github.com/ComputeCanada/easybuild-easyconfigs/blob/computecanada-master/easybuild/easyconfigs/l/libxsmm/
(see versions for 1.8 and 1.16.1)

The new recipe for libxsmm introduced in #10714 does not.

@boegel, should this be an EasyBlock which will apply different flags for different CPU architectures ?

[mboisson]

For what it's worth, the documentation on the page
https://github.com/hfp/libxsmm

does mention BLAS, so presumably it can benefit from MKL.

[bartoldeman]

We need the arch flags for sure when "cross" compiling, although we can now replace the 4 easyconfigs by one using a hook similar to what we do for OpenBLAS (in CC context).

I need to study this a bit more to see if it automatically picks up the native arch and what it does if no MKL is available.

[mboisson]

According to the doc, it seems to automatically fall back to (BLAS, i.e. MKL in our case) when the problem size is large enough, so it seems to be a good idea to have it in my opinion.

[mboisson]

It seems like whichever way we compile it, it does not link against MKL or BLAS, despite having undefined BLAS symbols like dgemm :

$ ldd /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/*.so | grep mkl
	libiomp5.so => /cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2020.1.217/compilers_and_libraries_2020.1.217/linux/compiler/lib/intel64_lin/libiomp5.so (0x00002b61261c3000)

$ for f in /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/*.so; do echo $f; objdump -T $f | grep dgemm_$; done
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/libxsmmext.so
0000000000000000      DF *UND*	0000000000000000              __real_dgemm_
0000000000006030 g    DF .text	0000000000000d90  Base        __wrap_dgemm_
0000000000003590  w   DF .text	0000000000000180  Base        dgemm_
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/libxsmmf.so
0000000000000000      DF *UND*	0000000000000000              libxsmm_dgemm_
0000000000000000      DF *UND*	0000000000000000              libxsmm_blas_dgemm_
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/libxsmmgen.so
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/libxsmmnoblas.so
0000000000001130  w   DF .text	0000000000000040  Base        dgemm_
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/intel2020/libxsmm/1.16.1/lib/libxsmm.so
0000000000000000      D  *UND*	0000000000000000              dgemm_
000000000003b3b0 g    DF .text	0000000000000080  Base        libxsmm_blas_dgemm_
0000000000035720  w   DF .text	0000000000000010  Base        __real_dgemm_
0000000000039e40 g    DF .text	0000000000000010  Base        libxsmm_dgemm_

It looks like it is when building against libxsmm that MKL/BLAS may be needed (to resolve this dgemm_ symbolf rom libxssm.so, or that one should link against libxsmmnoblas.so

[excitedbox]

MKL and anything INTEL cripples AMD CPUs. They also disabled xHost for Epyc in their compiler which would normally be used to set optimizations. Using ANYTHING Intel can cause a 200% slowdown on AMD CPUs.

You may think this was resolved since 2005 but this was just reported to still be the case as of YESTERDAY. Intel is still using dirty tricks to get ahead. They claim they just don't optimize for other CPUs but every time the community finds a fix they actively do something to remove it.

[mboisson]

This issue has nothing to do with AMD CPUs, we don't have those.

[hfp]

LIBXSMM does not "use" Intel MKL (or any other BLAS) except for "falling back" or for validation/samples/comparison. The fallback case is normally agnostic with respect to a specific BLAS implementation ("normally" depends on the OS, i.e., Linux is the norm). Any BLAS can serve the fallback and this is decided at link-time of the application (no need to decide when building LIBXSMM).

If the application (the Easybuild recipe of the application) decides to link against MKL, the application can benefit from MKL when LIBXSMM's fallback code-path is exercised. The fallback is only exercised if a matrix multiplication exceeds a certain problem-size (determining this case is negligible in case of a large matrix multiplication). However, an application can be further optimized if small matrix-multiplications can be batched, i.e., same problem-size is decided upfront whether to fall-back or not.

Further, the first paragraph of LIBXSMM's documentation states: "LIBXSMM is suitable for "build once and deploy everywhere", i.e., no special target flags are needed to exploit the available performance.". This means flags such as "AVX=2", "AVX=3 MIC=0", or similar flags are not needed. Indeed, the target is exploited at runtime depending on CPUID (which is in turn is purely based on feature-flags as opposed to vendor-check).