Build error with intel compiler & spack

[s-sajid-ali]

I've built openblas with spack using intel@18.0.3 and when I try to run a simple program it gives the following errors:

PATH_TO_OPENBLAS/lib/libopenblas.so: undefined reference to `for_cpystr'
PATH_TO_OPENBLAS/lib/libopenblas.so: undefined reference to `for_concat'

Here's how the linking looks like :

[sajid@xrmlite lib]$ pwd
/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/intel-18.0.3/openblas-0.3.4-meisjl5yp4xcjywvq3melryeyueubox2/lib
[sajid@xrmlite lib]$ ldd libopenblas.so
        linux-vdso.so.1 =>  (0x00007ffe29283000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b1aa8aac000)
        libimf.so => /opt/intel/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64/libimf.so (0x00002b1aa8dae000)
        libsvml.so => /opt/intel/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64/libsvml.so (0x00002b1aa9341000)
        libirng.so => /opt/intel/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64/libirng.so (0x00002b1aaac50000)
        libgcc_s.so.1 => /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-7.3.0-xyzezhjmbiebkjfoakso464rhfshlkyq/lib64/libgcc_s.so.1 (0x00002b1aaafc4000)
        libintlc.so.5 => /opt/intel/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64/libintlc.so.5 (0x00002b1aab1db000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b1aab44a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b1aab817000)
        /lib64/ld-linux-x86-64.so.2 (0x000055ba20f17000)

And for reference, here is the build log : [build.out] and the [build.env].
spack version: develop, at the latest commit.

[martin-frbg]

I see you already opened spack/spack#8767 for what appears to be the same problem some months ago. So far I can only concur that there seems to be something going wrong with the linking to libifcore, although -lifcore appears on the final link line in your build.out. I do not think that this is OpenBLAS' fault, perhaps your system (or the spack package) has an empty libifcore (or one linked to a different library) in its search path ?

[s-sajid-ali]

As per your request:

[sajid@xrmlite intel64_lin]$ pwd
/opt/intel/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin
[sajid@xrmlite intel64_lin]$ ldd libifcore.a
        not a dynamic executable
[sajid@xrmlite intel64_lin]$ ldd libifcore.so
        not a dynamic executable
[sajid@xrmlite intel64_lin]$ ldd libifcore.so.5
        linux-vdso.so.1 =>  (0x00007ffd5f9f8000)
        libimf.so => not found
        libm.so.6 => /lib64/libm.so.6 (0x00002b7a3fee1000)
        libsvml.so => not found
        libintlc.so.5 => not found
        libc.so.6 => /lib64/libc.so.6 (0x00002b7a401e4000)
        /lib64/ld-linux-x86-64.so.2 (0x0000558462648000)

I never expected this to be the issue. I'll look into why this happened then. Thanks for the tip.

[s-sajid-ali]

@martin-frbg : If I may ask one more question, is there a way to make openblas not try to do things like
icc -lifcore and instead do icc PATH_TO_LIB/libifcore.a ?

According to a thread I was reading, icc really seems to dislike linking Fortran libraries when compiling with icpc (and icc?) and prefers to be given the path to the static library.

If this is not something that can be done as a variation of openblas build, feel free to close the issue. Thanks in advance.

[martin-frbg]

I am not aware of this being set anywhere in the OpenBLAS build files (except in conjuction with linking libacml for a non-default benchmark target). Probably this can (or needs to) be handled viaCFLAGS or LDFLAGS settings, but as I do not use icc or ifort regularly I do not know any to suggest.

[martin-frbg]

Actually some platforms (OSX at least) use FC rather than CC for the final link of the dynamic libopenblas.so in exports/Makefile - perhaps this should be done whenever the compiler is Intel ?

According to another thread on the same forum it seems to be discouraged to specify the full path to a lib.

[s-sajid-ali]

According to thread_1 it's better to use full paths if using icc to link mixed language packages.

According to thread_2, it's ideal to use ifort to link mixed language packages and not pass full paths to ifort.

Quote from thread_2 : There's no need for this (specifying libraries to link); better to let the ifort driver choose which libraries to link, and let it take them from the same installation as the compiler (so don't specify the compiler library path with -L). Use the switch -cxxlib to ask the driver to link also to the standard C++ libraries, if needed. It is better to use ifort rather than icc or icpc for linking mixed language applications, because ifort can link the libraries for all 3 languages, whereas the other drivers do not know how to link Fortran libraries.

From build.out, we have :

[sajid@xrmlite OpenBLAS-0.3.5]$ cat spack-build.out  | grep ifcore                                                                                                            
/home/sajid/packages/spack/lib/spack/env/intel/icc -O2 -DMAX_STACK_ALLOC=2048 -wd981 -DF_INTERFACE_INTEL -fPIC -DNO_WARMUP -DMAX_CPU_NUMBER=32 -DMAX_PARALLEL_NUMBER=1 -DVERSI
ON=\"0.3.5\" -mavx2 -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME=\"_\" -DCHAR_CNAME=\"\" -DNO_AFFINITY -I..  -w -o linktest linktest.c ../libopenblas_haswell-r0.3.5.
so -L/opt/intel/compilers_and_libraries_2019.1.144/linux/mpi/intel64/libfabric/lib -L/opt/intel/compilers_and_libraries_2019.1.144/linux/ipp/lib/intel64 -L/opt/intel/compiler
s_and_libraries_2019.1.144/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2019.
1.144/linux/tbb/lib/intel64/gcc4.7 -L/opt/intel/compilers_and_libraries_2019.1.144/linux/daal/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2019.1.144/linux/daal/../tb
b/lib/intel64_lin/gcc4.4 -L/opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64_lin -L/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/
gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/lib/gcc/x86_64-pc-linux-gnu/8.2.0/ -L/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsj
lpqz2dm2qbbxzfyn/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../../lib64 -L/home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2q
bbxzfyn/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/opt/intel/compilers_and_libraries_20
19.1.144/linux/mpi/intel64/libfabric/lib/ -L/opt/intel/compilers_and_libraries_2019.1.144/linux/ipp/lib/intel64/ -L/opt/intel/compilers_and_libraries_2019.1.144/linux/compile
r/lib/intel64_lin/ -L/opt/intel/compilers_and_libraries_2019.1.144/linux/mkl/lib/intel64_lin/ -L/opt/intel/compilers_and_libraries_2019.1.144/linux/tbb/lib/intel64/gcc4.7/ -L
/opt/intel/compilers_and_libraries_2019.1.144/linux/daal/lib/intel64_lin/ -L/opt/intel/compilers_and_libraries_2019.1.144/linux/daal/../tbb/lib/intel64_lin/gcc4.4/ -L/home/sa
jid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../ -L/lib64 -L/lib/ -L/usr/lib
64 -L/usr/lib  -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lpthread -lsvml -lc -lirc_s -ldl -lc  -lifcore && echo OK.                                                  
[sajid@xrmlite OpenBLAS-0.3.5]$                                                                                                                                               

So it might be better to do this final linking using ifort instead of icc as explained here.

[martin-frbg]

Could you try changing line 145 in exports/Makefile, which currently reads
$(CC) $(CFLAGS) $(LDFLAGS) -shared -o ../$(LIBSONAME) \
to
$(FC) $(FFLAGS) $(LDFLAGS) -shared -o ../$(LIBSONAME) \
(that is, change the CC to FC and replace CFLAGS with FFLAGS on the first line following the test for "LSB"), just to see if this is sufficient to pass the proper parameters to ifort? (If that works, we could put an ifeq($(F_COMPILER), INTEL) around that whole ifeq(LSB)...else...endif block to just call ifort if it exists.)

[s-sajid-ali]

I forked the repo and built it with the suggested change. Everything works now! Thanks for the help.

Linking info :

[sajid@xrmlite lib]$ ldd libopenblas_haswell-r0.3.6.dev.so
        linux-vdso.so.1 =>  (0x00007ffcb95fd000)
        libm.so.6 => /lib64/libm.so.6 (0x00002af99ad23000)
        libifport.so.5 => /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64/libifport.so.5 (0x00002af99b025000)
        libifcoremt.so.5 => /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64/libifcoremt.so.5 (0x00002af99b253000)
        libimf.so => /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64/libimf.so (0x00002af99b5e8000)
        libsvml.so => /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64/libsvml.so (0x00002af99bb88000)
        libintlc.so.5 => /opt/intel/compilers_and_libraries_2019.1.144/linux/compiler/lib/intel64/libintlc.so.5 (0x00002af99d52b000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002af99d79e000)
        libc.so.6 => /lib64/libc.so.6 (0x00002af99d9ba000)
        libgcc_s.so.1 => /home/sajid/packages/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-sxbf4jq6ghmoybsjlpqz2dm2qbbxzfyn/lib64/libgcc_s.so.1 (0x00002af99dd87000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002af99dfa0000)
        /lib64/ld-linux-x86-64.so.2 (0x00005645c03cb000)

A simple test case which was failing before, works after the change in build recipe.

I'm also attaching the build-env and build-log for reference.