"__ieee128" is undefined on POWER with RedHat 8 and CUDA < 11

[branfosj]

The error is:

  RuntimeError: Encountered unknown error while testing nvcc:
  /rds/bear-apps/devel/eb-sjb-up/EL8/EL8-p9/software/GCCcore/8.3.0/include/c++/8.3.0/type_traits(335): error: identifier "__ieee128" is undefined
  /usr/include/bits/floatn.h(79): error: identifier "__ieee128" is undefined
  /usr/include/bits/floatn.h(82): error: invalid argument to attribute "__mode__"

We've looked at this for TensorFlow 2.3.1 in easybuilders/easybuild-easyblocks#2251 and #11859 - the debugging there leads to easybuilders/easybuild-easyblocks#2251 (comment)

Edit: FTR this is a GLIBC 2.26 issue: https://forums.developer.nvidia.com/t/request-add-nvcc-compatibility-with-glibc-2-26/53306

However, this is more widespread than just TensorFlow. Anywhere NVCC is passing flags back to GCC when we are building on POWER on RedHat 8 with CUDA < 11. (I expect that it'll also impact other OSes as well.)

I did limited testing and I've seen the error with (but there are likely other bits of software this impacts):

• TensorFlow
• PyTorch
• CuPy
• magma
• torchvision

The solution is to get NVCC to pass the -mno-float128 flag (and also -std=c++11 if that is not already there). Depending on where this has to be added varies how complicated doing that is.

The alternative is to build against newer toolchains - 2020a and later, where CUDA 11 is used.

[boegel]

@branfosj Is this something we could tackle in framework somehow?

[branfosj]

For some of the software we can alter the flags we pass to nvcc and that will be fine. Annoyingly some of the software hard codes the flags in a setup.py, Makefile, or similar and the only option then is to patch them.

magma: prebuildopts += 'CFLAGS="-mno-float128 -std=c++11 ${CFLAGS}" '

PyTorch - needs both the main PyTorch and gloo changed.

• PyTorch: TORCH_NVCC_FLAGS="-Xcompiler -mno-float128 "
• gloo: patch to add it in third_party/gloo/cmake/Cuda.cmake

TensorFlow, patch https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/t/TensorFlow/TensorFlow-2.3.1_disable-float128-on-power.patch

torchvision: preinstallopts = 'export NVCC_FLAGS="-std=c++14 -Xcompiler -mno-float128" && '

I've not looked into how we fix this in CuPy.

[branfosj]

I've looked at adding the -Xcompiler -mno-float128 into https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/toolchains/compiler/cuda.py but we do not use CUDA_CFLAGS or CUDA_CXXFLAGS in building magma and PyTorch.

For reference, the code change I used was:

diff --git a/easybuild/toolchains/compiler/cuda.py b/easybuild/toolchains/compiler/cuda.py
index 1758b6d..0353a85 100644
--- a/easybuild/toolchains/compiler/cuda.py
+++ b/easybuild/toolchains/compiler/cuda.py
@@ -27,8 +27,10 @@ Support for CUDA compilers as toolchain (co-)compiler.
 
 :author: Kenneth Hoste (Ghent University)
 """
+from distutils.version import LooseVersion
 
 from easybuild.tools.toolchain.compiler import Compiler
+from easybuild.tools.systemtools import POWER, get_cpu_architecture
 
 
 TC_CONSTANT_CUDA = "CUDA"
@@ -91,6 +93,11 @@ class Cuda(Compiler):
             'Xcompiler="%s"' % str(self.variables['CXXFLAGS']),
             'Xlinker="%s %s"' % (str(self.variables['LDFLAGS']), str(self.variables['LIBS'])),
         ]
+
+        cuda_version = self.get_software_version('CUDA')[0]
+        if LooseVersion(cuda_version) < LooseVersion('11.0') and POWER == get_cpu_architecture():
+            cuda_flags += ['Xcompiler="-mno-float128"']
+
         self.variables.nextend('CUDA_CFLAGS', cuda_flags)
         self.variables.nextend('CUDA_CXXFLAGS', cuda_flags)