LAMMPS-3Mar2020-intel-2020a fails on RHEL8 AMD (zen2), but LAMMPS-3Mar2020-intel-2019b not.

[hajgato]

I am trying to make LAMMPS 3Mar2020 with intel/2020a, but it fails on RHEL8 with AMD EPYC 7552. It seems 2019b works (OK the intel/2019a is a bit crippled as we use IMPI/2019.7.217, because RHEL8), but I do not understand what can cause the problem:

BlockingIOError: [Errno 11] Resource temporarily unavailable: '/kyukon/home/gent/400/vsc40003/.local/share/virtualenv/seed-app-data/v1.0.1/3.8/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip/_vendor/distlib/w32.exe' -> '/tmp/vsc40003/easybuild/LAMMPS/3Mar2020/intel-2020a-Python-3.8.2-kokkos/easybuild_obj/docenv/lib/python3.8/site-packages/pip/_vendor/distlib/w32.exe'

Could someone test it on RHEL8 with 2020a whether the problem can be reproduced?

You can see more details in #11577

[hajgato]

Adding configopts = "-DBUILD_DOC=off " solves the problem. I have the feeling that this problem specifically relates to our system.

[ocaisa]

I switched docs off as well, it creates a virtualenv and was just a general headache for us

[boegel]

Maybe the docs-related patch we use should be extended a bit?

[hajgato]

It seems to be that the error is indeed specific to our system:
I can only reproduce the error when the home (where the virtualenv is installed and causes the BlockingIOError) is on a GPFS partition.

[hajgato]

IBM confirmed: this is a Spectrum Scale (GPFS) bug, if a file is a multiple of 44 bytes then sendfile fails.

Applying this patch on Python 3.8 solves the problem (with minimum impact):

--- /usr/lib64/python3.8/shutil.py.orig	2020-10-30 17:40:08.176641322 +0100
+++ /usr/lib64/python3.8/shutil.py	2020-10-30 17:40:19.014622459 +0100
@@ -50,7 +50,7 @@
     import nt
 COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024
-_USE_CP_SENDFILE = hasattr(os, "sendfile") and sys.platform.startswith("linux")
+_USE_CP_SENDFILE = False
 _HAS_FCOPYFILE = posix and hasattr(posix, "_fcopyfile")  # macOS
 __all__ = ["copyfileobj", "copyfile", "copymode", "copystat", "copy", "copy2",

[boegel]

@hajgato Should we add that patch to recent Python easyconfigs?
I'd say GPFS is used commonly enough that we should consider this.

Is there an easy way that people can tell if they're affected by this GPFS bug?

[hajgato]

@boegel I kinda hope that we will have a fix soon for this on IBM side.