MPI jobs fail with intel toolchains after upgrade of EL8 Linux from 8.5 to 8.6

[OleHolmNielsen]

I'm testing the upgrade of our compute nodes from Almalinux 8.5 to 8.6 (the RHEL 8 clone similar to Rocky Linux).

We have found that all MPI codes built with any of the Intel toolchains intel/2020b or intel/2021b fail after the 8.5 to 8.6 upgrade. The codes fail also on login nodes, so the Slurm queue system is not involved.
The FOSS toolchains foss/2020b and foss/2021b work perfectly on EL 8.6, however.

My simple test uses the attached trivial MPI Hello World code running on a single node:

$ module load intel/2021b
$ mpicc mpi_hello_world.c
$ mpirun ./a.out

Now the mpirun command enters an infinite loop (running many minutes) and we see these processes with "ps":

/bin/sh /home/modules/software/impi/2021.4.0-intel-compilers-2021.4.0/mpi/2021.4.0/bin/mpirun ./a.out
mpiexec.hydra ./a.out

The mpiexec.hydra process doesn't respond to 15/SIGTERM and I have to kill it with 9/SIGKILL. I've tried to enable debugging output with

export I_MPI_HYDRA_DEBUG=1
export I_MPI_DEBUG=5

but nothing gets printed from this.

Question: Has anyone tried an EL 8.6 Linux with the Intel toolchain and mpiexec.hydra? Can you suggest how I may debug this issue?

OS information:

$ cat /etc/redhat-release
AlmaLinux release 8.6 (Sky Tiger)
$ uname -r
4.18.0-372.9.1.el8.x86_64

[ocaisa]

Quoting some discussion we've had on this in Slack:

no, its not a glibc issue afaics. If you use a RHEL8.5 kernel (with an uptodate RHEL8.6 system on the other side), intelmpi is working

[boegel]

Yikes...

@OleHolmNielsen Have you been in touch with Intel support on this?

@rscohn2 Any thoughts on this?

[OleHolmNielsen]

I didn't know that this issue is related to the updated RHEL 8.6 kernel, so I didn't contact Intel support yet. I've never been in touch with Intel compiler/libraries support before, so if someone else knows how to do that, could you kindly open an issue with them?
Thanks,
Ole

[boegel]

We ran into a silent hang issue several years ago too, details in hpcugent/vsc-mympirun#74

Any luck w.r.t. getting output when using mpirun -d?

[daRecall]

It seems (although nothing to be seen within the kernel release notes) that numa info has changed within the kernel.
intelmpi before version 2021.6.0 gets stuck.
using pstack, one can see, that the processes seem to hang within an infinite loop somewhere around ipl_detect_machine_topology
That happens even before mpiexec.hydra tries to do something with the to be called binary (might it be a.out or hostname).

[daRecall]

@boegel mpiexec.hydra does not know the -d parameter:

$> mpirun -d -np 2 hostname
[mpiexec@nrm095.hpc.itc.rwth-aachen.de] match_arg (../../../../../src/pm/i_hydra/libhydra/arg/hydra_arg.c:91): unrecognized argument d
[mpiexec@nrm095.hpc.itc.rwth-aachen.de] Similar arguments:
[mpiexec@nrm095.hpc.itc.rwth-aachen.de]          membind
[mpiexec@nrm095.hpc.itc.rwth-aachen.de]          debug
[mpiexec@nrm095.hpc.itc.rwth-aachen.de]          dac
[mpiexec@nrm095.hpc.itc.rwth-aachen.de]          disable-x
[mpiexec@nrm095.hpc.itc.rwth-aachen.de]          demux
[mpiexec@nrm095.hpc.itc.rwth-aachen.de] HYD_arg_parse_array (../../../../../src/pm/i_hydra/libhydra/arg/hydra_arg.c:128): argument matching returned error
[mpiexec@nrm095.hpc.itc.rwth-aachen.de] mpiexec_get_parameters (../../../../../src/pm/i_hydra/mpiexec/mpiexec_params.c:1356): error parsing input array
[mpiexec@nrm095.hpc.itc.rwth-aachen.de] main (../../../../../src/pm/i_hydra/mpiexec/mpiexec.c:1749): error parsing parameters

but it knows --debug, but the only thing you see, is the called command:

$> mpiexec.hydra --debug -np 2 hostname
[mpiexec@nrm095.hpc.itc.rwth-aachen.de] Launch arguments: /cvmfs/software.hpc.rwth.de/Linux/RH8/x86_64/intel/skylake_avx512/software/impi/2021.2.0-intel-compilers-2021.2.0/mpi/2021.2.0/bin//hydra_bstrap_proxy --upstream-host nrm095.hpc.itc.rwth-aachen.de --upstream-port 44829 --pgid 0 --launcher ssh --launcher-number 0 --base-path /cvmfs/software.hpc.rwth.de/Linux/RH8/x86_64/intel/skylake_avx512/software/impi/2021.2.0-intel-compilers-2021.2.0/mpi/2021.2.0/bin/ --tree-width 16 --tree-level 1 --time-left -1 --launch-type 2 --debug --proxy-id 0 --node-id 0 --subtree-size 1 --upstream-fd 7 /cvmfs/software.hpc.rwth.de/Linux/RH8/x86_64/intel/skylake_avx512/software/impi/2021.2.0-intel-compilers-2021.2.0/mpi/2021.2.0/bin//hydra_pmi_proxy --usize -1 --auto-cleanup 1 --abort-signal 9

[ocaisa]

Looking at https://community.intel.com/t5/Intel-oneAPI-HPC-Toolkit/bug-mpiexec-segmentation-fault/m-p/1183364, you can influence this with

I_MPI_HYDRA_TOPOLIB=ipl

(Ha, look who is posting the last comment in that link)

[daRecall]

using impi 2021.6.0, everything is working:

$> mpiexec.hydra --version
Intel(R) MPI Library for Linux* OS, Version 2021.6 Build 20220227 (id: 28877f3f32)
Copyright 2003-2022, Intel Corporation.

$> mpiexec.hydra -np 2 hostname
nrm095.hpc.itc.rwth-aachen.de
nrm095.hpc.itc.rwth-aachen.de

[daRecall]

@ocaisa that doesn't change anything, looping around the same function

[daRecall]

ahh, yes, I forgot, we have had an issue open with intel (case# 05472393) regarding the problem. Their first comment was as usual "are you trying the newest version?"
I did not visit ISC this year, but some of my colleagues did and they talked to some intel guys directly. Outcome was, with RHEL 8.6 and newer, old intelmpi is not working anymore.

[OleHolmNielsen]

Is there any chance that Red Hat will accept a bug report for the older IntelMPI versions not working?
This would require a deeper understanding of what changes in the new kernel does that breaks IntelMPI, so documenting the bug might be a challenge...

[stdweird]

@OleHolmNielsen kernel updates that break userspace are frwoned upon, so you can try to open a bugreport with redhat. they will at some point as you what they need, or point you to the release notes that say what has changed that broke this. they will probaby blame intel (and sounds like intel already fixed it, but doesn't want to backport it)

[OleHolmNielsen]

@stdweird Yes, but how do we get any error messages from mpiexec.hydra which can be reported to Red Hat?