Need advice on debugging an issue with 3.1.3 and -mca pml ob1

[mwheinz]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

Intel OPA build of OMPI v3.1.3

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Internal CI testing on a back-to-back OPA fabric.

Please describe the system on which you are running

• Operating system/version: RHEL 7.5, 7.6, SLES 12 SP3, SLES 12 SP4
• Computer hardware: Xeon Skylake or better
• Network type: OPA

---

Details of the problem

Recently got this issue from one of our testers saying that the following command line:

/usr/mpi/gcc/openmpi-3.1.3-hfi/bin/mpirun -H hds1fna5102.hd.intel.com,hds1fna5103.hd.intel.com,hds1fna5104.hd.intel.com,hds1fna5101.hd.intel.com --allow-run-as-root --mca oob tcp --mca pml ob1 --mca btl tcp,vader,self --mca btl_tcp_if_include ib0 -np 200 --map-by node --oversubscribe /usr/mpi/gcc/openmpi-3.1.3-hfi/tests/IMB-4.0/IMB-MPI1 Sendrecv -npmin 200 -iter 150 -iter_policy off

is hanging "unless they remove --mca pml ob1". Of course, looking at this command line, I'm pretty sure that if they do that they stop using OPA altogether.

Since I'm still trying to learn the OMPI internals, I'm unsure how to approach this, so a few questions:

1. Any chance this is a known issue?
2. Assuming it's not, any suggestions on how to debug a silent hang? (I mean, besides staring at the screen and hoping for inspiration to strike...)

[ggouaillardet]

The command you posted force the use of IP over the ib0 interface for inter node communications (and vader for intra node communications).

If you do not pass --mca pml ob1 then (native) OPA will likely be used either via libfabric or PSM2.

[mwheinz]

If you do not pass --mca pml ob1 then (native) OPA will likely be used either via libfabric or PSM2.

Interesting. Thanks for helping me understand what they're testing. Not sure why anyone would even want to use ipoib instead of native OPA....

[bosilca]

This is extremely unsettling as OB1 + TCP was our defacto working setup for all environments. Fortunately, I cannot reproduce such a deadlock on my systems, even when the processes are heavily oversubscribed, so this might be some issue with your particular setting.

Do you get any output from the test before the deadlock ?

[mwheinz]

Do you get any output from the test before the deadlock ?

From my current round of testing it appears that the 0-byte round completes but the 1-byte round never does:

[RHEL7.5 hdsmpriv01 20190723_1154 ~]# /usr/mpi/gcc/openmpi-3.1.3-hfi/bin/mpirun -H hdsmpriv01,hdsmpriv02 --allow-run-as-root --mca oob tcp --mca pml ob1 --mca btl tcp,vader,self --mca btl_tcp_if_include ib0 -np 400 --map-by node --oversubscribe /usr/mpi/gcc/openmpi-3.1.3-hfi/tests/IMB-4.0/IMB-MPI1 Sendrecv -npmin 400 -iter 200 -iter_policy off
#------------------------------------------------------------
#    Intel (R) MPI Benchmarks 2018 Update 1, MPI-1 part    
#------------------------------------------------------------
# Date                  : Tue Jul 23 12:38:39 2019
# Machine               : x86_64
# System                : Linux
# Release               : 3.10.0-862.el7.x86_64
# Version               : #1 SMP Wed Mar 21 18:14:51 EDT 2018
# MPI Version           : 3.1
# MPI Thread Environment: 


# Calling sequence was: 

# /usr/mpi/gcc/openmpi-3.1.3-hfi/tests/IMB-4.0/IMB-MPI1 Sendrecv -npmin 400 -iter 200 -iter_policy off
#                                                      

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT
# MPI_Op                         :   MPI_SUM  
#
#

# List of Benchmarks to run:

# Sendrecv

#-----------------------------------------------------------------------------
# Benchmarking Sendrecv 
# #processes = 400 
#-----------------------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]   Mbytes/sec
            0          200      2425.23      3888.08      3223.43         0.00

While running the test I also ran "while true; do ifstat ib0; sleep 1; done" on the other host. What I saw there was that the packet rate went to zero after the 0-byte round completed.

[bosilca]

Strange, as a 0-byte packet translate in a message in Open MPI, and thus requires the connections to be setup between the peers. So we can't blame the TCP connection setup, the issue should be somewhere else. Unfortunately I cannot reproduce it on my setup (I only have ipoib), I would really appreciate some help to understand what is going on. Does it happen on a non-oversubscribed run ? What is the smallest number of processes when this starts happening ? Can you find a stack trace of a debug build pinpointing to the deadlock ?

[mwheinz]

If you can instruct me on how to collect such a stack trace, I would be happy to do so - I've never quite figured out how to do that.

[mwheinz]

The primary thing pointing the finger at OMPI instead of OPA is that the same install using OMPI 2.1.2 does not experience the issue. (Which doesn't mean the problem isn't somehow in our own build of OMPI 3.1.3)

[bosilca]

To get the stack trace, you can attach with gdb to one of the processes using gdb -p **pib** and then generate the backtrace with the gdb command bt. However, if order for the trace to be human readable, you would need to have a debug build of OMPI.

Before going there, can you please confirm if this bug appears with a more recent version of OMPI stable (4.0.1) ?

[mwheinz]

To get the stack trace, you can attach with gdb to one of the processes using gdb -p **pib** and then generate the backtrace with the gdb command bt. However, if order for the trace to be human readable, you would need to have a debug build of OMPI.

Unfortunately, when I built a debug version of 3.1.3 the job doesn't start - it fails with this message:

[RHEL7.5 hdsmpriv01 20190724_1314 OFA_MPIS.RHEL75-x86_64.10.10.0.0.296]# /usr/mpi/gcc/openmpi-3.1.3-hfi/bin/mpirun -H hdsmpriv01.hd.intel.com,hdsmpriv02.hd.intel.com --allow-run-as-root --mca oob tcp --mca pml ob1 --mca btl tcp,vader,self --mca btl_tcp_if_include ib0 -np 400 --map-by node --oversubscribe /usr/mpi/gcc/openmpi-3.1.3-hfi/tests/IMB-4.0/IMB-MPI1 Sendrecv -npmin 400 -iter 200 -iter_policy off
[hdsmpriv01:56631] [[18566,0],0] ORTE_ERROR_LOG: Data unpack had inadequate space in file base/plm_base_launch_support.c at line 1319

Before going there, can you please confirm if this bug appears with a more recent version of OMPI stable (4.0.1) ?

Doing that now!

[mwheinz]

bosilca, I appreciate your help but I made a disturbing discovery - while I always get the issue with the "official" build of ompi 3.1.3, none of my developer builds have the problem.

Looks like I'm going to be studying the build logs looking for a discrepancy.

[bosilca]

@mwheinz another place you should look at is the system wide configuration files (${prefix-ompi-3.1}/etc/openmpi-mca-params.conf). It might contain unexpected restrictions.

[ggouaillardet]

@mwheinz that kind of issue can happen when mpirun and orted are not compatible. For example if you built mpirun with --enable-debug but orted was built without this option.

[mwheinz]

@ggouaillardet, @bosilca, So, from looking at the output of autogen and configure, the differences I've noticed between the official build machine and my personal dev machine is that the official machine has infinipath headers installed (probably not the cause, but easy to check, systemd-devel (libudev.h), and ltdl.h - which seems to be part of a special version of libtool?

There were other differences in the output but these were the ones that looked like they might impact the operation of ompi on a machine that might not have the same software installed - any thoughts?

Oh, and adding @jsquyres - - Does ompi still have infinipath users? It's been 7 years since I've seen a SusieQ and I helped develop them...

[jsquyres]

Also, we have seen problems with IPoIB before in the past (i.e., where the IPoIB stack wasn't 100% stable, and various versions of Open MPI worked/didn't work, depending on how each OMPI version specifically used the TCP stack). I would take --mca btl_tcp_if_include ib0 out altogether, just to remove that variable. Heck, you may even want to --mca btl_tcp_if_exclude ib0 --mca oob_tcp_if_exclude ib0 to make sure Open MPI doesn't use your IPoIB interfaces at all.

By "infinipath headers", I assume you mean that Open MPI was able to compile the PSM MTL?

@ggouaillardet correctly mentioned that --mca pml ob1 means that you have forced Open MPI to use the OB1 PML, which means that the BTLs are used for MPI point-to-point communications (e.g., the TCP BTL, which @bosilca is now worried about). If you remove --mca pml ob1, Open MPI should default back to the cm PML+PSM MTL. But you can force this, too, via --mca pml cm --mca mtl psm -- just to remove any question as to what transport was used.

Does ompi still have infinipath users?

That's a question only Intel can really answer... You guys effectively "own" the PSM and PSM2 MTLs; I don't think that anyone other than Intel would remove them.

[mwheinz]

That's a question only Intel can really answer... You guys effectively "own" the PSM and PSM2 MTLs; I don't think that anyone other than Intel would remove them.

Yeah, that's what I figured. Okay.

As to your other points, I agree that ipoib can be unreliable and, actually, that's the point of this particular test - rather than testing ompi, it's supposed to be testing ipoib. The finger got pointed at ompi, however, because the test stopped working when we shifted our builds to using 3.1.3 instead of 2.1.2.