MPI example on Ubuntu 18

[tomaslaz]

Version of Singularity:

2.6.1.-HEAD.9103f015

Expected behavior

Host: Fresh Ubuntu 18.04, Open MPI (3.1.3) install with default settings

Image (Created with NVIDIA HPCCM): Ubuntu 16.04, Open MPI (3.1.3) install with default settings, Mellanox OFED version 4.5-1.0.1.0

https://github.com/tomaslaz/HPCCM_testcases/blob/master/tests/mpi/mpi_point2point.def

Then I execute:

mpirun -np 4 singularity exec mpi_point2point.simg /usr/local/bin/point_to_point_mpi_send 

I would expect it to work and produce benchmarking results.

Actual behavior

The executions hangs and does not produce any output or errors.

Steps to reproduce behavior

The set up is described above. Executing:

mpirun -np 4 singularity exec mpi_point2point.simg /usr/local/bin/point_to_point_mpi_send 

[jscook2345]

A few questions:

Is this on a single node or multiple nodes?

Is the version of MPI in the container ABI compliant with the MPI version on the host(s)?

Can you run with the -d option provided to singularity for debug output and paste it in a comment here?

[tomaslaz]

Thank you for the reply jscook2345,

I am running on a single node (VirtuaBox)

-d output: log.txt

Could you please let me know how to check the ABI compatibility?

[tomaslaz]

I have done some more testing and found out that I have this issue when

Host is Ubuntu18.04/OpenMPI 3.1.3 and Singularity image is Ubuntu16.04/OpenMPI 3.1.3

but if

Host is Ubuntu18.04/OpenMPI 3.1.3 and Singularity image is Ubuntu18.04/OpenMPI 3.1.3

the problem does not occur.

[jscook2345]

I've opened an issue (open-mpi/ompi#6548) since it appears to be related to the OS + OpenMPI version. I came to this conclusion since I was able to reproduce this in a Vagrant VirtualBox VM without singularity being involved.

If you use a newer version of OpenMPI (4.X) everything seems to work across the board. So, for now, I'm closing this issue and will work with the OpenMPI folks on a resolution.

@tomaslaz Thanks for the report! If you wish to follow the work please subscribe to the issue I linked above.