import from mpi4py is causing python to crash on login nodes

[gppezzi]

The following mpi4py import on fancylogger is causing python to crash on login nodes (where we don't have a working MPI subsystem):

from mpi4py import MPI

The author of mpi4py suggested a couple of workarounds, the first being to precede the import with:

import mpi4py.rc
mpi4py.rc.initialize = False

This would fix the crash on the login nodes, but I suppose the following call to MPI.COMM_WORLD.Get_rank() would stop working on the compute nodes.

For keeping track of the discussion, I'm also pasting here another proposed solution by the author of mpi4py:

However, you could write your code this way:

from mpi4py import MPI
if MPI.Is_initialized():
rank_id = MPI.COMM_WORLD.Get_rank()
else:
rank_id = 0 # or None, and special case None values elsewhere in the logger code

Here we have some clues on how to fix the problem, however it is not clear to me how we can fix that in a way that it works on both compute and login nodes.

Any ideas?

[stdweird]

what is the error or exception that Get_rank() throws?

[boegel]

@stdweird: it never gets that far, the from mpi4py import MPI makes Python crash (not with an exception, but a hard crash), like this:

python -c "from mpi4py import MPI"
[Wed Feb  3 16:38:58 2016] [unknown] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(547): 
MPID_Init(203).......: channel initialization failed
MPID_Init(584).......:  PMI2 init failed: 1 
Aborted

We had similar problems with the QLogicMPI on gulpin waaaaaaay back.

[stdweird]

the second solution seems to be clean, i'll make aquick patch for that

[boegel]

Ah, sorry, I should've read the whole thing first...

If you call Get_rank with init'ing disabling, you get this (hard crash again):

kehoste@daint101:~> python                    
Python 2.7.10 (default, Jun 25 2015, 18:27:17) 
[GCC 4.8.2 20131016 (Cray Inc.)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mpi4py.rc
>>> mpi4py.rc.initialize = False
>>> from mpi4py import MPI
>>> MPI.COMM_WORLD.Get_rank()
Attempting to use an MPI routine before initializing MPICH
kehoste@daint101:~> 

[gppezzi]

@stdweird I dont see how just applying the second solution will fix the problem, since python crashes at the import statement as showed by @boegel

let me know if you want me to do some testing

[stdweird]

@gppezzi ah, i read your original post such that the second proposed solution was an actual solution. hmmm, what MPI is this? and what is mpi4py version?

[stdweird]

@gppezzi hmm, so your setup is such using the mpi4py is broken but available, and you want us to handle that? for any other module, we would simply say "fix/rebuild your module" 😄
i assume you can't just fix the underlying problem and that this is some site specific issue?

can you try

mkdir -p /tmp/brokenmpi4py/mpi4py
echo 'raise ImportError' >  /tmp/brokenmpi4py/mpi4py/__init__.py
export PYTHONPATH="/tmp/brokenmpi4py:$PYTHONPATH"
python -c 'from vsc.utils import fancylogger'

[boegel]

@stdweird: the problem is that import mpi4py is triggering an MPI_Init, while it shouldn't be.

For some MPI libraries (like the Cray MPICH one, and the QLogicMPI we had on gulpin in the old days), this is a problem.

[gppezzi]

@stdweird yes I can workaround this on my side, I was just reporting a bug that is triggered by loading vsc-base.

Even if the mpi subsystem is not enabled on login nodes, mpi4py all alone is not making python unusable, while if we load vsc-base, python crashes.

If you tell me that vsc-base is supposed only to work on compute nodes when mpi4py is available, it is fine for me I can redesign our python builds to not embed mpi4py.

[stdweird]

@gppezzi use the current workaround on your loginnodes, i'll add a environment to disable loading/trying to load mpi4py when the provided module is available but unusable so you can set this in e.g. the profile of your loginnodes.
vsc-base will try to use mpi4py when it's available, clearly this is too agressive for your setup. (although i consider it a problem to have known broken modules available)

[gppezzi]

@stdweird it would be nice to have this option to set a variable to avoid loading mpi4py, thanks a lot for taking a look into this issue!

We have many other modules that are available on the login nodes and only actually work on the compute nodes, so you can consider our environment completely broken ;-)

[boegel]

#217 adds support for picking up $FANCYLOGGER_IGNORE_MPI4PY, so EB should define that as $FANCYLOGGER_IGNORE_MPI4PY=1?