Python garbage collector and mpi interface interaction issues.

[roigcarlo]

It appears that for some unknown reason python does not call the destructor of the mpi module when executed with mpirun. The problem is mainly that the job never ends so it keeps consuming CPU time and also it interferes with any 3rd party tool that tries to monitor for the end of the process as for example profilers

Example case:

mpirun -np 16 python MainKratos.py

In this case we should call 16 instances of:

/// Destructor, finalizes MPI if necessary.
~PythonMPI()
{
    int MpiIsFinalized = 0;
    MPI_Finalized(&MpiIsFinalized);
	
    if (MpiIsFinalized == 0)
        MPI_Finalize();
}

But only 11 or 12 are called. This drives me to the conclusion that the garbage collector is (a garbage) not waiting until the function is called at the end of script. Or the problem lies in how boost-python handles the destruction of the class. I am not sure.

A proposed solution for this, which I am using right now would be to explicitly define a finalize function:

/// Destructor, finalizes MPI if necessary.
void finalize()
{
    int MpiIsFinalized = 0;
    MPI_Finalized(&MpiIsFinalized);
	
    if (MpiIsFinalized == 0)
        MPI_Finalize();
}

Which basically does the same as the destructor but is explicitly called at the end of the script ( ideally leaving some room for the python script to do more things )

from KratosMultiphysics import mpi    # mpi_Initialize() is done here

# Fancy code here

mpi.finalize()

# Some more fancy code here if possible

[RiccardoRossi]

it may be a python2 issue

http://stackoverflow.com/questions/14628486/why-arent-destructors-guaranteed-to-be-called-on-interpreter-exit

[pooyan-dadvand]

@roigcarlo would you please check if this issue is only python2?

[roigcarlo]

Ok, let me make a little toy program

[RiccardoRossi]

i suspect this may be related to this issue.

boostorg/python#128

if this is so, it is a deep boost-python problem, and i would suggest removing this issue from the release blockers

[pooyan-dadvand]

mmmm, I'm not sure.... But I also think that this would be known issue for this release until we investigate more the problem.

[roigcarlo]

I made a small example here: https://github.com/roigcarlo/KratosUtils/tree/test/mpi/boostpython-destructors/Mpi/BoostPythonDel

Seems to work fine for me. I've tried only a normal example and initializing the module inside a class, both of them behaved correctly...

The only difference I've done with respect kratos is the

int MpiIsFinalized = 0;
    MPI_Finalized(&MpiIsFinalized);
	
    if (MpiIsFinalized == 0)
        MPI_Finalize();

I just call MPI_Finalize() without checking anything.

[pooyan-dadvand]

If I understood correctly the problem is calling the finalize manually and not relying on destructor, am I right?