Open MPI 2.1.0: MPI_Finalize hangs because cuIpcCloseMemHandle fails

[Evgueni-Petrov-aka-espetrov]

Hi Open MPI,

Thank you very much for fixing #3042!

We want to switch from version 2.0.2 to 2.1.0 containing the fix but, if we do, our application starts hanging in MPI_Finalize.
From our point of view, this behavior is a regression in version 2.1.0 w.r.t version 2.0.2.

First, MPI_Finalize warns that cuIpcCloseMemHandle failed with the return value of 4 (CUDA_DEINITIALIZED), and then it prints the following messages in a loop:

[hostname:87484] Sleep on 87484
[hostname:87483] Sleep on 87483
[hostname:87478] 1 more process has sent help message help-mpi-common-cuda.txt / cuIpcCloseMemHandle failed
[hostname:87478] 1 more process has sent help message help-mpi-common-cuda.txt / cuIpcCloseMemHandle failed

...
Gdb shows the following stack:

#0  0x00007f12df23393d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007f12df2337d4 in __sleep (seconds=0)
    at ../sysdeps/unix/sysv/linux/sleep.c:137
#2  0x00007f12d59f63bd in cuda_closememhandle ()
   from /home/espetrov/sandbox/install_mpi/lib/libmca_common_cuda.so.20
#3  0x00007f12d55e93c9 in mca_rcache_rgpusm_finalize ()
   from /home/espetrov/sandbox/install_mpi/lib/openmpi/mca_rcache_rgpusm.so
#4  0x00007f12deca5b92 in mca_rcache_base_module_destroy ()
   from /home/espetrov/sandbox/install_mpi/lib/libopen-pal.so.20
#5  0x00007f12d435e57a in mca_btl_smcuda_del_procs ()
   from /home/espetrov/sandbox/install_mpi/lib/openmpi/mca_btl_smcuda.so
#6  0x00007f12d51e1042 in mca_bml_r2_del_procs ()
   from /home/espetrov/sandbox/install_mpi/lib/openmpi/mca_bml_r2.so
#7  0x00007f12dfaa2918 in ompi_mpi_finalize ()
   from /home/espetrov/sandbox/install_mpi/lib/libmpi.so.20

I am not sure but I would say that MPI_Finalize tries to close a remote memory handle after the remote MPI process unloaded libcuda.so.

Probably, getting CUDA_DEINITIALIZED from cuIpcCloseMemHandle is OK?
Our CUDA version is 7.5, CUDA driver version is 361.93.02.

Evgueni.

[jsquyres]

@sjeaugey This appears to be stuck in a CUDA call. Can you look into this?

[sjeaugey]

Indeed, looks like a problem on our part. I'll look into this.

[sjeaugey]

Maybe it's not even stuck. For some reason, there is a sleep(20) (!) for every IpcCloseMemHandle that fails. So if we're emptying our cache, there may be a lot of handles to close, and it can take a really long time !

I noticed yesterday my MTT was not finished in the morning, so I had to kill it but didn't have time to look into it. Same today. Maybe that's the reason.

I'm currently testing a fix ignoring CUDA_DEINITIALIZED return codes and most importantly removing the sleep(20).

[sjeaugey]

I couldn't manage to reproduce the bug so far.

I tested that patch :

diff --git a/opal/mca/common/cuda/common_cuda.c b/opal/mca/common/cuda/common_cuda.c
index 2ce3b20..d66f00b 100644
--- a/opal/mca/common/cuda/common_cuda.c
+++ b/opal/mca/common/cuda/common_cuda.c
@@ -1157,10 +1157,10 @@ int cuda_closememhandle(void *reg_data, mca_rcache_base_registration_t *reg)
     if (ctx_ok) {
         result = cuFunc.cuIpcCloseMemHandle((CUdeviceptr)cuda_reg->base.alloc_base);
         if (OPAL_UNLIKELY(CUDA_SUCCESS != result)) {
-            opal_show_help("help-mpi-common-cuda.txt", "cuIpcCloseMemHandle failed",
-                           true, result, cuda_reg->base.alloc_base);
-            opal_output(0, "Sleep on %d", getpid());
-            sleep(20);
+            if (CUDA_ERROR_DEINITIALIZED != result) {
+                opal_show_help("help-mpi-common-cuda.txt", "cuIpcCloseMemHandle failed",
+                true, result, cuda_reg->base.alloc_base);
+            }
             /* We will just continue on and hope things continue to work. */
         } else {
             opal_output_verbose(10, mca_common_cuda_output,

which compiles and works but since I can't reproduce the bug, I can't confirm it fixes the problem for sure.

@Evgueni-Petrov-aka-espetrov can you give it a try ?

[Evgueni-Petrov-aka-espetrov]

Thanks for the fix, @sjeaugey!
It works for our application.

[sjeaugey]

Thanks for sharing the result. @jsquyres is it OK for me to push that patch to master directly (since Evgueni confirmed it fixed the issue ?)

[rhc54]

Why not just put it in a branch and submit a PR like normal? It would allow the CI to ensure nothing broke outside of this environment.

[sjeaugey]

Sure -- just takes more time. I'll submit a PR.