MTT master: Segmentation faults in various Fortran_20.0_32_CentOS6.10 basic tests

[awlauria]

https://mtt.open-mpi.org/index.php?do_redir=3224

c_hello and c_ring are both segv'ing:

[ompi32:22230] *** Process received signal ***
[ompi32:22230] Signal: Segmentation fault (11)
[ompi32:22230] Signal code: Address not mapped (1)
[ompi32:22230] Failing at address: 0x86985a
[ompi32:22230] [ 0] [0xc4440c]
[ompi32:22230] [ 1] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmca_common_dstore.so.0(pmix_common_dstor_init+0x156)[0x40791a]
[ompi32:22230] [ 2] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/pmix/mca_gds_ds12.so(+0x12ac)[0x4122ac]
[ompi32:22230] [ 3] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libopen-pal.so.0(OPAL_MCA_PMIX4X_pmix_gds_base_select+0x1de)[0x6dac05]
[ompi32:22230] [ 4] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libopen-pal.so.0(OPAL_MCA_PMIX4X_pmix_rte_init+0x141b)[0x7480f6]
[ompi32:22230] [ 5] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libopen-pal.so.0(OPAL_MCA_PMIX4X_PMIx_Init+0x238)[0x6fe80d]
[ompi32:22230] [ 6] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmpi.so.0(ompi_rte_init+0x123)[0x47f48e]
[ompi32:22230] [ 7] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmpi.so.0(ompi_mpi_init+0x2c6)[0x561f5b]
[ompi32:22230] [ 8] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmpi.so.0(MPI_Init+0x89)[0x4cba80]
[ompi32:22230] [ 9] ./c_hello[0x80485ff]
[ompi32:22230] [10] /lib/libc.so.6(__libc_start_main+0xe8)[0x20ed28]
[ompi32:22230] [11] ./c_hello[0x8048551]
[ompi32:22230] *** End of error message ***[ompi32:22229] *** Process received signal ***
[ompi32:22229] Signal: Segmentation fault (11)
[ompi32:22229] Signal code: Address not mapped (1)
[ompi32:22229] Failing at address: 0xa4d85a[ompi32:22229] [ 0] [0xb0340c]
[ompi32:22229] [ 1] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmca_common_dstore.so.0(pmix_common_dstor_init+0x156)[0xe3091a]
[ompi32:22229] [ 2] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/pmix/mca_gds_ds12.so(+0x12ac)[0xdb52ac]
[ompi32:22229] [ 3] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libopen-pal.so.0(OPAL_MCA_PMIX4X_pmix_gds_base_select+0x1de)[0x8cfc05]
[ompi32:22229] [ 4] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libopen-pal.so.0(OPAL_MCA_PMIX4X_pmix_rte_init+0x141b)[0x93d0f6]
[ompi32:22229] [ 5] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libopen-pal.so.0(OPAL_MCA_PMIX4X_PMIx_Init+0x238)[0x8f380d]
[ompi32:22229] [ 6] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmpi.so.0(ompi_rte_init+0x123)[0x5bf48e]
[ompi32:22229] [ 7] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmpi.so.0(ompi_mpi_init+0x2c6)[0x6a1f5b]
[ompi32:22229] [ 8] /home/ompitest/scratches/2020-04-14/installs/tki8/install/lib/libmpi.so.0(MPI_Init+0x89)[0x60ba80]
[ompi32:22229] [ 9] ./c_hello[0x80485ff]
[ompi32:22229] [10] /lib/libc.so.6(__libc_start_main+0xe8)[0xb9cd28]
[ompi32:22229] [11] ./c_hello[0x8048551]
[ompi32:22229] *** End of error message
***--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node ompi32 exited on signal 11 (Segmentation
fault).

[rhc54]

@jsquyres Boris needs to see some debug info from Absoft - can you put him in contact with somebody there? We are unable to reproduce the problem, even when using the same OS version.

[rhc54]

Refs openpmix/openpmix#1720

[jsquyres]

Hey @cagoelz -- can you help out here? We're getting some errors in the MTT Absoft runs, but no one is able to reproduce them.

[cagoelz]

I will take a look and see what I can find out.

…

On 4/15/20 1:32 PM, Jeff Squyres wrote: Hey @cagoelz <https://github.com/cagoelz> -- can you help out here? We're getting some errors in the MTT Absoft runs, but no one is able to reproduce them. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7627 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJSNVJORBIHP6OSP2XCJRTRMXVSBANCNFSM4MHXR2EQ>.

[cagoelz]

Here are stack traces for c_hello and c_ring from the core files left behind in scratches/2020-04-15/installs/ompi-nightly-master--absoft--master-202004150242-708f945/tests/trivial/test_get__trivial ``` $ file core.29236 core.29236: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from './c_hello', real uid: 500, effective uid: 500, real gid: 500, effective gid: 500, execfn: './c_hello', platform: 'i686' $ gdb ./c_hello core.29236 (gdb) bt #0 0x007dc85a in ?? () #1 0x0041ecaa in OPAL_MCA_PMIX4X_pmix_pshmem_base_select () at base/pshmem_base_select.c:89 #2 0x004c691a in pmix_common_dstor_init (ds_name=0xf98964 "ds12", info=0x0, ninfo=0, lock_cb=0xf99d00, file_cb=0xf99e60) at dstore_base.c:1590 #3 0x00f972ac in ds12_init (info=0x0, ninfo=0) at gds_ds12_base.c:39 #4 0x00325c05 in OPAL_MCA_PMIX4X_pmix_gds_base_select (info=0x0, ninfo=0) at base/gds_base_select.c:86 #5 0x003930f6 in OPAL_MCA_PMIX4X_pmix_rte_init (type=1, info=0x0, ninfo=0, cbfunc=0x3473f7 <pmix_client_notify_recv>) at runtime/pmix_init.c:396 #6 0x0034980d in OPAL_MCA_PMIX4X_PMIx_Init (proc=0xafbc28, info=0x0, ninfo=0) at client/pmix_client.c:561 #7 0x009e448e in ompi_rte_init (pargc=0xbfc31350, pargv=0xbfc31354) at runtime/ompi_rte.c:552 #8 0x00ac6f5b in ompi_mpi_init (argc=1, argv=0xbfc31464, requested=0, provided=0xbfc31370, reinit_ok=false) at runtime/ompi_mpi_init.c:508 #9 0x00a30a80 in PMPI_Init (argc=0xbfc313c0, argv=0xbfc313c4) at pinit.c:67 #10 0x080485ff in main () ``` ``` $ file core.29113 core.29113: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from './c_ring', real uid: 500, effective uid: 500, real gid: 500, effective gid: 500, execfn: './c_ring', platform: 'i686' $ gdb ./c_ring core.29113 (gdb) bt #0 0x0064485a in ?? () #1 0x00f6ecaa in OPAL_MCA_PMIX4X_pmix_pshmem_base_select () at base/pshmem_base_select.c:89 #2 0x0023891a in pmix_common_dstor_init (ds_name=0x1ca964 "ds12", info=0x0, ninfo=0, lock_cb=0x1cbd00, file_cb=0x1cbe60) at dstore_base.c:1590 #3 0x001c92ac in ds12_init (info=0x0, ninfo=0) at gds_ds12_base.c:39 #4 0x00e75c05 in OPAL_MCA_PMIX4X_pmix_gds_base_select (info=0x0, ninfo=0) at base/gds_base_select.c:86 #5 0x00ee30f6 in OPAL_MCA_PMIX4X_pmix_rte_init (type=1, info=0x0, ninfo=0, cbfunc=0xe973f7 <pmix_client_notify_recv>) at runtime/pmix_init.c:396 #6 0x00e9980d in OPAL_MCA_PMIX4X_PMIx_Init (proc=0x80bc28, info=0x0, ninfo=0) at client/pmix_client.c:561 #7 0x006f448e in ompi_rte_init (pargc=0xbfb50450, pargv=0xbfb50454) at runtime/ompi_rte.c:552 #8 0x007d6f5b in ompi_mpi_init (argc=1, argv=0xbfb505a4, requested=0, provided=0xbfb50470, reinit_ok=false) at runtime/ompi_mpi_init.c:508 #9 0x00740a80 in PMPI_Init (argc=0xbfb50500, argv=0xbfb50504) at pinit.c:67 #10 0x080486d7 in main () ``` Anything else I can provide?

[cagoelz]

Possibly exceeding the default stack limit?

Working in scratches/2020-04-15/installs/ompi-nightly-master--absoft--master-202004150242-708f945/tests/trivial/test_get__trivial

$ source ../../../mpi_installed_vars.sh 
$ ulimit -s
10240
$ mpiexec -n 2 ./c_hello 
[ompi32:29858] *** Process received signal ***
[ompi32:29858] Signal: Segmentation fault (11)
[ompi32:29858] Signal code: Address not mapped (1)
[ompi32:29858] Failing at address: 0xab985a
   (output snipped)

$ ulimit -s unlimited
$ mpiexec -n 2 ./c_hello 
Hello, C world!  I am 0 of 2
Hello, C world!  I am 1 of 2

[rhc54]

Has to be something like stack corruption as it otherwise makes no sense.

[rhc54]

That's an awfully short stack limit - is there a reason for it? I don't believe that is what comes with CentOS by default - is it?

[cagoelz]

I did not do anything special when creating our 32 bit test bed, just a stock install with the extra packages for development. I just checked a 64 bit version of CentOS 6.10 that I set up a
week or so ago and it also has 10240k as the default stack limit. Out of curiosity, I checked
on 64 bit RHEL 8 system and the default stack limit there is 8192k.

[rhc54]

Just to clarify: are all the failures on 32-bit test beds?

[cagoelz]

Yes, all of the failures at Absoft are on the 32-bit test bed.

[karasevb]

According to the backtrace it fails at pmix_pshmem.finalize() line 89:

85        if (best_pri < priority) {
86            best_pri = priority;
87            /* give any prior module a chance to finalize */
88            if (NULL != pmix_pshmem.finalize) {
89                pmix_pshmem.finalize(); 
90            }
91            pmix_pshmem = *nmodule;
92            inserted = true;
93        }

In the case that we see, the pmix_pshmem.finalize pointer must be NULL. Looks like the finalize pointer is corrupted and assigned to unexpected address.

I cannot able to reproduce at my VirtualBox with the same 32bit OS, even setup ulimit -s 32 it works well.