Hang in mca_mpool_hugepage_module_init() on ARM64

[yosefe]

Background information

               Package: Open MPI mtt@hpc-arm-01 Distribution
                Open MPI: 2.1.2a1
  Open MPI repo revision: v2.1.1-55-g4d82554
   Open MPI release date: Unreleased developer copy
                Open RTE: 2.1.2a1
  Open RTE repo revision: v2.1.1-55-g4d82554
   Open RTE release date: Unreleased developer copy
                    OPAL: 2.1.2a1
      OPAL repo revision: v2.1.1-55-g4d82554
       OPAL release date: Unreleased developer copy

• Operating system/version: RedHat 7.2
• Computer hardware: aarch64 (ARM 64-bit)
• Network type: Infiniband

---

Details of the problem

A hang in ctxalloc test during MPI_Init.
Similar hang is observed in many other tests.
ctxalloc can be found here

mpirun -np 192 -mca btl_openib_warn_default_gid_prefix 0 \
  --bind-to core -mca pml ucx \
  -x UCX_NET_DEVICES=mlx5_0:1 -mca btl_openib_if_include mlx5_0:1 \
  -x UCX_TLS=rc,sm -mca opal_pmix_base_async_modex 0 \
   -mca mpi_add_procs_cutoff 100000 --map-by node \
ctxalloc 2 1500 100

stack trace:

Thread 1 (Thread 0x3ffb800a480 (LWP 27563)):
#0  0x000003ffb7ccfa74 in nanosleep () from /usr/lib64/libpthread.so.0
#1  0x000003ffb7911e0c in _opal_lifo_release_cpu () at ../opal/class/opal_lifo.h:195
#2  0x000003ffb7911e40 in opal_lifo_pop_atomic (lifo=0x620520) at ../opal/class/opal_lifo.h:210
#3  0x000003ffb7911fd0 in opal_free_list_get_st (flist=0x620520) at ../opal/class/opal_free_list.h:213
#4  0x000003ffb7911ff4 in opal_free_list_get (flist=0x620520) at ../opal/class/opal_free_list.h:225
#5  0x000003ffb7912264 in opal_rb_tree_init (tree=0x6204e0, comp=0x3ffb4671ae4 <mca_mpool_rb_hugepage_compare>) at class/opal_rb_tree.c:86
#6  0x000003ffb4671d44 in mca_mpool_hugepage_module_init (mpool=0x620410, huge_page=0x61ef30) at mpool_hugepage_module.c:107
#7  0x000003ffb4672a10 in mca_mpool_hugepage_open () at mpool_hugepage_component.c:166
#8  0x000003ffb7948ab4 in open_components (framework=0x3ffb7a34788 <opal_mpool_base_framework>) at mca_base_components_open.c:117
#9  0x000003ffb79489e4 in mca_base_framework_components_open (framework=0x3ffb7a34788 <opal_mpool_base_framework>, flags=MCA_BASE_OPEN_DEFAULT) at mca_base_components_open.c:65
#10 0x000003ffb79bd2a0 in mca_mpool_base_open (flags=MCA_BASE_OPEN_DEFAULT) at base/mpool_base_frame.c:89
#11 0x000003ffb7957d1c in mca_base_framework_open (framework=0x3ffb7a34788 <opal_mpool_base_framework>, flags=MCA_BASE_OPEN_DEFAULT) at mca_base_framework.c:174
#12 0x000003ffb7d59bb8 in ompi_mpi_init (argc=4, argv=0x3ffffffdc38, requested=0, provided=0x3ffffffda2c) at runtime/ompi_mpi_init.c:589
#13 0x000003ffb7d990e0 in PMPI_Init (argc=0x3ffffffdaac, argv=0x3ffffffdaa0) at pinit.c:66
#14 0x0000000000400c5c in main (argc=4, argv=0x3ffffffdc38) at ctxalloc.c:20

[shamisp]

Does it hang forever in nanosleep() ?

…

On Wed, Jun 14, 2017 at 3:20 AM, Yossi Itigin ***@***.***> wrote: Assigned #3697 <#3697> to @shamisp <https://github.com/shamisp>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3697 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACIe2L0EeuusZcT5vC-y9JeExQu5CW95ks5sD5fPgaJpZM4N5ijf> .

[yosefe]

Unlikely, this is a fixed 100ns interval"

static inline void _opal_lifo_release_cpu (void)
{
    /* NTH: there are many ways to cause the current thread to be suspended. This one
     * should work well in most cases. Another approach would be to use poll (NULL, 0, ) but
     * the interval will be forced to be in ms (instead of ns or us). Note that there
     * is a performance improvement for the lifo test when this call is made on detection
     * of contention but it may not translate into actually MPI or application performance
     * improvements. */
    static struct timespec interval = { .tv_sec = 0, .tv_nsec = 100 };
    nanosleep (&interval, NULL);
}

[jsquyres]

Can you provide more information about what exactly is hanging? I.e., this is clearly one stack trace of the hang, but it must be looping on something that never completes.

[shamisp]

@yosefe can I reproduce this with a single node ? Do I really need IB for this ? thanks

[bosilca]

You don't need IB for this, just force the load of hugepage mpool. Also, this seems to happen in a very peculiar place, during a lifo_pop operation, which in this particualr context use the conditional load/store of the ARM64. As our tests pass on ARM64, I don't think the issue is in the atomic itself, but instead in the way the rb tree is initialized or how the freelist is grown in this particular instance.

[shamisp]

is this a multi-thread run ? how many threads ?

[shamisp]

@yosefe , can you please replace

ompi/opal/class/opal_lifo.h

Line 226 in d7ebcca

opal_atomic_wmb ();

with
opal_atomic_mb()

[yosefe]

@shamisp it's looping in

ompi/opal/class/opal_lifo.h

Line 224 in d7ebcca

} while (!opal_atomic_sc_ptr (&lifo->opal_lifo_head.data.item, next));

, changing to atomic_mb() does not help.
it can be reproduced on 1 node.
disabling hugepage mpool moves the hange to another place - so it's something more fundamental:

(gdb) bt
#0  0x000003ffb7ccfa78 in nanosleep () from /usr/lib64/libpthread.so.0
#1  0x000003ffb7911e0c in _opal_lifo_release_cpu () at ../opal/class/opal_lifo.h:195
#2  0x000003ffb7911e40 in opal_lifo_pop_atomic (lifo=0x3ffb7a3b778 <mca_mpool_base_tree+64>) at ../opal/class/opal_lifo.h:210
#3  0x000003ffb7911fd0 in opal_free_list_get_st (flist=0x3ffb7a3b778 <mca_mpool_base_tree+64>) at ../opal/class/opal_free_list.h:213
#4  0x000003ffb7911ff4 in opal_free_list_get (flist=0x3ffb7a3b778 <mca_mpool_base_tree+64>) at ../opal/class/opal_free_list.h:225
#5  0x000003ffb7912264 in opal_rb_tree_init (tree=0x3ffb7a3b738 <mca_mpool_base_tree>, comp=0x3ffb79bdba0 <mca_mpool_base_tree_node_compare>) at class/opal_rb_tree.c:86
#6  0x000003ffb79bde28 in mca_mpool_base_tree_init () at base/mpool_base_tree.c:87
#7  0x000003ffb79bd378 in mca_mpool_base_open (flags=MCA_BASE_OPEN_DEFAULT) at base/mpool_base_frame.c:102
#8  0x000003ffb7957d1c in mca_base_framework_open (framework=0x3ffb7a34788 <opal_mpool_base_framework>, flags=MCA_BASE_OPEN_DEFAULT) at mca_base_framework.c:174
#9  0x000003ffb7d59bc0 in ompi_mpi_init (argc=4, argv=0x3ffffffe5c8, requested=0, provided=0x3ffffffe3bc) at runtime/ompi_mpi_init.c:589
#10 0x000003ffb7d990e8 in PMPI_Init (argc=0x3ffffffe43c, argv=0x3ffffffe430) at pinit.c:66
#11 0x0000000000400c5c in main (argc=4, argv=0x3ffffffe5c8) at ctxalloc.c:20

[yosefe]

command line on single node:

mpirun -np 1 -mca btl self --bind-to core --display-map -mca pml ob1 ./ctxalloc 2 1500 100

[kawashima-fj]

I can reproduce the hang on my ARM64 machine.

Open MPI: openmpi-v2.x-201706150321-b562082 (nightly snapshot tarball)
CPU: Cavium Thunder X (ARMv8 (64-bit))
OS: CentOS Linux release 7.2.1603 (AltArch)
Compiler: GCC 4.8.5 (in CentOS)
configure option: --enable-debug or --disable-debug
compiler option: -O0

I cannot reproduce the hang with -O1 or -O2 (either --enable-debug or --disable-debug).

Test programs in the Open MPI source tree also hang.

test/class/opal_lifo

(gdb) bt
#0  0x0000ffffb7d009a8 in __nanosleep_nocancel () from /lib64/libpthread.so.0
#1  0x0000000000401384 in _opal_lifo_release_cpu ()
    at ../../../../opal/class/opal_lifo.h:195
#2  0x00000000004013b8 in opal_lifo_pop_atomic (lifo=0xffffffffed80)
    at ../../../../opal/class/opal_lifo.h:210
#3  0x00000000004014f4 in thread_test (arg=0xffffffffed80)
    at ../../../../test/class/opal_lifo.c:50
#4  0x0000000000401ab8 in main (argc=1, argv=0xfffffffff018)
    at ../../../../test/class/opal_lifo.c:147

test/class/opal_fifo

(gdb) bt
#0  opal_fifo_pop_atomic (fifo=0xffffffffed68)
    at ../../../../opal/class/opal_fifo.h:238
#1  0x00000000004015d4 in thread_test (arg=0xffffffffed68)
    at ../../../../test/class/opal_fifo.c:51
#2  0x0000000000401e3c in main (argc=1, argv=0xfffffffff018)
    at ../../../../test/class/opal_fifo.c:184

test/class/ompi_rb_tree

(gdb) bt
#0  0x0000ffffb7a589a8 in __nanosleep_nocancel () from /lib64/libpthread.so.0
#1  0x0000ffffb7bfb338 in _opal_lifo_release_cpu ()
    at ../../../opal/class/opal_lifo.h:195
#2  0x0000ffffb7bfb36c in opal_lifo_pop_atomic (lifo=0xffffffffec90)
    at ../../../opal/class/opal_lifo.h:210
#3  0x0000ffffb7bfb4fc in opal_free_list_get_st (flist=0xffffffffec90)
    at ../../../opal/class/opal_free_list.h:213
#4  0x0000ffffb7bfb520 in opal_free_list_get (flist=0xffffffffec90)
    at ../../../opal/class/opal_free_list.h:225
#5  0x0000ffffb7bfb790 in opal_rb_tree_init (tree=0xffffffffec50, comp=0x4016c4 <comp_fn>)
    at ../../../opal/class/opal_rb_tree.c:86
#6  0x0000000000401c04 in test1 ()
    at ../../../../test/class/ompi_rb_tree.c:145
#7  0x0000000000402850 in main (argc=1, argv=0xfffffffff018)
    at ../../../../test/class/ompi_rb_tree.c:408

In the opal_lifo case, it is looping in do-while loop of the opal_lifo_pop_atomic function (OPAL_HAVE_ATOMIC_LLSC_PTR == 1 case).

In the opal_fifo case, it is looping in do-while loop of the opal_fifo_pop_atomic function (OPAL_HAVE_ATOMIC_LLSC_PTR == 1 case). Not continue case.

[hjelmn]

Does it run with --disable-builtin-atomics?

[kawashima-fj]

@hjelmn Builtin atomics are disabled by default in v2.x branch. I enabled BUILTIN_GCC (__atomic_*) by --enable-builtin-atomics. Then make check (which includes three test programs above) passed without hang. Both -O0 and -O2 were tested.

[shamisp]

@yosefe - what compiler version is used ?