Various benchmarks from OSU-Micro-Benchmarks/5.7.1-gompi-2021a-CUDA-11.3.1 segfault when using CUDA buffers

[casparvl]

Issue

Since 2021a the support for CUDA aware MPI communication has changed: rather than using a fosscuda toolchain, we now use a non-CUDA aware MPI, with a CUDA aware UCX (UCX-CUDA/1.10.0-GCCcore-10.3.0-CUDA-11.3.1). However, I'm seeing various (but not all) OSU tests fail with that setup with a segfault:

==== backtrace (tid:2888500) ====
 0 0x000000000002a160 ucs_debug_print_backtrace()  /tmp/jenkins/build/UCX/1.10.0/GCCcore-10.3.0/ucx-1.10.0/src/ucs/debug/debug.c:656
 1 0x0000000000012b20 .annobin_sigaction.c()  sigaction.c:0
 2 0x000000000016065c __memmove_avx_unaligned_erms()  :0
 3 0x0000000000052aeb non_overlap_copy_content_same_ddt()  opal_datatype_copy.c:0
 4 0x00000000000874b9 ompi_datatype_sndrcv()  ???:0
 5 0x00000000000e2d84 ompi_coll_base_alltoall_intra_pairwise()  ???:0
 6 0x000000000000646c ompi_coll_tuned_alltoall_intra_dec_fixed()  ???:0
 7 0x0000000000089a4f MPI_Alltoall()  ???:0
 8 0x0000000000402d40 main()  ???:0
 9 0x0000000000023493 __libc_start_main()  ???:0
10 0x000000000040318e _start()  ???:0
=================================
[gcn30:2888500] *** Process received signal ***
[gcn30:2888500] Signal: Segmentation fault (11)
[gcn30:2888500] Signal code:  (-6)
[gcn30:2888500] Failing at address: 0xb155002c1334
[gcn30:2888500] [ 0] /lib64/libpthread.so.0(+0x12b20)[0x1464e51a8b20]
[gcn30:2888500] [ 1] /lib64/libc.so.6(+0x16065c)[0x1464e4f3165c]
[gcn30:2888500] [ 2] /sw/arch/Centos8/EB_production/2021/software/OpenMPI/4.1.1-GCC-10.3.0/lib/libopen-pal.so.40(+0x52aeb)[0x1464e485daeb]
[gcn30:2888500] [ 3] /sw/arch/Centos8/EB_production/2021/software/OpenMPI/4.1.1-GCC-10.3.0/lib/libmpi.so.40(ompi_datatype_sndrcv+0x949)[0x1464e72454b9]
[gcn30:2888500] [ 4] /sw/arch/Centos8/EB_production/2021/software/OpenMPI/4.1.1-GCC-10.3.0/lib/libmpi.so.40(ompi_coll_base_alltoall_intra_pairwise+0x174)[0x1464e72a0d84]
[gcn30:2888500] [ 5] /sw/arch/Centos8/EB_production/2021/software/OpenMPI/4.1.1-GCC-10.3.0/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_alltoall_intra_dec_fixed+0x7c)[0x1464d494746c]
[gcn30:2888500] [ 6] /sw/arch/Centos8/EB_production/2021/software/OpenMPI/4.1.1-GCC-10.3.0/lib/libmpi.so.40(MPI_Alltoall+0x15f)[0x1464e7247a4f]
[gcn30:2888500] [ 7] osu_alltoall[0x402d40]
[gcn30:2888500] [ 8] /lib64/libc.so.6(__libc_start_main+0xf3)[0x1464e4df4493]
[gcn30:2888500] [ 9] osu_alltoall[0x40318e]
[gcn30:2888500] *** End of error message ***

Working/failing tests

I haven't run all OSU benchmarks, but a few that run without issues are:

mpirun -np 2 osu_latency -d cuda D D
mpirun -np 2 osu_bw -d cuda D D
mpirun -np 2 osu_bcast -d cuda D D

A few that produce the segfault are:

mpirun -np 2 osu_gather -d cuda D D
mpirun -np 2 osu_alltoall -d cuda D D

Note that for the osu_latency and osu_bw tests I get results that correspond with what can be expected from GPU direct RDMA, so they really seem to function properly.

Summary of discussion on EasyBuild Slack

This thread on OpenMPI seems to suggest that UCX has limited support for GPU operations (see here) and might only work for pt2pt, but not for collectives (see here). The relevant parts:

OK I would like to drag in @jsquyres for confirming this, because my impression from Jeff was that for this case Open MPI could still be CUDA-aware. @bureddy Are you saying collectives won't work but point-to-point could? Interesting...

Yes. it might change in the future when UCX handle all datatypes pack/unpack and collectives

The thread was from a while ago though, it is possible things have changed, but it would be unsure in which version (if anything changed).

If UCX indeed does not fully support operations on GPU buffers, then we might have to build OpenMPI with GPU support again (currently we build OpenMPI with UCX support, and then build UCX with CUDA support), i.e. build the smcuda BTL again. It's a bit odd, since it seemed OpenMPI wanted to move away from that (and towards using UCX). Plus, it reintroduces the original issue that we prefer not to have a fosscuda toolchain, because that causes duplication of a lot of modules (see the original discussion here). As @Micket suggested on the chat: if it proves to be needed, we could try to build the smcuda BTL seperately, as an add-on, like we do for UCX-CUDA.

Open questions:

• Can others confirm that the tests that fail for me also fail for them?
• Can we get confirmation from experts (e.g. on OpenMPI / UCX issue tracker) that indeed UCX still has this limited support?

[Micket]

How is this for e.g. fosscuda/2020b? Crashing by default?
Same crash if one specifies mpirun --mca pml ucx ... ? Does it work with mpirun --mca btl smcuda,....?

[casparvl]

How is this for e.g. fosscuda/2020b? Crashing by default?

We don't have that toolchain, but we did have 2020a. I just tested with that, and at least osu_latency and osu_alltoall work.

Same crash if one specifies mpirun --mca pml ucx ... ?

This does not change anything for me: results are the same both for my tests with 2021a and 2020a.

Does it work with mpirun --mca btl smcuda,....?

Not sure what else should be at the ... but: mpirun -np 2 --mca btl smcuda osu_alltoall -d cuda D D works for 2020a. It fails for 2021a because (as expected) it reports the requested component is not found.

[casparvl]

This OpenMPI FAQ seems to suggest MPI_alltoall should be supported by UCX.

[Micket]

This is actually becoming more worrying to me. I thought there was just UCX's cuda plugin, or OpenMPI's smcuda BTL. But if 2020a + --mca pml ucx is different from 2021a, then I don't know what it is..

[casparvl]

But if 2020a + --mca pml ucx is different from 2021a, then I don't know what it is..

yeah, this I also don't understand. Why would using UCX in 2020a make the CUDA stuff work, but it fails in 2021a?

I tried to build an OpenMPI with CUDA the 'traditional' way, so I now have a OpenMPI/4.1.1-GCC-10.3.0-CUDA-11.3.1. I figured, that should make the osu_alltoall test work again, just like it does in 2020a. But... same error!

So, I really need someone to reproduce my initial problem. We might be chasing something down that is specific to our system/installation somehow... So if anyone with GPU Direct RDMA capable hardware can install OSU-Micro-Benchmarks/5.7.1-gompi-2021a-CUDA-11.3.1 and try to run mpirun -np 2 --mca pml ucx osu_alltoall -d cuda D D, and let me know if they even see the same error, that would be great :)

[Micket]

Nothing stops openmpi from having an evil #ifdef CUDA inside it's core code outside of any plugins, but I really really hope that isn't the case. They would just totally screw things up for us. Also wouldn't explain why your "conventional" OpenMPI build still fails.

I have already reproduced the error. gather and alltoall crashes for me with 2021a + UCX-CUDA. The other 3 work.
I can't easily check the older toolchains.

I also played around a bit with additional flags

mpirun -np 2 --mca pml ucx -x UCX_TLS=rc,sm,cuda_copy,gdr_copy,cuda_ipc ./osu_latency D D

but the only thing I see is that "gdr_copy" which I suspect is just outdated documentation? It's uct-cuda plugin nowdays(?). I see the same warning about nonexistent gdr_copy on fosscuda/2020b where we aren't doing funny things.

[casparvl]

For me, this just works:

mpirun -np 2 --mca pml ucx -x UCX_TLS=rc,sm,cuda_copy,gdr_copy,cuda_ipc osu_latency D D
# OSU MPI-CUDA Latency Test v5.7.1
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
# Size          Latency (us)
0                       0.36
1                       1.49
2                       2.33
4                       2.33
8                       2.32
16                      1.50
32                      2.33
64                      2.49
128                     2.47
256                     2.75
512                     3.00
1024                    4.57
2048                    5.63
4096                    9.19
8192                   15.48
16384                   9.30
32768                   9.33
65536                   9.86
131072                 10.82
262144                 12.38
524288                 14.97
1048576                20.56
2097152                31.49
4194304                53.87

You mean you get warnings there? That might be related to the fact that gdr_copy also requires a kernel extension, maybe you guys don't have that running on the cluster? Anyway, it should be unrelated to the errors with osu_alltoall and friends...

[Micket]

Hm, well, all of this CUDA RDMA stuff requires kernel level support no?
I run Mellanox OFED 5.4 and have nvidia_peermem (the replacement for the deprecated nv_peer_mem)

[1642700936.729058] [alvis5-01:223304:0]    ucp_context.c:770  UCX  WARN  transport 'gdr_copy' is not available, please use one or more of: cma, cuda, cuda_copy, cuda_ipc, dc, dc_mlx5, dc_x, ib, mm, posix, rc, rc_mlx5, rc_v, rc_verbs, rc_x, self, shm, sm, sysv, tcp, ud, ud_mlx5, ud_v, ud_verbs, ud_x

[branfosj]

I did some testing and built some extra modules:

• OSU-Micro-Benchmarks/5.7-gompic-2020b - works
• OSU-Micro-Benchmarks/5.7.1-gompi-2021a-CUDA-11.3.1 - broken
• OSU-Micro-Benchmarks/5.8-gompi-2021a-CUDA-11.3.1 - broken
• OSU-Micro-Benchmarks/5.7.1-gompi-2021b-CUDA-11.4.1 - broken
• OSU-Micro-Benchmarks/5.8-gompi-2021a-CUDA-11.3.1-v2 - replacing UCX and UCX-CUDA with version 1.11.2 - broken
• OSU-Micro-Benchmarks/5.8-gompi-2021aa-CUDA-11.3.1 - replacing OpenMPI with one that has CUDA and UCX-CUDA 1.10.0 dependencies - works

[branfosj]

My conclusion from those is that we need OpenMPI to be built with CUDA as a dependency, to get us OpenMPI with cuda support and that the UCX CUDA is not sufficient here.

[Micket]

Those results match up with whether it has smcuda or not, which leaves the door open for a OMPI_MCA_mca_component_path + mca_btl_smcuda.so solution. However, it could also be something more intrusive, like
https://github.com/open-mpi/ompi/search?q=OPAL_CUDA_SUPPORT
that seems to infect a lot of code.

What I can't make sense of is;

I tried to build an OpenMPI with CUDA the 'traditional' way, so I now have a OpenMPI/4.1.1-GCC-10.3.0-CUDA-11.3.1. I figured, that should make the osu_alltoall test work again, just like it does in 2020a. But... same error!

which should have matched with OSU-Micro-Benchmarks/5.8-gompi-2021aa-CUDA-11.3.1 this error persist for @casparvl? Perhaps just some unrelated mistake was made?

[casparvl]

@Micket absolutely right. I just had another look, and I did something silly. We RPATH everything, and therefore, I should have recompiled the OSU-benchmarks. However, I forgot to do that, and simply loaded the CUDA-aware MPI. That, for obvious reasons, wasn't enough.

I'll recompile OSU and run another test, but I have no doubt it will confirm what @branfosj has found...

[casparvl]

The thing that still puzzles me is that this FAQ really seems to suggest a CUDA-aware UCX should be enough... https://www.open-mpi.org/faq/?category=runcuda#mpi-apis-cuda-ucx Once I've reran the OSU tests, I'll post on the OpenMPI issue page to ask for confirmation that the smcuda btl is really still needed.

[Micket]

If so, I think it would be interested to test this with

export OMPI_MCA_mca_component_path=additional/path/to/smcuda

to see if we can expand the paths to add, cf. @bartoldeman comments in
#12484

[casparvl]

It would actually create the possibility of re-adopting the fosscuda toolchain, but then it is foss + smcuda btl. That would be nice, because it would make foss a subtoolchain of fosscuda, and foss-based installations could be reused as dependencies for fosscuda, hence solving the duplication issue we had before...