Merging CUDA and non CUDA toolchains into one

[Micket]

As has been discussed multiple times in zoom and in chat, merging fosscuda+foss, and just using versionsuffixes for CUDA-variants of the (relatively few) easyconfigs that does have CUDA bindings.

One thing holding us back has been the CUDA support in MPI, which prevents us from moving into foss since it's part of the toolchain definitions itself, but now with UCX, it might be possible to get the best of both worlds (I'm going to just dismiss the legacy CUDA stuff in openmpi and just focus on UCX).

We want something that

1. works regardless of RPATH is used or not
2. can be opt int after foss(without CUDA) is already in place
3. supports all the RDMA goodies we have today in fosscuda.

Can it be done? Maybe; UCX has an environment variable for all the plugins it uses (ucx_info -f lists all variables):
We could introduce a a UCX-package (UCX-CUDA maybe?) which shadows non-CUDA UCX, and we start setting the environment variable;

UCX_MODULE_DIR='%(installdir)s/lib/ucx'

and, well, that should be it?

Example UCX-CUDA how i envision it:

easyblock = 'ConfigureMake'

name = 'UCX-CUDA'
version = '1.9.0'
local_cudaversion = '11.1.1'
versionsuffix = '-CUDA-%s' % local_cudaversion

homepage = 'http://www.openucx.org/'
description = """Unified Communication X
An open-source production grade communication framework for data centric
and high-performance applications
"""

toolchain = {'name': 'GCCcore', 'version': '10.2.0'}
toolchainopts = {'pic': True}

source_urls = ['https://github.com/openucx/ucx/releases/download/v%(version)s']
sources = ['%(namelower)s-%(version)s.tar.gz']
checksums = ['a7a2c8841dc0d5444088a4373dc9b9cc68dbffcd917c1eba92ca8ed8e5e635fb']

builddependencies = [
    ('binutils', '2.35'),
    ('Autotools', '20200321'),
    ('pkg-config', '0.29.2'),
]

osdependencies = [OS_PKG_IBVERBS_DEV]

dependencies = [
    ('UCX', version),
    ('numactl', '2.0.13'),
    ('CUDAcore', local_cudaversion, '', True),
    ('GDRCopy', '2.1', versionsuffix),
]

configure_cmd = "contrib/configure-release"
configopts = '--enable-optimizations --enable-cma --enable-mt --with-verbs '
configopts += '--without-java --disable-doxygen-doc '
configopts += '--with-cuda=$EBROOTCUDACORE --with-gdrcopy=$EBROOTGDRCOPY '

prebuildopts = 'unset CUDA_CFLAGS && unset LIBS && '
buildopts = 'V=1'

# Not a PATH since we want to replace it, not append to it
modextravars = {
    'UCX_MODULE_DIR': '%(installdir)s/lib/ucx',
}

sanity_check_paths = {
    'files': ['bin/ucx_info', 'bin/ucx_perftest', 'bin/ucx_read_profile'],
    'dirs': ['include', 'lib', 'share']
}

sanity_check_commands = ["ucx_info -d"]

moduleclass = 'lib'

We'd then just let a TensorFlow-2.4.1-foss-2020b-CUDA-11.1.1.eb have a dependency on UCX-CUDA (at least indirectly) which is probably the ugliesst thing with this approach.

This would remove the need for gcccuda, gompic, fosscuda, and they would all just use suffixes instead (and optionally depend on UCX-CUDA if they have some MPI parts).

@bartoldeman Is this at all close to the approach you envisioned?

[bartoldeman]

@Micket more-or-less. But I tested it out and ran into a couple of issues:

1. I found that UCX_MODULE_DIRdoes not override the default search path for modules, it merely adds a second alternative to $EBROOTUCX/lib/ucx (the first is derived at runtime from from the full path of $EBROOTUCX/lib/libucs.so.0. That could of course be patched in the UCX source code.
2. With RPATH, the single Open MPI is still linked to the main non-CUDA UCX libraries, so libucs.so.0 and co in UCX-CUDA would be there for nothing.
3. There is however an alternative which instead of UCX centric is Open MPI centric: via OMPI_MCA_mca_component_path you could add a directory with a mca_pml_ucx.so that links to a full UCX-CUDA. So you'd have this:

• gompi is a subtoolchain of gompic
• gompic depends on OpenMPI-CUDA and gompi components
• OpenMPI-CUDA only has the modified plugins and extends OMPI_MCA_mca_component_path, no libmpi etc.
• OpenMPI-CUDA depends on OpenMPI (which depends on plain UCX) and UCX-CUDA
  a bit complex though...

[bartoldeman]

Perhaps switching the two lines here:
https://github.com/openucx/ucx/blob/0477cce66118f6c9a65b8954878c0ee3a33b5035/src/ucs/sys/module.c#L121
makes a difference, will check...

[Micket]

Is 1. and 2. basically the same issue right? Were we to patch the order, then I suspect it would solve the RPATH issues? (or are
I was a bit lazy here and just reused the entire UCX build, but I'm really only after the new directory with UCX-modules; so one could do the same as you suggest for OpenMPI-CUDA.

Regarding 3. redefining gompi as a subtoolchain sounds a bit scary; can we even do that without wreaking havoc on all previous toolchains versions?

[bartoldeman]

I'm actually not even sure if non-CUDA UCX can be convinced to see the CUDA plugins.. will need to test some more.
The Open MPI approach via OMPI_MCA_mca_component_path works for sure (tested).

[akesandgren]

Unfortunately that won't be enough in the long run, when we get other things that use UCX directly and would then need the UCX-CUDA version...

[Micket]

We would definitely have a "UCX-CUDA" even with the OMPI_MCA_component_path approach, and I don't think there would be an issue depending on that directly if necessary with either approach

[bartoldeman]

Upon further investigation the UCX plugin architecture is not flexible enough for our purpose. The reason is that the list of plugins to load is set at configure time: for us this is:

#define uct_MODULES ":ib:rdmacm:cma:knem"

in config.h for non-CUDA and

#define uct_MODULES ":cuda:ib:rdmacm:cma:knem"

for CUDA. UCX parses this list to figure out which plugins to load.

Open MPI's plugin architecture is more flexible, though messing about with OMPI_MCA_mca_component_path would be a novelty in modules as far as I know, so tread carefully...

[bartoldeman]

Note about Intel MPI: this one goes via libfabric which is flexible enough (via FI_PROVIDER_PATH).
Note that we (with rpath) patch it a little in a hook already that modifies postinstallcmds adding this:

patchelf --set-rpath $EBROOTUCX/lib --force-rpath %(installdir)s/intel64/libfabric/lib/prov/libmlx-fi.so

we'd need to have two libmlx-fi.so copies, one linking to the non-CUDA UCX and one to the CUDA UCX (if the latter works properly at all)

[Micket]

So

0. expand OpenMPI to set OMPI_MCA_mca_component_path
1. UCX-CUDA (under GCCcore) like above (minus the pointless MODULE_DIR)
2. OpenMPI-CUDA (under gompi) that just contains one MCA library + OMPI_MCA_mca_component_path.
3. impi-CUDA (under iimpi?) that just contains libmlx-fi.so + FI_PROVIDER_PATH

[boegel]

Why does OpenMPI need to set OMPI_MCA_mca_component_path? (step 0)
Maybe I'm missing something...

[bartoldeman]

@boegel no, step 0 isn't necessary; only if you want OpenMPI-CUDA to prepend that would be slightly cleaner.
step 2 could set

OMPI_MCA_mca_base_component_path="$EBROOTOPENMPIMINUSCUDA/lib/openmpi:$EBROOTOPENMPI/lib/openmpi:$HOME/.openmpi/components"

or reading the source code at https://github.com/open-mpi/ompi/blob/92389c364df669822bb6d72de616c8ccf95b891c/opal/mca/base/mca_base_component_repository.c#L215 this is also possible:

OMPI_MCA_mca_base_component_path="$EBROOTOPENMPIMINUSCUDA/lib/openmpi:SYSTEM_DEFAULT:USER_DEFAULT"

[Micket]

So we have a working UCX + CUDA split working now. We should just decide on how workflow of how to put the easyconfigs on top of these now. Some loose suggestions have been floating around but nothing concrete. I can only recall hearing objections to all approaches so I doubt we can serve.
The suggestions I have myself, or have seen others make are:

1. The simplest approach we could take is to just depend on CUDAcore or UCX-CUDA if the software has MPI support. The cuda variants of software gets a versionsuffix = "-CUDA-%(cudaver)s". E.g. GROMACS-2021.3-foss-2021a.eb and GROMACS-2021.3-foss-2021a-CUDA-11.3.1.eb. For intel mpi I don't think it supports UCX-CUDA regardless so there they could just depend on CUDAcore. The test suite would ensure we don't mix multiple different versions. Upside: Simplicity. Also, it mirrors what we do at GCCcore with UCX-CUDA and NCCL. Downside: UCX-CUDA might be a bit obscure and people might not know to depend on it? (perhaps a test suite check would suffice to remedy that)

2. Add a trivial bundle "CUDA-11.3.1-GCCcore-10.3.0.eb" that depends on UCX-CUDA, and, maybe binutils. Then just depend on that and use a versionsuffix just like before. Upside: Perhaps simpler to remember that dependency name for those we don't realize what the purpose of UCX-CUDA is? Downside: Forces UCX-CUDA on everything even if it doesn't need UCX/MPI, especially for the intel side which can't even use UCX-CUDA (yet?).

3. Add a "CUDA" package that expands modulepath somewhere so it looks like more HMNS. Of course, the whole plan was to be able to reuse non CUDA foss dependencies here, so, it would somehow need to be below foss, so, I don't know what sort of mess we'd have to create to expand modulepath depending on what other modules you have loaded. We'd have to mess about with toolchain definitions and such. Upside: Adds a hierarchy for those who like that. Downside: I think it would create a fair bit of complexity

For option 2 and 3 we should probably also watch out for how the name CUDA would interact with the naming schemes and install locations.

[akesandgren]

As of Intel MPI 2019.6 (5 actually but there it is just a tech preview) it requires UCX (https://software.intel.com/content/www/us/en/develop/articles/improve-performance-and-stability-with-intel-mpi-library-on-infiniband.html), so from that point of view we should use UCX-CUDA just to make them identical.

I think option 1 is the cleanest one, and it also makes it clearer for users which module they want when they look for CUDA enabled stuff.

I don't really like the -CUDA-x.y versionsuffix myself but I'd still go for it.

For option 2, shouldn't that be one CUDA-x.y-GCCcore-x.y which depend on CUDAcore and one CUDA-x.y-gompi-z that depends on UCX-CUDA? I.e. depending on at what level the toolchain is it pulls in the required CUDA(core|UCX)?