topic suggestions for talk on OpenMPI and related libraries (UCX, PMIx, ...)

[boegel]

On Wed June 24th 2020 at 5pm CEST, a remote presentation by @jsquyres and @rhc54 will be held on Open MPi and related libraries including UCX, PMIx, PSM2, libfabric, verbs, etc., as well as the integration with other things like Slurm, CUDA, etc.

Details on attending the presentation (via WebEx) live will be shared via the EasyBuild mailing list & Slack channel closer to the date. We plan to record the presentation in full, and upload it to the EasyBuild YouTube channel shortly afterwards.

If you have any suggestions for specific topics you would like to see covered, please share them here!

[victorusu]

@boegel, it would be great if @rhc54 could mention the ucx support for aries network and/or Cray machines. Something in the lines of…

1. the status UCX the support for aries
2. if this part of the code is maintained
3. if they ever have performance numbers on the Crays

I would also be great if @jsquyres could mention OpenMPI and CUDA_MPS and well as OpenMPI experiences on Aries and/or Cray machines.

@boegel, what do you think?

[rhc54]

Urrr...I don't know anything about UCX or Aries, I'm afraid

…

On May 27, 2020, at 10:55 AM, victorusu ***@***.***> wrote: @boegel <https://github.com/boegel>, it would be great if @rhc54 <https://github.com/rhc54> could mention the ucx support for aries network and/or Cray machines. Something in the lines of… the status UCX the support for aries if this part of the code is maintained if they ever have performance numbers on the Crays I would also be great if @jsquyres <https://github.com/jsquyres> could mention OpenMPI and CUDA_MPS and well as OpenMPI experiences on Aries and/or Cray machines. @boegel <https://github.com/boegel>, what do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#630 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAST5YAGRERAOTMXBFW4B6DRTVHXPANCNFSM4NMJMLRQ>.

[boegel]

We should avoid making the scope of the talk too wide, so it's perfectly fine if you consider some things out of scope or outside your expertise @rhc54 and @jsquyres!

[jsquyres]

Neither @rhc54 nor myself are part of UCX/Mellanox; I think we'd be pretty hesitant to say anything definitive about it.

We were more thinking of providing an overview of the alphabet soup that is the ecosystem that is needed to build+install+run Open MPI, what the different parts are/what they are for, and explain what the different ecosystem support parts are (e.g., dependencies and libraries and the like). We weren't planning to spend a lot of time discussing any individual component, such as a specific networking library or platform.

The focus will likely be on what system/network admins need to know, such that the main parts of Open MPI, the main parts of PMIx, and the main parts of (the upcoming) PRRTE. Each of these three things, in turn, touches lots of other packages, so we'll touch on some of the ecosystems as well.

[stdweird]

@jsquyres one interesting thing might be recommendations and/or requirements when using libraries that are present in the OS and (ofcourse) available via EB.
eg slurm installed on the OS might require PMI and UCX libraries being present on the OS as dependencies, typically not the latest and greatest, but they will be present on the system.
and then EB build openmpi might have different versions of said libraries, so this might be tricky (or pointless?) to try to get right.
it's definitely no fun when e.g. certain UCX variables meant for the mpi application are being picked up by slurm/pmi/ucx and make it do stupid things.

[boegel]

Neither @rhc54 nor myself are part of UCX/Mellanox; I think we'd be pretty hesitant to say anything definitive about it.

We were more thinking of providing an overview of the alphabet soup that is the ecosystem that is needed to build+install+run Open MPI, what the different parts are/what they are for, and explain what the different ecosystem support parts are (e.g., dependencies and libraries and the like). We weren't planning to spend a lot of time discussing any individual component, such as a specific networking library or platform.

The focus will likely be on what system/network admins need to know, such that the main parts of Open MPI, the main parts of PMIx, and the main parts of (the upcoming) PRRTE. Each of these three things, in turn, touches lots of other packages, so we'll touch on some of the ecosystems as well.

@jsquyres That sounds really good to me!

Of course you won't be able to get in the nitty gritty details of all possible libraries, but having a good overview of what's important where would already be very valuable to many. Alphabet soup is spot on, especially with PRRTE coming into play (which is totally new to me).

I'm sure there will be many questions popping up during the Q&A after the talk as well, if those are too specific to answer in a reasonable amount of time, hopefully you'll be able to provide references for those aspects.

[bartoldeman]

I'm also curious about where libfabric is heading.

On an OmniPath cluster OpenMPI should use the cm pml but then can use either the psm2 mtl (which is what we're doing on the Cedar cluster by default), or else use the ofi mtl to use libfabric. However if CUDA support comes in only the psm2 mtl was usable, however now that libfabric supports CUDA too so is it now recommended to always use ofi?

The other funky slightly off-topic story here is that libfabric used to be able to wrap around UCX via the mlx provider which was dropped in upstream libfabric but this was then kept in the libfabric fork that Intel ships with Intel MPI,

[Falkor]

Excellent idea! Topic suggestions of interest on our side :

Slurm PMI[x] bindings and OpenMPI

I understand this was planned yet we would be very interested to see how the slurm bindings (and internal PMI[2] vs. external PMIx bindings) are handled within OpenMPI, ideally illustrated with the recommended diffs to apply on top of the default easyconfigs like easyconfigs/o/OpenMPI/OpenMPI-3.1.4-GCC-8.3.0.eb (here are the ones we apply.
We have also troubles with static vs shared library builds, it would be also interesting to get some info (and recommendations) on this topic.

Python PMI bindings

It may be out of scope for this talk (kindly advise) but a very annoying behaviour affects us for several parallel Python software depending indirectly on OpenMPI (or more precisely the Slurm PMI bindings), and we would appreciate to get some expert insights on this issue happening at the sanity check steps (i.e. upon invoking python -c 'import <name>' as defined by the [default extension filter for Python packages](https://easybuild.readthedocs.io/en/latest/Removed-functionality.html?highlight=exts_filter#extension-filter-template) i.e., exts_filter = ("python -c 'import %(ext_name)s'", ""))`. This raises typically the following error:

<prefix>/software/Python/3.7.4-GCCcore-8.3.0/bin/python -c "import <name>"" failed; output:
--------------------------------------------------------------------------
PMI2_Init failed to intialize.  Return code: 14

To mitigate this error, we have to customize exts_filter as follows to work with SLURM and PMI:

exts_filter = ("srun -n1 python -c 'import %(ext_name)s'", '')

And this is of course not sustainable as this induces custom easyconfigs which are time consuming and make us deviate from streamline easyconfigs. We would prefer to avoid that as this seems useless if we find the correct way to build the root Python easyconfigs with the appropriate bindings.

[boegel]

@Falkor Is that last question related to h5py (or things that depend on it)? See easybuilders/easybuild-easyconfigs#10687

[rhc54]

@Falkor I will briefly address your first question during the presentation, but let me pass along a couple of things in advance as the schedule for the meeting is getting quite long. First, please be aware that OMPI will no longer be supporting PMI-1 or PMI-2 beginning with the v5.0 release - only PMIx will be supported. Thus, if you want to launch OMPI v5+ jobs using srun, you will need to build Slurm with PMIx support. Since I'll be talking mostly about v5 and above, I won't spend much time on how OMPI used to deal with the PMI-1/2 abstraction.

I'm not sure I can address your second question as that seems more of a Slurm PMI2 issue. However, I can offer that unlike the earlier PMI libraries, PMIx offers native Python bindings for all its APIs beginning with the upcoming v4 release. Argonne and Intel combined to create them as PMIx is intended to be used at the application level and not just inside the MPI libraries. This might help alleviate the problems you are seeing, though I understand it would require some updating of the applications using the earlier PMIs.

[Falkor]

@boegel I saw the h5py issue, ours was happening upon building Theano-1.0.4-foss-2019b-Python-3.7.4. Investigations in progress, I'll eventually open a separate ticket.

@rhc54 Thx for your feedback. I assume you will still review briefly PMI1/2/X and that will be quite nice ;).
Otherwise, I would then have one more topic to suggest if not yet considered: how do you suggest to validate the full stack (OMPI / PMIx / UCX / etc.) in a systematic and automatic way ? "Generic" Frameworks like buildtest or ReFrame can help on some sample code, but may be you can point to smaller testing suites/projects dedicated to this validation on a larger set of code ? (ideally with Slurm launchers ;))

[boegel]

Closing this, as the series of EasyBuild Tech Talks on Open MPI was done last year; see https://github.com/easybuilders/easybuild/wiki/EasyBuild-tech-talks-I:-Open-MPI