MPI and RMPISNOW alterations for R 4.1.1

[heatherkellyucl]

IN: 04932896

1. In the modulefile for r/4.1.1-openblas/gnu-10.2.0, this line needs to be changed:

setenv           OMPI_MCA_btl tcp,sm,self 

to

setenv           OMPI_MCA_btl tcp,vader,self 

(No sm for OpenMPI 4). This may be able to be dropped entirely, because the mpirun wrapper for OpenMPI ought to be setting this correctly already.
This was a problem on Myriad, MPI within single node.

2. Check if RMPISNOW is giving this error for a user with no existing R profile and fix:

GERun: GErun command being run:
GERun:  mpirun RMPISNOW
Loading required package: utils
Error: cannot add binding of '.MPIrun' to the base environment
Execution halted

and see https://stackoverflow.com/questions/68198277/error-cannot-add-binding-of-first-to-the-base-environment for reason and fix syntax.
This was reported on Kathleen, 2-node job. Was fixed by copying RMPISNOW and RMPISNOW_profile to user's home dir and modifying based on that link.

https://cran.r-project.org/doc/manuals/r-devel/NEWS.html
R 4.1.0 had

The base environment and its namespace are now locked (so one can no longer add bindings to these or remove from these).

[heatherkellyucl]

Test jobs submitted with our RMPISNOW example to confirm failure.

[heatherkellyucl]

On Myriad with one node, I do get this error:

GERun:  mpirun RMPISNOW
Loading required package: utils
Error: cannot add binding of '.MPIrun' to the base environment

I do not get a btl error at this stage (from the comment in the module, that ought to only be a problem with single-node Rmpi on its own, which calls openmpi as a library and ignores mpirun - but still needs updating).

[balston]

I'm checking if R 4.0.2 has this problem using the same example files that we've used before for testing.

[balston]

R 4.0.2 (which uses Open MPI 3.1.5 and GNU 9.2.0) snow jobs on Myriad (12 cores) and Kathleen (80 cores/2 nodes) work without modifications.

[heatherkellyucl]

R 4.1.1

I get this error

As of version 3.0.0, the "sm" BTL is no longer available in Open MPI.

Efficient, high-speed same-node shared memory communication support in
Open MPI is available in the "vader" BTL.  To use the vader BTL, you
can re-run your job with:

    mpirun --mca btl vader,self,... your_mpi_application

If I run Rmpi only, with no mpirun.

[heatherkellyucl]

R 4.1.1

I'm getting this error when setting

export OMPI_MCA_btl='tcp,vader,self'
mpirun -np 1 R CMD BATCH test_rmpi.r test_rmpi_${NSLOTS}_${JOB_ID}.out

Open MPI detected an inbound MPI TCP connection request from a peer
that appears to be part of this MPI job (i.e., it identified itself as
part of this Open MPI job), but it is from an IP address that is
unexpected.  This is highly unusual.

The inbound connection has been dropped, and the peer should simply
try again with a different IP interface (i.e., the job should
hopefully be able to continue).

  Local host:          node-f00a-001
  Local PID:           291503
  Peer hostname:       node-f00a-001 ([[21106,1],0])
  Source IP of socket: 10.128.17.101
  Known IPs of peer:   
        10.34.4.101
        10.128.25.101
        192.168.122.1

Note this is on one node. It seems to be this issue open-mpi/ompi#5818 with tcp when there are multiple interfaces available so it isn't sending and receiving to itself on the same one...

Setting btl_tcp_if_include to a particular interface or subnet might work.

It gives the error and then hangs until it runs out of wallclock time.

[heatherkellyucl]

10.128.17.101 is ib0 on that node.

10.34.4.101 is eno1
10.128.25.101 is ib0:0
192.168.122.1 is virbr0

Maybe we want to tell it to use eno1 since IB is the storage network? (Again, this is local).

[heatherkellyucl]

I had better check what happens with 2 nodes on the Economics bit of Myriad.

Fails init:

It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_dpm_dyn_init() failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
[node-d97a-006:00462] *** An error occurred in MPI_Init
[node-d97a-006:00462] *** reported by process [2134441986,50]
[node-d97a-006:00462] *** on a NULL communicator
[node-d97a-006:00462] *** Unknown error
[node-d97a-006:00462] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[node-d97a-006:00462] ***    and potentially your MPI job)

[heatherkellyucl]

The 2-node Rmpi processes output these errors:

On head node

[1635856407.612162] [node-d97a-005:224742:0]         wireup.c:910  UCX  ERROR old: am_lane 0 wireup_lane 255 reachable_mds 0x1ab
[1635856407.791953] [node-d97a-005:224742:0]         wireup.c:918  UCX  ERROR old: lane[0]:  0:posix/memory.0 md[0] <proxy>  -> md[0]/posix    am am_bw#0
[1635856407.830051] [node-d97a-005:224742:0]         wireup.c:918  UCX  ERROR old: lane[1]: 12:cma/memory.0 md[7]            -> md[7]/cma      rma_bw#0
[1635856407.859858] [node-d97a-005:224742:0]         wireup.c:918  UCX  ERROR old: lane[2]:  8:dc_mlx5/mlx5_0:1.0 md[5]      -> md[5]/ib       rma_bw#1
[1635856407.894855] [node-d97a-005:224742:0]         wireup.c:910  UCX  ERROR new: am_lane 0 wireup_lane 2 reachable_mds 0x1ab
[1635856407.954865] [node-d97a-005:224742:0]         wireup.c:918  UCX  ERROR new: lane[0]:  0:posix/memory.0 md[0] <proxy>  -> md[0]/posix    am am_bw#0
[1635856407.987926] [node-d97a-005:224742:0]         wireup.c:918  UCX  ERROR new: lane[1]: 13:knem/memory.0 md[8]           -> md[8]/knem     rma_bw#0
[1635856408.023856] [node-d97a-005:224742:0]         wireup.c:918  UCX  ERROR new: lane[2]:  7:rc_mlx5/mlx5_0:1.0 md[5]      -> md[5]/ib       rma_bw#1 wireup  
[node-d97a-005:224742:0:225496]      wireup.c:1055 Fatal: endpoint reconfiguration not supported yet

On second node

[node-d97a-006.myriad.ucl.ac.uk:00596] pml_ucx.c:178  Error: Failed to receive UCX worker address: Not found (-13)
[node-d97a-006.myriad.ucl.ac.uk:00596] [[32569,2],57] ORTE_ERROR_LOG: Error in file dpm/dpm.c at line 493

[heatherkellyucl]

I think the 2-node problem is a similar device-choice issue.

For within-node Rmpi, setting these = happy and working.

export OMPI_MCA_btl='tcp,vader,self'
export OMPI_MCA_btl_tcp_if_include='eno1'

Does not fix it for 2-node.

[heatherkellyucl]

Oh, the R module also sets

setenv           OMPI_MCA_pml ob1 

which ought to be ucx here, I am pretty sure.

(ob1 is fine for single node, we set --mca mtl '^psm2' -mca pml ucx -mca btl ^usnic for multi-node OpenMPI 4 on Myriad).

[balston]

I think the ob1 setting was just copied from the R 4.0.2 module file.

[heatherkellyucl]

Still fails 2 node with ucx anyway.