[openmpi-4.1.2] No components were able to be opened in the pml framework

[Honggang-LI]

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

v4.1.2 (openmpi-4.1.2-1.el8)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

+ ./configure --prefix=/usr/lib64/openmpi --mandir=/usr/share/man/openmpi-x86_64 --includedir=/usr/include/openmpi-x86_64 --sysconfdir=/etc/openmpi-x86_64 --disable-silent-rules --enable-builtin-atomics --enable-mpi-cxx --enable-mpi-java --enable-mpi1-compatibility --with-sge --with-valgrind --enable-memchecker --with-hwloc=/usr --with-libevent=external --with-pmix=external

Please describe the system on which you are running

• Operating system/version:

rhel-8.6

• Computer hardware:

x64

• Network type:

RDMA (IB/IWARP/ROCE/OPA)

---

Details of the problem

Please describe, in detail, the problem that you are having, including the behavior you expect to see, the actual behavior that you are seeing, steps to reproduce the problem, etc. It is most helpful if you can attach a small program that a developer can use to reproduce your problem.

shell$ + [21-12-16 09:20:42] timeout 3m /usr/lib64/openmpi/bin/mpirun --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include mlx4_0:1 -mca mtl '^psm2,psm,ofi' -mca btl '^openib' -mca btl_openib_allow_ib 1 --mca mtl_openib_verbose 100 --mca btl_base_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES=mlx4_ib0 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 -hostfile /root/hfile_one_core -np 2 /usr/lib64/openmpi/bin/mpitests-IMB-MPI1 PingPong
[rdma-virt-00:79988] mca: base: components_register: registering framework btl components
[rdma-virt-00:79988] mca: base: components_register: found loaded component ofi
[rdma-virt-00:79988] mca: base: components_register: component ofi register function successful
[rdma-virt-00:79988] mca: base: components_register: found loaded component self
[rdma-virt-00:79988] mca: base: components_register: component self register function successful
[rdma-virt-00:79988] mca: base: components_register: found loaded component sm
[rdma-virt-00:79988] mca: base: components_register: found loaded component tcp
[rdma-virt-00:79988] mca: base: components_register: component tcp register function successful
[rdma-virt-00:79988] mca: base: components_register: found loaded component usnic
[rdma-virt-00:79988] mca: base: components_register: component usnic register function successful
[rdma-virt-00:79988] mca: base: components_register: found loaded component vader
[rdma-virt-00:79988] mca: base: components_register: component vader register function successful
[rdma-virt-00:79988] mca: base: components_open: opening btl components
[rdma-virt-00:79988] mca: base: components_open: found loaded component ofi
[rdma-virt-00:79988] mca: base: components_open: component ofi open function successful
[rdma-virt-00:79988] mca: base: components_open: found loaded component self
[rdma-virt-00:79988] mca: base: components_open: component self open function successful
[rdma-virt-00:79988] mca: base: components_open: found loaded component tcp
[rdma-virt-00:79988] mca: base: components_open: component tcp open function successful
[rdma-virt-00:79988] mca: base: components_open: found loaded component usnic
[rdma-virt-00:79988] mca: base: components_open: component usnic open function successful
[rdma-virt-00:79988] mca: base: components_open: found loaded component vader
[rdma-virt-00:79988] mca: base: components_open: component vader open function successful
[rdma-virt-00:79988] select: initializing btl component ofi
[rdma-virt-01:80287] mca: base: components_register: registering framework btl components
[rdma-virt-01:80287] mca: base: components_register: found loaded component ofi
[rdma-virt-01:80287] mca: base: components_register: component ofi register function successful
[rdma-virt-01:80287] mca: base: components_register: found loaded component self
[rdma-virt-01:80287] mca: base: components_register: component self register function successful
[rdma-virt-01:80287] mca: base: components_register: found loaded component sm
[rdma-virt-01:80287] mca: base: components_register: found loaded component tcp
[rdma-virt-01:80287] mca: base: components_register: component tcp register function successful
[rdma-virt-01:80287] mca: base: components_register: found loaded component usnic
[rdma-virt-01:80287] mca: base: components_register: component usnic register function successful
[rdma-virt-01:80287] mca: base: components_register: found loaded component vader
[rdma-virt-01:80287] mca: base: components_register: component vader register function successful
[rdma-virt-01:80287] mca: base: components_open: opening btl components
[rdma-virt-01:80287] mca: base: components_open: found loaded component ofi
[rdma-virt-01:80287] mca: base: components_open: component ofi open function successful
[rdma-virt-01:80287] mca: base: components_open: found loaded component self
[rdma-virt-01:80287] mca: base: components_open: component self open function successful
[rdma-virt-01:80287] mca: base: components_open: found loaded component tcp
[rdma-virt-01:80287] mca: base: components_open: component tcp open function successful
[rdma-virt-01:80287] mca: base: components_open: found loaded component usnic
[rdma-virt-01:80287] mca: base: components_open: component usnic open function successful
[rdma-virt-01:80287] mca: base: components_open: found loaded component vader
[rdma-virt-01:80287] mca: base: components_open: component vader open function successful
[rdma-virt-01:80287] select: initializing btl component ofi
[rdma-virt-00:79988] select: init of component ofi returned success
[rdma-virt-00:79988] select: initializing btl component self
[rdma-virt-00:79988] select: init of component self returned success
[rdma-virt-00:79988] select: initializing btl component tcp
[rdma-virt-00:79988] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[rdma-virt-00:79988] btl: tcp: Found match: 127.0.0.1 (lo)
[rdma-virt-00:79988] btl:tcp: Attempting to bind to AF_INET port 1024
[rdma-virt-00:79988] btl:tcp: Successfully bound to AF_INET port 1024
[rdma-virt-00:79988] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[rdma-virt-00:79988] btl:tcp: examining interface mlx4_roce
[rdma-virt-00:79988] btl:tcp: using ipv6 interface mlx4_roce
[rdma-virt-00:79988] btl:tcp: examining interface mlx4_ib0
[rdma-virt-00:79988] btl:tcp: using ipv6 interface mlx4_ib0
[rdma-virt-00:79988] btl:tcp: examining interface mlx4_ib0.8004
[rdma-virt-00:79988] btl:tcp: using ipv6 interface mlx4_ib0.8004
[rdma-virt-00:79988] btl:tcp: examining interface mlx4_ib0.8002
[rdma-virt-00:79988] btl:tcp: using ipv6 interface mlx4_ib0.8002
[rdma-virt-00:79988] btl:tcp: examining interface mlx4_roce.43
[rdma-virt-00:79988] btl:tcp: using ipv6 interface mlx4_roce.43
[rdma-virt-00:79988] btl:tcp: examining interface mlx4_roce.45
[rdma-virt-00:79988] btl:tcp: using ipv6 interface mlx4_roce.45
[rdma-virt-00:79988] btl:tcp: examining interface lab-bridge0
[rdma-virt-00:79988] btl:tcp: using ipv6 interface lab-bridge0
[rdma-virt-00:79988] select: init of component tcp returned success
[rdma-virt-00:79988] select: initializing btl component usnic
[rdma-virt-00:79988] btl:usnic: disqualifiying myself due to fi_getinfo(3) failure: No data available (-61)
[rdma-virt-00:79988] select: init of component usnic returned failure
[rdma-virt-00:79988] mca: base: close: component usnic closed
[rdma-virt-00:79988] mca: base: close: unloading component usnic
[rdma-virt-00:79988] select: initializing btl component vader
[rdma-virt-00:79988] select: init of component vader returned failure
[rdma-virt-00:79988] mca: base: close: component vader closed
[rdma-virt-00:79988] mca: base: close: unloading component vader
[rdma-virt-01:80287] select: init of component ofi returned success
[rdma-virt-01:80287] select: initializing btl component self
[rdma-virt-01:80287] select: init of component self returned success
[rdma-virt-01:80287] select: initializing btl component tcp
[rdma-virt-01:80287] btl: tcp: Searching for exclude address+prefix: 127.0.0.1 / 8
[rdma-virt-01:80287] btl: tcp: Found match: 127.0.0.1 (lo)
[rdma-virt-01:80287] btl:tcp: Attempting to bind to AF_INET port 1024
[rdma-virt-01:80287] btl:tcp: Successfully bound to AF_INET port 1024
[rdma-virt-01:80287] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_roce
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_roce
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_ib0
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_ib0
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_ib1
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_ib1
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_ib0.8004
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_ib0.8004
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_ib0.8002
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_ib0.8002
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_ib1.8003
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_ib1.8003
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_ib1.8005
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_ib1.8005
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_roce.43
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_roce.43
[rdma-virt-01:80287] btl:tcp: examining interface mlx4_roce.45
[rdma-virt-01:80287] btl:tcp: using ipv6 interface mlx4_roce.45
[rdma-virt-01:80287] btl:tcp: examining interface lab-bridge0
[rdma-virt-01:80287] btl:tcp: using ipv6 interface lab-bridge0
[rdma-virt-01:80287] select: init of component tcp returned success
[rdma-virt-01:80287] select: initializing btl component usnic
[rdma-virt-01:80287] btl:usnic: disqualifiying myself due to fi_getinfo(3) failure: No data available (-61)
[rdma-virt-01:80287] select: init of component usnic returned failure
[rdma-virt-01:80287] mca: base: close: component usnic closed
[rdma-virt-01:80287] mca: base: close: unloading component usnic
[rdma-virt-01:80287] select: initializing btl component vader
[rdma-virt-01:80287] select: init of component vader returned failure
[rdma-virt-01:80287] mca: base: close: component vader closed
[rdma-virt-01:80287] mca: base: close: unloading component vader
[rdma-virt-00:79988] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-virt-01:80287] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-virt-00:79988] common_ucx.c:304 posix/memory: did not match transport list
[rdma-virt-00:79988] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-virt-00:79988] common_ucx.c:304 self/memory0: did not match transport list
[rdma-virt-00:79988] common_ucx.c:304 tcp/mlx4_ib0: did not match transport list
[rdma-virt-00:79988] common_ucx.c:311 support level is none
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      rdma-virt-00
  Framework: pml
--------------------------------------------------------------------------
[rdma-virt-00:79988] PML ucx cannot be selected
[rdma-virt-01:80287] common_ucx.c:304 posix/memory: did not match transport list
[rdma-virt-01:80287] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-virt-01:80287] common_ucx.c:304 self/memory0: did not match transport list
[rdma-virt-01:80287] common_ucx.c:304 tcp/mlx4_ib0: did not match transport list
[rdma-virt-01:80287] common_ucx.c:311 support level is none
[rdma-virt-01:80282] 1 more process has sent help message help-mca-base.txt / find-available:none found
[rdma-virt-01:80282] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

v4.1.1 also has this issue. We revert c36d7459b6331c4da825cad5a64326e7c1a272aa for v4.1.1 to avoid this issue.

[yosefe]

@Honggang-LI seems like an issue with RDMA transports support in the UCX version on this system - since rc/mlx4_0:1 is not in the list of checked transports in common_ucx.c:304
Can you pls provide the output of:

1. rpm -qi ucx-ib - assuming UCX is installed from RPM
2. env UCX_MODULE_LOG_LEVEL=warn ucx_info -vd
3. ucx_info -b

[ggouaillardet]

FWIW, btl components are used when pml/ob1 is used, and mtl components are used when pml/cm is used. Since you force both pml/ucx and osc/ucx you do not need the mtl and btl* options.

[Honggang-LI]

3. ucx_info -b

[root@rdma-qe-14 ~]$ rpm -q ucx-ib
ucx-ib-1.11.2-2.el9.x86_64

[root@rdma-qe-14 ~]$ env UCX_MODULE_LOG_LEVEL=warn ucx_info -vd
[1641472459.969228] [rdma-qe-14:2648 :0]          module.c:69   UCX  WARN  ucs library path: /lib64/libucs.so.0
[1641472459.969240] [rdma-qe-14:2648 :0]          module.c:251  UCX  WARN  loading modules for ucs
# UCT version=1.11.2 revision 0000000
# configured with: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --disable-params-check --without-java --enable-cma --without-cuda --without-gdrcopy --with-verbs --without-cm --without-knem --with-rdmacm --without-rocm --without-xpmem --without-fuse3 --without-ugni
[1641472459.969394] [rdma-qe-14:2648 :0]          module.c:251  UCX  WARN  loading modules for uct
[1641472459.970215] [rdma-qe-14:2648 :0]          module.c:180  UCX  WARN  loaded /usr/lib64/ucx/libuct_ib.so.0.0.0 [0x563108657b40]
[1641472459.970224] [rdma-qe-14:2648 :0]          module.c:185  UCX  WARN  not calling constructor 'ucs_module_global_init' in /lib64/ucx/libuct_ib.so.0
[1641472459.970451] [rdma-qe-14:2648 :0]          module.c:180  UCX  WARN  loaded /usr/lib64/ucx/libuct_rdmacm.so.0.0.0 [0x56310865acf0]
[1641472459.970459] [rdma-qe-14:2648 :0]          module.c:185  UCX  WARN  not calling constructor 'ucs_module_global_init' in /lib64/ucx/libuct_rdmacm.so.0
[1641472459.970525] [rdma-qe-14:2648 :0]          module.c:180  UCX  WARN  loaded /usr/lib64/ucx/libuct_cma.so.0.0.0 [0x56310865c010]
[1641472459.970531] [rdma-qe-14:2648 :0]          module.c:185  UCX  WARN  not calling constructor 'ucs_module_global_init' in /lib64/ucx/libuct_cma.so.0
#
# Memory domain: tcp
#     Component: tcp
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: tcp
#         Device: lom_1
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 113.16/ppn + 0.00 MB/sec
#              latency: 5776 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: hfi1_opa0
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11895.50/ppn + 0.00 MB/sec
#              latency: 5206 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 6 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#      Transport: tcp
#         Device: lo
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 11.91/ppn + 0.00 MB/sec
#              latency: 10960 nsec
#             overhead: 50000 nsec
#            put_zcopy: <= 18446744073709551590, up to 6 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 0
#             am_short: <= 8K
#             am_bcopy: <= 8K
#             am_zcopy: <= 64K, up to 6 iov
#   am_opt_zcopy_align: <= 1
#         am_align_mtu: <= 0
#            am header: <= 8037
#           connection: to ep, to iface
#      device priority: 1
#     device num paths: 1
#              max eps: 256
#       device address: 18 bytes
#        iface address: 2 bytes
#           ep address: 10 bytes
#       error handling: peer failure, ep_check, keepalive
#
#
# Connection manager: tcp
#      max_conn_priv: 2064 bytes
#
# Memory domain: self
#     Component: self
#             register: unlimited, cost: 0 nsec
#           remote key: 0 bytes
#
#      Transport: self
#         Device: memory0
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 6911.00 MB/sec
#              latency: 0 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 8K
#             am_bcopy: <= 8K
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 0 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: sysv
#     Component: sysv
#             allocate: unlimited
#           remote key: 12 bytes
#           rkey_ptr is supported
#
#      Transport: sysv
#         Device: memory
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
#
# Memory domain: posix
#     Component: posix
#             allocate: unlimited
#           remote key: 24 bytes
#           rkey_ptr is supported
#
#      Transport: posix
#         Device: memory
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 12179.00 MB/sec
#              latency: 80 nsec
#             overhead: 10 nsec
#            put_short: <= 4294967295
#            put_bcopy: unlimited
#            get_bcopy: unlimited
#             am_short: <= 100
#             am_bcopy: <= 8256
#               domain: cpu
#           atomic_add: 32, 64 bit
#           atomic_and: 32, 64 bit
#            atomic_or: 32, 64 bit
#           atomic_xor: 32, 64 bit
#          atomic_fadd: 32, 64 bit
#          atomic_fand: 32, 64 bit
#           atomic_for: 32, 64 bit
#          atomic_fxor: 32, 64 bit
#          atomic_swap: 32, 64 bit
#         atomic_cswap: 32, 64 bit
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 8 bytes
#       error handling: ep_check
#
[1641472459.986952] [rdma-qe-14:2648 :0]          module.c:251  UCX  WARN  loading modules for uct_ib
[1641472459.988017] [rdma-qe-14:2648 :0]          module.c:251  UCX  WARN  loading modules for ucm
#
# Memory domain: hfi1_0
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#
#      Transport: ud_verbs
#         Device: hfi1_0:1
#  System device: 0000:04:00.0 (0)
#
#      capabilities:
#            bandwidth: 11794.23/ppn + 0.00 MB/sec
#              latency: 630 nsec
#             overhead: 105 nsec
#             am_bcopy: <= 4088
#             am_zcopy: <= 4088, up to 4 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 3952
#           connection: to ep, to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 5 bytes
#        iface address: 3 bytes
#           ep address: 6 bytes
#       error handling: peer failure, ep_check
#
#
#      Transport: rc_verbs
#         Device: hfi1_0:1
#  System device: 0000:04:00.0 (0)
#
#      capabilities:
#            bandwidth: 11794.23/ppn + 0.00 MB/sec
#              latency: 600 + 1.000 * N nsec
#             overhead: 75 nsec
#            put_bcopy: <= 8256
#            put_zcopy: <= 2G, up to 4 iov
#  put_opt_zcopy_align: <= 512
#        put_align_mtu: <= 4K
#            get_bcopy: <= 8256
#            get_zcopy: 65..2G, up to 4 iov
#  get_opt_zcopy_align: <= 512
#        get_align_mtu: <= 4K
#             am_bcopy: <= 8255
#             am_zcopy: <= 8255, up to 3 iov
#   am_opt_zcopy_align: <= 512
#         am_align_mtu: <= 4K
#            am header: <= 127
#           connection: to ep
#      device priority: 0
#     device num paths: 1
#              max eps: 256
#       device address: 5 bytes
#           ep address: 4 bytes
#       error handling: peer failure, ep_check
#
#
# Connection manager: rdmacm
#      max_conn_priv: 54 bytes
#
# Memory domain: cma
#     Component: cma
#             register: unlimited, cost: 9 nsec
#
#      Transport: cma
#         Device: memory
#  System device: <unknown>
#
#      capabilities:
#            bandwidth: 0.00/ppn + 11145.00 MB/sec
#              latency: 80 nsec
#             overhead: 400 nsec
#            put_zcopy: unlimited, up to 16 iov
#  put_opt_zcopy_align: <= 1
#        put_align_mtu: <= 1
#            get_zcopy: unlimited, up to 16 iov
#  get_opt_zcopy_align: <= 1
#        get_align_mtu: <= 1
#           connection: to iface
#      device priority: 0
#     device num paths: 1
#              max eps: inf
#       device address: 8 bytes
#        iface address: 4 bytes
#       error handling: peer failure, ep_check
#

[root@rdma-qe-14 ~]$  ucx_info -b
#define UCX_CONFIG_H              
#define ENABLE_BUILTIN_MEMCPY     1
#define ENABLE_DEBUG_DATA         0
#define ENABLE_MT                 0
#define ENABLE_PARAMS_CHECK       0
#define HAVE_1_ARG_BFD_SECTION_SIZE 0
#define HAVE_ALLOCA               1
#define HAVE_ALLOCA_H             1
#define HAVE_ATTRIBUTE_NOOPTIMIZE 1
#define HAVE_CLEARENV             1
#define HAVE_CPU_SET_T            1
#define HAVE_DC_DV                1
#define HAVE_DECL_ASPRINTF        1
#define HAVE_DECL_BASENAME        1
#define HAVE_DECL_BFD_GET_SECTION_FLAGS 0
#define HAVE_DECL_BFD_GET_SECTION_VMA 0
#define HAVE_DECL_BFD_SECTION_FLAGS 0
#define HAVE_DECL_BFD_SECTION_VMA 0
#define HAVE_DECL_CPU_ISSET       1
#define HAVE_DECL_CPU_ZERO        1
#define HAVE_DECL_ETHTOOL_CMD_SPEED 1
#define HAVE_DECL_FMEMOPEN        1
#define HAVE_DECL_F_SETOWN_EX     1
#define HAVE_DECL_IBV_ACCESS_ON_DEMAND 1
#define HAVE_DECL_IBV_ACCESS_RELAXED_ORDERING 1
#define HAVE_DECL_IBV_ADVISE_MR   1
#define HAVE_DECL_IBV_ALLOC_DM    1
#define HAVE_DECL_IBV_ALLOC_TD    1
#define HAVE_DECL_IBV_CMD_MODIFY_QP 0
#define HAVE_DECL_IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN 1
#define HAVE_DECL_IBV_CREATE_QP_EX 1
#define HAVE_DECL_IBV_CREATE_SRQ  1
#define HAVE_DECL_IBV_CREATE_SRQ_EX 1
#define HAVE_DECL_IBV_EVENT_GID_CHANGE 1
#define HAVE_DECL_IBV_EVENT_TYPE_STR 1
#define HAVE_DECL_IBV_EXP_ACCESS_ALLOCATE_MR 0
#define HAVE_DECL_IBV_EXP_ACCESS_ON_DEMAND 0
#define HAVE_DECL_IBV_EXP_ALLOC_DM 0
#define HAVE_DECL_IBV_EXP_ATOMIC_HCA_REPLY_BE 0
#define HAVE_DECL_IBV_EXP_CQ_IGNORE_OVERRUN 0
#define HAVE_DECL_IBV_EXP_CQ_MODERATION 0
#define HAVE_DECL_IBV_EXP_CREATE_QP 0
#define HAVE_DECL_IBV_EXP_CREATE_SRQ 0
#define HAVE_DECL_IBV_EXP_DCT_OOO_RW_DATA_PLACEMENT 0
#define HAVE_DECL_IBV_EXP_DEVICE_ATTR_PCI_ATOMIC_CAPS 0
#define HAVE_DECL_IBV_EXP_DEVICE_ATTR_RESERVED_2 0
#define HAVE_DECL_IBV_EXP_DEVICE_DC_TRANSPORT 0
#define HAVE_DECL_IBV_EXP_DEVICE_MR_ALLOCATE 0
#define HAVE_DECL_IBV_EXP_MR_FIXED_BUFFER_SIZE 0
#define HAVE_DECL_IBV_EXP_MR_INDIRECT_KLMS 0
#define HAVE_DECL_IBV_EXP_ODP_SUPPORT_IMPLICIT 0
#define HAVE_DECL_IBV_EXP_POST_SEND 0
#define HAVE_DECL_IBV_EXP_PREFETCH_MR 0
#define HAVE_DECL_IBV_EXP_PREFETCH_WRITE_ACCESS 0
#define HAVE_DECL_IBV_EXP_QPT_DC_INI 0
#define HAVE_DECL_IBV_EXP_QP_CREATE_UMR 0
#define HAVE_DECL_IBV_EXP_QP_INIT_ATTR_ATOMICS_ARG 0
#define HAVE_DECL_IBV_EXP_QP_OOO_RW_DATA_PLACEMENT 0
#define HAVE_DECL_IBV_EXP_QUERY_DEVICE 0
#define HAVE_DECL_IBV_EXP_QUERY_GID_ATTR 0
#define HAVE_DECL_IBV_EXP_REG_MR  0
#define HAVE_DECL_IBV_EXP_SEND_EXT_ATOMIC_INLINE 0
#define HAVE_DECL_IBV_EXP_SETENV  0
#define HAVE_DECL_IBV_EXP_WR_EXT_MASKED_ATOMIC_CMP_AND_SWP 0
#define HAVE_DECL_IBV_EXP_WR_EXT_MASKED_ATOMIC_FETCH_AND_ADD 0
#define HAVE_DECL_IBV_EXP_WR_NOP  0
#define HAVE_DECL_IBV_GET_ASYNC_EVENT 1
#define HAVE_DECL_IBV_GET_DEVICE_NAME 1
#define HAVE_DECL_IBV_LINK_LAYER_ETHERNET 1
#define HAVE_DECL_IBV_LINK_LAYER_INFINIBAND 1
#define HAVE_DECL_IBV_ODP_SUPPORT_IMPLICIT 1
#define HAVE_DECL_IBV_QPF_GRH_REQUIRED 1
#define HAVE_DECL_IBV_QUERY_DEVICE_EX 1
#define HAVE_DECL_IBV_QUERY_GID   1
#define HAVE_DECL_IBV_WC_STATUS_STR 1
#define HAVE_DECL_INOTIFY_ADD_WATCH 1
#define HAVE_DECL_INOTIFY_INIT    1
#define HAVE_DECL_IN_ATTRIB       1
#define HAVE_DECL_IPPROTO_TCP     1
#define HAVE_DECL_MADV_FREE       1
#define HAVE_DECL_MADV_REMOVE     1
#define HAVE_DECL_MLX5DV_CQ_INIT_ATTR_MASK_CQE_SIZE 1
#define HAVE_DECL_MLX5DV_CREATE_QP 1
#define HAVE_DECL_MLX5DV_DCTYPE_DCT 1
#define HAVE_DECL_MLX5DV_DEVX_SUBSCRIBE_DEVX_EVENT 1
#define HAVE_DECL_MLX5DV_INIT_OBJ 1
#define HAVE_DECL_MLX5DV_IS_SUPPORTED 1
#define HAVE_DECL_MLX5DV_OBJ_AH   1
#define HAVE_DECL_MLX5DV_QP_CREATE_ALLOW_SCATTER_TO_CQE 1
#define HAVE_DECL_MLX5DV_UAR_ALLOC_TYPE_BF 1
#define HAVE_DECL_MLX5DV_UAR_ALLOC_TYPE_NC 1
#define HAVE_DECL_POSIX_MADV_DONTNEED 1
#define HAVE_DECL_PR_SET_PTRACER  1
#define HAVE_DECL_SOL_SOCKET      1
#define HAVE_DECL_SO_KEEPALIVE    1
#define HAVE_DECL_SPEED_UNKNOWN   1
#define HAVE_DECL_STRERROR_R      1
#define HAVE_DECL_SYS_BRK         1
#define HAVE_DECL_SYS_IPC         0
#define HAVE_DECL_SYS_MADVISE     1
#define HAVE_DECL_SYS_MMAP        1
#define HAVE_DECL_SYS_MREMAP      1
#define HAVE_DECL_SYS_MUNMAP      1
#define HAVE_DECL_SYS_SHMAT       1
#define HAVE_DECL_SYS_SHMDT       1
#define HAVE_DECL_TCP_KEEPCNT     1
#define HAVE_DECL_TCP_KEEPIDLE    1
#define HAVE_DECL_TCP_KEEPINTVL   1
#define HAVE_DECL___PPC_GET_TIMEBASE_FREQ 0
#define HAVE_DEVX                 1
#define HAVE_DLFCN_H              1
#define HAVE_HW_TIMER             1
#define HAVE_IB                   1
#define HAVE_IBV_DM               1
#define HAVE_IN6_ADDR_S6_ADDR32   1
#define HAVE_INFINIBAND_MLX5DV_H  1
#define HAVE_INFINIBAND_TM_TYPES_H 1
#define HAVE_INOTIFY              1
#define HAVE_INTTYPES_H           1
#define HAVE_IP_IP_DST            1
#define HAVE_LIBGEN_H             1
#define HAVE_LIBRT                1
#define HAVE_LINUX_FUTEX_H        1
#define HAVE_LINUX_IP_H           1
#define HAVE_LINUX_MMAN_H         1
#define HAVE_MALLOC_H             1
#define HAVE_MALLOC_TRIM          1
#define HAVE_MEMALIGN             1
#define HAVE_MEMORY_H             1
#define HAVE_MLX5_HW              1
#define HAVE_MLX5_HW_UD           1
#define HAVE_MREMAP               1
#define HAVE_NETINET_IP_H         1
#define HAVE_NET_ETHERNET_H       1
#define HAVE_NUMA                 1
#define HAVE_NUMAIF_H             1
#define HAVE_NUMA_H               1
#define HAVE_ODP                  1
#define HAVE_ODP_IMPLICIT         1
#define HAVE_POSIX_MEMALIGN       1
#define HAVE_PREFETCH             1
#define HAVE_SCHED_GETAFFINITY    1
#define HAVE_SCHED_SETAFFINITY    1
#define HAVE_SIGACTION_SA_RESTORER 1
#define HAVE_SIGEVENT_SIGEV_UN_TID 1
#define HAVE_SIGHANDLER_T         1
#define HAVE_STDINT_H             1
#define HAVE_STDLIB_H             1
#define HAVE_STRERROR_R           1
#define HAVE_STRINGS_H            1
#define HAVE_STRING_H             1
#define HAVE_STRUCT_BITMASK       1
#define HAVE_STRUCT_DL_PHDR_INFO  1
#define HAVE_STRUCT_IBV_DEVICE_ATTR_EX_PCI_ATOMIC_CAPS 1
#define HAVE_STRUCT_IBV_TM_CAPS_FLAGS 1
#define HAVE_STRUCT_MLX5DV_CQ_CQ_UAR 1
#define HAVE_SYS_EPOLL_H          1
#define HAVE_SYS_EVENTFD_H        1
#define HAVE_SYS_STAT_H           1
#define HAVE_SYS_TYPES_H          1
#define HAVE_SYS_UIO_H            1
#define HAVE_TL_DC                1
#define HAVE_TL_RC                1
#define HAVE_TL_UD                1
#define HAVE_UCM_PTMALLOC286      1
#define HAVE_UNISTD_H             1
#define HAVE___CLEAR_CACHE        1
#define HAVE___CURBRK             1
#define HAVE___SIGHANDLER_T       1
#define IBV_HW_TM                 1
#define LT_OBJDIR                 ".libs/"
#define NVALGRIND                 1
#define PACKAGE                   "ucx"
#define PACKAGE_BUGREPORT         ""
#define PACKAGE_NAME              "ucx"
#define PACKAGE_STRING            "ucx 1.11"
#define PACKAGE_TARNAME           "ucx"
#define PACKAGE_URL               ""
#define PACKAGE_VERSION           "1.11"
#define STDC_HEADERS              1
#define STRERROR_R_CHAR_P         1
#define UCM_BISTRO_HOOKS          1
#define UCS_MAX_LOG_LEVEL         UCS_LOG_LEVEL_DEBUG
#define UCT_TCP_EP_KEEPALIVE      1
#define UCT_UD_EP_DEBUG_HOOKS     0
#define UCX_CONFIGURE_FLAGS       "--build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --disable-params-check --without-java --enable-cma --without-cuda --without-gdrcopy --with-verbs --without-cm --without-knem --with-rdmacm --without-rocm --without-xpmem --without-fuse3 --without-ugni"
#define UCX_MODULE_SUBDIR         "ucx"
#define VERSION                   "1.11"
#define restrict                  __restrict
#define test_MODULES              ":module"
#define ucm_MODULES               ""
#define ucs_MODULES               ""
#define uct_MODULES               ":ib:rdmacm:cma"
#define uct_cuda_MODULES          ""
#define uct_ib_MODULES            ""
#define uct_rocm_MODULES          ""
#define ucx_perftest_MODULES      ""
[root@rdma-qe-14 ~]$ 

[root@rdma-qe-14 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include hfi1_0:1 -mca pml ucx -mca mtl psm2 -x PSM2_PKEY=0x8001 --mca pml_base_verbose 100 -mca btl '^openib' -mca osc ucx -x UCX_NET_DEVICES=hfi1_opa0 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 hostname
rdma-qe-14.rdma.lab.eng.rdu2.redhat.com
rdma-qe-15.rdma.lab.eng.rdu2.redhat.com
[root@rdma-qe-14 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include hfi1_0:1 -mca pml ucx -mca mtl psm2 -x PSM2_PKEY=0x8001 --mca pml_base_verbose 100 -mca btl '^openib' -mca osc ucx -x UCX_NET_DEVICES=hfi1_opa0 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 mpitests-IMB-MPI1 Reduce -time 1.5
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] mca: base: components_register: registering framework pml components
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] mca: base: components_register: found loaded component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] mca: base: components_register: component ucx register function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] mca: base: components_open: opening pml components
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] mca: base: components_open: found loaded component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] mca: base: components_open: component ucx open function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] select: initializing pml component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] common_ucx.c:303 tcp/hfi1_opa0: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] common_ucx.c:303 self/memory0: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] common_ucx.c:303 posix/memory: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] common_ucx.c:303 cma/memory: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] common_ucx.c:311 support level is none
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] select: init returned failure for component ucx
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      rdma-qe-15
  Framework: pml
--------------------------------------------------------------------------
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:14485] PML ucx cannot be selected
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] mca: base: components_register: registering framework pml components
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] mca: base: components_register: found loaded component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] mca: base: components_register: component ucx register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] mca: base: components_open: opening pml components
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] mca: base: components_open: found loaded component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] mca: base: components_open: component ucx open function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] select: initializing pml component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] common_ucx.c:303 tcp/hfi1_opa0: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] common_ucx.c:303 self/memory0: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] common_ucx.c:303 posix/memory: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] common_ucx.c:303 cma/memory: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] common_ucx.c:311 support level is none
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02681] select: init returned failure for component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02676] 1 more process has sent help message help-mca-base.txt / find-available:none found
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02676] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

[yosefe]

The devices on this setup are Intel HFI, and AFAIK the OpenMPI component for using them should be mtl/ofi and not pml/ucx

[Honggang-LI]

The devices on this setup are Intel HFI, and AFAIK the OpenMPI component for using them should be mtl/ofi and not pml/ucx

Without c36d745, those parameters work for me. What parameters are needed for different RDMA hardware (OPA/IB/IWARP/ROCE) with c36d745 ?

[yosefe]

Right, because the goal of that commit was to enable UCX on IB/RoCE.
Can you try all removing mtl/pml/btl parameters and let OpenMPI select the default component?
maybe @jsquyres can also comment what should be used for OPA/IWARP

[Honggang-LI]

Right, because the goal of that commit was to enable UCX on IB/RoCE. Can you try all removing mtl/pml/btl parameters and let OpenMPI select the default component?

After removed mtl/pml/btl parameters, it seems openmpi selects openib as default component.

However, openib is deprecated. See https://www.open-mpi.org/software/ompi/major-changes.php

Is it right to select openib as default component?

[Honggang-LI]

After removing mtl/pml/btl parameters for OPA.

[root@rdma-qe-14 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include hfi1_0:1   -x PSM2_PKEY=0x8001 --mca pml_base_verbose 100  -x UCX_NET_DEVICES=hfi1_opa0 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 -mca orte_base_help_aggregate 0  mpitests-IMB-MPI1 Reduce -time 1.5
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   rdma-qe-14
  Local device: hfi1_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   rdma-qe-15
  Local device: hfi1_0
--------------------------------------------------------------------------
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: registering framework pml components
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: found loaded component v
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: component v register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: found loaded component cm
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: component cm register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: found loaded component monitoring
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: component monitoring register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: found loaded component ob1
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: component ob1 register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: found loaded component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_register: component ucx register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: opening pml components
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: found loaded component v
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: component v open function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: found loaded component cm
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: registering framework pml components
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: found loaded component v
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: component v register function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: found loaded component cm
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: component cm register function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: found loaded component monitoring
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: component monitoring register function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: found loaded component ob1
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: component ob1 register function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: component cm open function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: found loaded component monitoring
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: component monitoring open function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: found loaded component ob1
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: component ob1 open function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: found loaded component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: found loaded component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_register: component ucx register function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: opening pml components
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: found loaded component v
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: component v open function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: found loaded component cm
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: component cm open function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: found loaded component monitoring
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: component monitoring open function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: found loaded component ob1
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: component ob1 open function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: found loaded component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: components_open: component ucx open function successful
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: components_open: component ucx open function successful
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: component v not in the include list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: initializing pml component cm
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: component v not in the include list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: initializing pml component cm
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: init returned priority 40
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: component monitoring not in the include list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: initializing pml component ob1
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: init returned priority 20
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: initializing pml component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] common_ucx.c:303 tcp/hfi1_opa0: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] common_ucx.c:303 self/memory0: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] common_ucx.c:303 posix/memory: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] common_ucx.c:303 cma/memory: did not match transport list
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] common_ucx.c:311 support level is none
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: init returned failure for component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] selected cm best priority 40
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: component cm selected
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] select: component ob1 not selected / finalized
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: component v closed
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: unloading component v
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: component monitoring closed
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: unloading component monitoring
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: component ob1 closed
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: unloading component ob1
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] pml_ucx.c:268 mca_pml_ucx_close
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: component ucx closed
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: unloading component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: init returned priority 40
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: component monitoring not in the include list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: initializing pml component ob1
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: init returned priority 20
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: initializing pml component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] common_ucx.c:303 tcp/hfi1_opa0: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] common_ucx.c:303 self/memory0: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] common_ucx.c:303 posix/memory: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] common_ucx.c:303 cma/memory: did not match transport list
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] common_ucx.c:311 support level is none
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: init returned failure for component ucx
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] selected cm best priority 40
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: component cm selected
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] select: component ob1 not selected / finalized
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: component v closed
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: unloading component v
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: component monitoring closed
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: unloading component monitoring
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: component ob1 closed
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: unloading component ob1
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] pml_ucx.c:268 mca_pml_ucx_close
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: component ucx closed
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: unloading component ucx
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] check:select: PML check not necessary on self
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] check:select: checking my pml cm against process [[58670,1],0] pml cm
#----------------------------------------------------------------
#    Intel(R) MPI Benchmarks 2021.3, MPI-1 part
#----------------------------------------------------------------
# Date                  : Thu Jan  6 09:30:09 2022
# Machine               : x86_64
# System                : Linux
# Release               : 5.14.0-39.el9.x86_64
# Version               : #1 SMP PREEMPT Fri Dec 24 00:07:58 EST 2021
# MPI Version           : 3.1
# MPI Thread Environment: 


# Calling sequence was: 

# mpitests-IMB-MPI1 Reduce -time 1.5 

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT 
# MPI_Op                         :   MPI_SUM  
# 
# 

# List of Benchmarks to run:

# Reduce

#----------------------------------------------------------------
# Benchmarking Reduce 
# #processes = 2 
#----------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
            0         1000         0.04         0.05         0.05
            4         1000         0.16         1.74         0.95
            8         1000         0.17         1.76         0.97
           16         1000         0.20         1.82         1.01
           32         1000         0.18         1.79         0.99
           64         1000         0.17         1.80         0.99
          128         1000         0.18         1.84         1.01
          256         1000         0.19         1.92         1.05
          512         1000         0.20         1.99         1.10
         1024         1000         0.22         2.12         1.17
         2048         1000         0.27         2.40         1.34
         4096         1000         0.42         2.93         1.67
         8192         1000         0.70         3.93         2.31
        16384         1000         4.21         4.70         4.45
        32768         1000         6.27         7.53         6.90
        65536          640        21.40        28.00        24.70
       131072          320        31.72        40.07        35.90
       262144          160        51.12        56.48        53.80
       524288           80        76.59       103.03        89.81
      1048576           40       129.63       193.38       161.50
      2097152           20       240.30       374.87       307.59
      4194304           10       485.41       733.20       609.30


# All processes entering MPI_Finalize

[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: component cm closed
[rdma-qe-15.rdma.lab.eng.rdu2.redhat.com:16586] mca: base: close: unloading component cm
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: component cm closed
[rdma-qe-14.rdma.lab.eng.rdu2.redhat.com:02885] mca: base: close: unloading component cm

[Honggang-LI]

Test over Intel irdma.

[root@rdma-dev-30 ~]$ rpm -qa ucx*
ucx-1.11.2-2.el9.x86_64
ucx-debugsource-1.11.2-2.el9.x86_64
ucx-debuginfo-1.11.2-2.el9.x86_64
ucx-ib-1.11.2-2.el9.x86_64
ucx-rdmacm-1.11.2-2.el9.x86_64
ucx-cma-debuginfo-1.11.2-2.el9.x86_64
ucx-ib-debuginfo-1.11.2-2.el9.x86_64
ucx-rdmacm-debuginfo-1.11.2-2.el9.x86_64
ucx-devel-1.11.2-2.el9.x86_64
ucx-cma-1.11.2-2.el9.x86_64
[root@rdma-dev-30 ~]$ 
[root@rdma-dev-30 ~]$ 
[root@rdma-dev-30 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include irdma0:1 -mca mtl '^psm2,psm,ofi' -mca btl '^openib' --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES=i810_roce.45 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 mpitests-IMB-MPI1 Allgather -time 1.5
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] common_ucx.c:303 tcp/i810_roce.45: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] common_ucx.c:303 self/memory0: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] common_ucx.c:303 posix/memory: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] common_ucx.c:303 cma/memory: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] common_ucx.c:311 support level is none
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      rdma-dev-31
  Framework: pml
--------------------------------------------------------------------------
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53034] PML ucx cannot be selected
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] common_ucx.c:303 tcp/i810_roce.45: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] common_ucx.c:303 self/memory0: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] common_ucx.c:303 posix/memory: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] common_ucx.c:303 cma/memory: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61449] common_ucx.c:311 support level is none
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61444] 1 more process has sent help message help-mca-base.txt / find-available:none found
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61444] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[root@rdma-dev-30 ~]$ 
[root@rdma-dev-30 ~]$ 
[root@rdma-dev-30 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include irdma0:1  --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -x UCX_NET_DEVICES=i810_roce.45 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 mpitests-IMB-MPI1 Allgather -time 1.5
[rdma-dev-31][[39873,1],1][btl_openib_ini.c:171:opal_btl_openib_ini_query] Querying INI files for vendor 0x8086, part ID 5522
[rdma-dev-31][[39873,1],1][btl_openib_ini.c:196:opal_btl_openib_ini_query] Did not find corresponding INI values
--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            rdma-dev-31
  Device name:           irdma0
  Device vendor ID:      0x8086
  Device vendor part ID: 5522

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
[rdma-dev-31][[39873,1],1][btl_openib_ini.c:171:opal_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[rdma-dev-31][[39873,1],1][btl_openib_ini.c:187:opal_btl_openib_ini_query] Found corresponding INI values: default
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           rdma-dev-31
  Local device:         irdma0
  Local port:           1
  CPCs attempted:       rdmacm, udcm
--------------------------------------------------------------------------
[rdma-dev-30][[39873,1],0][btl_openib_ini.c:171:opal_btl_openib_ini_query] Querying INI files for vendor 0x8086, part ID 5522
[rdma-dev-30][[39873,1],0][btl_openib_ini.c:196:opal_btl_openib_ini_query] Did not find corresponding INI values
[rdma-dev-30][[39873,1],0][btl_openib_ini.c:171:opal_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[rdma-dev-30][[39873,1],0][btl_openib_ini.c:187:opal_btl_openib_ini_query] Found corresponding INI values: default
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_register: registering framework mtl components
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_register: found loaded component ofi
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_register: component ofi register function successful
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_register: found loaded component psm2
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_register: component psm2 register function successful
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_open: opening mtl components
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_open: found loaded component ofi
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_open: component ofi open function successful
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: components_open: found loaded component psm2
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: close: component psm2 closed
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: close: unloading component psm2
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_register: registering framework mtl components
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_register: found loaded component ofi
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_register: component ofi register function successful
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_register: found loaded component psm2
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_register: component psm2 register function successful
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_open: opening mtl components
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_open: found loaded component ofi
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_open: component ofi open function successful
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: components_open: found loaded component psm2
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: close: component psm2 closed
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: close: unloading component psm2
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca:base:select: Auto-selecting mtl components
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca:base:select:(  mtl) Querying component [ofi]
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca:base:select:(  mtl) Query of component [ofi] set priority to 25
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca:base:select:(  mtl) Selected component [ofi]
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] select: initializing mtl component ofi
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca:base:select: Auto-selecting mtl components
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca:base:select:(  mtl) Querying component [ofi]
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca:base:select:(  mtl) Query of component [ofi] set priority to 25
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca:base:select:(  mtl) Selected component [ofi]
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] select: initializing mtl component ofi
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mtl_ofi_component.c:365: mtl:ofi:provider: irdma0
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mtl_ofi_component.c:365: mtl:ofi:provider: irdma0
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] select: init returned success
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] select: component ofi selected
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] common_ucx.c:303 tcp/i810_roce.45: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] common_ucx.c:303 self/memory0: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] common_ucx.c:303 posix/memory: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] common_ucx.c:303 cma/memory: did not match transport list
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] common_ucx.c:311 support level is none
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] pml_ucx.c:268 mca_pml_ucx_close
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] select: init returned success
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] select: component ofi selected
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] common_ucx.c:303 tcp/i810_roce.45: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] common_ucx.c:303 self/memory0: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] common_ucx.c:303 sysv/memory: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] common_ucx.c:303 posix/memory: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] common_ucx.c:303 cma/memory: did not match transport list
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] common_ucx.c:311 support level is none
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] pml_ucx.c:268 mca_pml_ucx_close
#----------------------------------------------------------------
#    Intel(R) MPI Benchmarks 2021.3, MPI-1 part
#----------------------------------------------------------------
# Date                  : Thu Jan  6 09:37:34 2022
# Machine               : x86_64
# System                : Linux
# Release               : 5.14.0-39.el9.x86_64
# Version               : #1 SMP PREEMPT Fri Dec 24 00:07:58 EST 2021
# MPI Version           : 3.1
# MPI Thread Environment: 


# Calling sequence was: 

# mpitests-IMB-MPI1 Allgather -time 1.5 

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT 
# MPI_Op                         :   MPI_SUM  
# 
# 

# List of Benchmarks to run:

# Allgather

#----------------------------------------------------------------
# Benchmarking Allgather 
# #processes = 2 
#----------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
            0         1000         0.05         0.05         0.05
            1         1000         9.19        10.35         9.77
            2         1000         9.42        10.28         9.85
            4         1000         9.14        10.54         9.84
            8         1000         9.13        10.53         9.83
           16         1000         2.74        18.37        10.55
           32         1000         2.72        18.51        10.61
           64         1000         2.74        18.65        10.69
          128         1000        18.16        20.43        19.29
          256         1000        18.51        20.77        19.64
          512         1000        18.91        21.20        20.05
         1024         1000        19.18        21.53        20.36
         2048         1000        19.69        22.10        20.90
         4096         1000        21.33        21.57        21.45
         8192         1000        26.36        26.55        26.45
        16384         1000        28.49        28.72        28.60
        32768         1000        37.80        39.76        38.78
        65536          640        56.02        56.30        56.16
       131072          320        90.96        90.96        90.96
       262144          160       172.20       172.84       172.52
       524288           80       337.67       337.90       337.78
      1048576           40       664.69       665.27       664.98
      2097152           20      1785.03      1789.02      1787.03
      4194304           10      4295.49      4334.62      4315.06


# All processes entering MPI_Finalize

[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: close: component ofi closed
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61493] mca: base: close: unloading component ofi
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: close: component ofi closed
[rdma-dev-31.rdma.lab.eng.rdu2.redhat.com:53095] mca: base: close: unloading component ofi
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61488] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61488] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[rdma-dev-30.rdma.lab.eng.rdu2.redhat.com:61488] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port

[jsquyres]

@mwheinz Who's correct to ping about OPA questions these days?

[Honggang-LI]

Test over bnxt_re.

[root@rdma-dev-25 ~]$ rpm -qa ucx*
ucx-devel-1.11.2-2.el8.x86_64
ucx-debugsource-1.11.2-2.el8.x86_64
ucx-cma-debuginfo-1.11.2-2.el8.x86_64
ucx-cma-1.11.2-2.el8.x86_64
ucx-debuginfo-1.11.2-2.el8.x86_64
ucx-ib-debuginfo-1.11.2-2.el8.x86_64
ucx-1.11.2-2.el8.x86_64
ucx-ib-1.11.2-2.el8.x86_64
ucx-rdmacm-debuginfo-1.11.2-2.el8.x86_64
ucx-rdmacm-1.11.2-2.el8.x86_64

[root@rdma-dev-25 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include bnxt_re0:1 -mca mtl '^psm2,psm,ofi' -mca btl '^openib' --mca mtl_base_verbose 100 --mca btl_openib_verbose 100 -mca pml ucx -mca osc ucx -x UCX_NET_DEVICES=bnxt_roce.45 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 mpitests-IMB-MPI1 Gather -time 1.5
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] common_ucx.c:304 posix/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] common_ucx.c:304 self/memory0: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] common_ucx.c:304 tcp/bnxt_roce.45: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] common_ucx.c:304 cma/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] common_ucx.c:311 support level is none
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      rdma-dev-26
  Framework: pml
--------------------------------------------------------------------------
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69256] PML ucx cannot be selected
[root@rdma-dev-25 ~]$ 
[root@rdma-dev-25 ~]$ 
[root@rdma-dev-25 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include bnxt_re0:1  --mca mtl_base_verbose 100 --mca btl_openib_verbose 100  -x UCX_NET_DEVICES=bnxt_roce.45 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 mpitests-IMB-MPI1 Gather -time 1.5
[rdma-dev-26][[7336,1],1][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x14e4, part ID 5968
[rdma-dev-26][[7336,1],1][btl_openib_ini.c:196:opal_btl_openib_ini_query] Did not find corresponding INI values
--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            rdma-dev-26
  Device name:           bnxt_re0
  Device vendor ID:      0x14e4
  Device vendor part ID: 5968

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
[rdma-dev-26][[7336,1],1][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[rdma-dev-26][[7336,1],1][btl_openib_ini.c:188:opal_btl_openib_ini_query] Found corresponding INI values: default
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port.  As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host:           rdma-dev-26
  Local device:         bnxt_re0
  Local port:           1
  CPCs attempted:       rdmacm, udcm
--------------------------------------------------------------------------
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_register: registering framework mtl components
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_register: found loaded component ofi
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_register: component ofi register function successful
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_register: found loaded component psm2
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_register: component psm2 register function successful
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_open: opening mtl components
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_open: found loaded component ofi
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_open: component ofi open function successful
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: components_open: found loaded component psm2
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: close: component psm2 closed
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca: base: close: unloading component psm2
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca:base:select: Auto-selecting mtl components
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca:base:select:(  mtl) Querying component [ofi]
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca:base:select:(  mtl) Query of component [ofi] set priority to 25
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mca:base:select:(  mtl) Selected component [ofi]
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] select: initializing mtl component ofi
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] mtl_ofi_component.c:365: mtl:ofi:provider: bnxt_re0
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] select: init returned success
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] select: component ofi selected
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] common_ucx.c:304 posix/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] common_ucx.c:304 self/memory0: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] common_ucx.c:304 tcp/bnxt_roce.45: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] common_ucx.c:304 cma/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] common_ucx.c:311 support level is none
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69332] pml_ucx.c:268 mca_pml_ucx_close
[rdma-dev-25][[7336,1],0][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x14e4, part ID 5968
[rdma-dev-25][[7336,1],0][btl_openib_ini.c:196:opal_btl_openib_ini_query] Did not find corresponding INI values
[rdma-dev-25][[7336,1],0][btl_openib_ini.c:172:opal_btl_openib_ini_query] Querying INI files for vendor 0x0000, part ID 0
[rdma-dev-25][[7336,1],0][btl_openib_ini.c:188:opal_btl_openib_ini_query] Found corresponding INI values: default
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_register: registering framework mtl components
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_register: found loaded component ofi
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_register: component ofi register function successful
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_register: found loaded component psm2
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_register: component psm2 register function successful
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_open: opening mtl components
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_open: found loaded component ofi
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_open: component ofi open function successful
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: components_open: found loaded component psm2
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: close: component psm2 closed
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca: base: close: unloading component psm2
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca:base:select: Auto-selecting mtl components
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca:base:select:(  mtl) Querying component [ofi]
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca:base:select:(  mtl) Query of component [ofi] set priority to 25
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mca:base:select:(  mtl) Selected component [ofi]
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] select: initializing mtl component ofi
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] mtl_ofi_component.c:365: mtl:ofi:provider: bnxt_re0
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] select: init returned success
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] select: component ofi selected
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] common_ucx.c:304 posix/memory: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] common_ucx.c:304 self/memory0: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] common_ucx.c:304 tcp/bnxt_roce.45: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] common_ucx.c:304 cma/memory: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] common_ucx.c:311 support level is none
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61232] pml_ucx.c:268 mca_pml_ucx_close
#----------------------------------------------------------------
#    Intel(R) MPI Benchmarks 2021.3, MPI-1 part
#----------------------------------------------------------------
# Date                  : Thu Jan  6 22:52:23 2022
# Machine               : x86_64
# System                : Linux
# Release               : 4.18.0-356.el8.x86_64
# Version               : #1 SMP Wed Dec 15 18:07:11 EST 2021
# MPI Version           : 3.1
# MPI Thread Environment: 


# Calling sequence was: 

# mpitests-IMB-MPI1 Gather -time 1.5 

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT 
# MPI_Op                         :   MPI_SUM  
# 
# 

# List of Benchmarks to run:

# Gather
rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:rank1: Process connect/disconnect error: 8, opcode ce

rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:rank0: Process connect/disconnect error: 8, opcode ce

[rdma-dev-26:69332] *** Process received signal ***
[rdma-dev-26:69332] Signal: Aborted (6)
[rdma-dev-26:69332] Signal code:  (-6)
[rdma-dev-26:69332] [ 0] [rdma-dev-25:61232] *** Process received signal ***
[rdma-dev-25:61232] Signal: Aborted (6)
[rdma-dev-25:61232] Signal code:  (-6)
/lib64/libpthread.so.0(+0x12c20)[0x148b1124ac20]
[rdma-dev-26:69332] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x148b10ec1a4f]
[rdma-dev-26:69332] [ 2] /lib64/libc.so.6(abort+0x127)[0x148b10e94db5]
[rdma-dev-26:69332] [ 3] /lib64/libfabric.so.1(+0x1c8f5)[0x148b059798f5]
[rdma-dev-26:69332] [ 4] /lib64/libfabric.so.1(+0x15936a)[0x148b05ab636a]
[rdma-dev-26:69332] [ 5] /lib64/libfabric.so.1(+0x182a6b)[0x148b05adfa6b]
[rdma-dev-26:69332] [ 6] /lib64/libfabric.so.1(+0x183555)[0x148b05ae0555]
[rdma-dev-26:69332] [ 7] /lib64/libfabric.so.1(+0x183c8f)[0x148b05ae0c8f]
[rdma-dev-26:69332] [ 8] /lib64/libfabric.so.1(+0x184b02)[0x148b05ae1b02]
[rdma-dev-26:69332] [ 9] /lib64/libfabric.so.1(+0x14ff06)[0x148b05aacf06]
[rdma-dev-26:69332] [10] /lib64/libfabric.so.1(+0x17a546)[0x148b05ad7546]
[rdma-dev-26:69332] [11] /lib64/libfabric.so.1(+0x1851be)[0x148b05ae21be]
[rdma-dev-26:69332] [12] /lib64/libfabric.so.1(+0x157aea)[0x148b05ab4aea]
[rdma-dev-26:69332] [13] /lib64/libfabric.so.1(+0x1360d6)[0x148b05a930d6]
[rdma-dev-26:69332] [14] /lib64/libfabric.so.1(+0x1495f7)[0x148b05aa65f7]
[rdma-dev-26:69332] [15] /usr/lib64/openmpi/lib/openmpi/mca_mtl_ofi.so(+0xd7d3)[0x148af974c7d3]
[rdma-dev-26:69332] [16] /usr/lib64/openmpi/lib/openmpi/mca_pml_cm.so(+0x251e)[0x148b0446551e]
[rdma-dev-26:69332] [17] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_coll_base_sendrecv_actual+0x86)[0x148b11e408f6]
[rdma-dev-26:69332] [18] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_coll_base_allgather_intra_two_procs+0x8d)[0x148b11e3f70d]
[rdma-dev-26:69332] [19] /usr/lib64/openmpi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgather_intra_dec_fixed+0x4e)[0x148af37dfefe]
[rdma-dev-26:69332] [20] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_comm_split_with_info+0xcc)[0x148b11dbc15c]
[rdma-dev-26:69332] [21] /usr/lib64/openmpi/lib/libmpi.so.40(MPI_Comm_split+0x73)[0x148b11dfd093]
[rdma-dev-26:69332] [22] mpitests-IMB-MPI1(+0x4b45c)[0x555d651c445c]
[rdma-dev-26:69332] [23] mpitests-IMB-MPI1(+0x4ae7e)[0x555d651c3e7e]
[rdma-dev-26:69332] [24] mpitests-IMB-MPI1(+0x69b25)[0x555d651e2b25]
[rdma-dev-26:69332] [25] mpitests-IMB-MPI1(+0x15894)[0x555d6518e894]
[rdma-dev-26:69332] [26] /lib64/libc.so.6(__libc_start_main+0xf3)[0x148b10eadca3]
[rdma-dev-26:69332] [27] mpitests-IMB-MPI1(+0x1407e)[0x555d6518d07e]
[rdma-dev-26:69332] *** End of error message ***
[rdma-dev-25:61232] [ 0] /lib64/libpthread.so.0(+0x12c20)[0x1530e310bc20]
[rdma-dev-25:61232] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x1530e2d82a4f]
[rdma-dev-25:61232] [ 2] /lib64/libc.so.6(abort+0x127)[0x1530e2d55db5]
[rdma-dev-25:61232] [ 3] /lib64/libfabric.so.1(+0x1c8f5)[0x1530d37e08f5]
[rdma-dev-25:61232] [ 4] /lib64/libfabric.so.1(+0x15936a)[0x1530d391d36a]
[rdma-dev-25:61232] [ 5] /lib64/libfabric.so.1(+0x182a6b)[0x1530d3946a6b]
[rdma-dev-25:61232] [ 6] /lib64/libfabric.so.1(+0x183555)[0x1530d3947555]
[rdma-dev-25:61232] [ 7] /lib64/libfabric.so.1(+0x183c8f)[0x1530d3947c8f]
[rdma-dev-25:61232] [ 8] /lib64/libfabric.so.1(+0x184b02)[0x1530d3948b02]
[rdma-dev-25:61232] [ 9] /lib64/libfabric.so.1(+0x14ff06)[0x1530d3913f06]
[rdma-dev-25:61232] [10] /lib64/libfabric.so.1(+0x17a546)[0x1530d393e546]
[rdma-dev-25:61232] [11] /lib64/libfabric.so.1(+0x1851be)[0x1530d39491be]
[rdma-dev-25:61232] [12] /lib64/libfabric.so.1(+0x157aea)[0x1530d391baea]
[rdma-dev-25:61232] [13] /lib64/libfabric.so.1(+0x1360d6)[0x1530d38fa0d6]
[rdma-dev-25:61232] [14] /lib64/libfabric.so.1(+0x1495f7)[0x1530d390d5f7]
[rdma-dev-25:61232] [15] /usr/lib64/openmpi/lib/openmpi/mca_mtl_ofi.so(+0xd7d3)[0x1530cb6137d3]
[rdma-dev-25:61232] [16] /usr/lib64/openmpi/lib/openmpi/mca_pml_cm.so(+0x251e)[0x1530d22cc51e]
[rdma-dev-25:61232] [17] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_coll_base_sendrecv_actual+0x86)[0x1530e3d018f6]
[rdma-dev-25:61232] [18] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_coll_base_allgather_intra_two_procs+0x8d)[0x1530e3d0070d]
[rdma-dev-25:61232] [19] /usr/lib64/openmpi/lib/openmpi/mca_coll_tuned.so(ompi_coll_tuned_allgather_intra_dec_fixed+0x4e)[0x1530c96d7efe]
[rdma-dev-25:61232] [20] /usr/lib64/openmpi/lib/libmpi.so.40(ompi_comm_split_with_info+0xcc)[0x1530e3c7d15c]
[rdma-dev-25:61232] [21] /usr/lib64/openmpi/lib/libmpi.so.40(MPI_Comm_split+0x73)[0x1530e3cbe093]
[rdma-dev-25:61232] [22] mpitests-IMB-MPI1(+0x4b45c)[0x5584bbac445c]
[rdma-dev-25:61232] [23] mpitests-IMB-MPI1(+0x4ae7e)[0x5584bbac3e7e]
[rdma-dev-25:61232] [24] mpitests-IMB-MPI1(+0x69b25)[0x5584bbae2b25]
[rdma-dev-25:61232] [25] mpitests-IMB-MPI1(+0x15894)[0x5584bba8e894]
[rdma-dev-25:61232] [26] /lib64/libc.so.6(__libc_start_main+0xf3)[0x1530e2d6eca3]
[rdma-dev-25:61232] [27] mpitests-IMB-MPI1(+0x1407e)[0x5584bba8d07e]
[rdma-dev-25:61232] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61227] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61227] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61227] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node rdma-dev-25 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
[root@rdma-dev-25 ~]$ 
[root@rdma-dev-25 ~]$ 
[root@rdma-dev-25 ~]$ timeout --preserve-status --kill-after=5m 3m mpirun -hostfile /root/hfile_one_core -np 2 --allow-run-as-root --map-by node -mca btl_openib_warn_nonexistent_if 0 -mca btl_openib_if_include bnxt_re0:1 -mca mtl '^psm2,psm,ofi' -mca btl '^openib' --mca mtl_base_verbose 100 --mca btl_openib_verbose 100  -x UCX_NET_DEVICES=bnxt_roce.45 --mca osc_ucx_verbose 100 --mca pml_ucx_verbose 100 mpitests-IMB-MPI1 Gather -time 1.5
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] mca: base: components_register: registering framework mtl components
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] mca: base: components_open: opening mtl components
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] common_ucx.c:304 posix/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] common_ucx.c:304 self/memory0: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] common_ucx.c:304 tcp/bnxt_roce.45: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] common_ucx.c:304 cma/memory: did not match transport list
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] common_ucx.c:311 support level is none
[rdma-dev-26.rdma.lab.eng.rdu2.redhat.com:69413] pml_ucx.c:268 mca_pml_ucx_close
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] mca: base: components_register: registering framework mtl components
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] mca: base: components_open: opening mtl components
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] pml_ucx.c:197 mca_pml_ucx_open: UCX version 1.11.2
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] common_ucx.c:304 posix/memory: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] common_ucx.c:304 sysv/memory: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] common_ucx.c:304 self/memory0: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] common_ucx.c:304 tcp/bnxt_roce.45: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] common_ucx.c:304 cma/memory: did not match transport list
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] common_ucx.c:311 support level is none
[rdma-dev-25.rdma.lab.eng.rdu2.redhat.com:61283] pml_ucx.c:268 mca_pml_ucx_close
#----------------------------------------------------------------
#    Intel(R) MPI Benchmarks 2021.3, MPI-1 part
#----------------------------------------------------------------
# Date                  : Thu Jan  6 22:53:15 2022
# Machine               : x86_64
# System                : Linux
# Release               : 4.18.0-356.el8.x86_64
# Version               : #1 SMP Wed Dec 15 18:07:11 EST 2021
# MPI Version           : 3.1
# MPI Thread Environment: 


# Calling sequence was: 

# mpitests-IMB-MPI1 Gather -time 1.5 

# Minimum message length in bytes:   0
# Maximum message length in bytes:   4194304
#
# MPI_Datatype                   :   MPI_BYTE 
# MPI_Datatype for reductions    :   MPI_FLOAT 
# MPI_Op                         :   MPI_SUM  
# 
# 

# List of Benchmarks to run:

# Gather

#----------------------------------------------------------------
# Benchmarking Gather 
# #processes = 2 
#----------------------------------------------------------------
       #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
            0         1000         0.05         0.06         0.05
            1         1000        10.79        26.55        18.67
            2         1000         2.76        18.13        10.44
            4         1000         2.80        20.35        11.58
            8         1000         2.78        18.84        10.81
           16         1000         2.79        18.42        10.61
           32         1000         2.79        20.24        11.52
           64         1000         2.81        19.29        11.05
          128         1000         2.81        18.88        10.84
          256         1000         2.83        20.99        11.91
          512         1000         2.83        20.89        11.86
         1024         1000         2.88        20.94        11.91
         2048         1000         2.93        21.92        12.43
         4096         1000         3.06        24.08        13.57
         8192         1000         3.31        28.95        16.13
        16384         1000         3.86        30.34        17.10
        32768         1000         4.89        38.29        21.59
        65536          640        32.52        70.28        51.40
       131072          320        55.09       104.64        79.86
       262144          160        66.78       134.52       100.65
       524288           80        92.12       183.03       137.57
      1048576           40       143.13       277.06       210.10
      2097152           20       238.41       497.76       368.08
      4194304           10       416.10      1852.40      1134.25


# All processes entering MPI_Finalize

[root@rdma-dev-25 ~]$ 

[mwheinz]

@mwheinz Who's correct to ping about OPA questions these days?

Short answer - still myself or @BrendanCunningham.

Slightly longer answer, I've been complaining about UCX and openinb jumping ahead of OPA on OPA fabrics for quite a while now. It's one of the reasons we're still recommending OMPI 4.0.x. @Honggang-LI - what happens if you exclude UCX and openib with ^ucx,openib ?

@Honggang-LI , also be aware that if you don't explicitly specify the pt2pt BTL you will end up using twice the hardware resources when running with the OFI BTL. This will limit the # of ranks you can run per node.

[Honggang-LI]

@Honggang-LI - what happens if you exclude UCX and openib with ^ucx,openib ?

Please provide the entire bash command you want I to run.

[Honggang-LI]

@mwheinz, ping?