osc rdma/tcp: Wrong answer with c_post_start in MTT

[awlauria]

MTT failure example

Test source

Run command (single node):

./exports/bin/mpirun --np 2 --mca osc rdma --mca btl tcp,self ./c_post_start

I already triaged similar. This is new with the support for osc/rdma with btl/tcp to master and v5.0.x.

[wzamazon]

Hi, @awlauria

I tried to reproduced this error, but I was not able to reproduce.

I noticed that the failure is on IBM powerpc64le, which I do not have access to.

On a system with intel CPU, I did not encounter any error.

On a system with AWS graviton 2 CPU, I encountered a different error (about an error in "MPI_Win_Create"). I will continue debug this issue but I doubt it is the same issue reported here.

I wonder whether you were able to see this error outside powerpc64le? Is there a way for me to reproduce the error?

[wzamazon]

Hi, I made a mistake when testing on graviton 2 : I was not using open mpi master branch but was using open mpi 4.1.2.

Once I started using ompi master on graviton 2, I was able to reproduce the data validation error. Though to a lesser degree of the IBM CI. On IBM CI, the test fail consistently and each time there are ~16,000 error. On graviton 2, the test fails about 10% of the time, and when it fails there is usually 1 or 2 data error.

Anyway, I think the reproducer I have now is good enough for me to continue to work on this issue.

[wzamazon]

With the help from @bwbarrett, I think I know the root cause of this problem, which is related to completion semantics requirement of use_cpu_atomics.

use_cpu_atomics is an optimization, which mix use NIC's completion and CPU atomics. It is used when all processes are on single node. In this case, the actual messages are delivered by selected btl, but the counters to indicate completion (such as num_complete_msgs) are updated using CPU atomics (via shm of osc/rdma).

Such an implementation requires the btl's completion to be delivery complete. Which means when btl report the put operation is completed, the data must have arrived at the target buffer.

Btl/tcp's rdma does not satisfy this requirement. BTL/TCP use send to emulated put. When btl/tcp report the put operation to be finished, the data has been sent to the target rank. However, it is not guaranteed the the message has been processed by the target rank.

When use_cpu_atomics is used with btl/tcp (on single nodes), the counter num_complete_msgs is updated using cpu atomics after btl/tcp report the put is finished. However, because btl/tcp does not support delivery complete, it can happen that num_complete_msgs is updated before the message was delivered, hence the data corruption.

One thing worth noting is that this is a new issue (NOT a regression), because ompi 5.0.x is the first time osc/rdma can use btl/tcp. Prior to 5.0.x, btl/tcp does not support put, so osc/rdma cannot use it.

IMO, the solution to this issue is to stop using the optimization use_cpu_atomics if the selected btl does not support delivery complete.

Lacking some context, I do not quite understand the motivation of use_cpu_atomics. If we are already on single node, why not use btl/vader for everything?

[hjelmn]

use_cpu_atomics is intended when HCA and CPU atomics can be mixed. The only network I know of where this is the case is Infinipath/Omnipath where the atomic is done with the CPU. The completion semantics there are different than when using btl/tcp. So, the best fix is to remove the MCA_BTL_ATOMIC_SUPPORTS_GLOB flag from here:

https://github.com/open-mpi/ompi/blob/master/opal/mca/btl/base/btl_base_am_rdma.c#L1117

I can probably do that later today but if someone beats me to it I can review the PR.

[hjelmn]

I can look at making AM-RDMA support use_cpu_atomics later. It is a useful optimization when the network has high latency as is the case with anything TCP.

[wzamazon]

@hjelmn If the two processes are already on the same node, why would they even use tcp (instead of vader) to communicate?

[bwbarrett]

@wzamazon in the test case you're running, TCP is being used because you set the BTL list to only include tcp,self. So there's no shared memory btl active.

[wzamazon]

I see. So use_cpu_atomics is to accelerate the scenario that for some reason, user must use certain btl to transfer message.

[bosilca]

I don't understand how such an optimization would work with the vader BTL on a machine without SMSC support. Can someone clarify this for me please.

[hjelmn]

@bosilca It wouldn't. The optimization was focused on two scenarios: 1) all on-node with MPI_Win_allocate (if for some reason osc/sm is not wanted), and 2) btl/openib with globally visible atomics (now btl/ofi). Was not intended for any of the AM-RDMA scenarios. Not sure why I added that flag.