Flaky multi-node tests

[koivunej]

Running the pubsub tests repeatedly like:

while target/debug/deps/pubsub-c48a8707038c37a4 publish_between_two_nodes; do echo; echo '...'; echo; done

(Your executable name will vary, use cargo test -- publish_between_two_nodes to find out)

With all logging at level at trace this does produce at least two kinds of failures. Seen errors:

• [2020-04-02T09:41:35Z DEBUG libp2p_tcp] Dropped TCP connection to undeterminate peer when the 10s timeout started
• [2020-04-02T09:41:08Z DEBUG libp2p_secio] error during secio handshake IoError(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) even after sending pubsub rpc's back and forth but before the 10s timeout started

In a highly unscientific "lets run these in multiple shells until they fail, then inspect reason" both seem to happen as often. I think I'll just reference this comment in a tracking issue, this might already be better handled with the just released 0.17 of libp2p. Being resilient on disconnects in the test is one possibility but I am used to running tests in Linux and Windows over loopback without much of an issue. Then again, my systems have probably never been as loaded as any public CI server.

By running other tests than the pubsub_between_two_nodes they can also fail but I didn't recover any information on those as 5 tests were executed in parallel and only one of them turned logging on so let it be a small data point.

Originally posted by @koivunej in #133 (comment)

[koivunej]

For some reason the tests have started getting better. This might be because of libp2p 0.19 or just random luck with the other builds running on gha.

[ljedrz]

Currently the exchange_block test likes to fail; I'll look into it.

[ljedrz]

I've found a semi-reliable way of reproducing the issue; I'm suspecting insufficient node cleanup to be the root cause.

[ljedrz]

Ok, so it might require a bit of a broader solution, as even though my fix seems to be doing the trick, it sometimes causes some async deadlocking, which is not ideal either 😄. I have some ideas, will explore them shortly.

[koivunej]

These have not been flaky at all recently, nor sure is that because of GHA architecture or #307 for example. Lets reopen we get more of these.