Can’t run end-to-end tests concurrently

[geigerzaehler]

With our current setup for end-to-end tests in is not possible to run tests concurrently. This does not show at the moment but prevents us from adding more tests.

Symptoms

Trying to run tests concurrently may result in spurious watch_extrinsic stream terminated errors. These errors mask the fact that the transaction is invalid.

Cause

The cause for this is the account nonce. We use the ClientT::sign_and_submit_message method to submit messages in tests. This method fetches the account nonce from the chain. Since there is only one account with funds on the test chain all tests use this account. If two tests run concurrently it might happen that they issue the request for the account nonce at the same time and get the same nonce back. This results in the two tests submitting a transaction with the same nonce. Only one of the transaction will be accepted and the other one will fail.

Stop-gap solution

To work around this we can run the end-to-end tests with the --threads=1 flag on CI and document this properly

Long-term solution

In the long term, the best solution for this would probably be to create a mutex for accounts so that only one test submits a transaction at a time. We should also add more accounts to the test ledger so that the tests can use one of a variety of accounts.

[NunoAlexandre]

Top, @geigerzaehler!

For the record, we need --test-threads, not --threads. A good SO thread on this: https://stackoverflow.com/questions/44947914/how-to-limit-the-number-of-test-threads-in-cargo-toml

This is fixed and ready for review at #215.

[NunoAlexandre]

@geigerzaehler just found rust-lang/rust#42684, from where the serial-test results. https://crates.io/crates/serial_test
This seems like a great solution since we only need to mark the specific tests we want to run sequentially with #[serial]. This seems like a better quick fix since it doesn't affect all the tests. Will give it a try.