`cargo test --all` should run tests in parallel

[rocallahan]

Currently if you have a workspace with a lot of crates, cargo test --all runs those tests one crate at a time. This is slow if you have a lot more cores than individual crates can saturate. It would speed things up a lot if cargo could run tests for multiple crates in parallel.

[alexcrichton]

This'd currently be a breaking change I think (at least amongst Cargo's own test suite). In that sense I don't think we can do this backwards compatibly by default, but we can of course provide an option to do so!

The original thinking was that each test binary would be internally parallelized, but that doesn't always have the best utilization as I'm sure you're seeing

[henrikno]

This also affects projects having lots of integration tests, as those are not run in parallel since they are treated as separate crates.
Simple test case just to demonstrate: https://github.com/henrikno/rust-test-bench/tree/master.
I expected it to take around 1 second with sufficient threads.

A workaround would be to concatenate all the tests into one megatest.rs. Or I guess you could use cargo read-manifest to enumerate them and feed them to xargs/parallel.

cargo read-manifest | jq -r '.targets[]  | select ( .kind | index("test")) | .name' | parallel -j10 cargo test --test {}

[Dushistov]

Yeah, I thought this this the whole idea to run cargo test against the whole workspace - to run all tests from all crates via one job server that can utilize all cpu cores.

[repi]

This is become more and more of a bottleneck for us also in our workspace, we have around 25 crates / integration tests and with some tests in all of them which does run really quite slowly. Could refactor everything into a single crate with the tests in it but it is a lot nicer to have the testing code close to the actual code.

I also expected cargo test --workspace to parallelize all the test running of all the crates, but if that is not possible to have as default, having it as a setting would work

[rocallahan]

FWIW in our project we long ago moved almost all tests to a single toplevel crate. Not only does that work around this issue, it also works around scaling issues when building lots of test binaries in a project with long crate dependency chains. See https://robert.ocallahan.org/2018/10/problems-scaling-large-multi-crate-rust.html for more information about that.

[repi]

Thanks, good to know. We were seeing a lot of compilation & linking overhead from small integration test binaries as well slowing things down.

For these type of use cases, having Cargo build all crates with dynamic linking (cc EmbarkStudios/rust-ecosystem#13) that the integration tests would use would be most helpful, and doesn't need a stable ABI. But that is a whole other huge topic so I'll leave it at that, and would still be slower than a single test crate.

[nagisa]

This is not terribly difficult to work around by implementing your own test runner for now (I could get it done in 150 lines of python). I feel like this ought to mean its not terribly difficult to implement in cargo either, but there are caveats from my experience in implementing a custom test runner:

1. I believe cargo currently simply presents the output from the test runner directly. If the test suites were run in parallel, things would get quite jumbled!
   • This means cargo should run the test runners with --format=json;
   • And present the test output in its own way, much like it currently does for compilation progress.
2. There are complications with doctests:
   • As per usual™;
   • Similar considerations as for regular test suites, but rustdoc does not currently support outputting test results as json – feature development there will be necessary.
3. And custom/missing test harnesses;
   • Unclear how to handle those?

[nagisa]

(I really want this to happen. I have multiple test suites in the private workspace that have tests running in excess of 5 minutes in my workspace – running the suites in parallel took down testing times from 30+ minutes down to a more reasonable 7 minutes or so)

[nagisa]

Another thing: ideally tests should be run alongside building of code in a pipelined fashion, but that complicates presentation even further.

[gilescope]

I'd be more than happy to have buffered stdout/stderrs/presentation if it meant that more things happened in parallel. I don't entirely buy the argument that json output is a blocker. It would be nice but I can't see how it is essential? (I have lots of cores now and want to use them)

[gilescope]

(Given every test in the tests subdir becomes its own crate and gets run one at a time strongly hints that we should either stop doing that or optimize that use case)

What's MVP for this feature? Is there anything stopping someone providing a PR that adds an extra flag and does this this weekend? It looks like this loop here would need to be changed to be in parallel:

cargo/src/cargo/ops/cargo_test.rs

Line 85 in d56b42c

} in compilation.tests.iter()

I'd love to test out someone's PR if they've got a free weekend?

(We can leave doc tests as being serial for now. Any improvement on the existing situation is a win)