Make `cargo test` usable for nearcore developers

[matklad]

I've noticed that I never run cargo test --workspace locally. I do want to have an ability to quickly check if I broke something somewhere, but the usability of our current cargo test is abysmal.

Specifically:

• Tests output a lot to the terminal, making terminal scrollback buffer useless.
• Test suite is prohibitively slow. It takes 15 wall-clock minutes / 12 cpu minutes on a 24-core machine.

I suggest immediate AIs:

• fixing the tests such that the output of cargo test is never polluted unless the test actually fails. I think using TestWriter is one possible way to do this
• moving all obviously slow tests to expensive_tests
• revisiting this issue after this easy wins are achieved.

Longer term, I think it might sense to do the following:

• replace expensive_tests with run-time check (see the example at the end of "Make Tests Fast" section of this post).
• add two tiers of expensive_tests -- nightly_only_tests (> 5 min) and ci_only_tests (>10s, <5min). Local cargo t skips ci_only_tests by default, but they are run by CI for every commit that gets into master
• document the expectations about tests run-time and actively fix regressions.
• aim for really short test turn-around time. To give an aspiration goal, in rust-analyzer cargo t takes 5s wall-clock/35s cpu by default and 30s/100s when including ci only tests.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[bowenwang1996]

@matklad this is fixed right?

[matklad]

Not at all:

• running cargo test still outputs megabytes of DEBUG-level logging to the terminal,
• it still takes 15 minutes to complete on a 12-core machine