ExitCode of the underlying test process is ignored

[ViktorHofer]

I believe there should be an option to check the ExitCode of the underlying test process. Creating reports from test runs that didn't succeed could mean the execution stopped prematurely and results are spoiled. Thoughts?

[tonerdo]

This is exactly what happens by default with the MSBuild package, I got a lot of feedback that they'd still want to generate results even for a test failure so that's why the console package does that

[ViktorHofer]

I see. Just looking at coverlet.console itself as a separate product it would still be nice to make that configurable.

[MarcoRossignoli]

After merge new logging capabilities we should at least log a warning

[ViktorHofer]

Assuming we added logs or a switch now that this is closed?

[MarcoRossignoli]

Mmm we added log infrastructure but nothing related to this specific case afaik /cc @tonerdo

[tonerdo]

@ViktorHofer yeah, we added logging. Efforts (like #329) to overhaul Coverlet's underlying approach will render a switch redundant in the long run. Also, adding a switch to coverlet.console will lead to requests for similar in coverlet.msbuild which is not possible at the current time.

[adrianhara]

It would be great to have such an option in coverlet console. Imagine you have a lot of tests which take a long time to run. With the current approach a build pipeline would have to run tests twice (once the tests + once coverlet) to ensure the build breaks if any tests fail. If coverlet would expose an option to allow breaking the build if any tests fail, then the tests could be run just once as part of coverlet, potentially saving a lot of time.

[ViktorHofer]

The issue with that approach is that you aren't testing what you're shipping. Coverlet instruments the source assemblies so that the IL Code is different (mainly larger). Even though this probably doesn't cause issues in most cases, there could be different behaviors in time critical scenarios / races.

[MarcoRossignoli]

Confirm @ViktorHofer concern happen sometime #511 (comment)

[adrianhara]

It's a valid point and for sure the option is not for everyone, but it would still be nice to have it 😸