differenciate between errors and failures for junitlauncher

[bodewig]

Try to tell errors from failures in junitlaucher's formatters.

@Vampire and @jaikiran I'd like to have a second pair of eyes since I'm not familiar with the JUnit5 API at all.

[jaikiran]

Hello Stefan, as far as I remember, JUnit5 doesn't differentiate between a test error and failure. So for the junitlauncher task I had decided not to try an come up with our own definition of error vs failures. I haven't checked if that has changed in JUnit5 (I'll take a look at their documentation this week).

Having said that, if we want to differentiate between exception types as errors vs failures, irrespective of what JUnit5 does then I think we can do that if there's a demand for that. If it's to make the junit report output match with what we do for the junit task, then that might not be the best motivation. Because then developers who are more familiar with JUnit 5 might find it odd that Ant reports the failure as an error where as JUnit 5 itself doesn't do so.

I will take a more detailed look at the change and the JUnit 5 documentation in the coming days.

[bodewig]

Many thanks @jaikiran that's why I asked for your input.

For background please see @Vampire 's https://bz.apache.org/bugzilla/show_bug.cgi?id=69687 (and maybe https://bz.apache.org/bugzilla/show_bug.cgi?id=69685 as well. If I understand Björn correctly the test integrations of IDEs are more inline with what we have done in the past with junit.

Since I haven't used JUnit5 for a very long time I'm not really sure how it works in recent Java IDEs (which I haven't used in a long time either :-) ).

[Vampire]

Hm, yeah, JUnit Platform indeed does not differentiate between failure and error anymore.
I have no idea why, maybe @marcphilipp could shed some light on why this is not done anymore.

Gradle indeed does also not do this differentiation yet, but reports both cases as failure.

IntelliJ IDEA, when running the tests itself and not through Gradle delegation, does do the separation.

Seems like a big cross-project inconsistency mess. :-(

For me as user it was majorly confusing, that errors are reported as failures when using Ant, while when using IntelliJ IDEA integration they were shown as error.

So maybe it is appropriate to follow suite with JUnit logic and not differentiate them.
But then at least your own consumers of the file (the XSLs) should expect that it might be missing, as the generated HTML report looks broken if there is not even an errors attribute.

[marcphilipp]

Hm, yeah, JUnit Platform indeed does not differentiate between failure and error anymore. I have no idea why, maybe @marcphilipp could shed some light on why this is not done anymore.

Gradle indeed does also not do this differentiation yet, but reports both cases as failure.

Since both are signaled via exceptions, we decided to simplify things when we started work on JUnit 5 (or JUnit Lambda as it was called back in the day). If you want to differentiate "failures" from "errors", you can check whether the Throwable is an instance of AssertionError.