[SUREFIRE-2298] @Nested test classes are not reported properly

[jira-importer]

Leonard Brünings opened SUREFIRE-2298 and commented

Given this example test you can see that the @Nested test classes are not reported correctly. In 3.5.2 and earlier all executions land in the TEST-BaseNestedTest$Inner.xml and with the changes in 3.5.3 they are distributed in all files. However, the nested classes belong to the enclosing class and the reporting should reflect that, so it would be more accurate to report the nested executions as part of the enclosing tests (see below).

Marc Philipp please chime in.

 

import org.junit.jupiter.api.*;

abstract class BaseNestedTest {
    @Test
    void outerTest() {
    }
    
    @Nested
    class Inner {

        @Test
        void innerTest() {
        }
    }
}

class FirstNestedTest extends BaseNestedTest {
}

class SecondNestedTest extends BaseNestedTest {
}

This was most likely caused by 5aeca19

3.5.2

<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="FirstNestedTest" time="0.019" tests="0" errors="0" skipped="0" failures="0">
</testsuite>

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="SecondNestedTest" time="0.004" tests="0" errors="0" skipped="0" failures="0">
</testsuite>

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="BaseNestedTest$Inner" time="0.001" tests="2" errors="0" skipped="0" failures="0">
  <testcase name="outerTest" classname="FirstNestedTest" time="0.007"/>
  <testcase name="innerTest" classname="BaseNestedTest$Inner" time="0.0"/>
  <testcase name="innerTest" classname="BaseNestedTest$Inner" time="0.0"/>
  <testcase name="outerTest" classname="SecondNestedTest" time="0.001"/>
</testsuite>

---

3.5.3

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="FirstNestedTest" time="0.019" tests="1" errors="0" skipped="0" failures="0">
  <testcase name="outerTest" classname="FirstNestedTest" time="0.007"/>
</testsuite>

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="SecondNestedTest" time="0.003" tests="1" errors="0" skipped="0" failures="0">
  </properties>
  <testcase name="outerTest" classname="SecondNestedTest" time="0.001"/>
</testsuite>

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="BaseNestedTest$Inner" time="0.001" tests="1" errors="0" skipped="0" failures="0">
  <testcase name="innerTest" classname="BaseNestedTest$Inner" time="0.001"/>
  <testcase name="innerTest" classname="BaseNestedTest$Inner" time="0.0"/>
</testsuite>

 

---

Expected reporting

 

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="FirstNestedTest" time="0.019" tests="1" errors="0" skipped="0" failures="0">
  <testcase name="outerTest" classname="FirstNestedTest" time="0.007"/>
  <testcase name="innerTest" classname="BaseNestedTest$Inner" time="0.001"/>
</testsuite>

 <?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="SecondNestedTest" time="0.003" tests="1" errors="0" skipped="0" failures="0">
  </properties>
  <testcase name="outerTest" classname="SecondNestedTest" time="0.001"/>
  <testcase name="innerTest" classname="BaseNestedTest$Inner" time="0.0"/>
</testsuite>

 

---

Affects: 3.5.3

Issue Links:

• SUREFIRE-1643 JUnit 5 in parallel execution mode confuses Surefire reports
  ("is caused by")
• SUREFIRE-2299 Missing tests inside Surefire report for Cucumber scenarios

Remote Links:

• GitHub Pull Request #828
  

0 votes, 7 watchers

[jira-importer]

Marc Philipp commented

I agree with Leonard Brünings's assessment of the situation.

[jira-importer]

Pavlo Shevchenko commented

This change also breaks the reporting of Cucumber tests.

Steps to reproduce:

1. Set up the sample Cucumber project using "cucumber-archetype" (ref):
2. For Surefire 3.5.2, the console output will be:

[INFO] Running hellocucumber.RunCucumberTestScenario: The example                       # hellocucumber/example.feature:3
  Given an example scenario                 # hellocucumber.StepDefinitions.anExampleScenario()
  When all step definitions are implemented # hellocucumber.StepDefinitions.allStepDefinitionsAreImplemented()
  Then the scenario passes                  # hellocucumber.StepDefinitions.theScenarioPasses()
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.065 s -- in hellocucumber.RunCucumberTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 

, and the XML report ("TEST-hellocucumber.RunCucumberTest.xml") will contain a single test case:

<testcase name="An example - The example" classname="An example" time="0.028"> 

1. With Surefire 3.5.3, on the other hand, the output will say that no tests were run:

[INFO] Running hellocucumber.RunCucumberTestScenario: The example                       # hellocucumber/example.feature:3
  Given an example scenario                 # hellocucumber.StepDefinitions.anExampleScenario()
  When all step definitions are implemented # hellocucumber.StepDefinitions.allStepDefinitionsAreImplemented()
  Then the scenario passes                  # hellocucumber.StepDefinitions.theScenarioPasses()
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.078 s -- in hellocucumber.RunCucumberTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 

, and the XML report will contain no test cases.

[jira-importer]

Pavlo Shevchenko commented

Furthermore, feature failures will no longer cause the build to fail:

 

[INFO] Running hellocucumber.RunCucumberTestScenario: The example                       # hellocucumber/example.feature:3
  Given an example scenario                 # hellocucumber.StepDefinitions.anExampleScenario()
  When all step definitions are implemented # hellocucumber.StepDefinitions.allStepDefinitionsAreImplemented()
  Then the scenario passes                  # hellocucumber.StepDefinitions.theScenarioPasses()
      java.lang.RuntimeException: boom!
          at hellocucumber.StepDefinitions.theScenarioPasses(StepDefinitions.java:19)
          at ✽.the scenario passes(classpath:hellocucumber/example.feature:6)
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 s -- in hellocucumber.RunCucumberTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- jar:3.4.1:jar (default-jar) @ hellocucumber ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /Users/pshevche/misc/tmp/cucumber_surefire/reproducer/hellocucumber/target/hellocucumber-1.0.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS 

[jira-importer]

Piotr Zygielo commented

pshevche I think this (cucumber) is covered by SUREFIRE-2299.

[jira-importer]

Olivier Lamy commented

I have started a fix with this PR https://github.com/apache/maven-surefire/pull/828 

[jira-importer]

Ulli Hafner commented

I am wondering why your expectations are that the outer test results should be reported in the nested class XML and not in the corresponding class XML? From my perspective this expectation is wrong for 3.5.2 and 3.5.3, or am I missing something? Or is my use case a completely different thing because I do not inheritance?

I have a test like:

class SourcePrinterTest {
     @Test 
     void testOuter() {
     }

     @Nested 
     class ColumnMarkerTest {
          @Test 
          void testInner() {
          }
     }

This results in 0 tests in SourcePrinterTest.xml but 2 tests in SourcePrinterTest$ColumnMarkerTest.xml. I would expect to have 1 test in SourcePrinterTest.xml and 1 test in SourcePrinterTest$ColumnMarkerTest.xml.