Rerun results incorrectly collated with junit-platform-suite

[mpkorstanje]

Affected version

3.5.2

Bug description

When the junit-platform-suite engine is used in combination with the cucumber-junit-platform-engine and rerunFailingTestsCount re-run results are incorrectly collated. Originally reported as cucumber/cucumber-jvm#3008 but equally strange results can be reproduced with junit-jupiter. Working reproducer for everything below can be found at https://github.com/mpkorstanje/surefire-rerun.

Given two suites:

@Suite
@IncludeEngines("junit-jupiter")
@SelectClasses(Template.class)
public class SuiteATest {

}

@Suite
@IncludeEngines("junit-jupiter")
@SelectClasses(Template.class)
public class SuiteBTest {

}

And test targeted specifically crafted such that it will only fail for one suite (a bit contrived, but makes the reproducer more minimal).

@ExtendWith(Template.MyTestExtension.class)
class Template {

    private static String id;
    
    @Test
    void test(){
        System.out.println(id);
        if (id.contains("SuiteBTest")) {
            Assertions.fail();
        }
    }
    
    public static class MyTestExtension implements BeforeEachCallback {

        @Override
        public void beforeEach(ExtensionContext context) throws Exception {
            id = context.getUniqueId();
        }
    }
}

When executing

mvn clean test -Dsurefire.rerunFailingTestsCount=1

Then:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running a.SuiteBTest
[INFO] Running a.Template
[engine:junit-platform-suite]/[suite:a.SuiteBTest]/[engine:junit-jupiter]/[class:a.Template]/[method:test()]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.029 s <<< FAILURE! -- in a.Template
[ERROR] a.Template.test -- Time elapsed: 0.015 s <<< FAILURE!
org.opentest4j.AssertionFailedError
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:34)
        at org.junit.jupiter.api.Assertions.fail(Assertions.java:119)
        at a.Template.test(Template.java:23)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 s -- in a.SuiteBTest
[INFO] Running a.SuiteATest
[INFO] Running a.Template
[engine:junit-platform-suite]/[suite:a.SuiteATest]/[engine:junit-jupiter]/[class:a.Template]/[method:test()]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s -- in a.Template
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s -- in a.SuiteATest
[INFO] Running a.SuiteBTest
[INFO] Running a.Template
[engine:junit-platform-suite]/[suite:a.SuiteBTest]/[engine:junit-jupiter]/[class:a.Template]/[method:test()]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in a.Template
[ERROR] a.Template.test -- Time elapsed: 0.002 s <<< FAILURE!
org.opentest4j.AssertionFailedError
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:34)
        at org.junit.jupiter.api.Assertions.fail(Assertions.java:119)
        at a.Template.test(Template.java:23)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s -- in a.SuiteBTest
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] a.Template.test
[ERROR]   Run 1: Template.test:23
[INFO]   Run 2: PASS
[ERROR]   Run 3: Template.test:23
[INFO] 
[INFO] 
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.115 s
[INFO] Finished at: 2025-06-04T20:19:15+02:00
[INFO] ------------------------------------------------------------------------

So there are two tests:

• [engine:junit-platform-suite]/[suite:a.SuiteATest]/[engine:junit-jupiter]/[class:a.Template]/[method:test()]
• [engine:junit-platform-suite]/[suite:a.SuiteBTest]/[engine:junit-jupiter]/[class:a.Template]/[method:test()]

Of these SuiteBTest is consistently failing yet the build was a success.

In the report we also see

[WARNING] Flakes: 
[WARNING] a.Template.test
[ERROR]   Run 1: Template.test:23
[INFO]   Run 2: PASS
[ERROR]   Run 3: Template.test:23

So it would appear that the results of Template.test() from both suites are incorrectly collated together.

The same result can be seen when using the JUnit Platform Suite Engine in combination with Cucumber

Feature: Belly

  Scenario: a few cukes
    Given I have 42 cukes in my belly

package a;

import io.cucumber.java.en.Given;
import org.assertj.core.api.Assertions;
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.Suite;

import static io.cucumber.junit.platform.engine.Constants.GLUE_PROPERTY_NAME;

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("belly.feature")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "a")
public class RunCucumberATest {

    @Given("I have {int} cukes in my belly")
    public void I_have_cukes_in_my_belly(int cukes) {
        // Always fail
        Assertions.fail();
    }
}

package b;

import io.cucumber.java.en.Given;
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.Suite;

import static io.cucumber.junit.platform.engine.Constants.GLUE_PROPERTY_NAME;

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("belly.feature")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "b")
public class RunCucumberBTest {

    @Given("I have {int} cukes in my belly")
    public void I_have_cukes_in_my_belly(int cukes) {
        // Always pass
    }
}

mvn clean test -Dsurefire.rerunFailingTestsCount=1

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running a.RunCucumberATest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.082 s <<< FAILURE! -- in a.RunCucumberATest
[ERROR] Belly.a few cukes -- Time elapsed: 0.043 s <<< FAILURE!
java.lang.AssertionError: 
        at a.RunCucumberATest.I_have_cukes_in_my_belly(RunCucumberATest.java:21)
        at ✽.I have 42 cukes in my belly(classpath:belly.feature:4)

[INFO] Running b.RunCucumberBTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s -- in b.RunCucumberBTest
[INFO] Running a.RunCucumberATest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.006 s <<< FAILURE! -- in a.RunCucumberATest
[ERROR] Belly.a few cukes -- Time elapsed: 0.002 s <<< FAILURE!
java.lang.AssertionError: 
        at a.RunCucumberATest.I_have_cukes_in_my_belly(RunCucumberATest.java:21)
        at ✽.I have 42 cukes in my belly(classpath:belly.feature:4)

[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] Belly.a few cukes
[ERROR]   Run 1: 
[INFO]   Run 2: PASS
[ERROR]   Run 3: 
[INFO] 
[INFO] 
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.315 s
[INFO] Finished at: 2025-06-04T20:27:53+02:00
[INFO] ------------------------------------------------------------------------

A passing build that shouldn't be.

Notes:

1. Can't be reproduced on the latest surefire because of With cucumber integration tests not counted in total test summary in surefire-plugin 3.5.3 #834.
2. Surefire internally seems to assume that classname and name uniquely identify a test. And seems to derive these from TestDescriptor.getDisplayName. However getDisplayName does not provide such guarantees. Instead TestDescriptor.getUniqueId should be used.