Introduce support for declarative test suites

[smoyer64]

Overview

As requested in issue #681 (Introduce junit-platform-suite-api module), the annotations used for test discovery and selection in the JUnitPlatform runner have been refactored into the junit-platform-suite-api. This allows a purely JUnit 5 test suite to be defined declaratively, though it currently requires an alternate engine (as opposed to an extension).

This issue describes a generalized JUnit platform suite that's suitable for the Jupiter test engine as well as any others that might be chosen. It's my opinion that unit tests rarely require test suites, but it becomes far more common when integration, system and acceptance tests are being executed due to the very common integration with external systems that should only be initialized before the entire test run. Examples of resources that can be safely shared among tests are database connection pools (or really any connection pool), test frameworks that require significant set-up time (e.g. GWT test suites eliminate 20s of HtmlUnit set-up time for each test - it's only incurred once at the beginning of each suite).

The examples shown below indicate how the @Suite annotation can be used in conjunction with those already in the junit-platform-suite-api module. Originally I'd also defined the @BeforeSuite and @AfterSuite annotations but after further consideration decided that since suites are really just an arbitrary collection of indeterminately nested test containers, it would be better if the @BeforeAll and @AfterAll annotations were contextual (nested in the same way the hierarchy would be presented in the test tree). I've also assumed that these annotations no longer require the static modifier as discussed elsewhere.

Imagine if you will a set of acceptance tests that have a custom reporting mechanism that should be configured for the entire test run. Included within this outer suite is a set of tests that require a database connection pool and a different set of tests that require an LDAP connection pool. The top-level suite definition for the acceptance tests might look something like:

@Suite
@IncludeEngines("jupiter-engine") // not required if there's only one engine on the classpath
@IncludeClassNamePatterns(".*AcceptanceTestSuite$")
public class AcceptanceTestSuite {

    @BeforeAll
    void setUp() {
        ... // code to set-up custom test reporting
    }

    @AfterAll() {
    void tearDown() {
        ... // code to tear-down custom test reporting
    }

}

There are two notable characteristics of this test suite:

1. It specifies which engine will be used for the entire container/test hierarchy (assuming there is more than one).
2. It includes potentially numerous other test suites as sub-containers which can each have their own contextual set-up and tear-down.

A child suite for tests that require a database connection pool might look something like this:

@Suite
@IncludeTags({"integration", "database"})
@IncludeClassNamePatterns(".*IT$")
public class DatabaseAcceptanceTestSuite {

    @BeforeAll
    void setUp() {
        ... // code to set-up the database connection pool
    }

    @AfterAll() {
    void tearDown() {
        ... // code to tear-down the database connection pool
    }

}

If run under Maven, test classes that end in "IT" would be run using the maven-failsafe-plugin. If there are other classes of integration tests not run during acceptance testing it would also be necessary to make sure the acceptance test classes were not run (as they'd fail without the suite's set-up methods).

Feature Request

Provide the means for arbitrarily nested, declarative tests suites in JUnit 5 including the execution of set-up and tear-down methods

Related Issues

• Introduce support for programmatic test suites #330
• Introduce @BeforeSuite and @AfterSuite annotations #456
• Improve JavaDoc for junit-platform-suite-api annotations #746
• Introduce an extension point to allow test container creation #687 (perhaps superseded by this issue)
• Introduce support for parallel test execution in declarative test suites #964
• Eclipse IDE issue
• Introduce file-based declarative test suite support #1243

Deliverables

• TBD

[mlevison]

Please consider a new name other than suites - so that people with JUnit 3 history don't accidentally mis-use these for Unit Testing. Cheers Mark

[sbrannen]

Tentatively slated for M5.

[sbrannen]

@smoyer64, thanks for creating the ticket!

I think there are two main topics we need to sort out here:

1. suites within the JUnit Jupiter engine
2. declarative suite configuration at the Platform level

If I understand correctly, you are currently focusing on # 1; whereas, I would first like to focus on # 2.

Status Quo

• We have generic suite API annotations.
• We have a JUnit 4 based runner that supports them.

The Future

• Ideally we should have Platform-level support for them as well.
• If it makes sense to have additional support in JUnit Jupiter for "suites", we should consider that as well.

My Vision

I think it would be most beneficial to the entire community if the JUnit Platform provides support for declarative test suites, akin to what the JUnitPlatform runner already does, but without any ties to any testing framework.

In other words, the declarative test suite support should be analogous to using the ConsoleLauncher or JUnit Platform Gradle plugin.

This can be achieved by moving the internals for processing such suite annotations and building the corresponding LauncherDiscoveryRequest from the JUnitPlatform runner to a new home in the launcher module. This new "prediscovery" code could then be used to transparently add selectors and filters to the request before any engines are asked to discover. For example, this could potentially be performed in the DefaultLauncher.

In this manner, all test engines could benefit from being able to be configured by a generic suite (which is already supported in build scripts). From the perspective of a TestEngine, the presence of selectors and filters added from the suite declaration would be no different than those added via a build script.

[sbrannen]

This allows a purely JUnit 5 test suite to be defined declaratively, though it currently requires an alternate engine (as opposed to an extension).

My above proposal avoids exactly such a need.

If we support suites at the platform level, there is no need for any additional test engines.