Running a test suite causes all @BeforeClass instances to be run

[metasonic]

I have the following setup, where I group a series of classes based on the area they target.
The problem is that with version 1.9.2 and above this setup is no longer working correctly, instead of taking each class at a time, and running everything in it (@BeforeClass, , @test, @test,, @afterclass) the setul will run all the @BeforeClass methods from all the test classes before going back to running one class at a time, so (@BeforeClass1, @BeforeClass2, @BeforeClass3, @BeforeClass4, @BeforeClass5 , @Test1Class1, @Test2Class1...)

This will cause an error if i don;t have a @BeforeClass in any of my tests

@RunWith(Categories.class)
@IncludeCategory(WebSanity1.class)
@ExcludeCategory(WebSanity1_Tablet.class)
@SuiteClasses({ AllTests.class, })
public class WebSanity1_One {
}

[janschaefer]

could you provide a minimal example that reproduces the error?

[metasonic]

It looks like the issue is caused by @RunWith(DataProviderRunner.class)
Please see attached example https://www.dropbox.com/s/u7hd586vot7l2qz/example.rar?dl=0

[aaschmid]

Hi @metasonic,

hm ... I am aware of the behavior that @BeforeClass are executed before every test dataprovider. This is because in v1.9.2 I introduced that @BeforeClass runs before the dataproviders such that any results of an @BeforeClass can be used to generate the dataproviders. Do you have a problem with that? Please let me know which kind of problem, maybe there is a solution for this?

But you write about an error if you don't have any @BeforeClass. What do you mean by that? Even your example does work, with the above problems about running all @BeforeClass before any @Test method.

Cheers,
Andreas

[metasonic]

Here is a link with an example related to the error
The issue is caused by the fact that TestTwo class is included in the suite but does not have a @BeforeClass, but has a @RunWith(DataProviderRunner.class), while the other one TestOne has a @BeforeClass but is not included in the suite
@RunWith(DataProviderRunner.class)
@category({ MainTests.class })
public class TestTwo {

// @BeforeClass
// public static void beforeClass() {
// System.out.println("@BeforeClass for TestTwo");
// }
https://www.dropbox.com/s/xn2wihqzno4kw60/eampleError.rar?dl=0

We are doing selenium related tests, and use the @BeforeClass to open a browser, in this case the fact that all @BeforeClass are run at the start is an issue. It would be great if there is any way we could avoid this and keep the old flow.

[aaschmid]

I am very sorry, but I get no error running AllTests and running SuperTests causes an NoTestsRemainException with and without an @RunWith(DataProviderRunner.class) annotation.

However, the problem with @BeforeClass is serious and I am thinking about fixing it by getting back to the behavior before v1.9.2. I am just a bit uncertain because this breaks backwards compatibility ...
Still have to think about it. Is it urgent or better do you urgently need a feature of a version >= 1.9.2?

[metasonic]

It's not an urget issue but it's a blocker for what can be considered further updates, features, improvements.
The issue in that example is that (hope i'm not mistaking) with the older version running Super Tests won't cause any error.
At the moment it appears to run the @BeforeClass from a test that is no longer contained in the Test Suit. Will try to provide a better example.

[metasonic]

It's not an urget issue but it's a blocker for what can be considered further updates, features, improvements.
The issue in that example is that (hope i'm not mistaking) with the older version running Super Tests won't cause any error.
At the moment it appears to run the @BeforeClass from a test that is no longer contained in the Test Suit. Will try to provide a better example.

[aaschmid]

Is there any update according to an exmample @metasonic ?

[metasonic]

hey, sorry for the high delay, i no longer have access to that code, will try to replicate it but don't know if i will be able to do so.