Add Before and After Parameterized Run annotations

[bigmikef]

Currently there is no simple way for a method to be invoked before or after a Parameterized instance is ran.

Said another way, along with @BeforeClass and @afterclass there should be @BeforeParameterRun @AfterParameterRun. Each fires before/after a Parameterized instance is ran.

[Yishai]

How about a parameter on the @before annotation itself: @before(true) or @before(onEachParameter = true). Defaulted to false, of course.

[bigmikef]

The goal of this request is to have a place to put code that runs before and after a parameter run executes. Not before each test in a given parameter run. Thus the order would be BP BT T1p1 AT BT T2p1 AT BT T3p1 AT AP BP BT T1p2 AT BT T2p2 AT ... where

• BP is the before parameter run method @BeforeParameterRun
• BT is the before test method @before
• Txpy is the xth test, on Parameter y @test eg T2P3 is test 2 on parameter 3
• AT is the after test method @after
• AP is the after parameter run @AfterParameterRun

I had this implemented on my github, however had to take it down due to issues with my employer. I issued a pull to David Saff and he included it in his branch, but it was taken out later. I'll look for the revision number where the code was included after this message.

[bigmikef]

Looks like the merge was to a tree of dsaff around April 3, 2009. In the comments you can see a merge from bigmikef. It is way out of date, but did work at the time.

[alexeyOnGitHub]

guys, "after / before" for every parameterized run would be very useful.

[matthewfarwell]

This can be done using rules, in particular ExternalResource. The implemented methods are called for each parameter:

@RunWith(Parameterized.class)
public class ParameterRuleTest {
    @Parameters()
    public static Iterable<Object[]> data() {
        return Arrays.asList(new Object[]{}, new Object[]{});
    }

    @Rule
    public ExternalResource externalResource = new ExternalResource() {
        protected void before() throws Throwable { System.out.println("before"); }
        protected void after() { System.out.println("after"); }
    };

    @Test public void myTest1() { System.out.println("test"); }
}

This produces

before
test
after
before
test
after

If this is acceptable, please close this issue.

[coreyjv]

@matthewfarwell I may be misunderstanding your response, but in the example you provided that rule would run before and after each test for each instance created by the parameterized runner.

What I think @bigmikef wanted was the capability to be able to run a before and after method once for each instance created by the parameterized runner. I don't think @ClassRule can be used in this case because that runs in a static context and is only executed once for all parameterized instances.

[coreyjv]

I wouldn't mind taking the initiative to implement this since it would be beneficial in my current project. My proposal is below:

Summary of Implementation:

The traditional architecture of JUnit is a instance of a Java class per test method. In order to implement the requirements above I believe the Parameterized runner needs to be modified to create one Java instance for each parameterized instance.

The primary reason for this is so the logic executed before and after each parameterized run is local to the class and not put in some global scope with a static modifier. In my particular use case I have setup and tear down that would require use of parameters that are unique to each parameterized instance.

Additions / Modifications:

org.junit.runners.Parameterized

• Add a new annotation named ParameterRule.
• Add a new protected inner class that extends TestClassRunnerForParameters named something like SingleInstanceRunnerForParameters.
• Instantiate TestClassRunnerForParameters or SingleInstanceRunnerForParameters depending if ParameterRule exists or not.

org.junit.runners.BlockJUnit4ClassRunner

• Move the creation of the test object out of methodBlock. Provide the test object as a parameter instead.
  • The main reason for this is so extended classes could pass their own instance into methodBlock. Otherwise if someone wants to extend BlockJUnit4ClassRunner and keep most of the functionality they have to copy the following methods: withRules, withMethodRules, withTestRules and getMethodRules
• If moving the creation of the test object out of methodBlock is not acceptable then changing the following methods from private to protected would reduce duplication of code:
  • withRules
  • withMethodRules
  • withTestRules
  • getMethodRules

org.junit.runners.Parameterized.SingleInstanceRunnerForParameters

• Add a method named withParameterRules that would execute rules annotated by ParameterRule.
• Execute withParameterRules in an overridden version of classBlock

If there is a preference to provide an example of this implementation I can on my fork, but I wanted to discuss the design before I invested a significant amount of time.

[Tibor17]

@coreyjv what's wrong with @matthewfarwell `s answer?

[coreyjv]

@Tibor17 , see my response to his answer: https://github.com/KentBeck/junit/issues/45#issuecomment-11293228

His implementation is executed before and after each test method. In my case, my setup and tear down is significantly expensive which is why I want to execute it before and after each parameterized run.

For example if I have a test class that when executed once with a common setup (located in a @ClassRule currently) the entire class executes and completes the tests in ~20 seconds. However, if I move the setup to a method level rule it takes approximately ~240 seconds to execute and complete. That's just for one class. In my situation that increase in time is not acceptable.

Theoretically I could place all my tests in a single method to emulate the required behavior but I don't think that is a maintainable test design.

[coreyjv]

@Tibor17, I'm not sure if notifications go out on updates so I thought I'd another comment.

@dsaff, any thoughts on this proposal?

[Tibor17]

@coreyjv
Basically what you want is something like @BeforeParams and @AfterParams executed only once in lifecycle. Am I right?

[coreyjv]

@Tibor17, essentially yes. I think I'd use a single annotation that allowed rules to execute.

[Tibor17]

If I used the BeforeClass (BC) and AC, then would it look like this ?
BC BP BT T1p1 AT BT T2p1 AT BT T3p1 AT AP BP BT T1p2 AT BT T2p2 AT AC

[coreyjv]

@Tibor17, yes I believe what you've presented is accurate.

[Tibor17]

@coreyjv
I think this feature you want might be interesting. What is your real use case?

[coreyjv]

@Tibor17
I'm currently working on building out a web browser testing suite using Selenium to navigate and inspect page content which is fed to JUnit for test case execution.

My needs resulted in a JUnit test class per page to test for things such as:

• Does a field exist?
• Is it read only?
• Is the label text "X"?

These pages may be at any arbitrary depth inside the application so I put the navigational logic to get to the destination page in a "before" rule (currently @ClassRule). Ideally, I'd like to run the same test with different parameters, which are read and used in in the "setup." These parameters would be items like user identifier and password credentials and other particulars that are relevant to getting to the destination page.

I'll quote a previous response I wrote as to why I want to do it per "lifecycle":

For example if I have a test class that when executed once with a common setup (located in a @ClassRule currently) the entire class executes and completes the tests in ~20 seconds. However, if I move the setup to a method level rule it takes approximately ~240 seconds to execute and complete. That's just for one class. In my situation that increase in time is not acceptable.

[Tibor17]

@coreyjv
means that the params have to be visible within the newly annotated setup as well.
this gives the entire overview of the needs.
Go on.

[coreyjv]

@Tibor17, correct the parameters have to be visible in the setup. What kind of additional clarification or details would you like me to provide?

I think the overview and my previous comments provide a good picture of my need (you can see the mailing list as well) without going into the specifics of the application which I think would add limited value.

[Tibor17]

@coreyjv
Maybe you should now push a pull and include there the people in this issue as well.

[coreyjv]

@Tibor17
That's fine with me I was just looking for an initial approval on the "design" before I proceed.

[Tibor17]

@coreyjv
I am keen to see some test which describes the use, even if the runner's impl is incomplete.
What would the others say about the test...

[coreyjv]

@Tibor17

What would the others say about the test...

Are you looking for me to post a sample test class? It's really be pretty boring since it is just like any other parameterized test (where the parameters are read from a database for example) and "life cycle" setup would be used in place of @ClassRule or @rule setup. As I've previously mentioned a simple case would be setup which used different combinations of usernames, passwords and application roles.

I don't think it's that out of the ordinary. It is just in this case the setup is prohibitively expensive to do before and after each method.

[Tibor17]

@coreyjv

What about doing this in your test

@BeforeParameter
public void static setup() { // flag this call in some variable and assert it's set }

[coreyjv]

@Tibor17
As I mentioned previously this setup needs to be parameterized and would be unique to each set of parameters returned by @parameters so I don't think that approach would work.

[Tibor17]

@coreyjv
I think you already have some snapshot code in your mind.
Do you still want to discuss it here or you want to open a pull?

[coreyjv]

@Tibor17
I'll put it together and submit a pull request. I may not get it done until after the New Year but I will see what I can do. Thanks for the discussion I appreciate it.

[dsaff]

Thoughts:

1. We should get this to happen, and I'm happy to work with interested parties.
2. We can't change the signature of any protected methods. But extracting a protected methodBlock(FrameworkMethod method, Object test) sounds like a very good idea.
3. When writing the ParameterRule interface, keep in mind that I think we need to improve how Rule and ClassRule work. https://docs.google.com/document/d/1B6Z45BSGsY08rZqe2KUZTJ3RVsnUE1-HcXjl5MFm9ls/edit helps to explain where I'd like to see things moving forward: ParameterRule could especially benefit from this kind of rulefactory approach, since it would allow easy reuse of things like TemporaryFolder.
4. I think we can't unilaterally change the instantiation pattern of Parameterized, because current tests may depend on the one-instance-per-test behavior. How can we work around this?

a. Add an annotation or annotation parameter to enable the different behavior.
b. Create a subclass of Parameterized with the different behavior.
c. Another thought would be to create a new instance of the test class for each test, but pass that instance the same parameter object array, which could be operated on by a parameter rule. This would probably be most rewarding if the parameters were bundled into a single object. A potentially crazy idea would be to attach the ParameterRule to the parameter class itself...

I think I prefer c to b, and b to a, but open to discussion.

[coreyjv]

@dsaff

We should get this to happen, and I'm happy to work with interested parties.

Thank you!

We can't change the signature of any protected methods. But extracting a protected methodBlock(FrameworkMethod method, Object test) sounds like a very good idea.

I implemented my requirements this morning, and I ended up not modifying this method signature (I went with the approach of changing certain internal methods from private --> protected).

When writing the ParameterRule interface, keep in mind that I think we need to improve how Rule and ClassRule work. https://docs.google.com/document/d/1B6Z45BSGsY08rZqe2KUZTJ3RVsnUE1-HcXjl5MFm9ls/edit helps to explain where I'd like to see things moving forward: ParameterRule could especially benefit from this kind of rulefactory approach, since it would allow easy reuse of things like TemporaryFolder.

I'll take a look at this and keep it in mind.

I think we can't unilaterally change the instantiation pattern of Parameterized, because current tests may depend on the one-instance-per-test behavior. How can we work around this?

My implementation is conditional on the existence of a @ParameterRule annotation. If it detects a method or field is annotated with this annotation it will change the instantiation pattern of Parameterized.

a. Add an annotation or annotation parameter to enable the different behavior.

See above.

b. Create a subclass of Parameterized with the different behavior.

This could most definitely be implemented as a subclass and I would be fine with that as well. My thought process was to work with the existing class because that seemed to be the intent of the originator.

c. Another thought would be to create a new instance of the test class for each test, but pass that instance the same parameter object array, which could be operated on by a parameter rule. This would probably be most rewarding if the parameters were bundled into a single object. A potentially crazy idea would be to attach the ParameterRule to the parameter class itself...

I'm not sure I follow completely, but could I still achieve the requirement of having a common setup that is executed once per array of parameters returned from the @parameters method?

Would you prefer I submit a pull request so we have actual code to discuss?

[dsaff]

@coreyjv, yes, I think if you have code you're happy to share, that's a great next step. Thanks!

[coreyjv]

@dsaff

Sounds good, I'll try and get a pull request together in the next couple of days.

[ouellettetech]

@dsaff
@coreyjv

Was there ever any resolution to this issue? We have some Junit tests that use the parameter option to specify different browsers, such as IE/Firefox/Chrome, and it was launching a new browser for each test, but to increase the speed of running the tests I was trying to switch to starting and stopping the browsers around a class of tests. If I use the beforeClass and afterClass then there is too much time between one of the parameters finishing and closing the browser, so the remote web driver times out. I would like to use this AfterParameterRun approach, but I see a lot of discussion, and this issue leading to a couple pull requests, but It doesn't look like anything was ever merged into main?
Thanks,
Bradley

[sharpau]

I too have a case where I'd like to see an 'AfterParameter' function.

[bileshg]

@coreyjv Is this functionality in the pipeline because we are running Parameterized Test Cases where we need such annotations. Both @BeforeParam and @AfterParam are required.

[coreyjv]

@bileshg --

Per @dsaff this request was 'mothballed' a few years ago now. I defer to him if he wants to resurface it again. Additionally, I'm not sure if this is something that would be more appropriate for JUnit 5 now (I haven't kept up with the progress there).

[bileshg]

@coreyjv Ok. Thanks for responding. Actually, I needed it for something else - Getting failed test case count as per parameter. Take a look at this question.

[dstaraster]

I see that "@BeforeParam/@AfterParam for Parameterized runner #1435" got merged and added to the 4.13 release notes. Does that mean there's going to be a 4.13 release? When? I'm in dire need of this functionality! I didn't see it in JUnit 5, so I'm wondering when I can expect @BeforeParam/@AfteParam to be available.

[kcooney]

@dstaraster currently most of the JUnit team is focusing on releasing JUnit 5.0. I hope that soon after that we can start on the 4.13 release.

[raubel]

@kcooney I'm also very interested in these @BeforeParam and @AfterParam annotations (and unfortunately not yet ready to migrate my thousands of tests to JUnit5).
Is the 4.13 release still expected?

[kcooney]

@raubel I opened #1496 to discuss a 4.13 release.

JUnit5 can run JUnit4-style tests, so you wouldn't need to migrate all of your tests to use JUnit5 features.