Repository navigation
Is there a way to output something when the retry succeeds? #205
Description
I have a huge test suite and occasionally on CI builds my test runners go out of memory and fail. This plugin is a great stopgap while I try to diagnose the memory issues.
However, it's super confusing for developers who are casually working in the codebase - the build output makes it look like the test failed. Or worse, there might be 10 retryable failures and one "real" failure and it's hard to pinpoint the actual problem.
Here's some sample output from a recent build:
> Task :test
DeleteActionJspTest > test_content() FAILED
java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:38675/WEB-INF/views/mobile/facets/actions/DeleteAction.jsp
at jsp.test.framework.core.JspServer.render(JspServer.java:71)
at jsp.test.framework.core.JspTestCase.renderJspToElement(JspTestCase.java:111)
at webapp.testframework.CartJspTestCase.lambda$runTests$6(CartJspTestCase.java:174)
at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:185)
at webapp.testframework.CartJspTestCase.runTests(CartJspTestCase.java:172)
at webapp.testframework.CartJspTestCase.runSharedTestsWithDevice(CartJspTestCase.java:163)
at webinf.views.common.facets.actions.DeleteActionJspTest.test_content(DeleteActionJspTest.java:27)
Caused by:
java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:38675/WEB-INF/views/mobile/facets/actions/DeleteAction.jsp
at sun.net.[www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1902)](http://www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1902))
at sun.net.[www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)](http://www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500))
at jsp.test.framework.core.JspServer.render(JspServer.java:69)
... 6 more
6934 tests completed, 1 failed, 3 skipped
There were failing tests. See the report at: file:///local/p4clients/pkgbuild-vU1LG/workspace/src/CartExperienceWebapp/build/brazil-unit-tests/index.html
> Task :preCoverageReport
it would be great if I could insert a line right after the failure saying something like "DeleteActionJspTest passed on retry #1" or similar. Is that already possible? If not, how hard is it to code up?
(would also be nice to be able to print a "10 tests passed on retry" or similar in the summary part)
I'm open to contributing the code if you are aligned with this being a good idea. Let me know, thanks!
Activity
pshevche commentedon Apr 2, 2024
@devvmh , thank you for making this proposal. Have you looked at Gradle Build Scans: https://scans.gradle.com/?
It seems like they can help you answer your questions and will also provide a more user-friendly interface than a console output.
Here is an example for a flaky test in JUnit codebase, which shows all "flaky" tests in a build (i.e., ones that passed on retry): https://ge.junit.org/s/tcwmbmjybjktg/tests/overview?outcome=FLAKY. You can also navigate to individual methods and get details about individual retries: https://ge.junit.org/s/tcwmbmjybjktg/tests/task/:junit-jupiter-engine:test/details/org.junit.jupiter.engine.extension.TimeoutExtensionTests$SeparateThread/oneThreadStuckForeverAndOtherTestsInSameThread()?top-execution=1
Feel free to reopen the issue if you feel like the Build Scan does not cover your use case.
devvmh commentedon Apr 3, 2024
Build Scans don't cover my use case. For me, as the application owner, it's not too hard to learn about this edge case and remember it. However, there's a population of developers (around 100-200 per year, with some repeat offenders and many new each year) who are contributing more casually, or even once total in their lifetime. For them, they are overwhelmed with documentation and wikis to read to learn about contribution, so being aware of this problem is the issue. Build scans might help me target the flaky tests, but I already know which tests are flaky (it's the JSP tests that spin up a Tomcat server, and the issue is with the test framework I'm using, not with any individual test).
For them, they are likely to be exposed to this problem for the first time by seeing the sample build output I shared above, which is very confusing. I could add a target right before :test to say "hey, check build scans" or just explain this problem, but seems a bit heavy handed to me
(by the way, the problem has gotten better since I posted this last August, and I'm not really sure why. I might end up dropping the plugin and just relying on retrying the full build when a test is flaky)
So summarizing
pshevche commentedon Apr 3, 2024
@devvmh , I see; thank you for explaining your use case. I understand first-time contributors will likely pay little attention to the Build Scan, as it requires a certain habit to be developed. We have a couple of requests in the queue regarding adding callbacks to the plugin, allowing users to perform arbitrary actions on retries or when they complete. I linked this issue to the tracking epic: #271. I feel like you'd be able to achieve your use case by logging something in the callback action.
devvmh commentedon Sep 20, 2024
I learned more about this since April and I found that my company doesn't allow the usage of https://scans.gradle.com/ :(