-
-
Notifications
You must be signed in to change notification settings - Fork 31.9k
gh-130474: Create and implement flaky resource #130489
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some tests are not necessarily flaky. You should also update the command-line of libregretest python -m test --help
Misc/NEWS.d/next/Tests/2025-02-23-21-20-52.gh-issue-130474.dEckCL.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Bénédikt Tran <[email protected]>
Oh, the docs are failing. I thought that we had documented it but no. My bad. |
Misc/NEWS.d/next/Tests/2025-02-23-21-20-52.gh-issue-130474.dEckCL.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, so I'm not sure we should skip testSend
for now because while it's marked as flaky, I've never seen it fail beforehand.
OTOH, test_stress_modifying_handlers
is only flaky on free-threaded build so, if possible, we should not mark test_stress_modifying_handlers
as flaky at all (testing on the regular would not be flaky at all!) Sorry for missing the point before but for my defense it's 1 AM here!
As for test_selflink
, it's the only test that is really flaky actually and skipped (it's usually not flaky on an idle system, it's flaky on a stressed system). I would leave marking flaky tests in a separate PR (except for test_selflink
).
Co-authored-by: Bénédikt Tran <[email protected]>
Co-authored-by: Bénédikt Tran <[email protected]>
Misc/NEWS.d/next/Tests/2025-02-23-21-20-52.gh-issue-130474.dEckCL.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Adam Turner <[email protected]>
Looks ok, but Victor is probably the best one to review changes to libregrtest. A |
Reading the linked issue in the comment for |
It is though? at least according to my readings of #109959 (comment). Anyway, the thing is that it's still flaky as it randomly fails. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dislike this change, If a test is known to be flaky, it should be either fixed or removed.
I suggest to remove test_selflink() in test_glob.
I created #130551 to remove the flaky test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that this "flaky" resource is a good idea.
I created #130551 to remove the flaky test.
I merged my PR.
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
In light of #130474 (comment), I think it was a bad idea (my bad idea). At first, it seemed great but hiding a flaky test behind a decorator wouldn't help solving the problem, we could even forget about it! So I'll close the issue and this PR as the flaky tests I thought were flaky are being fixed (most of them happen on the free-threaded build and seem to be real issues, namely we should have written the tests differently). Thanks for the work though and sorry again! |
I've created the resource and implemented it in some obvious places where comments pointed to flaky tests. There were certain tests I left alone as although they were marked flaky by inline comments, there were other decorators addressing this, for example certain tests that were marked as being flaky without the GIL are already marked with the
@support.requires_gil_enabled
decorator which I think makes more sense.As with other resources, the use-case for decorating tests is just:
With this set-up, in CI environments we can configure the CI to run flaky tests separately and to automatically retry failed flaky tests::
python -m test --use=flaky --rerun