[CI] Automatic rerun of certain test failures

[gante]

What does this PR do?

Adds pytest-rerunfailures to our push CI. This should help stabilize our CI and allow faster dev cycles 🤗

In a nutshell, if a test failure has a substring in the newly added FLAKY_TEST_FAILURE_PATTERNS variable (.circleci/create_circleci_config.py), that test is repeated up to 5 times. FLAKY_TEST_FAILURE_PATTERNS in this PR includes the following categories of failures:

1. 'OSError' / 'Timeout' -- generic transient machine/connection errors
2. 'HTTPError.*502' / 'HTTPError.*504' -- hub HTTP failures
3. "AssertionError: Tensor-likes are not close!" -- unlucky torch.testing.assert_close cases
4. [several] -- exceptions thrown after a hub failure handling

Thank you for the suggestion @Wauplin 💛

---

Follow up: remove some uses of @is_flaky(), since they serve the same purpose:

1. Tests with old @is_flaky() additions that I couldn't reproduce with --flake-finder --flake-runs 1000
2. Tests with @is_flaky() that were tagged because one of the criteria in the list above

See diff in this commit

---

Corresponding PR for docker image update: #36427

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Rocketknight1]

A little confused about this line - is this to ensure the tests get fixed now rather than waiting for a release?

[gante]

No, this is to build the dev image correctly :)

Previous version of these dockerfiles: we installed from pypi's transformers, the latest release.
This version: we install from the latest commit in the REF branch, REF being the branch that pushes a commit.

In other words, these images were not affected by the changes in setup.py until a release, even when building new dev images. This pattern exists in all docker images, but was missing on these. As a result, before the changes in the dockerfiles, CI crashed because these images didn't have the new testing requirement :)

[ydshieh]

I think we need to add pytest-rerunfailures to

extras["testing"] = (

and make sure all docker files (used for circleci test jobs) have testing.

Am I right ?

Nice for

git+https://github.com/huggingface/transformers.git@${REF}

[gante]

I think we need to add pytest-rerunfailures to extras["testing"]

@ydshieh good catch! I added it to the PR that builds the dev image, but forgot to add it here. Added ✅

[gante]

image pushed in #36427 , merging

[ArthurZucker]

thanks all for working on this!