feat(ci): extend shuffle-tests with automatic detect-test-pollution#108286
Draft
feat(ci): extend shuffle-tests with automatic detect-test-pollution#108286
Conversation
github-actions
bot
added
the
Scope: Backend
joshuarli
changed the title
…revent test pollution Partition flag validation in `_cleanup()` was happening after `_validate_and_setup_environment()`, which sets `os.environ["_SENTRY_CLEANUP"]`. When validation raised a ClickException, the env var was left set, causing subsequent tests to skip error event deletion (the env var makes group deletion skip `ErrorEventsDeletionTask`).
The test_put_self_hosted_superuser_access_allowed test sets auth.allow-registration to 1 via the API, which persists in the database across tests when using --reuse-db. This pollutes test_customer_domain_feature which expects only organizations:create and system:multi-region in features, but gets an unexpected auth:register feature.
…uffle-tests-across-shards
…huffle-tests-across-shards
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This extends the shuffle-tests job to, on test failure, run
detect-test-pollution.pywhich automatically bisects down to the polluting test.That bisection process would otherwise tie up a local dev environment for ~1 hour (numerous pytest runs needed) and cannot be parallelized (unless we have disposable dev environment VMs - which is basically what CI is today).
This'll make shuffle-tests much, much, much more actionable and I expect to very quickly burn down any test pollution.
This also:
Manually triggered workflow runs are here (start from #22)