-
Notifications
You must be signed in to change notification settings - Fork 765
[CI] Add new github commands for quarantining/disabling tests #13558
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
0e039d5
[CI] Add new github commands for quarantining/disabling tests
radical 858677d
Update README.md
radical 3839057
Update apply-test-attributes.yml
radical 7f753d2
Update .github/workflows/apply-test-attributes.yml
radical 5382599
Fix shell injection vulnerabilities and null check in workflow
radical 05d51e4
Pin actions/github-script to SHA for security best practices
radical 9a92868
Replace execSync with spawnSync for git config commands
radical f5e923b
Use os.tmpdir() fallback for cross-platform temp directory compatibility
radical 6cab6e3
Update ci.yml
radical 221fbda
add zizmor ignore
radical 7662f7f
Use runId instead of timestamp for branch name uniqueness
radical 9abe2fa
Add to copilot instructions
radical File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # GitHub Workflows | ||
|
|
||
| ## Quarantine/Disable Test Workflow | ||
|
|
||
| The `quarantine-test.yml` workflow allows repository maintainers to quarantine, unquarantine, disable, or enable tests directly from issue or PR comments. | ||
|
|
||
| ### Commands | ||
|
|
||
| | Command | Description | Attribute Used | | ||
| |---------|-------------|----------------| | ||
| | `/quarantine-test` | Mark test(s) as quarantined (flaky) | `[QuarantinedTest]` | | ||
| | `/unquarantine-test` | Remove quarantine from test(s) | Removes `[QuarantinedTest]` | | ||
| | `/disable-test` | Disable test(s) due to an active issue | `[ActiveIssue]` | | ||
| | `/enable-test` | Re-enable previously disabled test(s) | Removes `[ActiveIssue]` | | ||
|
|
||
| ### Syntax | ||
|
|
||
| ``` | ||
| /quarantine-test <test-name(s)> <issue-url> [--target-pr <pr-url>] | ||
| /unquarantine-test <test-name(s)> [--target-pr <pr-url>] | ||
| /disable-test <test-name(s)> <issue-url> [--target-pr <pr-url>] | ||
| /enable-test <test-name(s)> [--target-pr <pr-url>] | ||
| ``` | ||
|
|
||
| ### Parameters | ||
|
|
||
| | Parameter | Required | Description | | ||
| |-----------|----------|-------------| | ||
| | `<test-name(s)>` | Yes | One or more test method names (space-separated) | | ||
| | `<issue-url>` | For quarantine/disable | URL of the GitHub issue tracking the problem | | ||
| | `--target-pr <pr-url>` | No | Push changes to an existing PR instead of creating a new one | | ||
|
|
||
| ### Examples | ||
|
|
||
| #### Quarantine a flaky test (creates new PR) | ||
| ``` | ||
| /quarantine-test MyTestClass.MyTestMethod https://github.com/dotnet/aspire/issues/1234 | ||
| ``` | ||
|
|
||
| #### Quarantine multiple tests | ||
| ``` | ||
| /quarantine-test TestMethod1 TestMethod2 TestMethod3 https://github.com/dotnet/aspire/issues/1234 | ||
| ``` | ||
|
|
||
| #### Quarantine a test and push to an existing PR | ||
| ``` | ||
| /quarantine-test MyTestMethod https://github.com/dotnet/aspire/issues/1234 --target-pr https://github.com/dotnet/aspire/pull/5678 | ||
| ``` | ||
|
|
||
| #### Unquarantine a test (creates new PR) | ||
| ``` | ||
| /unquarantine-test MyTestClass.MyTestMethod | ||
| ``` | ||
|
|
||
| #### Unquarantine and push to an existing PR | ||
| ``` | ||
| /unquarantine-test MyTestMethod --target-pr https://github.com/dotnet/aspire/pull/5678 | ||
| ``` | ||
|
|
||
| #### Disable a test due to an active issue | ||
| ``` | ||
| /disable-test MyTestMethod https://github.com/dotnet/aspire/issues/1234 | ||
| ``` | ||
|
|
||
| #### Enable a previously disabled test | ||
| ``` | ||
| /enable-test MyTestMethod | ||
| ``` | ||
|
|
||
| #### Comment on a PR to push changes to that PR | ||
| When you comment on a PR (not an issue), the workflow will automatically push changes to that PR's branch instead of creating a new PR. You can override this by specifying `--target-pr`. | ||
|
|
||
| ### Behavior | ||
|
|
||
| 1. **Permission Check**: Only users with write access to the repository can use these commands. | ||
| 2. **Processing Indicator**: The workflow adds an 👀 reaction to your comment when it starts processing. | ||
| 3. **Status Comments**: The workflow posts comments to indicate: | ||
| - ⏳ Processing started | ||
| - ✅ Success (with link to created/updated PR) | ||
| - ℹ️ No changes needed (test already in desired state) | ||
| - ❌ Failure (with error details) | ||
|
|
||
| ### Target PR Behavior | ||
|
|
||
| | Context | `--target-pr` specified | Result | | ||
| |---------|-------------------------|--------| | ||
| | Comment on Issue | No | Creates new PR from `main` | | ||
| | Comment on Issue | Yes | Pushes to specified PR | | ||
| | Comment on PR | No | Pushes to that PR's branch | | ||
| | Comment on PR | Yes | Pushes to specified PR (overrides) | | ||
|
|
||
| ### Restrictions | ||
|
|
||
| - The `--target-pr` URL must be from the same repository | ||
| - Cannot push to PRs from forks | ||
| - Cannot push to closed PRs | ||
| - The PR branch must not be protected in a way that prevents pushes | ||
|
|
||
| ### Concurrency | ||
|
|
||
| The workflow uses concurrency groups based on the issue/PR number to prevent race conditions when multiple commands are issued on the same issue. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.