[CI] Add new github commands for quarantining/disabling tests #13558
Conversation
This PR introduces GitHub issue/PR comment commands for managing test attributes: ## New Commands ### Quarantine Tests (flaky tests) - `/quarantine-test <test-name> <issue-url>` - Add [QuarantinedTest] attribute - `/unquarantine-test <test-name>` - Remove [QuarantinedTest] attribute ### Disable Tests (ActiveIssue) - `/disable-test <test-name> <issue-url>` - Add [ActiveIssue] attribute - `/enable-test <test-name>` - Remove [ActiveIssue] attribute ## Usage Commands can be used on: - **Issues**: Creates a new PR with the changes - **Pull Requests**: Pushes changes directly to the PR branch - **With --target-pr**: Pushes to a specified PR regardless of where command is posted ## Examples ``` /quarantine-test Namespace.Class.TestMethod dotnet#1234 /disable-test Namespace.Class.FlakyTest dotnet#5678 /unquarantine-test Namespace.Class.TestMethod /enable-test Namespace.Class.TestMethod ``` ## Implementation Details - Uses QuarantineTools from tools/QuarantineTools for source modifications - Supports both quarantine mode ([QuarantinedTest]) and activeissue mode ([ActiveIssue]) - Validates user has write access before executing - Posts progress comments and links to resulting PR
radical
added
the
area-engineering-systems
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 13558Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 13558" |
There was a problem hiding this comment.
Pull request overview
This PR adds a new GitHub Actions workflow that allows repository maintainers to manage test attributes (quarantine/disable tests) directly through issue or PR comments using slash commands. The workflow supports four commands: /quarantine-test, /unquarantine-test, /disable-test, and /enable-test, which apply [QuarantinedTest] and [ActiveIssue] attributes respectively.
Key Changes
- Implements a comprehensive GitHub Actions workflow with permission checks, argument parsing, and PR creation/update logic
- Supports flexible targeting: creates new PRs from main, pushes to current PR, or pushes to a specified PR via
--target-pr - Includes detailed documentation explaining command syntax, parameters, examples, and behavior
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| .github/workflows/apply-test-attributes.yml | New workflow implementing slash commands for test attribute management with argument parsing, permission validation, QuarantineTools execution, and PR automation |
| .github/workflows/README.md | Documentation for the new workflow commands including syntax, parameters, examples, and behavior specifications |
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
- Replace execSync with spawnSync for git commands to prevent shell injection - Add runGit() helper that passes args as array (no shell interpretation) - Add null check for pr.head.repo when detecting fork PRs - Addresses Copilot review feedback on PR dotnet#13558
| // Branch name is safe: actionVerb is from controlled lookup (Quarantine/Disable/etc), timestamp is numeric | ||
| const branchName = `automated/${actionVerb.toLowerCase()}-test-${timestamp}`; |
There was a problem hiding this comment.
The comment mentions that actionVerb is "from controlled lookup (Quarantine/Disable/etc)" and timestamp is numeric, explaining safety. However, while actionVerb comes from a controlled lookup, it's worth noting that the toLowerCase() call converts it to lowercase before use in the branch name. Consider adding a comment that the branch name format is: automated/{quarantine|unquarantine|disable|enable}-test-{timestamp} to make this more explicit.
| echo "$OUTPUT" | ||
|
|
||
| # Save output for failure comment (escape for GitHub Actions) | ||
| EOF="EOF_$(date +%s%N)" |
There was a problem hiding this comment.
The heredoc delimiter uses a timestamp-based unique identifier to prevent collisions. However, the date command with %N (nanoseconds) is not portable to all Unix systems (e.g., macOS's date doesn't support %N). Since this workflow runs on ubuntu-latest, it works fine, but if portability is needed, consider using $RANDOM or $$ (process ID) instead of %N.
For consistency with the rest of the script and to follow safer practices, use the runGit helper (which uses spawnSync) for git config commands instead of execSync. This avoids shell injection vulnerabilities as a defense-in-depth measure, even though the current hardcoded strings were safe.
2 participants
This PR introduces GitHub issue/PR comment commands for managing test attributes:
New Commands as comments
Quarantine Tests (flaky tests)
/quarantine-test <test-name> <issue-url>- Add [QuarantinedTest] attribute/unquarantine-test <test-name>- Remove [QuarantinedTest] attributeDisable Tests (ActiveIssue)
/disable-test <test-name> <issue-url>- Add [ActiveIssue] attribute/enable-test <test-name>- Remove [ActiveIssue] attributeUsage
Commands can be used on:
Examples
Implementation Details