Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 68 additions & 1 deletion .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,74 @@ These switches can be repeated to run tests on multiple classes or methods at on

Example: `[QuarantinedTest("..issue url..")]`

- To quarantine or unquarantine tests, use the tool in `tools/QuarantineTools/QuarantineTools.csproj`.
### Quarantine/Unquarantine via GitHub Commands (Preferred)

Use these commands in any issue or PR comment. They require write access to the repository.

```bash
# Quarantine a flaky test (creates a new PR)
/quarantine-test Namespace.Type.Method https://github.com/dotnet/aspire/issues/1234

# Quarantine multiple tests at once
/quarantine-test TestMethod1 TestMethod2 https://github.com/dotnet/aspire/issues/1234

# Quarantine and push to an existing PR
/quarantine-test TestMethod https://github.com/dotnet/aspire/issues/1234 --target-pr https://github.com/dotnet/aspire/pull/5678

# Unquarantine a test (creates a new PR)
/unquarantine-test Namespace.Type.Method

# Unquarantine and push to an existing PR
/unquarantine-test TestMethod --target-pr https://github.com/dotnet/aspire/pull/5678
```

When you comment on a PR, the changes are automatically pushed to that PR's branch (no need for `--target-pr`).

### Quarantine/Unquarantine via Local Tool

For local development, use the QuarantineTools directly:

```bash
# Quarantine a test
dotnet run --project tools/QuarantineTools -- -q -i https://github.com/dotnet/aspire/issues/1234 Full.Namespace.Type.Method

# Unquarantine a test
dotnet run --project tools/QuarantineTools -- -u Full.Namespace.Type.Method
```

## Disabled tests (ActiveIssue)

- Tests that consistently fail due to a known bug or infrastructure issue are marked with the `ActiveIssue` attribute.
- These tests are completely skipped until the underlying issue is resolved.
- Use this for tests that are **blocked**, not for flaky tests (use `QuarantinedTest` for flaky tests).

Example: `[ActiveIssue("https://github.com/dotnet/aspire/issues/1234")]`

### Disable/Enable via GitHub Commands (Preferred)

```bash
# Disable a test due to an active issue (creates a new PR)
/disable-test Namespace.Type.Method https://github.com/dotnet/aspire/issues/1234

# Disable and push to an existing PR
/disable-test TestMethod https://github.com/dotnet/aspire/issues/1234 --target-pr https://github.com/dotnet/aspire/pull/5678

# Enable a previously disabled test (creates a new PR)
/enable-test Namespace.Type.Method

# Enable and push to an existing PR
/enable-test TestMethod --target-pr https://github.com/dotnet/aspire/pull/5678
```

### Disable/Enable via Local Tool

```bash
# Disable a test with ActiveIssue
dotnet run --project tools/QuarantineTools -- -q -m activeissue -i https://github.com/dotnet/aspire/issues/1234 Full.Namespace.Type.Method

# Enable a test (remove ActiveIssue)
dotnet run --project tools/QuarantineTools -- -u -m activeissue Full.Namespace.Type.Method
```

## Outerloop tests

Expand Down
101 changes: 101 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# GitHub Workflows

## Quarantine/Disable Test Workflow

The `apply-test-attributes.yml` workflow allows repository maintainers to quarantine, unquarantine, disable, or enable tests directly from issue or PR comments.

### Commands

| Command | Description | Attribute Used |
|---------|-------------|----------------|
| `/quarantine-test` | Mark test(s) as quarantined (flaky) | `[QuarantinedTest]` |
| `/unquarantine-test` | Remove quarantine from test(s) | Removes `[QuarantinedTest]` |
| `/disable-test` | Disable test(s) due to an active issue | `[ActiveIssue]` |
| `/enable-test` | Re-enable previously disabled test(s) | Removes `[ActiveIssue]` |

### Syntax

```
/quarantine-test <test-name(s)> <issue-url> [--target-pr <pr-url>]
/unquarantine-test <test-name(s)> [--target-pr <pr-url>]
/disable-test <test-name(s)> <issue-url> [--target-pr <pr-url>]
/enable-test <test-name(s)> [--target-pr <pr-url>]
```

### Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `<test-name(s)>` | Yes | One or more test method names (space-separated) |
| `<issue-url>` | For quarantine/disable | URL of the GitHub issue tracking the problem |
| `--target-pr <pr-url>` | No | Push changes to an existing PR instead of creating a new one |

### Examples

#### Quarantine a flaky test (creates new PR)
```
/quarantine-test MyTestClass.MyTestMethod https://github.com/dotnet/aspire/issues/1234
```

#### Quarantine multiple tests
```
/quarantine-test TestMethod1 TestMethod2 TestMethod3 https://github.com/dotnet/aspire/issues/1234
```

#### Quarantine a test and push to an existing PR
```
/quarantine-test MyTestMethod https://github.com/dotnet/aspire/issues/1234 --target-pr https://github.com/dotnet/aspire/pull/5678
```

#### Unquarantine a test (creates new PR)
```
/unquarantine-test MyTestClass.MyTestMethod
```

#### Unquarantine and push to an existing PR
```
/unquarantine-test MyTestMethod --target-pr https://github.com/dotnet/aspire/pull/5678
```

#### Disable a test due to an active issue
```
/disable-test MyTestMethod https://github.com/dotnet/aspire/issues/1234
```

#### Enable a previously disabled test
```
/enable-test MyTestMethod
```

#### Comment on a PR to push changes to that PR
When you comment on a PR (not an issue), the workflow will automatically push changes to that PR's branch instead of creating a new PR. You can override this by specifying `--target-pr`.

### Behavior

1. **Permission Check**: Only users with write access to the repository can use these commands.
2. **Processing Indicator**: The workflow adds an 👀 reaction to your comment when it starts processing.
3. **Status Comments**: The workflow posts comments to indicate:
- ⏳ Processing started
- ✅ Success (with link to created/updated PR)
- ℹ️ No changes needed (test already in desired state)
- ❌ Failure (with error details)

### Target PR Behavior

| Context | `--target-pr` specified | Result |
|---------|-------------------------|--------|
| Comment on Issue | No | Creates new PR from `main` |
| Comment on Issue | Yes | Pushes to specified PR |
| Comment on PR | No | Pushes to that PR's branch |
| Comment on PR | Yes | Pushes to specified PR (overrides) |

### Restrictions

- The `--target-pr` URL must be from the same repository
- Cannot push to PRs from forks
- Cannot push to closed PRs
- The PR branch must not be protected in a way that prevents pushes

### Concurrency

The workflow uses concurrency groups based on the issue/PR number to prevent race conditions when multiple commands are issued on the same issue.
Loading