Skip to content

chore: add Github live sync integration tests #542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FelixTJDietrich merged 7 commits into from
Nov 9, 2025
Merged

chore: add Github live sync integration tests #542

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
41192cf
chore: add github integration test
FelixTJDietrich Nov 7, 2025
3e4d64e
Merge branch 'develop' into chore/add-sync-github-integration-test
FelixTJDietrich Nov 7, 2025
2d3b7d0
update docs
FelixTJDietrich Nov 7, 2025
c53085a
update
FelixTJDietrich Nov 7, 2025
39cc6dc
fix ci
FelixTJDietrich Nov 7, 2025
18601e2
fix reflection
FelixTJDietrich Nov 9, 2025
16d9a9f
final changes
FelixTJDietrich Nov 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,7 @@ venv/
ENV/
env.bak/
venv.bak/
/server/application-server/src/test/resources/application-github-integration-local.yml

# Spyder project settings
.spyderproject
Expand Down
50 changes: 50 additions & 0 deletions docs/contributor/testing.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,3 +96,53 @@ Available examples include `label.created`, `repository.created`, `create`, `pus

Use the optional `-Dgroups=unit` or `-Dgroups=integration` flags once category support lands.

## GitHub live sync integration tests

Some regression scenarios can only be validated against GitHub itself. We ship a focused suite that exercises the live GitHub App installation and verifies end-to-end sync behaviour (repository metadata, labels, milestones, and teams).

### Prerequisites

1. **Sandbox installation** – the `Hephaestus IntegrationTests` GitHub App must be installed in a sandbox organisation you control. The tests create and delete repositories, milestones, labels, and teams on each run.
2. **Credentials** – provide both a GitHub App private key and a Personal Access Token with the following scopes:
- `repo` (full)
- `admin:org`
- `read:packages`
3. **Local config file** – copy the template that lives alongside the tests:

```bash
cd server/application-server/src/test/resources
cp application-github-integration-local.example.yml application-github-integration-local.yml
```

Fill in the placeholders with the sandbox organisation slug, the installation id, and either an inline PEM key (`github.app.privateKey`) or a readable `privateKeyLocation`. Keep this file out of version control—it is already listed in `.gitignore`.
Comment on lines +113 to +117
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Nov 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Verify configuration property names match implementation.

The section references an inline github.app.privateKey property and privateKeyLocation. Confirm these property names match the actual application configuration schema and that the documentation accurately reflects supported configuration options.

🏁 Script executed:

#!/bin/bash
# Description: Verify GitHub app property names in configuration

# Search for property definitions and usage
rg "github\.app\.privateKey|privateKeyLocation" --type=java --type=yaml -B 2 -A 2 | head -50

Length of output: 5687

Update documentation to use the correct property name github.app.privateKeyLocation.

The documentation references privateKeyLocation, but the actual configuration property is github.app.privateKeyLocation. Update line 117 to use the full qualified property name for consistency with the inline github.app.privateKey reference.

🧰 Tools
🪛 LanguageTool

[uncategorized] ~117-~117: The official name of this software platform is spelled with a capital “H”.
Context: ...ation id, and either an inline PEM key (github.app.privateKey) or a readable `private...

(GITHUB)

🤖 Prompt for AI Agents 
In docs/contributor/testing.mdx around lines 113 to 117, the docs mention the
config key `privateKeyLocation` but the correct fully-qualified property is
`github.app.privateKeyLocation`; update the text on line 117 to reference
`github.app.privateKeyLocation` (so it matches the inline
`github.app.privateKey` reference) and ensure the example and explanatory
sentence use the fully qualified name.


Alternatively, export the matching environment variables:

```bash
export GH_IT_APP_ID=2250297
export GH_IT_APP_PAT=ghp_xxx... # PAT with the scopes above
export GH_IT_INSTALLATION_ID=93512943
export GH_IT_ORGANIZATION=HephaestusTest
export GH_IT_APP_PRIVATE_KEY_PATH=/absolute/path/to/private-key.pem
```

Use either the file _or_ the environment variables; the suite checks both and aborts if key material is missing.

### Running the suite

From `server/application-server/` run:

```bash
./mvnw test -Pgithub-integration-tests -Dgroups=github-integration
```

The `github-integration-tests` profile clears the default exclusion so the live suite can execute; the `-Dgroups` flag narrows execution to just those classes. Run the regular unit/integration suites separately via `./mvnw test`.

The run takes roughly two minutes and prints the GitHub artefacts it provisions. Clean-up is handled automatically, but if a failure interrupts execution you can safely delete any `hephaestus-it-*` repositories, milestones, or teams that remain in the sandbox.

### Troubleshooting

- **Skipping because of missing credentials** – check the console output; the base test class verifies that the App id, private key, PAT, and installation id are all present before executing.
- **Hub4j rate-limit failures** – the suite creates several entities per run. Prefer a dedicated sandbox organisation so you do not clash with production automation limits.
- **Longer runtimes** – each suite bootstraps a Testcontainers PostgreSQL instance and provisions GitHub resources. Expect higher runtimes than the pure Testcontainers integration tests; avoid running them on every PR and instead use them before releases or when touching the GitHub sync layer.

210 changes: 210 additions & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,10 @@
"devDependencies": {
"@openapitools/openapi-generator-cli": "2.16.3",
"npm-run-all": "^4.1.5",
"patch-package": "^8.0.1",
"prettier": "3.5.1",
"prettier-plugin-java": "2.6.7",
"prettier-plugin-kotlin": "2.1.0",
"shx": "0.3.4"
}
}
}
12 changes: 10 additions & 2 deletions server/application-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@
<!-- Test Configuration -->
<maven.test.failure.ignore>false</maven.test.failure.ignore>
<maven.test.skip>false</maven.test.skip>
<hephaestus.surefire.timeout>60</hephaestus.surefire.timeout>
<hephaestus.surefire.timeout>180</hephaestus.surefire.timeout>
<hephaestus.githubIntegrationExcludedGroups>github-integration</hephaestus.githubIntegrationExcludedGroups>
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Nov 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Verify test group exclusion strategy and profile activation semantics.

The approach uses a property-driven exclusion model:

  • Default: githubIntegrationExcludedGroups=github-integration excludes GitHub integration tests from Surefire
  • Profile github-integration-tests: activated by presence of groups property, sets excludedGroups to empty

However, the integration is unclear:

  1. The profile activation property <name>groups</name> is generic and matches any -Dgroups=... flag, but doesn't explicitly control which groups to run
  2. Setting excludedGroups to empty allows all tests to run, but relies on integration tests being properly tagged with @Tag("github-integration")
  3. Contrast with Failsafe (line 452), which uses <groups>integration</groups> to explicitly include tests

Please verify:

  • Are GitHub integration tests consistently tagged with @Tag("github-integration")?
  • Does setting an empty excludedGroups string behave as expected in Surefire?
  • Is the activation on the groups property sufficient, or should the profile be activated more explicitly?

Run the following to verify test tagging and the profile setup:

Also applies to: 438-438, 562-572

🏁 Script executed:

#!/bin/bash
# Description: Verify GitHub integration tests are tagged correctly and profile activation works

# Find all test files under gitprovider/github/
echo "=== GitHub integration test files ==="
find server/application-server/src/test/java -path '*/github/*' -name '*Test.java' -o -name '*IntegrationTest.java' | head -20

# Check for @Tag("github-integration") annotations in these files
echo "=== Checking for @Tag(\"github-integration\") annotations ==="
rg -n '@Tag\s*\(\s*["\x27]github-integration["\x27]\s*\)' server/application-server/src/test/java/de/tum/in/www1/hephaestus/gitprovider/github/

# Verify that integration test files exist and check their structure
echo "=== Sample of integration test classes ==="
find server/application-server/src/test/java -path '*/github/*' -name '*IntegrationTest.java' -exec head -30 {} \; | head -60

Length of output: 6291

I need to verify the inheritance model for the @tag annotation and check the pom.xml profile configuration more closely.

Now I need to check the pom.xml configuration to understand the profile setup and the Surefire/Failsafe configuration more clearly.

🏁 Script executed:

#!/bin/bash
# Extract pom.xml sections around the mentioned lines for analysis

echo "=== Maven profiles section (around line 438) ==="
sed -n '430,450p' server/application-server/pom.xml

echo ""
echo "=== Maven profiles section (around line 562-572) ==="
sed -n '555,580p' server/application-server/pom.xml

echo ""
echo "=== Search for profile 'github-integration-tests' ==="
rg -n -A15 'github-integration-tests' server/application-server/pom.xml

Length of output: 2479

Now I need to check the JUnit 5 version in use and verify the test inheritance chain:

Now I need to check the inheritance chain and JUnit version to understand the complete picture:

🏁 Script executed:

#!/bin/bash
# Check JUnit version and test inheritance chain

echo "=== JUnit Jupiter version in pom.xml ==="
rg -n 'junit-jupiter-api|junit-jupiter-engine' server/application-server/pom.xml | head -5

echo ""
echo "=== BaseGitHubIntegrationTest source ==="
cat server/application-server/src/test/java/de/tum/in/www1/hephaestus/gitprovider/github/BaseGitHubIntegrationTest.java

echo ""
echo "=== BaseIntegrationTest source (first 40 lines) ==="
find server/application-server/src/test/java -name 'BaseIntegrationTest.java' -exec head -40 {} \;

Length of output: 6278

Confirm tagging issue: GitHub integration tests are NOT properly tagged for exclusion.

The concrete GitHub integration test classes (GitHubInstallationMessageHandlerIntegrationTest, GitHubIssueMessageHandlerIntegrationTest, etc.) extend BaseIntegrationTest and inherit only @Tag("integration"). They do NOT inherit @Tag("github-integration") because BaseGitHubIntegrationTest is separate and not in their inheritance chain.

Since Surefire's default excludedGroups=github-integration targets the "github-integration" tag specifically, these tests will NOT be excluded by default and will run during normal unit test phases—contrary to the intended behavior of keeping GitHub integration tests separate unless explicitly activated.

The inheritance structure should be:

  • Concrete tests → BaseGitHubIntegrationTest → BaseIntegrationTest

Or each concrete test must carry an explicit @Tag("github-integration") annotation.

🤖 Prompt for AI Agents 
In server/application-server/pom.xml around line 39, the Maven Surefire
exclusion for github-integration groups is ineffective because concrete GitHub
integration tests don't carry the "github-integration" tag (they only inherit
@Tag("integration") from BaseIntegrationTest); update the test inheritance or
annotations so Surefire excludes work: either make concrete test classes extend
BaseGitHubIntegrationTest (so inheritance becomes Concrete →
BaseGitHubIntegrationTest → BaseIntegrationTest) or add explicit
@Tag("github-integration") to each concrete GitHub integration test class; apply
the chosen fix consistently to all GitHub integration test classes so they are
excluded by the existing excludedGroups setting.


<!-- Coverage Configuration (disabled by default) -->
<skipCoverage>true</skipCoverage>
Expand Down Expand Up @@ -433,7 +434,8 @@
<skipAfterFailureCount>5</skipAfterFailureCount>
<!-- Forked JVM timeout settings to prevent hanging -->
<forkedProcessTimeoutInSeconds>${hephaestus.surefire.timeout}</forkedProcessTimeoutInSeconds>
<forkedProcessExitTimeoutInSeconds>10</forkedProcessExitTimeoutInSeconds>
<forkedProcessExitTimeoutInSeconds>60</forkedProcessExitTimeoutInSeconds>
<excludedGroups>${hephaestus.githubIntegrationExcludedGroups}</excludedGroups>
</configuration>
</plugin>

Expand Down Expand Up @@ -557,5 +559,11 @@
<hephaestus.surefire.timeout>180</hephaestus.surefire.timeout>
</properties>
</profile>
<profile>
<id>github-integration-tests</id>
<properties>
<hephaestus.githubIntegrationExcludedGroups></hephaestus.githubIntegrationExcludedGroups>
</properties>
</profile>
</profiles>
</project>
Loading
Loading