🌱 Automate solution server test #657
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alejandro Brugarolas <[email protected]>
Signed-off-by: Alejandro Brugarolas <[email protected]>
Signed-off-by: Alejandro Brugarolas <[email protected]>
WalkthroughThis change introduces a comprehensive end-to-end test suite for validating the integration of a solution server with a VSCode extension and an MCP backend. New TypeScript enums and interfaces are added to support configuration and response modeling. The VSCode page object and related utilities are refactored and extended to facilitate settings manipulation, extension installation, and solution request workflows. A new test suite automates the process of running analysis, requesting fixes, accepting/rejecting solutions, and verifying backend state via MCP client queries. Supporting documentation and configuration files are updated to reflect new features and file renames. Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Assessment against linked issues
Assessment against linked issues: Out-of-scope changes
Possibly related PRs
Suggested reviewers
Note 🔌 MCP (Model Context Protocol) integration is now available in Early Access!Pro users can now connect to remote MCP servers under the Integrations page to get reviews and chat conversations that understand additional development context. 📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (8)
tests/e2e/pages/base-page.page.ts (1)
31-33: Prefer event-driven waits over fixed timeoutsA hard-coded sleep (
waitForTimeout) is brittle and lengthens the suite unnecessarily on fast machines. Consider waiting for a deterministic UI state (e.g. selector visible, IPC event, workspace ready) instead of an arbitrary 3-5 s pause.- public async waitDefault() { - await this.window.waitForTimeout(process.env.CI ? 5000 : 3000); - } + public async waitUntilReady() { + await this.window.waitForSelector('#workbench.parts.statusbar', { state: 'visible', timeout: 10000 }); + }Retain the fallback timeout only where a reliable signal is unavailable.
tests/package.json (1)
122-123: Move AI/MCP SDKs todevDependencies
@anthropic-ai/sdkand@modelcontextprotocol/sdkare used exclusively by the E2E test harness and are not required at runtime by the published extension. Declaring them under"dependencies"bloats the production install size.- "dependencies": { + "dependencies": { … - "yauzl": "^2.10.0", - "@anthropic-ai/sdk": "^0.57.0", - "@modelcontextprotocol/sdk": "^1.17.1" + "yauzl": "^2.10.0" + }, + "devDependencies": { + "@anthropic-ai/sdk": "^0.57.0", + "@modelcontextprotocol/sdk": "^1.17.1" },tests/e2e/enums/configuration-options.enum.ts (1)
1-3: Considerconst enum(or literal type) for zero-runtime costWith a single key, a regular
enumemits extra JS. Aconst enum(orexport const SolutionServerEnabled = '…' as const) keeps type-safety while tree-shaking the runtime object.-export enum ConfigurationOptions { - SolutionServerEnabled = 'konveyor.solutionServer.enabled', -} +export const enum ConfigurationOptions { + SolutionServerEnabled = 'konveyor.solutionServer.enabled', +}Use plain enums only when you need runtime iteration.
tests/global.setup.ts (1)
7-10: Consider extracting hardcoded repository values to constants.The hardcoded repository URL and name should be extracted to constants for better maintainability and reusability across the test suite.
+const DEFAULT_TEST_REPO_URL = 'https://github.com/konveyor-ecosystem/coolstore'; +const DEFAULT_TEST_REPO_NAME = 'coolstore'; + async function globalSetup() { console.log('Running global setup...'); - const vscodeApp = await VSCode.init( - 'https://github.com/konveyor-ecosystem/coolstore', - 'coolstore' - ); + const vscodeApp = await VSCode.init(DEFAULT_TEST_REPO_URL, DEFAULT_TEST_REPO_NAME);tests/e2e/pages/configuration.page.ts (1)
16-16: Extract hardcoded extension name to a constant.The extension name
@ext:konveyor.konveyor-aiis hardcoded. Consider extracting it to a constant for better maintainability.Add a constant at the top of the file:
const KONVEYOR_EXTENSION_FILTER = '@ext:konveyor.konveyor-ai';Then update line 16:
- await searchInput.pressSequentially('@ext:konveyor.konveyor-ai'); + await searchInput.pressSequentially(KONVEYOR_EXTENSION_FILTER);tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts (1)
128-133: Remove console.log from test code.Debug logging should be removed from committed test code.
- console.log( - await analysisView - .locator(`#javax-to-jakarta-import-00001-${accept ? 'accepted' : 'rejected'}-solutions`) - .innerText() - );tests/mcp-client/mcp-client.model.ts (2)
97-97: Replace 'any' type with a more specific type.Using 'any' type reduces type safety. Consider using a more specific type or generic constraint.
- params: any, + params: Record<string, unknown>,
105-105: Remove console.log from production code.Debug logging should not be in production code. Consider using a proper logging framework or removing it.
- console.log(result);
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
tests/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (15)
tests/e2e/enums/configuration-options.enum.ts(1 hunks)tests/e2e/enums/fix-types.enum.ts(1 hunks)tests/e2e/pages/base-page.page.ts(1 hunks)tests/e2e/pages/configuration.page.ts(1 hunks)tests/e2e/pages/vscode.page.ts(5 hunks)tests/e2e/tests/analyze_coolstore.test.ts(1 hunks)tests/e2e/tests/configure-and-run-analysis.test.ts(1 hunks)tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts(1 hunks)tests/e2e/utilities/vscode-commands.utils.ts(1 hunks)tests/global.setup.ts(2 hunks)tests/mcp-client/mcp-client-responses.model.ts(1 hunks)tests/mcp-client/mcp-client.model.ts(1 hunks)tests/package.json(1 hunks)tests/playwright.config.ts(1 hunks)webview-ui/src/components/ViolationIncidentsList.tsx(1 hunks)
🧰 Additional context used
🧠 Learnings (11)
📓 Common learnings
Learnt from: djzager
PR: konveyor/editor-extensions#645
File: vscode/src/webviewMessageHandler.ts:575-577
Timestamp: 2025-08-01T13:53:20.774Z
Learning: The TOGGLE_AGENT_MODE action handler in vscode/src/webviewMessageHandler.ts correctly calls toggleAgentMode() without parameters, as the toggleAgentMode function in vscode/src/utilities/configuration.ts has been updated to internally handle reading and toggling the agent mode configuration value.
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/e2e/pages/vscode.pages.ts:169-174
Timestamp: 2025-07-21T15:15:59.771Z
Learning: In tests/e2e/pages/vscode.pages.ts, the hardcoded 15-second wait in the runAnalysis() method and the 600000ms (10-minute) timeout for the "Run Analysis" button are intentionally set to accommodate performance differences between Linux and Windows environments where tests run slower. These timeouts should not be reduced or replaced with element-based waits without understanding the cross-platform testing requirements.
📚 Learning: in tests/e2e/pages/vscode.pages.ts, the hardcoded 15-second wait in the runanalysis() method and the...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/e2e/pages/vscode.pages.ts:169-174
Timestamp: 2025-07-21T15:15:59.771Z
Learning: In tests/e2e/pages/vscode.pages.ts, the hardcoded 15-second wait in the runAnalysis() method and the 600000ms (10-minute) timeout for the "Run Analysis" button are intentionally set to accommodate performance differences between Linux and Windows environments where tests run slower. These timeouts should not be reduced or replaced with element-based waits without understanding the cross-platform testing requirements.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.tstests/e2e/tests/analyze_coolstore.test.tstests/playwright.config.tstests/e2e/pages/configuration.page.tstests/e2e/tests/solution-server/solution-server-analysis-validation.test.tstests/global.setup.tstests/e2e/pages/vscode.page.ts
📚 Learning: the toggle_agent_mode action handler in vscode/src/webviewmessagehandler.ts correctly calls toggleag...
Learnt from: djzager
PR: konveyor/editor-extensions#645
File: vscode/src/webviewMessageHandler.ts:575-577
Timestamp: 2025-08-01T13:53:20.774Z
Learning: The TOGGLE_AGENT_MODE action handler in vscode/src/webviewMessageHandler.ts correctly calls toggleAgentMode() without parameters, as the toggleAgentMode function in vscode/src/utilities/configuration.ts has been updated to internally handle reading and toggling the agent mode configuration value.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.tstests/e2e/tests/analyze_coolstore.test.tstests/e2e/pages/configuration.page.tstests/global.setup.tstests/e2e/pages/vscode.page.ts
📚 Learning: in the tests/kai-evaluator/report/single-run-controller.js file, the insights-grid dom element is ha...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/kai-evaluator/report/single-run-controller.js:6-24
Timestamp: 2025-07-21T15:20:52.480Z
Learning: In the tests/kai-evaluator/report/single-run-controller.js file, the insights-grid DOM element is hardcoded in the HTML and never removed, so null checks for getElementById calls are not necessary.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.ts
📚 Learning: in the test/e2e/fixtures/provider-configs.fixture.ts file, using llmproviders.openai for openai-comp...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/e2e/fixtures/provider-configs.fixture.ts:41-42
Timestamp: 2025-07-16T16:08:20.126Z
Learning: In the test/e2e/fixtures/provider-configs.fixture.ts file, using LLMProviders.openAI for OpenAI-compatible services (like the PARASOL_PROVIDER) is expected and intentional behavior. The Parasol service uses ChatOpenAI with a custom baseURL, making it OpenAI-compatible.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.tstests/e2e/tests/analyze_coolstore.test.ts
📚 Learning: in test/e2e/fixtures/provider-configs.fixture.ts, the parasol_provider intentionally uses llmprovide...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/e2e/fixtures/provider-configs.fixture.ts:41-42
Timestamp: 2025-07-16T16:08:19.410Z
Learning: In test/e2e/fixtures/provider-configs.fixture.ts, the PARASOL_PROVIDER intentionally uses LLMProviders.openAI because it exposes an OpenAI-compatible API with a custom baseURL. This is a valid pattern for custom LLM services that implement OpenAI-compatible interfaces.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.tstests/e2e/tests/analyze_coolstore.test.ts
📚 Learning: in playwright test setups, global setup functions automatically catch and report errors. adding try-...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/global.setup.ts:4-6
Timestamp: 2025-07-21T15:25:01.028Z
Learning: In Playwright test setups, global setup functions automatically catch and report errors. Adding try-catch blocks around operations like VSCode.init() in global setup would result in duplicate error logging since Playwright already handles and reports these errors clearly. The framework's built-in error handling is sufficient for global setup functions.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.tstests/e2e/tests/analyze_coolstore.test.tstests/global.setup.tstests/e2e/pages/vscode.page.ts
📚 Learning: the electron package is required for the test environment in the konveyor/editor-extensions project,...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/package.json:11-122
Timestamp: 2025-07-16T12:49:03.553Z
Learning: The electron package is required for the test environment in the konveyor/editor-extensions project, specifically for the test/package.json file.
Applied to files:
tests/e2e/tests/configure-and-run-analysis.test.tstests/e2e/tests/analyze_coolstore.test.tstests/package.jsontests/global.setup.tstests/e2e/utilities/vscode-commands.utils.tstests/e2e/pages/vscode.page.ts
📚 Learning: security vulnerabilities flagged in package.json reviews may be related to the main package.json fil...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/package.json:11-122
Timestamp: 2025-07-16T12:49:03.553Z
Learning: Security vulnerabilities flagged in package.json reviews may be related to the main package.json file rather than test-specific package.json files in the konveyor/editor-extensions project.
Applied to files:
tests/package.jsontests/e2e/utilities/vscode-commands.utils.ts
📚 Learning: in playwright's electron support, automatic screenshots on test failure (configured via the screensh...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/playwright.config.ts:21-21
Timestamp: 2025-07-16T15:43:15.469Z
Learning: In Playwright's Electron support, automatic screenshots on test failure (configured via the screenshot: 'only-on-failure' setting) are not yet supported, even though manual screenshots can be taken programmatically. This limitation is tracked in GitHub issue #8208 which remains open as of 2025. The automatic screenshot-on-failure feature that works for web browsers does not work for Electron applications.
Applied to files:
tests/e2e/pages/base-page.page.tstests/e2e/pages/vscode.page.ts
📚 Learning: in the kai-evaluator build utils, the `isbuildable` function should resolve to `false` for build fai...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/kai-evaluator/utils/build.utils.ts:5-19
Timestamp: 2025-07-16T12:40:49.553Z
Learning: In the kai-evaluator build utils, the `isBuildable` function should resolve to `false` for build failures rather than rejecting, as build failures are expected and informative in this context - the function is designed to check buildability, not execute builds.
Applied to files:
tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts
🔇 Additional comments (7)
tests/mcp-client/mcp-client-responses.model.ts (1)
1-13: Interfaces look goodTyped response models match the fields returned by the solution server and provide clear contracts for the MCP client.
tests/e2e/tests/configure-and-run-analysis.test.ts (1)
2-2: LGTM! Import path correctly updated.The import path has been properly updated to reflect the refactored VSCode class location.
tests/e2e/tests/analyze_coolstore.test.ts (1)
2-2: LGTM! Import path correctly updated.The import path has been properly updated to align with the VSCode class refactoring.
tests/e2e/enums/fix-types.enum.ts (1)
1-5: LGTM! Well-structured enum for fix types.The enum is cleanly implemented with meaningful names and sequential numeric values that align with the solution server workflow requirements.
tests/playwright.config.ts (1)
33-36: LGTM! Consider if dependencies are needed.The new Playwright project configuration follows the established pattern and correctly targets solution server tests. Consider whether this project should depend on 'configure-and-run-analysis' like the 'analysis-tests' project does, if the solution server tests require a configured environment.
tests/global.setup.ts (1)
1-3: LGTM! Import updates align with refactoring.The import path corrections and new utility function import properly reflect the codebase refactoring.
tests/e2e/pages/vscode.page.ts (1)
74-77: All callers updated forinitparameter order change
The only usage intests/global.setup.tspasses the URL first and the directory second, matching the new(repoUrl?, repoDir?)signature. No further changes required.
tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts
Outdated
Show resolved
Hide resolved
Signed-off-by: Alejandro Brugarolas <[email protected]>
Signed-off-by: Alejandro Brugarolas <[email protected]>
sshveta
reviewed
Aug 6, 2025
tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts
Outdated
Show resolved
Hide resolved
tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts
Outdated
Show resolved
Hide resolved
tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts
Outdated
Show resolved
Hide resolved
tests/e2e/tests/solution-server/solution-server-analysis-validation.test.ts
Outdated
Show resolved
Hide resolved
Signed-off-by: Alejandro Brugarolas <[email protected]>
Signed-off-by: Alejandro Brugarolas <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
tests/e2e/pages/configuration.page.ts (2)
21-35: Consider unifying configuration methods as suggested.Per the previous review comment, consider creating a unified method that handles different types (boolean, string, string[]) to reduce code duplication. You could use method overloading or a discriminated union type approach.
Example approach:
public async setConfiguration( configuration: ConfigurationOptions, value: boolean | string | string[] ) { const window = this.vsCode.getWindow(); if (typeof value === 'boolean') { await window.getByLabel(configuration).setChecked(value); } else if (typeof value === 'string') { // Determine if it's input or dropdown based on element type const element = window.getByLabel(configuration); if (await element.evaluate(el => el.tagName === 'SELECT')) { await element.selectOption({ value }); } else { await element.fill(value); } } else if (Array.isArray(value)) { // Handle array values for multi-select // Implementation depends on UI component } }
16-16: Extract the hardcoded extension filter as a constant.The extension filter string
'@ext:konveyor.konveyor-ai'could be extracted as a constant for better maintainability.+const KONVEYOR_EXTENSION_FILTER = '@ext:konveyor.konveyor-ai'; + export class Configuration { public constructor(private readonly vsCode: VSCode) {} public static async open(vsCode: VSCode) { // ... - await searchInput.pressSequentially('@ext:konveyor.konveyor-ai'); + await searchInput.pressSequentially(KONVEYOR_EXTENSION_FILTER);tests/e2e/tests/solution-server/analysis-validation.test.ts (1)
113-113: Extract hardcoded timeout values as constants.The 35000ms timeout is hardcoded. Consider extracting timeouts as named constants for better maintainability.
+const SOLUTION_CONFIRMATION_TIMEOUT = 35000; + test.describe(`Solution server analysis validations`, () => { // ... async function requestFixAndAssertSolution(accept: boolean) { // ... - ).not.toBeVisible({ timeout: 35000 }); + ).not.toBeVisible({ timeout: SOLUTION_CONFIRMATION_TIMEOUT });
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
tests/e2e/enums/configuration-options.enum.ts(1 hunks)tests/e2e/pages/base.page.ts(1 hunks)tests/e2e/pages/configuration.page.ts(1 hunks)tests/e2e/pages/vscode.page.ts(5 hunks)tests/e2e/tests/solution-server/analysis-validation.test.ts(1 hunks)tests/e2e/utilities/download.utils.ts(0 hunks)
💤 Files with no reviewable changes (1)
- tests/e2e/utilities/download.utils.ts
✅ Files skipped from review due to trivial changes (1)
- tests/e2e/pages/base.page.ts
🧰 Additional context used
🧠 Learnings (9)
📓 Common learnings
Learnt from: abrugaro
PR: konveyor/editor-extensions#657
File: tests/global.setup.ts:22-27
Timestamp: 2025-08-05T16:16:56.005Z
Learning: In the Playwright test framework for konveyor/editor-extensions, the global setup in tests/global.setup.ts is designed to handle only essential initialization like extension installation verification. Individual tests are responsible for configuring GenAI when needed, rather than doing it globally. The openAnalysisView() method is used as a fast verification mechanism to ensure the extension was successfully installed, which is more efficient than configuring GenAI globally for all tests.
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/e2e/pages/vscode.pages.ts:169-174
Timestamp: 2025-07-21T15:15:59.771Z
Learning: In tests/e2e/pages/vscode.pages.ts, the hardcoded 15-second wait in the runAnalysis() method and the 600000ms (10-minute) timeout for the "Run Analysis" button are intentionally set to accommodate performance differences between Linux and Windows environments where tests run slower. These timeouts should not be reduced or replaced with element-based waits without understanding the cross-platform testing requirements.
Learnt from: djzager
PR: konveyor/editor-extensions#645
File: vscode/src/webviewMessageHandler.ts:575-577
Timestamp: 2025-08-01T13:53:20.774Z
Learning: The TOGGLE_AGENT_MODE action handler in vscode/src/webviewMessageHandler.ts correctly calls toggleAgentMode() without parameters, as the toggleAgentMode function in vscode/src/utilities/configuration.ts has been updated to internally handle reading and toggling the agent mode configuration value.
📚 Learning: in the playwright test framework for konveyor/editor-extensions, the global setup in tests/global.se...
Learnt from: abrugaro
PR: konveyor/editor-extensions#657
File: tests/global.setup.ts:22-27
Timestamp: 2025-08-05T16:16:56.005Z
Learning: In the Playwright test framework for konveyor/editor-extensions, the global setup in tests/global.setup.ts is designed to handle only essential initialization like extension installation verification. Individual tests are responsible for configuring GenAI when needed, rather than doing it globally. The openAnalysisView() method is used as a fast verification mechanism to ensure the extension was successfully installed, which is more efficient than configuring GenAI globally for all tests.
Applied to files:
tests/e2e/pages/configuration.page.tstests/e2e/tests/solution-server/analysis-validation.test.tstests/e2e/pages/vscode.page.ts
📚 Learning: in tests/e2e/pages/vscode.pages.ts, the hardcoded 15-second wait in the runanalysis() method and the...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/e2e/pages/vscode.pages.ts:169-174
Timestamp: 2025-07-21T15:15:59.771Z
Learning: In tests/e2e/pages/vscode.pages.ts, the hardcoded 15-second wait in the runAnalysis() method and the 600000ms (10-minute) timeout for the "Run Analysis" button are intentionally set to accommodate performance differences between Linux and Windows environments where tests run slower. These timeouts should not be reduced or replaced with element-based waits without understanding the cross-platform testing requirements.
Applied to files:
tests/e2e/pages/configuration.page.tstests/e2e/tests/solution-server/analysis-validation.test.tstests/e2e/pages/vscode.page.ts
📚 Learning: the toggle_agent_mode action handler in vscode/src/webviewmessagehandler.ts correctly calls toggleag...
Learnt from: djzager
PR: konveyor/editor-extensions#645
File: vscode/src/webviewMessageHandler.ts:575-577
Timestamp: 2025-08-01T13:53:20.774Z
Learning: The TOGGLE_AGENT_MODE action handler in vscode/src/webviewMessageHandler.ts correctly calls toggleAgentMode() without parameters, as the toggleAgentMode function in vscode/src/utilities/configuration.ts has been updated to internally handle reading and toggling the agent mode configuration value.
Applied to files:
tests/e2e/pages/configuration.page.tstests/e2e/pages/vscode.page.ts
📚 Learning: in playwright test framework for konveyor/editor-extensions, playwright's built-in ui interaction me...
Learnt from: abrugaro
PR: konveyor/editor-extensions#657
File: tests/e2e/pages/configuration.page.ts:21-25
Timestamp: 2025-08-05T16:17:54.293Z
Learning: In Playwright test framework for konveyor/editor-extensions, Playwright's built-in UI interaction methods like getByLabel() and setChecked() already provide comprehensive error handling and will throw meaningful errors if elements are not found or operations fail. Additional try-catch blocks around these methods are unnecessary as Playwright handles errors automatically.
Applied to files:
tests/e2e/pages/configuration.page.tstests/e2e/pages/vscode.page.ts
📚 Learning: in playwright test setups, global setup functions automatically catch and report errors. adding try-...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: tests/global.setup.ts:4-6
Timestamp: 2025-07-21T15:25:01.028Z
Learning: In Playwright test setups, global setup functions automatically catch and report errors. Adding try-catch blocks around operations like VSCode.init() in global setup would result in duplicate error logging since Playwright already handles and reports these errors clearly. The framework's built-in error handling is sufficient for global setup functions.
Applied to files:
tests/e2e/pages/configuration.page.tstests/e2e/pages/vscode.page.ts
📚 Learning: in playwright's electron support, automatic screenshots on test failure (configured via the screensh...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/playwright.config.ts:21-21
Timestamp: 2025-07-16T15:43:15.469Z
Learning: In Playwright's Electron support, automatic screenshots on test failure (configured via the screenshot: 'only-on-failure' setting) are not yet supported, even though manual screenshots can be taken programmatically. This limitation is tracked in GitHub issue #8208 which remains open as of 2025. The automatic screenshot-on-failure feature that works for web browsers does not work for Electron applications.
Applied to files:
tests/e2e/pages/configuration.page.tstests/e2e/pages/vscode.page.ts
📚 Learning: in the kai-evaluator build utils, the `isbuildable` function should resolve to `false` for build fai...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/kai-evaluator/utils/build.utils.ts:5-19
Timestamp: 2025-07-16T12:40:49.553Z
Learning: In the kai-evaluator build utils, the `isBuildable` function should resolve to `false` for build failures rather than rejecting, as build failures are expected and informative in this context - the function is designed to check buildability, not execute builds.
Applied to files:
tests/e2e/tests/solution-server/analysis-validation.test.ts
📚 Learning: the electron package is required for the test environment in the konveyor/editor-extensions project,...
Learnt from: abrugaro
PR: konveyor/editor-extensions#591
File: test/package.json:11-122
Timestamp: 2025-07-16T12:49:03.553Z
Learning: The electron package is required for the test environment in the konveyor/editor-extensions project, specifically for the test/package.json file.
Applied to files:
tests/e2e/pages/vscode.page.ts
🧬 Code Graph Analysis (1)
tests/e2e/pages/vscode.page.ts (2)
tests/e2e/pages/base.page.ts (1)
BasePage(4-34)tests/e2e/utilities/vscode-commands.utils.ts (1)
installExtension(13-42)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Build (windows)
- GitHub Check: Build (linux)
- GitHub Check: Build (macos)
🔇 Additional comments (1)
tests/e2e/enums/configuration-options.enum.ts (1)
1-5: Note the namespace inconsistency between configuration keys.The
SolutionEffortconfiguration useskonveyor.kainamespace while the other solution server configurations usekonveyor.solutionServer. This might be intentional based on the extension's architecture, but ensure this is the intended organization.
Signed-off-by: Alejandro Brugarolas <[email protected]>
…into 641-automate-solution-server-test
Signed-off-by: Alejandro Brugarolas <[email protected]>
pranavgaikwad
previously approved these changes
Aug 7, 2025
sshveta
reviewed
Aug 8, 2025
Signed-off-by: Alejandro Brugarolas <[email protected]>
sshveta
approved these changes
Aug 8, 2025
pranavgaikwad
approved these changes
Aug 8, 2025
This was referenced Aug 12, 2025
This was referenced Aug 25, 2025
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #641
This PR introduces an initial test case for the solution server.
It assumes that the solution server is deployed and accessible via the default URL.
The test verifies the success rate by directly hitting the endpoint after requesting, accepting, and rejecting a solution.
It also checks that upon accepting a solution, the best hint is updated, returning a relevant hint for the specified ruleset and violation.
Additionally, this PR introduces a class to modify the extension's configurations, along with some minor refactors to use composition and prevent the VSCode class from growing indefinitely.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Refactor
Style
Chores