[Feature] Add Gradle Tooling API test event stream for delegated Gradle tests

[wenytang-ms]

Summary

Add a Gradle Tooling API (GTA) based test event pipeline for Delegate Test to Gradle, so Gradle test delegation can provide real-time, structured test results without depending on BSP project import state or JUnit XML parsing as the primary mechanism.

The current implementation uses a conservative BSP-first strategy and falls back to direct Gradle execution plus JUnit XML parsing when BSP test delegation is unavailable. That fallback unblocks non-BSP / Buildship / failed-BSP-import projects, but XML is batch-oriented and cannot provide the same real-time or structured experience as an event stream.

Goals

• Make Gradle delegated test execution independent from BSP import state.
• Provide real-time test status updates for non-BSP projects.
• Use Gradle Tooling API test progress events as the long-term default event source.
• Preserve the current XML fallback as a compatibility path while the GTA implementation matures.
• Avoid re-running tests or hiding real build/test failures when a primary path fails after execution starts.

Non-goals

• Do not remove the current BSP path in the first iteration.
• Do not remove the XML fallback until the GTA path is proven stable.
• Do not attempt to fully model every JUnit/TestNG/dynamic-test edge case in the first milestone.
• Do not introduce behavior that retries tests after a real test/build failure.

Proposed execution strategy

Long term, the test delegation engine should become:

Delegate Test to Gradle
  -> GTA test event stream when supported
  -> XML fallback when GTA is unavailable or unsupported

During migration, a safer staged strategy can be:

Delegate Test to Gradle
  -> existing BSP path if available
  -> GTA test event path when BSP is unavailable
  -> XML fallback when neither BSP nor GTA event streaming is available

Milestones

1. Design and capability investigation

• Identify where the Gradle server invokes Gradle Tooling API today.
• Confirm how to register test progress listeners during delegated test builds.
• Define the minimum event model needed by the extension:
  • event kind: started, finished, output
  • descriptor id and parent descriptor id
  • project path and task path
  • class name, method name, display name
  • result state, duration, message, stack trace
• Decide how capability detection/versioning should work so older servers/extensions degrade safely.
• Confirm how cancellation and debug attach should interact with the GTA path.

2. Implement GTA event streaming end to end

• Add the server-side Gradle Tooling API test listener.
• Convert Gradle test descriptors/results into the new gRPC event model.
• Include project/task metadata to avoid multi-project result collisions.
• Add extension-side mapping from GTA events to vscode-java-test status events.
• Preserve pending-item finalization so requested tests never remain stuck in Running.
• Keep XML fallback for cases where GTA events are unavailable or unsupported.

Initial scope should cover:

• test started
• test passed
• test failed
• test errored
• test skipped
• duration and basic failure message/stack trace

Follow-up enhancements can add:

• per-test stdout/stderr where Gradle supports TestOutputEvent
• structured failure cause chains where Gradle supports them
• richer nested/dynamic test hierarchy

3. Mapping, debug, and compatibility hardening

• Define stable test id mapping rules for:
  • class-level runs
  • method-level runs
  • parameterized test invocations
  • nested classes
  • dynamic tests / synthetic children
  • multi-project builds with identical fully-qualified class names
• Prefer mapping results back to explicitly requested test ids.
• Reuse the existing debug attach flow initially.
• Prevent multiple Gradle Test tasks from racing on the same debug port.
• Avoid falling back to XML after a GTA run has already started executing tests, unless the failure is a known capability/precondition failure before execution.

4. Test coverage and rollout

• Add server tests for Gradle test event conversion.
• Add extension unit tests for GTA event to TestRunner event mapping.
• Add integration tests for non-BSP project test delegation.
• Cover multi-project, parameterized, nested, skipped, failed, errored, cancellation, and build-failure cases where practical.
• Roll out behind capability detection first.
• Keep XML fallback enabled during initial rollout.
• Switch to GTA-first only after tests and dogfooding show stable behavior.

Acceptance criteria

• Non-BSP Gradle projects can use Delegate Test to Gradle with real-time test status updates.
• BSP-imported projects do not regress.
• Test failures are reported without duplicate execution.
• Requested test items always receive a terminal state.
• Multi-project builds do not mix results between projects with identical class names.
• XML fallback remains available when GTA test events are not supported.
• CI includes meaningful unit and integration coverage for the GTA path.

Related context

• Current mitigation: BSP-first delegation with XML fallback when BSP is unavailable.
• Related issues:
  • Cannot run tests with the plugin #1802
  • When project is imported by Buildship, give a clear message what to do to use test delegation vscode-java-test#1726
  • Run Test -> Delegate to Gradle does not work vscode-java-test#1771
  • Support Gradle test delegation vscode-java-test#1045

[github-actions]

Hi @wenytang-ms, I'm an AI Support assistant here to help with your issue. While the team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

This feature request is an enhancement to the existing Gradle test delegation. There’s no complete implementation yet, but you may find the following discussions useful:

• Support Gradle test delegation vscode-java-test#1045: initial “Delegate to Gradle” support in VS Code (adds basic Gradle test delegation but does not yet include real-time event streaming)
  Support Gradle test delegation vscode-java-test#1045 (comment)

• Add support for delegating test and run/debug execution build-server-for-gradle#119: discussion on capturing Gradle test output in the Test Results view (notes absence of streaming output collection)
  Add support for delegating test and run/debug execution build-server-for-gradle#119 (comment)

Other references with low confidence

- microsoft/vscode-java-test#1701: mentions upcoming delegated test execution (no real-time events yet) https://github.com/microsoft/vscode-java-test/issues/1701#issuecomment-2191212874

The team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!

[wenytang-ms]

Why GTA is worth pursuing compared with the BSP-based delegated test path

The main goal of this feature is not to replace XML parsing first. XML should remain a compatibility fallback. The more important architectural question is whether delegated Gradle test execution should continue to depend on the BSP/build-server-for-gradle path, or whether vscode-gradle should own Gradle test execution directly through the Gradle Tooling API (GTA).

In the current ecosystem, delegated Gradle testing spans multiple components:

• Java Test Runner / Test Explorer owns test discovery, test item identity, run/debug profiles, and user-facing lifecycle.
• JDT LS provides Java project context, source/classpath model, discovery support, and breakpoint/source mapping.
• vscode-java-debug owns debugger attach and breakpoint handling once the test JVM starts with JDWP enabled.
• vscode-gradle should own Gradle execution and Gradle-originated runtime events.
• build-server-for-gradle / BSP currently participates in delegated Gradle test execution and event reporting, but it also has broader project model/build-target responsibilities.

The proposed GTA path should preserve the external contract currently used by Java Test Runner while removing delegated test execution from the BSP import/build-target precondition.

BSP-based delegated test path

The BSP path is valuable because it already provides a protocolized test execution and event callback model. However, it couples delegated test execution to the project being successfully recognized/imported as a Gradle build server project.

Conceptually, the current BSP path looks like this:

Java Test Runner / Test Explorer
        -> vscode-gradle delegated test command
        -> Gradle BSP / build-server-for-gradle
        -> Gradle test execution
        -> BSP test events
        -> Test Explorer status updates

This means a user can have a project where Gradle itself can run tests, but delegated test execution still fails before the test starts because the BSP project/build-target precondition is not satisfied. In that failure mode, the problem is not Gradle test execution itself; it is the dependency on the BSP integration state.

GTA-based delegated test path

The GTA path keeps Gradle execution inside vscode-gradle and uses Tooling API test progress events as the runtime event source:

Java Test Runner / Test Explorer
        -> vscode-gradle GTA runner
        -> Gradle Tooling API BuildLauncher
        -> Gradle test execution
        -> Tooling API test events
        -> GradleTestEvent gRPC stream
        -> Test Explorer status updates

This does not require GTA to call BSP, and BSP should not call GTA. They should be separate execution backends selected by the Gradle test runner orchestration layer. A single test run should have exactly one active event source: BSP, GTA, or fallback parsing.

Relationship to build-server-for-gradle#119

microsoft/build-server-for-gradle#119 is not directly fixed by the initial GTA protocol PR, but it is an important historical baseline for the delegated Gradle execution experience. That issue tracked the original BSP-based support for delegating test and run/debug execution to Gradle, including scenarios such as Micronaut/Testcontainers where tests need to run through the build tool.

If GTA is intended to become the long-term delegated Gradle test backend, it should treat the capabilities discussed in that issue as parity requirements, not as unrelated BSP-only behavior.

Parity checklist from the BSP delegated execution work

Before GTA can replace or deprecate the BSP delegated-test path, it should cover the relevant capabilities that users already expect from the BSP-based implementation.

Capability from BSP delegated execution	Why it matters	GTA requirement
Delegate tests to Gradle	Some tests only work correctly when executed by Gradle, for example tests depending on Gradle-managed classpath, plugins, generated sources, environment, Testcontainers, or Docker setup	GTA must execute through Gradle Test tasks, not through an independent JUnit launcher
Delegate debug tests to Gradle	Users expect breakpoints to work while still running tests through Gradle	GTA must reuse the existing runBuild / GradleBuildRunner JDWP injection and debug-port lifecycle
Preserve Java Test Runner profile integration	Users select Delegate Test to Gradle from Test Explorer profiles	GTA should remain an implementation backend for the same delegated Gradle profile, not introduce a competing user workflow
Preserve Test Explorer lifecycle	Existing users expect queued/running/passed/failed/skipped state updates in Test Explorer	GTA events must map back to the Java Test Runner-owned test items and lifecycle
Report test stdout/stderr into test results	build-server-for-gradle#119 explicitly called out that Gradle output was not collected into the Test Results log	GTA should use Tooling API TestOutputEvent / GradleTestEvent.OUTPUT to provide equivalent or better output support
Support build-tool-specific test environments	Micronaut/Testcontainers examples highlight that plain IDE execution may not reproduce Gradle's runtime environment	GTA must keep Gradle as the source of execution truth and avoid bypassing Gradle task configuration
Support multi-project Gradle builds	BSP build targets currently provide project boundaries	GTA should carry or infer enough Gradle project/task context to avoid cross-project collisions
Preserve cancellation and failure semantics	Users should not see duplicate execution or fallback reruns after a real Gradle failure	GTA orchestration should fallback only before execution starts; once Gradle execution starts, failures should be reported as failures
Keep room for delegated run/main support	#119 also discusses delegating run / main method execution to Gradle	The current issue can focus on tests, but the runBuild event-stream design should not block future run/debug delegation improvements

What complexity can be converged

If GTA becomes the primary delegated Gradle test backend, it can reduce the amount of logic that exists only because delegated tests currently depend on BSP state or on secondary result reconstruction.

Current complexity in BSP-oriented flow	Why it exists	GTA convergence point
BSP import/build-target precondition checks	Delegated execution requires the project to be known to BSP	GTA can execute through Gradle Tooling API even when BSP import is unavailable
BSP test callback forwarding	Test status is delivered through BSP callbacks	Test status can be delivered through the existing Gradle server runBuild stream
Split ownership between BSP execution and vscode-gradle fallback execution	Some projects go through BSP, others need local Gradle execution fallback	vscode-gradle can own Gradle execution consistently for delegated tests
Backend-specific status repair logic	BSP, fallback parsing, and Test Explorer each need alignment	A single GradleTestEvent -> TestItem mapper can become the central status layer
Ambiguous fallback semantics	It is easy to accidentally rerun or mix event sources	Orchestration can explicitly select one backend before execution starts

This does not remove all complexity. The important remaining complexity is test identity mapping:

GradleTestEvent -> Java Test Runner / VS Code TestItem

That mapper still needs to handle class-level requests, method-level requests, parameterized tests, nested tests, dynamic tests, Kotlin/Spock naming, inherited tests, custom display names, and unmatched Gradle descriptors. The benefit is that this complexity becomes localized and unit-testable instead of being spread across BSP preconditions, BSP callbacks, fallback execution, stale-result handling, and status aggregation.

What GTA should preserve from the BSP contract

If the long-term direction is to deprecate or reduce the BSP delegated-test path, GTA must preserve the external behavior that other Java extensions rely on.

Area	BSP currently contributes	GTA must cover
Test execution request	Receives delegated run/debug requests and executes Gradle tests	Convert Java Test Runner requests into equivalent Gradle task arguments and filters
Test events	Streams started/passed/failed/skipped/output events	Stream Tooling API test events and map them back to the same Test Explorer items
Test identity	Works with Java Test Runner/JDT LS discovered test items	Do not create a competing test tree; map Gradle descriptors to existing test item ids
Debug	Starts test JVMs in a way that Java Debug can attach	Reuse existing runBuild / GradleBuildRunner JDWP injection and debug-port handling
Project context	Uses BSP build targets to relate requests to Gradle projects	Use vscode-gradle project/task metadata and Java Test Runner request context without requiring BSP
Cancellation/lifecycle	Reports run cancellation/completion through the active backend	Keep cancellation and completion in the same runBuild stream
Multi-project behavior	BSP build targets provide project boundaries	GTA event mapping should carry/use enough Gradle project/task context to avoid cross-project collisions

Interaction with JDT LS, Java Test Runner, and Java Debug

GTA should not replace Java Test Runner or JDT LS discovery. The intended ownership should remain:

JDT LS / Java Test Runner:
  - discover tests
  - own Test Explorer items and test item ids
  - provide run/debug requests

vscode-gradle / GTA:
  - execute Gradle test tasks
  - stream Gradle runtime test events
  - map Gradle events back to the requested Java Test Runner items

For debugging, GTA should be transparent to vscode-java-debug. The key requirement is that the Gradle test worker JVM is started with the expected JDWP arguments and that VS Code attaches to the same port:

Java Test Runner debug request
        -> vscode-gradle GTA runner
        -> Gradle Test task with JDWP injected
        -> vscode-java-debug attaches
        -> GTA events update Test Explorer state

Debug parity should explicitly cover breakpoint hits, attach timing with suspend=y, cancellation without leaked debug sessions, no JDWP port race for multi-task debug runs, and useful failed-test events while debugging.

Rollout strategy

GTA should not immediately delete or bypass every existing path. A conservative rollout can keep current behavior while introducing GTA as a capability-gated backend:

1. Preserve BSP-first behavior initially if needed for compatibility.
2. If BSP is unavailable before execution starts, try GTA when the server/client capability exists.
3. If GTA is unavailable before execution starts, use the existing fallback path.
4. If any backend already started Gradle execution and the build/test fails, do not rerun with another backend.

Once GTA has parity across run/debug/mapping/multi-project scenarios, the preferred model can become:

1. GTA as the primary delegated Gradle test backend.
2. Fallback only for compatibility or capability gaps.
3. BSP delegated-test execution as legacy fallback, or no longer part of the main delegated-test flow.

This does not mean build-server-for-gradle is globally unnecessary. BSP may still be useful for project import, model, build target, classpath, and related IDE integration. The scope here is specifically delegated Gradle test execution and test event reporting.

Expected payoff

The short-term implementation adds one backend, so the immediate goal is not code deletion. The long-term payoff is that delegated Gradle test execution can converge around a single Gradle-owned runtime stream:

Before:
  BSP delegated execution and callbacks
  + separate fallback execution/result reconstruction
  + backend-specific status repair logic

After:
  GTA execution/event stream as primary path
  + one localized event-to-TestItem mapper
  + fallback only for compatibility gaps

This should improve reliability for projects where Gradle can run tests but BSP import is unavailable, preserve real-time test feedback without depending on BSP state, and reduce long-term maintenance cost by moving delegated Gradle test execution into vscode-gradle's Gradle Tooling API execution path.