Split local and Helix test runners by jaredpar · Pull Request #85859 · dotnet/roslyn

jaredpar · 2026-09-30T22:44:55Z

Summary

Keep RunTests responsible for local execution and introduce RunHelix for Helix scheduling, payload generation, and submission.
Share discovery, common options, and process/logging helpers through a globbed Compile link, without adding a shared-library project. Use TestRunner, TestRunner.RunTests, and TestRunner.Helix namespaces.
Update solution/filter membership, PrepareTests payload packaging, and Helix pipeline invocations. Keep Test.cmd/test.sh and currently local CI/integration execution on RunTests.
Separate CLI options and reject unrecognized arguments; remove ineffective Helix --out/--logs switches.
Add offline regression coverage and update runner documentation.

Validation

RunTests, RunHelix, and PrepareTests build with -p:RunAnalyzersDuringBuild=true, with zero warnings/errors.
pwsh -NoProfile -File eng\test-test-runners.ps1 -configuration Debug: 174 offline assertions pass.
Real local compiler tests: four passed in sequential mode via Test.cmd, and four passed in normal mode with XML/HTML results.
An intentionally failing xUnit fixture produced one failed test and runner exit code 1.
Verified linked Compile items, formatting, source encodings, pipeline syntax, and whitespace.

Full Helix artifact-generation coverage was skipped locally because symbolic-link permission is unavailable. Remote Windows/Unix execution will be validated by Azure DevOps. This PR remains draft while CI is monitored and any necessary fixes are pushed.

Microsoft Reviewers: Open in CodeFlow

Share discovery and common runner infrastructure through linked sources, route Helix pipelines to RunHelix, and add offline regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-pipelines · 2026-09-30T22:45:17Z

Azure Pipelines: Successfully started running 2 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run.

Rename the shared base to Options and derived types to RunTestOptions and RunHelixOptions; update file names, callers, and reflection-based regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Split the obsolete ITestExecutor file into TestExecutionOptions, TestResultInfo, and TestResult files without changing behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jaredpar · 2026-10-01T02:03:39Z

CI follow-up: fixed the failure-reporting crash in 728961e. RunTests now creates the configured log directory before writing failed-test output. The new regression reproduced the DirectoryNotFoundException before the fix; the analyzer build and 177 offline assertions pass afterward.

The three BuildHost template-test failures in build 1618955 are separate. The identical "BuildHost process is not responding" / "RPC channel closed; process exiting" signature also occurs on main: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1618719&view=ms.vss-test-web.build-test-results-tab&runId=44830552&resultId=291232 . This is evidence of a pre-existing failure, not a claim that the current checks are green.

Monitoring the new revision in roslyn-CI build 1619011 and roslyn-integration-CI build 1619012. Keeping this PR draft.

jaredpar · 2026-10-01T03:39:28Z

/azp run roslyn-CI

azure-pipelines · 2026-10-01T03:43:57Z

No pipelines are associated with this pull request.

azure-pipelines · 2026-10-01T03:45:47Z

Azure Pipelines: Successfully started running 1 pipeline(s).

jaredpar · 2026-10-01T16:04:18Z

@dibarbet PTAL

Copilot

Copilot review overview

🟢 Approval recommended

The split is coherent and validated end-to-end; only a minor unused-type cleanup remains.

Review effort: Balanced
Findings: 1

Open (1)

Remove unused TestExecutionOptions type · New

What changed in this PR

Splits local test execution from Helix submission while sharing discovery and utility sources.

Changes:

Adds the dedicated RunHelix executable.
Separates local and Helix CLI options and pipelines.
Adds payload handling, regression coverage, and documentation.

A non-blocking comment identifies the unused TestExecutionOptions type.

File	Description
`.github/memory/TESTING_STRATEGY.md`	Documents the runner split.
`src/Tools/TestRunnerCommon/ProcessRunner.cs`	Moves process utilities to shared namespace.
`src/Tools/TestRunnerCommon/Options.cs`	Adds shared CLI options.
`src/Tools/TestRunnerCommon/Logger.cs`	Shares logging support.
`src/Tools/TestRunnerCommon/ConsoleUtil.cs`	Shares console utilities.
`src/Tools/TestRunnerCommon/AssemblyInfo.cs`	Shares assembly metadata.
`src/Tools/TestRunnerCommon/AssemblyDiscovery.cs`	Extracts assembly discovery.
`src/Tools/RunTests/TestRunner.cs`	Specializes local execution.
`src/Tools/RunTests/TestResultInfo.cs`	Extracts execution results.
`src/Tools/RunTests/TestResult.cs`	Extracts enriched results.
`src/Tools/RunTests/TestExecutionOptions.cs`	Adds an unused options type.
`src/Tools/RunTests/RunTests.csproj`	Links shared runner sources.
`src/Tools/RunTests/RunTestOptions.cs`	Defines local-only options.
`src/Tools/RunTests/README.md`	Documents local execution.
`src/Tools/RunTests/Program.cs`	Removes Helix submission.
`src/Tools/RunTests/ProcessUtil.cs`	Updates namespace.
`src/Tools/RunTests/ProcessTestExecutor.cs`	Removes Helix-specific behavior.
`src/Tools/RunTests/ProcDumpUtil.cs`	Updates namespace.
`src/Tools/RunTests/Options.cs`	Removes combined options.
`src/Tools/RunTests/ITestExecutor.cs`	Removes obsolete combined types.
`src/Tools/RunTests/FileUtil.cs`	Updates namespace.
`src/Tools/RunTests/DumpCollector.cs`	Updates namespace and encoding.
`src/Tools/RunHelix/TypeInfo.cs`	Adds Helix scheduling metadata.
`src/Tools/RunHelix/TestHistoryManager.cs`	Uses Helix-specific options.
`src/Tools/RunHelix/RunHelixOptions.cs`	Defines submission options.
`src/Tools/RunHelix/RunHelix.csproj`	Adds the Helix executable.
`src/Tools/RunHelix/README.md`	Documents submission behavior.
`src/Tools/RunHelix/Program.cs`	Adds the Helix entry point.
`src/Tools/RunHelix/HelixTestRunner.cs`	Owns payload generation and submission.
`src/Tools/RunHelix/AzdoClient.cs`	Moves history client to Helix namespace.
`src/Tools/RunHelix/AssemblyScheduler.cs`	Moves partition scheduling to RunHelix.
`src/Tools/PrepareTests/README.md`	Documents both runners.
`src/Tools/PrepareTests/MinimizeUtil.cs`	Packages RunHelix output.
`Roslyn.slnx`	Adds RunHelix.
`Ide.slnf`	Adds RunHelix.
`Compilers.slnf`	Adds RunHelix.
`eng/test-test-runners.ps1`	Adds offline regression entry point.
`eng/test-runners/RegressionHarness.cs`	Tests options, discovery, scheduling, and payloads.
`eng/pipelines/test-windows-job.yml`	Submits through RunHelix.
`eng/pipelines/test-unix-job.yml`	Submits through RunHelix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🔵 Needs a closer look

The regression suite is not executed by CI, and RunHelix’s documented host requirement contradicts the macOS pipeline configuration.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Remove unused TestExecutionOptions type

Previously missed (2)

In code that hasn't changed since last review

Regression suite is not invoked by any project or CI pipeline

eng/test-test-runners.ps1:72

This regression suite is only referenced by its documentation; no project or pipeline invokes it, so these assertions will not fail CI when runner parsing, discovery, scheduling, or payload generation regresses. Please wire this command into an existing correctness job after the runner builds (or move the checks into an automatically executed test project).

Same-OS requirement conflicts with Linux-to-macOS pipeline stage

src/Tools/RunHelix/README.md:14

The same-OS-family requirement contradicts the repository's macOS stage: azure-pipelines.yml:342-350 runs this submission template on ubuntuPool while targeting HelixMacOsQueueName. Please either document Linux-to-macOS submission as supported or change that pipeline/implementation so the stated requirement is true.

Split local and Helix test runners

ab9b4ed

Share discovery and common runner infrastructure through linked sources, route Helix pipelines to RunHelix, and add offline regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jaredpar and others added 3 commits September 30, 2026 17:24

Clarify shared and executable-specific options names

9f7b295

Rename the shared base to Options and derived types to RunTestOptions and RunHelixOptions; update file names, callers, and reflection-based regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Match test execution model filenames to their types

20d50c5

Split the obsolete ITestExecutor file into TestExecutionOptions, TestResultInfo, and TestResult files without changing behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Create failure log directory before reporting test failures

728961e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jaredpar marked this pull request as ready for review October 1, 2026 16:04

jaredpar requested a review from a team as a code owner October 1, 2026 16:04

Copilot AI balanced review requested due to automatic review settings October 1, 2026 16:04

Copilot started reviewing on behalf of jaredpar October 1, 2026 16:04 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread src/Tools/RunTests/TestExecutionOptions.cs Outdated

Remove unused test execution options

455f917

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 1, 2026 17:06

Copilot started reviewing on behalf of jaredpar October 1, 2026 17:06 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Navigation Menu

Split local and Helix test runners - #85859

Split local and Helix test runners#85859
jaredpar wants to merge 5 commits into
dotnet:mainfrom
jaredpar:jaredpar-runtests-project-split

jaredpar commented Sep 30, 2026 •

edited by dotnet-policy-service Bot

Loading

Uh oh!

azure-pipelines Bot commented Sep 30, 2026

Uh oh!

jaredpar commented Oct 1, 2026

Uh oh!

jaredpar commented Oct 1, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

jaredpar commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jaredpar commented Sep 30, 2026 • edited by dotnet-policy-service Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Microsoft Reviewers: Open in CodeFlow

Uh oh!

azure-pipelines Bot commented Sep 30, 2026

Uh oh!

jaredpar commented Oct 1, 2026

Uh oh!

jaredpar commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

jaredpar commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jaredpar commented Sep 30, 2026 •

edited by dotnet-policy-service Bot

Loading

jaredpar commented Oct 1, 2026 •

edited

Loading