Split integration tests to single tfm and multi tfm project

[nohwnd]

Description

Add categories of tests, so we can run the most basic integration tests. Avoid running compatibility tests in PR pipeline (currently they run in the main pipeline only, but should change to run in separate nightly task, to avoid failing in the internal pipeline when we need to iterate fast.)

Split tests into 2 projects to make it explicit where we run tests in child process and where we need to test the code that is loaded into the test project.

Related issue

Related #15470

[Copilot]

Pull request overview

This PR restructures integration testing by separating single-TFM acceptance/integration tests from multi-TFM “in-proc” library integration tests, and centralizes shared integration-test build/setup logic.

Changes:

• Moved/centralized integration-test asset build + dotnet patching into IntegrationTestBuild (TestUtilities) and wired test assembly initialization to it.
• Split integration test projects: Acceptance.IntegrationTests becomes single-TFM, and a new multi-targeted Library.IntegrationTests project is introduced.
• Added new build script switches/filters to support smoke/compatibility/performance test slices.

Reviewed changes

Copilot reviewed 58 out of 87 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
test/Microsoft.TestPlatform.TestUtilities/Microsoft.TestPlatform.TestUtilities.csproj	Adds package references needed by shared test build/utilities code.
test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs	New centralized build/setup for integration tests (assets restore/build, compatibility matrix, dotnet patching).
test/Microsoft.TestPlatform.TestUtilities/CompatibilityDataSourceAttribute.cs	Introduces a shared “marker” base attribute for compatibility data sources.
test/Microsoft.TestPlatform.Library.IntegrationTests/README.MD	Documents the purpose of the new in-proc multi-TFM integration test project.
test/Microsoft.TestPlatform.Library.IntegrationTests/Properties/AssemblyInfo.cs	Enables MSTest parallelization for the new project.
test/Microsoft.TestPlatform.Library.IntegrationTests/Microsoft.TestPlatform.Library.IntegrationTests.csproj	New multi-targeted integration test project definition.
test/Microsoft.TestPlatform.Library.IntegrationTests/BannedSymbols.txt	Adds banned API list for the new integration test project.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TranslationLayerTests/RunTests.cs	Adds Smoke categorization to a specific test.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TestPlatformNugetPackageTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TestCaseFilterTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TelemetryTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SkipIOutOfProcessTestOnNetFrameworkConditionAttribute.cs	Removes the NetFx duplication-skip attribute implementation.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SelfContainedAppTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RunsettingsTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ResultsDirectoryTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RecursiveResourcesLookupTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/README.MD	Documents the purpose of the (now single-TFM) acceptance integration tests project.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ProcessesInteractionTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PostProcessingTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PortableNugetPackageTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PlatformTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/MultitargetingTestHostTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Microsoft.TestPlatform.Acceptance.IntegrationTests.csproj	Changes Acceptance.IntegrationTests to single-TFM and removes some package/framework-specific refs.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/LoggerTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ListExtensionsTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/FrameworkTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/FilePatternParserTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/WrapperCompatibilityDataSource.cs	Switches compatibility data sources to derive from the new marker base attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/TesthostCompatibilityDataSource.cs	Switches compatibility data sources to derive from the new marker base attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/TestDataSourceAttribute.cs	Changes AddData to accept TestDataRow<T> (to carry categories/metadata).
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/RunnnerInfo.cs	Updates to use StringUtils.IsNullOrEmpty.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/RunnerCompatibilityDataSource.cs	Switches compatibility data sources to derive from the new marker base attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/MSTestCompatibilityDataSource.cs	Switches compatibility data sources to derive from the new marker base attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/CustomCompatibilityDataSource.cs	Switches compatibility data sources to derive from the new marker base attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension/CompatibilityRowsBuilder.cs	Returns TestDataRow<RunnerInfo> and stamps Compatibility category.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionThreadApartmentStateTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionTests.cs	Removes NetFx duplication-skip condition attribute; adds a new smoke-categorized test.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/EventLogCollectorTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetTestTests.cs	Removes NetFx duplication-skip condition attribute; adds Smoke categorization.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetTestMSBuildOutputTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetHostArchitectureVerifierTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetArchitectureSwitchTests.Windows.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DiscoveryTests.cs	Removes NetFx duplication-skip condition attribute; adds Smoke categorization.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DisableAppdomainTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DifferentTestFrameworkSimpleTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DebugAssertTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectorTests.Coverlet.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectorAttachmentsProcessorsFactoryTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectionTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/CodeCoverageTests.cs	Removes NetFx duplication-skip condition attribute.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/CodeCoverageAcceptanceTestBase.cs	Uses StringUtils.IsNullOrEmpty instead of string.IsNullOrEmpty.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Build.cs	Simplifies acceptance test assembly initialization to call the shared build helper.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/BlameDataCollectorTests.cs	Removes NetFx duplication-skip condition attribute.
src/vstest.console/Friends.cs	Grants internals access to the new Library.IntegrationTests assembly.
src/Microsoft.TestPlatform.Common/Friends.cs	Grants internals access to the new Library.IntegrationTests assembly.
eng/build.ps1	Adds new CLI switches and constructs MSTest filter/test-parameter arguments.
TestPlatform.sln	Adds new test project and updates solution metadata.
Build.cs	Adds a solution-level Build entrypoint that calls the centralized integration-test build helper.

Comments suppressed due to low confidence (3)

test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1

• SkipIntegrationTestBuild is logged but does not actually skip the build (no return / early-exit). This makes the switch ineffective and can still trigger expensive restores/builds. Add an early return after logging (or wrap the build logic in an if (!skipBuild) block).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• Parameter name is misspelled (BuildCompatiblity). The build script sets BuildCompatibility, so this will always read as false and compatibility assets will never build. Rename the parameter key to BuildCompatibility (and update any related log messages using the misspelling).
  test/Microsoft.TestPlatform.TestUtilities/CompatibilityDataSourceAttribute.cs:1
• CompatibilityDataSourceAttribute is added under the TestUtilities project but inherits from TestDataSourceAttribute<RunnerInfo>, which (per this PR) is defined under test/Microsoft.TestPlatform.Acceptance.IntegrationTests/Extension. This creates a tight coupling between a shared utilities project and acceptance-test-specific extension types. Consider moving CompatibilityDataSourceAttribute next to TestDataSourceAttribute (same project), or moving the shared TestDataSourceAttribute/RunnerInfo abstractions into TestUtilities so dependencies flow one way.

[Copilot]

Pull request overview

Copilot reviewed 93 out of 94 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (6)

test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1

• SkipIntegrationTestBuild is logged but the method does not return, so the build still proceeds. If this flag is intended to prevent the expensive asset build (as suggested by the build script), add an early return when skipBuild is true.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• The test-parameter name has a typo (BuildCompatiblity). In eng/build.ps1 the parameter is set as BuildCompatibility, so this will never be read as true and compatibility assets won’t build. Rename the parameter key here to match the script (and keep spelling consistent everywhere).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• PATH is being prepended using a hard-coded ; separator, which will break PATH on non-Windows where the separator is :. Use Path.PathSeparator (or OS-conditional logic) to join PATH entries.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• generatedSln is set to CompatibilityTestAssets.slnx, but dotnet new sln typically creates a .sln file unless explicitly configured otherwise. If the file extension doesn’t match what is created, subsequent dotnet sln/restore/build calls will fail. Consider aligning generatedSln with the actual output (or pass an explicit argument/format that produces .slnx).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• generatedSln is set to CompatibilityTestAssets.slnx, but dotnet new sln typically creates a .sln file unless explicitly configured otherwise. If the file extension doesn’t match what is created, subsequent dotnet sln/restore/build calls will fail. Consider aligning generatedSln with the actual output (or pass an explicit argument/format that produces .slnx).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• Correct the comment typo: “I any” → “If any”.

[Copilot]

Pull request overview

Copilot reviewed 92 out of 93 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (7)

test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1

• The test parameter name is misspelled (BuildCompatiblity) and doesn’t match the name being set from the build script (BuildCompatibility). This will cause compatibility asset build to never run even when -compatibilityTest is requested; rename the parameter key (and the log message) to BuildCompatibility consistently.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• The test parameter name is misspelled (BuildCompatiblity) and doesn’t match the name being set from the build script (BuildCompatibility). This will cause compatibility asset build to never run even when -compatibilityTest is requested; rename the parameter key (and the log message) to BuildCompatibility consistently.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• When SkipIntegrationTestBuild is true, the code logs but continues into the mutex/build path anyway. If the intent is to skip the expensive one-time build in split jobs, this should return early (or otherwise bypass the build steps) to make the flag effective.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, the mutex is not acquired, but ReleaseMutex() is still called in finally, which will throw and can mask the original TimeoutException. Track whether the mutex is owned before releasing (e.g., only release if createdNew is true or gotOne is true).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, the mutex is not acquired, but ReleaseMutex() is still called in finally, which will throw and can mask the original TimeoutException. Track whether the mutex is owned before releasing (e.g., only release if createdNew is true or gotOne is true).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, the mutex is not acquired, but ReleaseMutex() is still called in finally, which will throw and can mask the original TimeoutException. Track whether the mutex is owned before releasing (e.g., only release if createdNew is true or gotOne is true).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• This prepends to PATH using ;, which is Windows-specific. Use Path.PathSeparator to keep PATH well-formed on Unix-like agents (even if you usually invoke dotnet via absolute path, other invoked tools may rely on PATH).

[Copilot]

Pull request overview

Copilot reviewed 92 out of 93 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (8)

test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1

• When SkipIntegrationTestBuild is true the method currently only logs but continues into the build path. To actually skip the shared build (as intended for split CI jobs), return immediately after logging or wrap the remaining logic in an if (!skipBuild) block.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• The test parameter name is misspelled (BuildCompatiblity) and does not match the build script which sets BuildCompatibility. As a result, compatibility builds will never be enabled via --test-parameter. Rename the parameter key consistently (and update the associated log message strings).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, the code throws but still calls ReleaseMutex() in finally without owning the mutex, which can raise ApplicationException and mask the original timeout. Track whether the mutex was successfully acquired (e.g., acquired = createdNew || gotOne) and only call ReleaseMutex() when acquired.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, the code throws but still calls ReleaseMutex() in finally without owning the mutex, which can raise ApplicationException and mask the original timeout. Track whether the mutex was successfully acquired (e.g., acquired = createdNew || gotOne) and only call ReleaseMutex() when acquired.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, the code throws but still calls ReleaseMutex() in finally without owning the mutex, which can raise ApplicationException and mask the original timeout. Track whether the mutex was successfully acquired (e.g., acquired = createdNew || gotOne) and only call ReleaseMutex() when acquired.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• generatedSln is hardcoded to .slnx, but dotnet new sln typically produces a .sln unless explicitly configured. This can cause subsequent dotnet sln/restore/build calls to fail because the solution path doesn't exist. Consider using .sln or passing an explicit format option (and keep the extension consistent everywhere).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• generatedSln is hardcoded to .slnx, but dotnet new sln typically produces a .sln unless explicitly configured. This can cause subsequent dotnet sln/restore/build calls to fail because the solution path doesn't exist. Consider using .sln or passing an explicit format option (and keep the extension consistent everywhere).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• Copying files into compatibilityProjectDir will fail for nested paths (e.g., Properties\\AssemblyInfo.cs) because the destination subdirectories are not created before File.Copy. Create Path.GetDirectoryName(fullPath) (when non-null) before copying, or replace this loop with a directory-copy routine that preserves structure.

[Copilot]

Pull request overview

Copilot reviewed 94 out of 95 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (5)

test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1

• The SkipIntegrationTestBuild flag logs that the build is being skipped, but the method continues and will still take the mutex and run the build steps. Return early after the common setup when skipBuild is true (or gate the build section on !skipBuild) so the flag actually has effect.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• The test parameter name is misspelled (BuildCompatiblity). In eng/build.ps1 the parameter is set as BuildCompatibility, so compatibility builds will never be enabled. Rename this key to BuildCompatibility (and keep spelling consistent across all producers/consumers).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• If WaitOne(...) times out, a TimeoutException is thrown and ReleaseMutex() is still called in finally, but the mutex won't be owned by this thread—ReleaseMutex() will throw and can mask the real timeout. Track whether the mutex was successfully acquired and only release when owned (e.g., acquired = createdNew || gotOne).
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBuild.cs:1
• Copying files into fullPath can throw DirectoryNotFoundException for nested paths because parent directories aren’t created. Create Path.GetDirectoryName(fullPath) first (when non-null) and Directory.CreateDirectory(...) before calling File.Copy.
  test/Microsoft.TestPlatform.TestUtilities/IntegrationTestEnvironment.cs:1
• Correct the grammar in the assert message: “You need to changed” → “You need to change”.