Consolidate Helix shards for quick test assemblies via MethodLimitMultiplier#54115
Merged
MichaelSimons merged 6 commits intoMay 6, 2026
Merged
Conversation
…tiplier
Add per-assembly MethodLimitMultiplier metadata support to reduce wasted
Helix compute on assemblies whose shards all complete very quickly. The
multiplier increases the methods-per-shard threshold, producing fewer but
larger work items and reducing per-item setup overhead.
Applied to three assembly groups:
- Microsoft.CodeAnalysis.NetAnalyzers (x5)
- Microsoft.TemplateEngine.* / TemplateSearch.* (x10)
- Microsoft.DotNet.Api{Compatibility,Diff,SymbolExtensions} (x5)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TE assemblies are already single-shard on FullFramework at x10. The x20 bump collapses the remaining 2-shard assemblies on TestBuild legs (Cli.UnitTests, Core.UnitTests, Orchestrator) to single shards. Api group bumped to x10 and expanded to include GenAPI.Tests and PackageValidation.Tests for additional shard savings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MichaelSimons
force-pushed
the
consolidate-helix-shards
branch
from
bbc8430 to
db5ebf7
Compare
This was referenced Apr 28, 2026
Open
This was referenced Apr 29, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a per-assembly Helix sharding knob for test projects so selected fast-running assemblies can run in fewer, larger work items. In the SDK test infrastructure, this lets UnitTests.proj tune shard density through item metadata instead of hard-coding assembly-specific behavior deeper in the task implementation.
Changes:
- Adds
MethodLimitMultiplierhandling toSDKCustomCreateXUnitWorkItemsWithTestExclusionand uses it to scale the scheduler’s per-shard method limit. - Applies multiplier metadata in
UnitTests.projfor NetAnalyzers, TemplateEngine/TemplateSearch, and selected API test assemblies. - Keeps the existing default shard thresholds intact for assemblies that do not opt into the new metadata.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
test/UnitTests.proj |
Adds MethodLimitMultiplier metadata to selected SDKCustomXUnitProject items to reduce Helix shard counts for targeted test groups. |
test/HelixTasks/SDKCustomCreateXUnitWorkItemsWithTestExclusion.cs |
Reads the new metadata, validates it, and applies the scaled method limit when partitioning test assemblies into Helix work items. |
marcpopMSFT
approved these changes
May 5, 2026
Update the XML docs on the XUnitProjects property to include the optional MethodLimitMultiplier metadata, keeping the documented inputs in sync with the implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/ba-g known intermitent failure in netanalyzer tests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
MethodLimitMultipliermetadata option forSDKCustomXUnitProjectitems to consolidate Helix test shards for assemblies that produce many very short work items. This reduces per-item overhead (payload download, agent allocation, setup, teardown) and makes us better citizens of the shared Helix queues.How it works
AssemblySchedulerpartitions each test assembly into shards based on amethodLimitthreshold (default 16, or 32 for FullFramework).MethodLimitMultipliersimply multiplies that threshold, producing fewer, larger shards. It is set via MSBuild item metadata inUnitTests.proj— no new config files.Changes
SDKCustomCreateXUnitWorkItemsWithTestExclusion.cs— Reads optionalMethodLimitMultipliermetadata; warns on invalid valuesUnitTests.proj— Applies multipliers to three assembly groups:Results
Comparison of latest PR validation run (build 1407886) against main (build 1405738).
Linux x64
Windows x64
macOS x64
FullFramework Windows
Overall
Each eliminated work item avoids the fixed per-item overhead of agent allocation, payload download, environment setup, and teardown. While this overhead is non-trivial, it varies by queue and machine and is difficult to measure precisely from a single comparison.
¹ TemplateEngine and Api assemblies are now at one shard per assembly — the minimum possible without cross-assembly consolidation. Further shard reduction for these groups would require changes to the Helix sharding model itself.