[9.0-preview.7] Microsoft.DotNet.MSBuildWorkloadSdkResolver fails with dotnet workload or build on Windows

[martincostello]

Describe the bug

Somewhere between 9.0.100-preview.7.24365.2 and 9.0.100-preview.7.24365.10 (diff) of the .NET SDK, something has started to cause dotnet workload restore and dotnet build to fail with the following exception on Windows (but not on Linux or macOS):

Unhandled exception: Microsoft.Build.Exceptions.InvalidProjectFileException: SDK Resolver Failure: "The SDK resolver "Microsoft.DotNet.MSBuildWorkloadSdkResolver" failed while attempting to resolve the SDK "Microsoft.NET.SDK.WorkloadAutoImportPropsLocator". Exception: "System.IO.FileNotFoundException: Workload manifest microsoft.net.sdk.aspire: 8.0.1/8.0.100 from workload version 9.0.100-preview.7.24367.5 was not installed. Running "dotnet workload repair" may resolve this.
   at Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider.GetManifests()
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.LoadManifestsFromProvider(IWorkloadManifestProvider manifestProvider)
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests()
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetInstalledWorkloadPacksOfKind(WorkloadPackKind kind)+MoveNext()
   at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, IWorkloadManifestProvider manifestProvider, IWorkloadResolver workloadResolver)
   at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, String dotnetRootPath, String sdkVersion, String userProfileDir, String globalJsonPath)
   at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.WorkloadSdkResolver.Resolve(SdkReference sdkReference, SdkResolverContext resolverContext, SdkResultFactory factory)
   at Microsoft.Build.BackEnd.SdkResolution.SdkResolverService.TryResolveSdkUsingSpecifiedResolvers(IReadOnlyList`1 resolvers, Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio, SdkResult& sdkResult, IEnumerable`1& errors, IEnumerable`1& warnings)""  C:\Program Files\dotnet\sdk\9.0.100-preview.7.[24](https://github.com/martincostello/costellobot/actions/runs/9972238641/job/27555071107?pr=1544#step:9:25)367.5\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props
   at Microsoft.Build.Shared.ProjectErrorUtilities.ThrowInvalidProject(String errorSubCategoryResourceName, IElementLocation elementLocation, String resourceName, Object[] args)
   at Microsoft.Build.Shared.ProjectErrorUtilities.ThrowInvalidProject[T1](IElementLocation elementLocation, String resourceName, T1 arg0)
   at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImportsFromUnescapedImportExpressionConditioned(String directoryOfImportingFile, ProjectImportElement importElement, List`1& projects, SdkResult& sdkResult)
   at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImports(String directoryOfImportingFile, ProjectImportElement importElement, SdkResult& sdkResult)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
   at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
   at Microsoft.Build.Evaluation.Evaluator`4.Evaluate()
   at Microsoft.Build.Evaluation.Evaluator`4.Evaluate(IEvaluatorData`4 data, Project project, ProjectRootElement root, ProjectLoadSettings loadSettings, Int[32](https://github.com/martincostello/costellobot/actions/runs/9972238641/job/27555071107?pr=1544#step:9:33) maxNodeCount, PropertyDictionary`1 environmentProperties, ILoggingService loggingService, IItemFactory`2 itemFactory, IToolsetProvider toolsetProvider, IDirectoryCacheFactory directoryCacheFactory, ProjectRootElementCacheBase projectRootElementCache, BuildEventContext buildEventContext, ISdkResolverService sdkResolverService, Int32 submissionId, EvaluationContext evaluationContext, Boolean interactive)
   at Microsoft.Build.Execution.ProjectInstance.Initialize(ProjectRootElement xml, IDictionary`2 globalProperties, String explicitToolsVersion, String explicitSubToolsetVersion, Int32 visualStudioVersionFromSolution, BuildParameters buildParameters, ILoggingService loggingService, BuildEventContext buildEventContext, ISdkResolverService sdkResolverService, Int32 submissionId, Nullable`1 projectLoadSettings, EvaluationContext evaluationContext, IDirectoryCacheFactory directoryCacheFactory)
   at Microsoft.Build.Execution.ProjectInstance..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectCollection projectCollection, Nullable`1 projectLoadSettings, EvaluationContext evaluationContext, IDirectoryCacheFactory directoryCacheFactory, Boolean interactive)
   at Microsoft.Build.Execution.ProjectInstance..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion)
   at Microsoft.DotNet.Workloads.Workload.Restore.WorkloadRestoreCommand.RunTargetToGetWorkloadIds(IEnumerable`1 allProjects)
   at Microsoft.DotNet.Workloads.Workload.Restore.WorkloadRestoreCommand.Execute()
   at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
   at System.CommandLine.ParseResult.Invoke()
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

I can repro this as recently as 9.0.100-preview.7.24367.5:

• Example for dotnet workload restore
• Example for dotnet build

Looking at the diff, the culprits could be either #41562 or #39991.

To Reproduce

1. Clone martincostello/project-euler@6727a6c
2. Run build.ps1 in the root of the repository

[Forgind]

This certainly looks bad!

I tried to reproduce it, and I'm having a little trouble. Any thoughts on what I'm doing wrong?

Windows --> windows powershell

I cloned your repo, checked out that commit, and verified that it included the global.json change to update the SDK.

I tried building, and it failed because of Join-Path. Apparently windows powershell is a bit out-of-date, and it doesn't permit Join-Path with more than 2 inputs, so I cobbled together several Join-Paths wherever you used one for > 2 arguments, and now it builds...

But it builds successfully with one warning that doesn't look relevant.

> ./build.ps1
The required version of the .NET SDK is not installed. Expected 9.0.100-preview.7.24367.5.
dotnet-install: Remote file https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.7.24367.5/dotnet-sdk-9.0.100-preview.7.24367.5-win-x64.zip size is 265904062 bytes.
dotnet-install: Downloaded file https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.7.24367.5/dotnet-sdk-9.0.100-preview.7.24367.5-win-x64.zip size is 265904062 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Extracting the archive.
dotnet-install: Binaries of dotnet can be found in C:\bugs\42211\project-euler\.dotnetcli\
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install/windows#dependencies
dotnet-install: Installed version is 9.0.100-preview.7.24367.5
dotnet-install: Installation finished
Building solution...

Welcome to .NET 9.0!
---------------------
SDK Version: 9.0.100-preview.7.24367.5

Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate, run 'dotnet dev-certs https --trust'
Learn about HTTPS: https://aka.ms/dotnet-https

----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
Restore complete (27.9s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  ProjectEuler succeeded (32.3s) → artifacts\bin\ProjectEuler\release\ProjectEuler.dll
  ProjectEuler.Benchmarks succeeded (6.1s) → artifacts\bin\ProjectEuler.Benchmarks\release\ProjectEuler.Benchmarks.dll
  ProjectEuler.Tests succeeded (5.1s) → artifacts\bin\ProjectEuler.Tests\release\ProjectEuler.Tests.dll

Build succeeded in 72.6s
Running tests...
Restore complete (1.2s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  ProjectEuler succeeded (0.2s) → artifacts\bin\ProjectEuler\release\ProjectEuler.dll
  ProjectEuler.Benchmarks succeeded (0.8s) → artifacts\bin\ProjectEuler.Benchmarks\release\ProjectEuler.Benchmarks.dll
  ProjectEuler.Tests succeeded (0.3s) → artifacts\bin\ProjectEuler.Tests\release\ProjectEuler.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.8.2+699d445a1a (64-bit .NET 9.0.0-preview.7.24365.1)
[xUnit.net 00:00:00.78]   Discovering: ProjectEuler.Tests
[xUnit.net 00:00:01.13]   Discovered:  ProjectEuler.Tests
[xUnit.net 00:00:01.14]   Starting:    ProjectEuler.Tests
The specified maximum value is invalid.
The specified maximum value is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified total is invalid.
The specified total is invalid.
At least 2 arguments are required.
At least 1 argument is required.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number of sides is invalid.
The specified number of sides is invalid.
The specified maximum value is invalid.
The specified maximum value is invalid.
The specified number of factors is invalid.
[xUnit.net 00:00:04.06]       Too slow.
The specified number of factors is invalid.
The specified number of divisors is invalid.
The specified maximum number is invalid.
The specified maximum number is invalid.
The specified maximum value is invalid.
The specified maximum value is invalid.
The specified number of sides is invalid.
The specified number of sides is invalid.
The specified number of sides is invalid.
The specified number of divisors is invalid.
The specified number of digits is invalid.
The specified number of digits is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
The specified number is invalid.
[xUnit.net 00:00:24.66]   Finished:    ProjectEuler.Tests
  ProjectEuler.Tests test succeeded with 1 warning(s) (29.2s)
    C:\bugs\42211\project-euler\.dotnetcli\sdk\9.0.100-preview.7.24367.5\Microsoft.TestPlatform.targets(46,5): warning : [xUnit.net 00:00:04.06]     Puzzle092_Returns_Correct_Solution [SKIP]

Test summary: total: 565, failed: 0, succeeded: 564, skipped: 1, duration: 29.1s
Build succeeded with 1 warning(s) in 33.1s

I suspect you have a workload set in an unexpected place, but I'd love to know more about it.

Here's my best guess:
We've had a baseline workload set officially being produced for a while now, but it hasn't really had any effect, partially because it's unconditionally in the 8.0.100 folder, and we never looked there. dsplaisted's refactoring now permits looking across feature bands, and it found that workload set, but that workload set was never properly installed, which means it didn't come with all the manifests it was supposed to have, and that made this fail.

Mostly coincidentally, I think your current problem will be mitigated by #41996 but only because that PR basically says "if you didn't ask for a workload version, we shouldn't give you a workload version," which means we wouldn't go looking for a manifest that was never installed.

Can you verify that you have a file named *.workloadset.json under a folder named 9.0.100-preview.7.24367.5 in your dotnet directory (under sdk-manifests\8.0.100\workloadsets, I think, but I haven't checked yet) and that it specifies the 8.0.1/8.0.100 version of microsoft.net.sdk.aspire? And can you try manually deleting that (along with parent folder(s) until they're not empty) and see if that resolves your issue?

[martincostello]

I typically just have the dailies ingest in the background in CI, rather than play around with them locally (unless I want to play with a brand new API that's just dropped). As these PRs are a failing, I haven't been pulling them down locally as I just pull from the target branch (dotnet-nightly).

Maybe it's something about the default SDK versions installed on the Windows GitHub Actions runners and actions/setup-dotnet works?

From the logs:

"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command & 'D:\a\_actions\actions\setup-dotnet\v4\externals\install-dotnet.ps1' -SkipNonVersionedFiles -Runtime dotnet -Channel LTS
dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Runtime/8.0.7/dotnet-runtime-8.0.7-win-x64.zip size is 33315858 bytes.
dotnet-install: Downloaded file https://dotnetcli.azureedge.net/dotnet/Runtime/8.0.7/dotnet-runtime-8.0.7-win-x64.zip size is 33315858 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Extracting the archive.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install/windows#dependencies
dotnet-install: Installed version is 8.0.7
dotnet-install: Installation finished
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command & 'D:\a\_actions\actions\setup-dotnet\v4\externals\install-dotnet.ps1' -SkipNonVersionedFiles -Version 9.0.100-preview.7.24367.5
dotnet-install: Remote file https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.7.24367.5/dotnet-sdk-9.0.100-preview.7.24367.5-win-x64.zip size is 265904062 bytes.
dotnet-install: Downloaded file https://dotnetbuilds.azureedge.net/public/Sdk/9.0.100-preview.7.24367.5/dotnet-sdk-9.0.100-preview.7.24367.5-win-x64.zip size is 265904062 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Extracting the archive.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install/windows#dependencies
dotnet-install: Installed version is 9.0.100-preview.7.24367.5
dotnet-install: Installation finished

[Forgind]

We're having issues reproducing this error. I mentioned what I did, and @baronfel looked into your github action and tried to reproduce it from there but also had difficulty.

It sounds like you haven't actually reproduced this locally. Can you try and see what comes of it? If it reproduces for you, we can try to see what's different between our setup and yours, and if it doesn't, we can look into what's different between CI and local builds.

Part of what's confusing is that the manifest that it references apparently should be installed already, so the fact that it can't find it means a poor error message, a broken install, or we're looking in the wrong spot. I was at least able to verify that the size of the SDK CI installed was the same as the size of the SDK that I installed, though it could've theoretically failed in extraction.

[martincostello]

I can't repro this locally either. If I checkout the branch and run build.ps1 it works and I can a file at the path .dotnetcli\sdk-manifests\9.0.100-preview.7\workloadsets\9.0.100-preview.7.24367.5\baseline.workloadset.json with the following content:

{
"Microsoft.NET.Sdk.Android": "34.99.0-preview.6.340/9.0.100-preview.6",
"Microsoft.NET.Sdk.iOS": "17.2.9714-net9-p6/9.0.100-preview.6",
"Microsoft.NET.Sdk.MacCatalyst": "17.2.9714-net9-p6/9.0.100-preview.6",
"Microsoft.NET.Sdk.macOS": "14.2.9714-net9-p6/9.0.100-preview.6",
"Microsoft.NET.Sdk.Maui": "9.0.0-preview.6.24327.7/9.0.100-preview.6",
"Microsoft.NET.Sdk.tvOS": "17.2.9714-net9-p6/9.0.100-preview.6",
"Microsoft.NET.Workload.Mono.ToolChain.Current": "9.0.0-preview.7.24365.1/9.0.100-preview.7",
"Microsoft.NET.Workload.Emscripten.Current": "9.0.0-preview.7.24352.2/9.0.100-preview.7",
"Microsoft.NET.Workload.Emscripten.net6": "9.0.0-preview.7.24352.2/9.0.100-preview.7",
"Microsoft.NET.Workload.Emscripten.net7": "9.0.0-preview.7.24352.2/9.0.100-preview.7",
"Microsoft.NET.Workload.Emscripten.net8": "9.0.0-preview.7.24352.2/9.0.100-preview.7",
"Microsoft.NET.Workload.Mono.ToolChain.net6": "9.0.0-preview.7.24365.1/9.0.100-preview.7",
"Microsoft.NET.Workload.Mono.ToolChain.net7": "9.0.0-preview.7.24365.1/9.0.100-preview.7",
"Microsoft.NET.Workload.Mono.ToolChain.net8": "9.0.0-preview.7.24365.1/9.0.100-preview.7",
"Microsoft.NET.Sdk.Aspire": "8.0.1/8.0.100",
}

Something that does stand out here (well, I had a squiggle when I opened the file in VS Code) regardless of whether it's actually related to this issue, I notice that the content isn't strictly valid JSON due to the trailing comma.

[martincostello]

I see the comma is catered for, so it's not that:

sdk/src/Resolvers/Microsoft.NET.Sdk.WorkloadManifestReader/WorkloadSet.cs

Line 72 in 18985b1

AllowTrailingCommas = true,

[martincostello]

This is what's in the corresponding C:\Program Files\dotnet\sdk-manifests\8.0.100 folder on my local machine:

[martincostello]

Looks like the GitHub Actions runner comes with 8.0.0 pre-installed.

[martincostello]

I also noticed that between the last successful build with the prior working version and the latest, the GitHub Actions runner image has changed from 20240707.1.0 to 20240714.1.0 (changes).

I re-ran it to pick up the new runner image to see if that is the motivating factor, but it still passes.

[martincostello]

Messing about in this PR it seems that the GitHub Actions runner has aspire 8.0.0/8.0.100 installed on it initially:

Installed Workload Id      Manifest Version       Installation Source
---------------------------------------------------------------------
maui-windows               8.0.61/8.0.100         VS 17.10.35027.167 
maccatalyst                17.2.8053/8.0.100      VS 17.10.35027.167 
ios                        17.2.8053/8.0.100      VS 17.10.35027.167 
android                    34.0.95/8.0.100        VS 17.10.35027.167 
wasm-tools                 8.0.7/8.0.100          VS 17.10.35027.167 
aspire                     8.0.0/8.0.100          VS 17.10.35027.167 
Use `dotnet workload search` to find additional workloads to install.

Then if I run actions/setup-dotnet for both 8.0.100 and 9.0.100-preview.7.24367.22 then I end up with 8.0.2/8.0.100:

Installed Workload Id      Manifest Version       Installation Source            
---------------------------------------------------------------------------------
android                    34.0.113/8.0.100       SDK 8.0.300, VS 17.10.35027.167
aspire                     8.0.2/8.0.100          SDK 8.0.300, VS 17.10.35027.167
ios                        17.2.8078/8.0.100      SDK 8.0.300, VS 17.10.35027.167
maccatalyst                17.2.8078/8.0.100      SDK 8.0.300, VS 17.10.35027.167
maui-windows               8.0.61/8.0.100         SDK 8.0.300, VS 17.10.35027.167
wasm-tools                 8.0.7/8.0.100          SDK 8.0.300, VS 17.10.35027.167
Use `dotnet workload search` to find additional workloads to install.

[martincostello]

I also notice that the sorting there isn't stable - is it worth me opening a PR to add a sort to WorkloadListCommand.Execute()?

[martincostello]

I can't work out an obvious way to get 8.0.1/8.0.100 manually as a work around. It starts out with 8.0.0, but any command I seem to run related to workloads then wants to download 8.0.2 unless I add --skip-manifest-update.

[MichalStrehovsky]

I'm also running into this even with a super vanilla setup: https://github.com/MichalStrehovsky/bug42211/pulls

There's almost nothing in the repo, just a hello world console app and a hello world Github action setup.