Updating a `workload` during the "patch Tuesday" release window corrupts most dotnet commands on Linux and macOS

[StudioLE]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When dotnet workload install wasm-tools (or any other workload) is run on a linux or macOS machine during the "patch Tuesday" release window it corrupts most of the dotnet commands.

This issue was first noted on a GitHub Actions build but subsequently experienced by multiple users on their local machines.

Command and output:

$ dotnet workload install wasm-tools

Skipping NuGet package signature verification.
Skipping NuGet package signature verification.
Installing workload manifest microsoft.net.sdk.android version 34.0.95...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8043...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8043...
Installing workload manifest microsoft.net.sdk.maui version 8.0.7...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043...
Installing workload manifest microsoft.net.workload.mono.toolchain.current version 8.0.4...
Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12...
Workload installation failed. Rolling back installed packs...
Installation rollback failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Workload installation failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Error: Process completed with exit code 1.

Expected Behavior

Expected to be able to update a workload at any time without it corrupting the local system.

Steps To Reproduce

Execute the following during a "patch Tuesday" release window when some but apparently not all dependencies have been released.

dotnet workload install wasm-tools

Exceptions (if any)

No response

.NET Version

8.0.202

Anything else?

Developers should not have to have knowledge of the .NET release process. The tooling should have checks built in to ensure that any dependencies it's installing are actually available.

The tools should either:

• Fail with a warning that not all dependencies of the next version are available, or
• Install the existing working version - ie as 8.0.4 was only partially released the tool should have fallen back to 8.0.3

[martincostello]

The 8.0.4 release is in the process of being published as we speak as part of Patch Tuesday, but it's not an atomic process and takes some time to complete, culminating with it being available from the .NET website for download.

Does your CI pin all of your dependencies to specific versions, or do you have some floating versions somewhere? If the latter, then if the CI runs while a new monthly release is being published, then there will be a window of time where the versions aren't coherent and may be incompatible with each other, like you've experienced.

[StudioLE]

The issue isn't constrained to my CI environment.

I made the error of running dotnet workload install wasm-tools on my local (Linux) machine and now it appears most dotnet related commands are now failing and I've no idea how to remove the broken wasm-tools because the tools to install/uninstall depend on dotnet.

Failing

dotnet new console
Error: Failed to retrieve template packages from provider 'Optional workloads'.
Details: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Error: Failed to retrieve template packages from provider 'Optional workloads'.
Details: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Error: Failed to retrieve template packages from provider 'Optional workloads'.
Details: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Error: Failed to retrieve template packages from provider 'Optional workloads'.
Details: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
The template "Console App" was created successfully.

Processing post-creation actions...
Restoring /e/Repos/C#/Test/Test.csproj:
/e/Repos/C#/Test/Test.csproj : error MSB4242: SDK Resolver Failure: "The SDK resolver "Microsoft.DotNet.MSBuildWorkloadSdkResolver" failed while attempting to resolve the SDK "Microsoft.NET.Sdk". Exception: "Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
/e/Repos/C#/Test/Test.csproj : error MSB4242:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests()
/e/Repos/C#/Test/Test.csproj : error MSB4242:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.Create(IWorkloadManifestProvider manifestProvider, String dotnetRootPath, String sdkVersion, String userProfileDir)
/e/Repos/C#/Test/Test.csproj : error MSB4242:    at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, String dotnetRootPath, String sdkVersion, String userProfileDir, String globalJsonPath)
/e/Repos/C#/Test/Test.csproj : error MSB4242:    at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.WorkloadSdkResolver.Resolve(SdkReference sdkReference, SdkResolverContext resolverContext, SdkResultFactory factory)
/e/Repos/C#/Test/Test.csproj : error MSB4242:    at Microsoft.Build.BackEnd.SdkResolution.SdkResolverService.TryResolveSdkUsingSpecifiedResolvers(IReadOnlyList`1 resolvers, Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio, SdkResult& sdkResult, IEnumerable`1& errors, IEnumerable`1& warnings)""
Restore failed.
Post action failed.
Manual instructions: Run 'dotnet restore'

dotnet workload install wasm-tools
Unhandled exception: Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests()
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.Create(IWorkloadManifestProvider manifestProvider, String dotnetRootPath, String sdkVersion, String userProfileDir)
   at Microsoft.DotNet.Workloads.Workload.Install.WorkloadResolverFactory.Create(String globalJsonStartDir)
   at Microsoft.DotNet.Workloads.Workload.InstallingWorkloadCommand..ctor(ParseResult parseResult, IReporter reporter, IWorkloadResolverFactory workloadResolverFactory, IInstaller workloadInstaller, INuGetPackageDownloader nugetPackageDownloader, IWorkloadManifestUpdater workloadManifestUpdater, String tempDirPath)
   at Microsoft.DotNet.Workloads.Workload.Install.WorkloadInstallCommand..ctor(ParseResult parseResult, IReporter reporter, IWorkloadResolverFactory workloadResolverFactory, IInstaller workloadInstaller, INuGetPackageDownloader nugetPackageDownloader, IWorkloadManifestUpdater workloadManifestUpdater, String tempDirPath, IReadOnlyCollection`1 workloadIds)
   at Microsoft.DotNet.Cli.WorkloadInstallCommandParser.<>c.<ConstructCommand>b__6_0(ParseResult parseResult)
   at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

dotnet workload uninstall wasm-tools
Unhandled exception: Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests()
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.Create(IWorkloadManifestProvider manifestProvider, String dotnetRootPath, String sdkVersion, String userProfileDir)
   at Microsoft.DotNet.Workloads.Workload.Install.WorkloadResolverFactory.Create(String globalJsonStartDir)
   at Microsoft.DotNet.Workloads.Workload.Uninstall.WorkloadUninstallCommand..ctor(ParseResult parseResult, IReporter reporter, IWorkloadResolverFactory workloadResolverFactory, INuGetPackageDownloader nugetPackageDownloader)
   at Microsoft.DotNet.Cli.WorkloadUninstallCommandParser.<>c.<ConstructCommand>b__4_0(ParseResult parseResult)
   at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

dotnet workload list                
Unhandled exception: Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests()
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.Create(IWorkloadManifestProvider manifestProvider, String dotnetRootPath, String sdkVersion, String userProfileDir)
   at Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper..ctor(Boolean isInteractive, VerbosityOptions verbosity, String targetSdkVersion, Nullable`1 verifySignatures, IReporter reporter, IWorkloadInstallationRecordRepository workloadRecordRepo, String currentSdkVersion, String dotnetDir, String userProfileDir, IWorkloadResolver workloadResolver)
   at Microsoft.DotNet.Workloads.Workload.List.WorkloadListCommand..ctor(ParseResult parseResult, IReporter reporter, IWorkloadInstallationRecordRepository workloadRecordRepo, String currentSdkVersion, String dotnetDir, String userProfileDir, String tempDirPath, INuGetPackageDownloader nugetPackageDownloader, IWorkloadManifestUpdater workloadManifestUpdater, IWorkloadResolver workloadResolver)
   at Microsoft.DotNet.Cli.WorkloadListCommandParser.<>c.<ConstructCommand>b__6_0(ParseResult parseResult)
   at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

dotnet workload repair
Unhandled exception: Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests()
   at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.Create(IWorkloadManifestProvider manifestProvider, String dotnetRootPath, String sdkVersion, String userProfileDir)
   at Microsoft.DotNet.Workloads.Workload.Install.WorkloadResolverFactory.Create(String globalJsonStartDir)
   at Microsoft.DotNet.Workloads.Workload.Repair.WorkloadRepairCommand..ctor(ParseResult parseResult, IReporter reporter, IWorkloadResolverFactory workloadResolverFactory, IInstaller workloadInstaller, INuGetPackageDownloader nugetPackageDownloader)
   at Microsoft.DotNet.Cli.WorkloadRepairCommandParser.<>c.<ConstructCommand>b__5_0(ParseResult parseResult)
   at System.CommandLine.Invocation.InvocationPipeline.Invoke(ParseResult parseResult)
   at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, TimeSpan startupTime, ITelemetry telemetryClient)

Working:

dotnet tool list --global
Package Id            Version             Commands        
----------------------------------------------------------

[martincostello]

When I said "CI", what I meant "build and source control" rather than specifically just your Continuous Integration environment.

I'm not aware of a way to get dotnet workload install to use a specific version. According to the documentation it's appears to be tied to the .NET SDK version used.

The .NET 8.0.4 release changes were merged a few minutes ago (dotnet/core#9262), my own personal automation has even kicked in updating my repositories (martincostello/alexa-london-travel#1150), so if you re-run now or in a few minutes, you'll probably find the problem will have resolved itself.

As I originally noted though, unless there's a way for your code to pin all the versions (e.g. with a global.json file, etc.), then there will be a period during a new release when the versions won't all be aligned/available until the rollout is completed.

[StudioLE]

I believe I've managed to cleanup the mess dotnet workload install wasm-tools made of my system, but it was incredibly hacky.

Basically, I replaced /usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4 with the 8.0.3 version.

cd /usr/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/
rm -rf 8.0.4
cp -r 8.0.3 8.0.4

I also updated the dotnet sdk to 8.0.203 but no idea if that was necessary.

This allowed me to finally run

dotnet workload uninstall wasm-tools
dotnet workload update

Which seems to have stabilised things as this now works:

dotnet new console
The template "Console App" was created successfully.

[KalleOlaviNiemitalo]

I'm not aware of a way to get dotnet workload install to use a specific version.

It seems something like that is being added for .NET SDK 8.0.300: dotnet/sdk#37681 (--version from the command line), dotnet/sdk#39644 (workloadVersion from global.json).

[MaximG1234]

Thanks heaps @StudioLE, I ran into this issue with MAUI and following your suggested steps worked for me. I don't think the SDK version matters as I already had 8.0.203 installed.

My specific error was: Unhandled exception: Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current'

[StudioLE]

@MaximG1234 Out of interest what action was it that led to the corrupted state? Were you also installing or updating a workload during the .NET team's "patch Tuesday" update window?

[MaximG1234]

@StudioLE I suppose it must have been, I initially attempted to install MAUI workloads at around maybe 12am-1AM Melbourne, Australia time yesterday/today.

[MaximG1234]

The exact command was: dotnet workload install maui

PS: This was on a mac BTW so your title could potentially be further adjusted.

[martincostello]

This issue isn't ASP.NET Core specific from what I can tell as it looks like it would affect any workload.

It looks like #55032 (comment) mentions issues that will either improve or fix this in a future SDK feature release.

I'm not sure if we should transfer the issue to dotnet/sdk, or close this as an effective duplicate of the issues those fix PRs relate to.

[StudioLE]

@martincostello I've updated the title and the initial bug description to more accurately describe the issue following the discussion.

It looks like #55032 (comment) mentions issues that will either improve or fix this in a future SDK feature release.

I don't believe either dotnet/sdk#37681 or dotnet/sdk#39644 would resolve the issue. They might work to pin to a specific workload version as you noted but they don't resolve the issue itself.

This issue isn't ASP.NET Core specific from what I can tell as it looks like it would affect any workload.
...
I'm not sure if we should transfer the issue to dotnet/sdk, or close this as an effective duplicate of the issues those fix PRs relate to.

It's not a duplicate of dotnet/sdk#37681 or dotnet/sdk#39644 but it does seem there's probably a different repository that it's more appropriate for this to be transferred to.