RestoreTask randomly fails after upgrading to latest version

[akoeplinger]

NuGet Product Used

dotnet.exe

Product Version

6.11.0-rc.90

Worked before?

NuGet in dotnet sdk 9.0-preview4

Impact

It's more difficult to complete my work

Repro Steps & Context

Since we updated to a dotnet 9.0-preview6 SDK we started seeing RestoreTask failing randomly in the VMR build: dotnet/sdk#41477 (comment)

After the fix in NuGet/NuGet.Client#5845 we bumped again (i.e. we're now using a very recent nuget.client) we now see these messages

D:\a_work\1\s.dotnet\sdk\9.0.100-preview.6.24311.23\NuGet.targets(170,5): The feed 'dotnet-public [https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json]' lists package 'System.Text.Json.7.0.3' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [D:\a_work\1\s\src\nuget-client\build\restorehelper.targets]

or

D:\a_work\1\s.dotnet\sdk\9.0.100-preview.6.24311.23\NuGet.targets(170,5): error : The feed 'dotnet-public [https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json]' lists package 'Castle.Core.5.1.1' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [D:\a_work\1\s\src\efcore\EFCore.sln]

I've attached some binlogs: binlogs.zip

It's happening for random packages and given we only started seeing this when bumping the dotnet SDK (and thus nuget) I think it's more likely a nuget bug than an issue with the AzDO feed.

Is there some way we can enable more logging?

/cc @nkolev92 @kartheekp-ms

Verbose Logs

No response

[lewing]

We're seeing this and #13551 repeatedly in CI is there something we can do to help get this resolved

cc @jebriede @aortiz-msft

[jeffkl]

I think what's going on is Azure DevOps Artifacts Services public upstreams is causing this. When a package is restored from a feed with upstream(s) for the first time, the account doing the restore must have Contributor permissions to populate the package. If the user is not a Contributor, the package will not be made available, and the server returns an HTTP 404.

https://learn.microsoft.com/en-us/azure/devops/artifacts/how-to/set-up-upstream-sources?view=azure-devops&tabs=nuget

The NuGet repo recently hit this problem and we had to get permissions fixed on the feed to get unblocked. I guess CI agents need to be using an account with the correct permissions in order to populate feeds? Can you confirm if this workflow is using upstream feeds?

[akoeplinger]

The dotnet-public feed doesn't have upstreams.

[jeffkl]

@nkolev92 do you know what's going on? It looks like the same symptoms where the feed lists a particular version but then NuGet receives a 404 when trying to download the package.

[akoeplinger]

Looking at the code this is the only place where PackageNotFoundProtocolException is thrown: https://github.com/NuGet/NuGet.Client/blob/20f05435be385abfe74737b6433dc80fd3b3b504/src/NuGet.Core/NuGet.Protocol/Utility/FindPackagesByIdNupkgDownloader.cs#L69-L88

Following ProcessNupkgStreamAsync feels like there might be a race condition hidden somewhere :)

[lewing]

Also tracking in dotnet/dnceng#3100 and dotnet/runtime#103526

[nkolev92]

@lewing, @akoeplinger

I'll reach out offline as well, but if possible, can anyone try using .NET 9, P4 and seeing if the problems persist.

[am11]

#13551 is tracked as dotnet/runtime#103526. Retry always fixes the error so it's intermittent but consistently failing couple of builds a day.