Nuget restore/package download C# API

[wli3]

This was raised during development of dotnet tools. The decision was "try to use a temp project and restore the package for now". However, after using this mechanism for 2 years. We still wish to have a C# API to do restore, especially considering we would reuse the dotnet tool install component to install dotnet SDK workloads. Dotnet SDK workloads installation experience needs to be bulletproof or the user will be blocked using Xamarin.

Issues for existing temp project approach

1. Multiple IO operation and process hops caused unreliability and they are very hard to reproduce and resolve. First a file needs to be written to the disk. And a restore in a separate process need to write more files. The following 2 issues are likely caused by race conditions. However, I cannot reproduce them. The cleanup of the temporary files is also tricky.

   a. .NET 5: Unable to install some .NET tools (dotnet tool install) · Issue #14547 · dotnet/sdk (github.com)

   b. Failed to install tool package ... · Issue #11459 · dotnet/sdk (github.com)

2. Hard-to-act-on error messages. Because the communication between SDK and nuget restore is via the command line. When error happens, SDK can only redirect the output of nuget and SDK cannot give a more specific guidance for the user to act on. That results the long list of "probably this happened." https://docs.microsoft.com/en-us/dotnet/core/tools/troubleshoot-usage-issues#executable-file-not-found

3. Environment leak in. Because there is an explicit msbuild run. Environment variables get leaked in. There is no good way to prevent them.

   a. `dotnet tool install ...` fails if environment variable RESTOREPROJECTS is set · Issue #10798 · dotnet/sdk (github.com)

4. Escape unescaped madness. Because the communication is done via the command line. SDK need to carefully escape/unescape characters. It is not trivial and error prone.

Requirement

It should be a replacement for the temp project. It should select correct TFM and generate asset.json. However, if we can get an API that can simply download package to a specific location, that would fit net6.0 requirement. But it won't help us making tools experience better.

[wli3]

@nkolev92 @aortiz-msft we hope to discuss this in next Monday sync meeting

[nkolev92]

I think the requirements for the workload download are not equivalent and as such need different solutions.

Tools rely on detecting configuration, but also doing asset selection, while it's not clear that workload download will need any type of asset selection.

I think migrating to use tool restore without a project file is possible, but we need to evaluate the work.

A more generic API for downloading also technically exists, but there's many gotchas depending on the requirements.

However, if we can get an API that can simply download package to a specific location, that would fit net6.0 requirement

I think that's not enough for tools. You need asset selection, which is why restore was the preferred implementation.

@wli3 Before the meeting, it'd be helpful if you can describe the workload scenarios in more detail.

[wli3]

Agree. Workload scenario has less requirement than tools. For workloads, it need to support input of: package id, version range, location to restore to, RestoreRootConfigDirectory -- path where nuget starts to probe configuration (like nuget config), optional path for nuget config, sources urls as an array. This is what I can think of so far. It does not require nuget to filter assets and produce asset.json like dotnet tools. And we should have C# exception when error happens so SDK could do recovery or issue helpful messages.

The main goal is to download nuget packages while no need for SDK team to reimplement http retry, caching and supports for custom/multiple/local feed etc.

However, it is highly desired that it also supports tools scenario. Or we will have 2 different package download system (3 if you count the package download in msbuild) inside SDK.

[nkolev92]

Thanks for clarifying the scenario @wli3, that helps.

[dsplaisted]

As part of the API, it would be nice to have a way to report download progress.

[nkolev92]

As part of the API, it would be nice to have a way to report download progress.

We have log reporting for individual files, but not for progress on a single file.