Improve performance of workload installation

[jonathanpeppers]

Is your feature request related to a problem?

dotnet workload install maui seems to take about 6 minutes on the Mac hosted pool, while the dotnet-install script only takes ~13 seconds. See an example here:

https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=5972136&view=logs&jobId=96c3d711-2288-532e-87f7-1a666089720f&j=96c3d711-2288-532e-87f7-1a666089720f&t=907c59ba-a277-5b2e-83ee-ef1e51b38284

Is there maybe some performance issues to look into here? We could also look into reducing packs in the maui workload if that would help.

This problem might be worse on Windows because of the use of .msi, we can get an example of that if needed.

Describe the solution you'd like

If dotnet workload install maui only took a couple minutes, that would be great.

[jonathanpeppers]

Ok, so some details on this issue.

Here is an example of when only the NuGet.org source is used:

https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=6041692&view=logs&j=96c3d711-2288-532e-87f7-1a666089720f

So using multiple NuGet sources seems to influence the workload install command quite a bit? It goes from 6m 17s to 1m 40s.

[joeloff]

Yeah, NuGet's performance can degrade quickly when multiple feeds are being searched. Since individual packs don't have dependencies like traditional packages, we might be able to make gains if the downloads happened in parallel since that should be when the feeds are being searched. Parallel installs probably won't make a difference and even if it did, it would only benefit file based installs since MSIs must be executed sequentially.

[joeloff]

So, MSI installs will be slower than package based installations. There's a lot that happens. Every file is backed by a separate component, every component gets its own entry in the registry, which does add additional I/O. And becasue we don't support upgrades for workloads, we have to do full uninstalls/installs, which when you have large MSIs with lots of files can take very long.

[marcpopMSFT]

The performance of the install is affected by a few things:

• How many installers are being run
• How many files are getting installed and their size
• How many feeds are configured
• If there is any cleanup being done of prior installs

I don't think we can do much about feeds. @jonathanpeppers , am I correct that this was a clean machine without existing files and the feed was the primary source of pain? Is it worth a flag to generate output for the performance of the nuget search, download, and install pieces?

[jonathanpeppers]

@marcpopMSFT yes, this should have been a clean install. It was the Mac Hosted Pool on Azure DevOps, fresh install of .NET 6, followed by dotnet workload install maui.

If the performance of workload commands is something being worked on for .NET 7 (or future), let me know if I can help. We could try attaching dotnet trace and archiving a .nettrace or .speedscope file without having to make code changes or new flags.

[joeloff]

@jonathanpeppers the dotnet-install script only installs the SDK/runtimes. That's significantly smaller compared to the maui workload I think. Do you recall how many feeds were involved in the original build?

[jonathanpeppers]

I retained the build forever when I filed this, can you see the full log here?

https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=5972136&view=logs&j=96c3d711-2288-532e-87f7-1a666089720f&t=907c59ba-a277-5b2e-83ee-ef1e51b38284&l=11

At the time, the command was:

dotnet workload install maui --verbosity diag --from-rollback-file https://aka.ms/dotnet/maui/rc.1.json --source https://pkgs.dev.azure.com/dnceng/public/_packaging/darc-pub-dotnet-runtime-323bf2dd/nuget/v3/index.json --source https://pkgs.dev.azure.com/dnceng/public/_packaging/darc-pub-dotnet-emsdk-572aeedc/nuget/v3/index.json --source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json --source https://api.nuget.org/v3/index.json

So we had a feed for dotnet/runtime, emsdk, dotnet6, and then NuGet.org.

[joeloff]

I can access the full log, thank you

[jonathanpeppers]

@joeloff is there a way to:

1. cache the downloaded .nupkg files in a folder somewhere
2. subsequent workload commands can use the cached files instead of downloading again

This would help speed up CI and local development for some of our repos. Is there a way to do something like this?

[joeloff]

There is a --download-to-cache option (hidden). Also, for MSI based installs we do create a protected cache under ProgramData. I can't recall why we had this option hidden - it could easily be share across feature bands and having it globally configurable would make sense I think. Maybe something to chat about

[grendello]

@joeloff I tried --download-to-cache and --from-cache, but alas, on Linux it fails to work because while --download-to-cache saves all nugets in files with lower-case names, --from-cache expects to find files with the original file name case. Not sure if I should file a separate issue for this? The options being hidden, are they "officially" supported?

[jeromelaban]

As of .NET 7.0, installing workloads for ios, android, catalyst and macos takes about 1.5 minutes to complete on an azure devops hosted agents. Is there some additional work planned to improve this further?

[marcpopMSFT]

It's gated on the number and size of the packages. I wonder if maybe we could parallelize the nuget downloads to get some performance though that would end up with network/io bottlenecks. For windows admin installs, the msi installs would have to be serial so there are limits of what we could improve there other than ensuring the downloads are gated by the installs of the prior packs.

@joeloff want to try doing the download in parallel and see what happens?

Probably the biggest improvement would be workloads adopting workload pack groups which would combine workload packs into fewer numbers of nugets.

[Peppe426]

Takes > 6 min to run: run: dotnet workload install maui

Workflow file

Name: .NET
on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build:
    runs-on: windows-latest

    steps:
    - uses: actions/checkout@v3
    - name: Setup .NET
      uses: actions/setup-dotnet@v3
      with:
        dotnet-version: 7.0.x              
    - name: Install .NET MAUI Workload     
      run: dotnet workload install maui   
      
    - name: Restore dependencies
      working-directory: ./MyRepo
      run: dotnet restore
    - name: Build
      working-directory: ./MyRepo
      run: dotnet build --no-restore
    - name: Test
      working-directory: ./MyRepo
      run: dotnet test --no-build --verbosity normal

[jonathanpeppers]

@Peppe426 my understanding is the windows-latest images have .NET MAUI preinstalled now.

Are you running the workload commands because you want a newer/specific version of the maui workload?

[Peppe426]

I did not know about windows-latest having the .NET maui preinstalled, can I just leave out the run: dotnet workload install maui ?

[jonathanpeppers]

They list it on the image here:

https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md

[cmw9706]

Just tested on windows-latest and did not install maui and it failed?

To build this project, the following workloads must be installed: maui-tizen

[mip1983]

I have similarly terrible performance with workload install, in my case it spends a long time on 'Installing workloads: wasm-tools' (app is blazor web). I've tried switching the agent from ubuntu to windows latest, it doesn't seem to help. It's wildly variable, 3-9 minutes.