.NET 8: Enable .NET product to be optimized by making SDK construction coherent

[richlander]

We want to enable various optimizations for the .NET product (particularly in containers). These optimizations require that each product layer includes unique binaries. This is not the case today. The SDK layer contains binaries that are present in lower layers.

A subsequent investigation further discovered that there is a lot of extra cruft in the SDK that shouldn't be there at all, and that may or not be a problem for the optimizations.

The following are the near-term optimizations. We may enable others over time.

• version bubbles
• composite images

Problem

The primary issue is that the SDK is a composition of apps. These apps are packaged as portable NuGet packages and then exploded into an SDK layout, without any processing to limit the payload to what's required for Windows x64 or Linux Arm64, for example. In addition, some of the apps have package references to the NuGet versions of assemblies that are in the Microsoft.NETCore.App layer.

We need the following characteristic for SDK apps (like dotnet-watch), either exactly as defined or analogous to it:

• published as framework-dependent, TFM- and RID-specific
• No PackageReferences (at least for the net* TFM) to NuGet packages that overlap with assemblies in dotnet/shared.
• A shared assembly model to enable single instancing common assemblies used by the SDK (think of this as a "software suite sharing mode", which should be built as a generic feature, not an SDK only one).

With that model in place, we can optimize the product in weird an wonderful ways with confidence.

Examples:

• Generate whole product in one version bubble, enabling better startup and throughput performance.​
• Enable consistent set of optimizations and application of training data across all binaries.​
• Functionally, be certain that our testing of System.Reflection.Metadata (for example) accrues to all scenarios.

Evidence

Here are some simple queries of assemblies that should not be in a Linux x64 container. These queries should all return zero results.

% docker run --rm mcr.microsoft.com/dotnet/sdk:6.0 find /usr/share/dotnet/sdk | grep System\. | wc -l   
     236
% docker run --rm mcr.microsoft.com/dotnet/sdk:6.0 find /usr/share/dotnet/sdk | grep runtimes/win | wc -l
      23
% docker run --rm mcr.microsoft.com/dotnet/sdk:6.0 find /usr/share/dotnet/sdk | grep /net4 | wc -l
     279

This change could reduce the size of the SDK by between 50-100MB.

We know that there are other files, within the SDK layer, that are duplicated. The queries above should not be considered exhaustive.

[ghost]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[richlander]

I wrote a tool (and captured the results) that better captures the coherence challenges via the various relationship arks.

https://gist.github.com/richlander/366a62d8877e856731e8cb284eb2bd70#gistcomment-3752543

[mthalman]

The addition of the dotnet-format tool for 6.0 Preview 7 (45 MB) just exacerbates this problem.

[richlander]

Ouch! That's more evidence that we don't have a sustainable model.

[richlander]

This will likely get solved via dotnet/source-build#2911, specifically the move to a vertical build. We'll see.

[richlander]

Closing, leaving #41128 as the tracking issue.