Produce SDK-only archive file without shared frameworks

[mthalman]

Currently the SDK is shipped as a monolithic archive file that includes the SDK and shared frameworks (Microsoft.NETCore.App, Microsoft.AspNetCore.App, Microsoft.WindowsDesktop.App). But there is a need for the ability to construct a .NET installation piecemeal.

A scenario where this would be used is with Docker images. As part of the changes in dotnet/dotnet-docker#1351 and dotnet/dotnet-docker#1814, the SDK image will now be based on the ASP.NET image, which itself is based on the runtime image. This kind of layering means that each framework will be installed individually. The runtime image installs Microsoft.NETCore.App and dotnet.exe. The ASP.NET image builds on top of the runtime image by also installing Microsoft.AspNetCore.App. The SDK image builds on top of the ASP.NET image by also installing the SDK and Microsoft.WindowsDesktop.App. This layered approach requires that the SDK and frameworks be made available as separate archive files so they can be retrieved individually.

Another scenario is the concept of a single SDK that can be shipped with multiple runtime versions.

Both scenarios are discussed here: dotnet/dotnet-docker#1482 (comment).

Without this kind of componentization of the SDK archive, it requires implementers of piecemeal installation to have very specific knowledge of the .NET installation folder structure so that it only pulls out the components it needs. For example, the command to retrieve just the SDK-only components from the SDK archive file would look something like this: tar -C /usr/share/dotnet -oxzf dotnet.tar.gz ./packs ./sdk ./templates. This kind of dependency with consumers is fragile and can easily lead to breakage if changes are made to the folder structure.

The proposal is to produce an archive file of the SDK that contains only the components that are SDK-specific and none of the shared frameworks. This would essentially consist of the following folders: packs, sdk, templates.

Another thing to consider is LICENSE.TXT and ThirdPartyNotices.txt, related to dotnet/installer#7043, which should probably be moved into a relevant sub-folder.

This work should also be done in conjunction with https://github.com/dotnet/wpf/issues/2865. Without the WindowsDesktop package being available as a separate archive file, having an SDK-only archive file doesn't provide much benefit for Windows scenarios.

cc @MichaelSimons, @richlander @mmitche

[mthalman]

@marcpopMSFT - Just wanted to make you aware that we've now authored and shipped Docker images that expose the unfortunate need to be selective about what is pulled out of the SDK archive file. Example: https://github.com/dotnet/dotnet-docker/blob/86586d6f1823014ae033513d5dce4130d8fe4ce1/5.0/sdk/alpine3.11/amd64/Dockerfile#L30

It would be ideal if this could be implemented for 5.0 to avoid the need for this kind of hacky workaround.

[marcpopMSFT]

We are planning on looking at the installation of the sdk in the .net 6 timeframe and this request seems to dovetail with those plans. @wli3 and @KathleenDollard are working on those plans. .net 5 looks pretty tight at the moment for us so I don't think we can take on additional work though we'll be revisiting our plans in a month or too based on what progress we've made to our committed work.

[wli3]

Although I'd like to understand why "the SDK image will now be based on the ASP.NET image, which itself is based on the runtime image".

The SDK zip should have an aspnet runtime and a base runtime. Why we cannot base the image on sdk zip?

Also, we don't test against arbitrary runtime with a certain SDK at all. And it is only tested against one runtime which is in the bundle.

[mthalman]

Although I'd like to understand why "the SDK image will now be based on the ASP.NET image, which itself is based on the runtime image".

The SDK zip should have an aspnet runtime and a base runtime. Why we cannot base the image on sdk zip?

We can. It's just not ideal to do that because it requires hard-coding to get the SDK-specific components out of the sdk zip file. If the contents or folder structure changes, then this could lead to a breaking change for consumers, such as our Dockerfiles, that need to do this sort of thing.

Also, we don't test against arbitrary runtime with a certain SDK at all. And it is only tested against one runtime which is in the bundle.

All that's being said here is that this sort of componentization would help to support such a scenario. There's nothing saying that it needs to be supported. There has been discussion of this scenario though.

[wli3]

We can. It's just not ideal to do that because it requires hard-coding to get the SDK-specific components out of the sdk zip file.

What component do you need to remove? SDK need the base runtime to function, so the only thing can be removed is aspnet runtime.

this sort of componentization would help to support such a scenario.

That depends. Currently, SDK has a hard dependency on the latest runtime(the time it ships). And aspnet runtime version need to be in sync with base runtime to avoid dll hell. For 6.0, we plan to make aspnet runtime separately installable. But I think we will still bundle a base runtime. That goes back to my question about, why do you need to separate out the base runtime?

[mthalman]

What component do you need to remove? SDK need the base runtime to function, so the only thing can be removed is aspnet runtime.

The hierarchy of the Docker images is this:
Runtime <- ASP.NET <- SDK

The runtime image only installs the runtime bits. The ASP.NET image only installs the ASP.NET bits but it "inherits" the runtime bits from the runtime base image. The SDK image only installs the SDK bits but it "inherits" the ASP.NET and runtime bits from its base image. So the runtime bits are there, we just don't need the runtime bits contained in the SDK zip. All the build versions are in sync because of how each base image is referenced.

[MichaelSimons]

What component do you need to remove? SDK need the base runtime to function, so the only thing can be removed is aspnet runtime.

As Matt pointed out in this discussion, there are also scenarios in which we also want to install the SDK without the associated templates and targeting packs. This is because we want to build SDK images with the latest SDK version but are designed for building apps that target older runtime versions. In this scenario we don't want the latest templates and targeting packs, rather we will acquire the templates and targeting packs for the older target version. In this scenario we will be installing two runtimes - the one for the SDK to use plus the older target runtime version.

[wli3]

@KathleenDollard

The hierarchy of the Docker images is this:
Runtime <- ASP.NET <- SDK

Any reason need to base on a runtime image first instead of base on SDK entirely?

This is because we want to build SDK images with the latest SDK version but are designed for building apps that target older runtime versions. In this scenario we don't want the latest templates and targeting packs, rather we will acquire the templates and targeting packs for the older target version.

some not connected thoughts

1. this is a should work but we have zero test for them.
2. on technical side. Creating another package flavor of SDK is very expensive and confusing. I hope we can leverage 6.0 work. But we haven't considered separating runtime and templates completely yet, especially cross different versions.
3. If we ever disassemble SDK into runtime and templates. I don't think the API will be easier than just moving the folders around. One reason of we don't like moving folders around is we don't have confidence of that it will work -- we still don't, if we build an API around it.

[MichaelSimons]

The hierarchy of the Docker images is this:
Runtime <- ASP.NET <- SDK

Any reason need to base on a runtime image first instead of base on SDK entirely?

Yes, with Docker, layer sharing is a very important concept. If a developer or CI system pulls the sdk, aspnet, and runtime images for the same version, they would have one copy of the runtime layer on disk. Given the total size of the SDK, this is something we provide of significant value.

[marcpopMSFT]

We're talking to the VS setup team and there may be some changes coming down that will help with this as we have a similar ask for msbuild without the sdk. @rainersigwald and I are collecting the details and will see if that would help not just msbuild but this scenario as well.