Approach for using Crossgen2

[richlander]

I was reading through #1775. It seems that there are a bunch of choices and trade-offs to make.

To date, we valued the following:

• Small (as possible) images.
• Creating dependencies between images to single instance binaries in docker layers as much as possible (for example, the aspnet image depends on the runtime one. That benefits in cases where you use both images. We intend to do more of this over time. The biggest win is getting the SDK to depend on the aspnet image (which then in turn depends on runtime). That would help docker-based builds alot!

Going forward, we are going to have to make more tradeoffs, and likely give up on some desirable scenarios.

I see the following rough-hewn options:

• Each layer is monolithic and optimized for itself (aspnet includes but does not depend on runtime; this matches the current state of the PR). Basically, aggregation not inheritance at every layer.
• The runtime uses composite images and nothing else. ASP.NET and the SDK build on top of the runtime (and the sdk might build on top of aspnet) and are in the same version bubble as the runtime but do not use composite images. Same with apps that build on top.
• Self-contained apps can generate fully composite and linked apps. (one layer on top of runtime-deps). This option stands along from the rest. It's not an alternative, but just another option for people to opt-into and support.

I'm attracted to the second option as the default story. My guess is that it would be a sweet spot, assuming crossgen supports it. This option enables the new crossgen2 features, while keeping us on our current path (described at the start of this issue).

/cc @davidwrighton, @MichaelSimons

[davidwrighton]

I see a fourth option which is the runtime uses composite images and aspnet does as well, but they do not use a unified composite image, but rather one built locally to the particular image. I believe there is a bit of size cost there, but I would like to measure it before I dismiss it out of hand.

All of this presumes that we can use the composite image for all of the scenarios, especially the SDK scenarios.

[richlander]

K. I had an assumption that we could only have one composite image. If we can have multiple, then it's really a variation on the second option. Cool.

Is the issue with the SDK that Roslyn uses a different version of System.Collections.Immutable or System.Reflection.Metdata, or both?

[richlander]

Obviously, let's perf test all the options, and then choosing among them will be easier.

[davidwrighton]

Yeah, the issue is around the assemblies in the standard runtime that are overridden. The same problem exists in asp.net, as more of their assemblies are overrideable, which might push us back to your initial strategy with option 2. I believe since we only document a small number of assemblies as overrideable in the runtime we might be able to finesse the issue with just a runtime image that doesn't include the problematic assemblies, but I'm not sure.

[richlander]

I'd like to say "don't do that if you want to use these images" or at least start out with that model. The nice things about Docker is that we have a lot more freedom to change these images in servicing than we do the mainline build.

[richlander]

I also forgot a major point I was wanting to make earlier around size. We want these images to be as small as possible. So, the thing we ABSOLUTELY DO NOT WANT is a build of the runtime that has IL+R2R images and then a composite R2R file beside it. We want IL binaries + a composite R2R file.

We were talking earlier about if and how we should change the build to accommodate this scenario. I think the main thing we want is a build that isn't crossgen'd (for Linux x64 and Linux ARM64). That will give us the most flexibility, will presumably be easier/quicker to process, and have the least size footprint. I am sure there are ways to get to the same result, but this particular ask seems like it might be the sweet spot.

I'd also like an SDK (and I think they already have this) that is just the CLI (msbuild, nuget, the CLI verbs and friends) but no ASP.NET or runtime. That will make it much easier to layer it on top of these other layers we have been discussing.

With both the ASP.NET and SDK layers, I'm wondering if we could have just one build that works on x64 and ARM64, and naturally, no R2R.

[davidwrighton]

There is no particular benefit to producing a non-R2R'd official build. The binaries produced by such a build when fed into composite image creation, are recreated without the R2R data, but with a marker that indicates that R2R data should be found in an associated composite image.

[richlander]

I hear you saying that there is no benefit to the final artifacts (they will be the same). Is there a benefit to processing time? We re-build the container images multiple times a month (and there are many of them) so processing time is a metric we care about.