Produce composite crossgen2 image for entire aspnet docker contents

[davidwrighton]

Initial stab at producing composite image for aspnet docker container

• I had difficulty getting versioning to work without doing it all at once including producing an SDK image, this feels like something that should be better
• Crossgen2 is currently only published as a nuget package, so my technique for acquiring it is to dotnet publish some dummy app using crossgen2, then use the nuget image cache to find crossgen2. This isn't pretty, and it makes the build dependent on nuget package server install latency, but it does work.
• Docker multistage build has been used to control the size of the final image to something in the range of reasonable
• The produced image is 152MB, 86MB of which is the composite shared image with just the R2R code in it. Previous efforts on size reduction of the R2R images are not reflected in this number, and will need to be rebuilt
• No testing has been performed to see if this actually works. Crossgen2 R2R is currently relatively untested on ASP.NET scenarios, and composite mode is completely untested in that scenario

[davidwrighton]

@richlander @MichaelSimons This is my intial stab at docker file creation for the aspnet scenario. Its not pretty, but it appears to work. Unfortunately it seems to have uncovered a bug in crossgen2 in that compilation is very time consuming. (~5 minutes on my machine) This time is spent mostly after finishing the actual JIT compilation and the system is single threaded, so we clearly have a non O(N) algorithm in there somewhere that is causing some havok.

[MichaelSimons]

Thanks @davidwrighton!

I'll work on the following:

1. Clean up the Dockerfile a little.
2. Run the unit tests.
3. Update the samples.
4. Split this across the runtime and aspnet image flavors.

What is the planned acquisition experience for the crossgen2 tool?

• part of the base SDK?
• an optional SDK component?
• global tool?
• other?

[davidwrighton]

I'm concerned about your task 4 that you called out here. For best performance we need to actually build 1 single composite image with crossgen that serves as the implementation for both the runtime and aspnet components of the image with the current tech. We could potentially split that up, but it will cause the collective composite image size to be larger. Can you explain what sort of work you expect to do here, and what the benefits would be?

In general, the crossgen2 acquisition experience for customers is expected to be through nuget for this release (The developer specifies a special property which injects an extra nuget package reference which contains the tool). For the docker build... well, no planning has been done, as we didn't have an expectation on how we would be doing the building. I expect we could probably package up the contents of the nuget package in a tarball for easier acquisition.

[davidwrighton]

Of course, we may need to move this all to official build, as this build process doesn't sign the resulting binaries and will in fact destroy any signatures that may exist on every managed dll in the container.

[MichaelSimons]

Yeah - that is going to be a blocker.

[MichaelSimons]

For best performance we need to actually build 1 single composite image with crossgen that serves as the implementation for both the runtime and aspnet components of the image with the current tech.

Can you explain more of what you mean by best performance? Is this startup, CPU usage, disk size or all of the above?

The .NET Core docker images build upon each other. runtime-deps -> runtime -> aspnet -> SDK. The SDK image is not currently based on the aspnet image but that is the desire. The reason for this is to optimize disk usage and reduce pull times for scenarios in which multiple image variants are used on a single machine. With the way the images are constructed today, if you have the runtime and aspnet image pulled, you only contain one copy of the runtime. This allows higher disk density when using out images.

[davidwrighton]

So, what composite R2R does is that it builds a collection of .NET dlls at once into a single binary blob that contains all of the code, and then rewrites all of the input managed .NET dlls so that they have a reference to that single shared composite image. There are two benefits to this system.

1. The compiler is permitted to inline from method to method even across .dll boundaries. (This is not typically true for R2R code.) This could easily be enabled by having a runtime composite image and a separate ASPNet compiled image, as long as we can guarantee that the aspnet composite is built with a known runtime image that will also be used at runtime) This has significant impact on throughput of applications before the system is able to rejit to higher tiered code, and put us in a better position to in the future have code which we know is good enough in the R2R image that it doesn't need to tier up, which should improve the rate at which applications achieve best throughput performance.

2. The compiler is able to compile generics such as List into a location where the runtime will find them. This is difficult to do without having as large a view as possible into the set of assemblies in use. This is the driving factor for pushing a crossgen strategy that produces one image shared across runtime and aspnet. This improves startup, and also provides better throughput for untiered performance at a cost to file size.

The benefits here primarily accrue to startup, and early process throughput, and thus are intimately intertwined with issues such as docker image size, which are also large drivers of startup cost and site density.

There are a number of significant drawbacks to composite images though, and those really need to be called out and explored.

1. Composite images fix exact versions of multiple binaries in place. At the moment the application must not override any of the dlls in the runtime/aspnet frameworks. At the moment, I believe this makes the composite image unuseable for scenarios that involve running Roslyn (the C# compiler), or WPF.

2. Composite images are expected to eventually provide better performance wins the group of assemblies grows larger. Here I put together an experiment with merging all of the runtime/aspnet assemblies. I would really like to see startup perf numbers which compare that with a layered docker approach. The primary customer driving the implementation of the feature is expecting to merge every assembly loaded into their application including the runtime assemblies into 1 exceptionally large image. (This will accumulate >1000 files on disk together).

Overall, there are a lot of questions here, and at the moment I would actually expect these composite images to be provided in addition to the existing non-composite images as they will break some applications, and are not a completely drop in replacement in all situations.

[davidwrighton]

@MichaelSimons If you think this is a good start to the process, could you commit this change to the branch? I don't have write permission here.