Latest Dockerfile for .NET Core 2.2 is using older version of MSBuild

[gabrielsadaka]

Steps to reproduce the issue

1. Create Dockerfile using mcr.microsoft.com/dotnet/core/sdk:2.2 as base image

Expected behavior

Uses latest version of MSBuild (16.2.32702+c4012a063)

Actual behavior

Uses older version of MSBuild (16.0.450+ga8dc7f1d34) which has a bug in Roslyn that is triggering the following build error that has since been fixed in later version of MSBuild dotnet/roslyn#34720.

Additional information (e.g. issue happens only occasionally)

This only started happening with the latest version of the Docker image published for the .NET Core SDK 2.2.207. The previous docker image was using .NET Core SDK 2.2.402 which has the fixed version of MSBuild.

Not sure if this is an issue with the image or the latest SDK published but to resolve this issue the SDK included in the latest 2.2 image should have the latest version of MSBuild to resolve the issue we are experiencing.

[mthalman]

cc @rainersigwald

[mthalman]

@livarcocc, @KathleenDollard, @leecow - There's a regression in the version of MSBuild that's available in the SDK between the previous version 2.2.402 and the latest drop 2.2.207. The Docker images were serviced to include 2.2.207 but that caused a regression from the previous version of the images. It seems that 2.2.207 was only serviced for fixes but not upgraded to use the latest tooling that was available in 2.2.402.

[leecow]

Adding @nguerrera as Livar is on vacation and this needs some thought now. This is a result of the 2.2.4xx and 2.1.8xx feature trains hitting the end of the tracks because we stopped delivering updates to VS 16.2.

[nguerrera]

@leecow Can you set up some time to chat on this?

The only options I see are:

1. Continue to produce 2.2.4xx, and 2.1.8xx but this would not align with our strategy to reduce the number of releases.

2. Work to make the docker image you pull to target a given runtime have latest SDK + appropriate runtimes, templates, targeting packs. I think this needs to be the long term direction, but the CLI experiences aren't quite there yet to make this as transparent as it could be. @richlander and I have discussed being able to, for example, configure defaults for target framework on dotnet new, etc. I think that sort of thing factors in.

A compromise of doing 1 for now while building towards 2 makes sense to me ignoring costs, but the costs of two more monthly releases are very high for us.

[nguerrera]

cc @mmitche

[richlander]

We discussed this earlier today. Let me see if I remember what we talked about ...

Let's start with our goals (beyond the obvious):

• You can run code in an SDK image. dotnet test is a good example, but there are plenty of others.
• You have access to the latest tools (that's this issue).
• We have a good and intuitive experience for developers using in-support versions.

Today, we only offer one SDK version in Visual Studio. That's largely not observable. When we switch 3.0 to 3.1 and then to 5.0 in VS, no one should really notice. You'll still have the runtimes you need installed and it is all good. Containers don't work that way, in two dimensions:

• If we stop supporting a given tag, thousands of people need to update their Dockerfiles to a supported one. This is expected for EOL releases but not for in-support releases.
• Docker images don't compose. With VS, you can rely on the fact that the right runtime versions are installed globally and accessible as new SDK versions become available. Docker images contain what they contain, end of story.

We should think of this primarily outside Docker and then figure out how it fits with Docker. These are the two obvious ends-of-the-spectrum options:

• We produce .NET Core SDKs for each runtime version for the life of that runtime, and always include the latest tools. We use this SDK to produce container images for that runtime version. This is the best experience.
• We produce only one SDK, which matches the latest supported runtime version. This is similar to our approach to VS. For Docker, we produce an SDK image for each supported runtime versions. It contains the matching runtime version + the latest SDK. This isn't a bad option. The primary negative is that the SDK docker image grows by the size of a second runtime (the one needed by the SDK).

This will sound like a simple statement, but it isn't: If we're not going to choose the second option, we should choose the first. I say that because there are other options in between that I didn't list. My feeling is that if we're not going to fully the one SDK version model then we should produce the SDKs people need even if it is expensive.

We see millions of docker pulls on .NET Core images every month (yes, we have data). I don't know how many of them are the SDK, but I would assume >50%. We've been doing a bunch of customer calls this week and we're hearing with nearly 100% of server-side developers that they are using containers, that they use the Microsoft-provided images and most do their builds (and presumably testing) in containers.

Definitely happy to talk about this further. This framing is (obviously) how I think about this topic.

[KathleenDollard]

Why do we think this issue is an issue in our design? 2.2.207 was released Tuesday. The issue says that 2.2.207 has an issue that was not present in 2.2.402. 2.2.402 was released earlier. Am I missing something - this looks like a simple bug.

Rich's discussion is relevant, regardless of whether that is the issue here. The second approach is more compatible with our simplification plans. For Docker, that is a larger image on dev machines only, and it seems worth it. We could potentially do some fancy layering, as only the one or two runtimes would be updated monthly, most of the time.

For folks watching, that simplification makes releases and testing easier, but a primary reason is regain sanity by having higher version numbers always be more recent - no decoder ring. Today 2.2.2xxx is the same SDK bits as 2.1.6xx (no corresponding 3.0) complies with 2.2 instead of 2.1 runtime. Etc.

[richlander]

For Docker, that is a larger image on dev machines only,

What leads you to that conclusion?

We could potentially do some fancy layering, as only the one or two runtimes would be updated monthly, most of the time.

Can you elaborate on that? I'm not seeing an obvious plan there.

The issue says that 2.2.207 has an issue that was not present in 2.2.402. 2.2.402 was released earlier. Am I missing something - this looks like a simple bug.

Great question. I don't know the answer. That would be good to know. My reason for writing the long answer (which is probably obvious) is to get all the context down so that everyone was working from the same assumptions and options.

[nguerrera]

The issue says that 2.2.207 has an issue that was not present in 2.2.402. 2.2.402 was released earlier. Am I missing something - this looks like a simple bug.

Great question. I don't know the answer. That would be good to know.

2.2.207 fixes 2.2.x runtime bugs, but has 16.0.x tools.

2.2.402 still has 2.2.x runtime bugs as there was no 2.2.403 built with the runtime fixes, but has 16.2.x tools

[richlander]

Ah. That wedges us / our users. As an aside, is there a 2.2 SDK with 16.3 tools?

We have had multiple complaints in the past when the SDK in the SDK image was stale in some way. This is an example of that. Developers will always run into challenges when we have these combinations. There are so many users of these images that they reliably run into incompatibilities or bugs that are present in them (if there are).

Do you have a proposal on how to resolve this, @nguerrera? I'm most interested in a long term systemic fix.

Are we releasing fixes in December? I'm guessing not, but I do not know.

[nguerrera]

There is no 2.2 SDK with 16.3 tools.

The systemic long term fix is to provide images that have the latest SDK + the older runtime you want to target (+ templates and targeting packs for older runtime, and maybe some convenience configuration to make the older runtime a default for things like dotnet new.) I already stated this above

I don't think this is the right forum for discussing upcoming servicing schedule, I'll point you to where you can check it offline.

[richlander]

Do those convenience options work already (like the dotnet new example)? That's an example of where this configuration would bleed into the UX.

Ya, my feeling is that we're either going to have to make a significant investment on SDK production (which will result in them being even more confusing) or we adopt this latest SDK plan. While I really dislike the size increase that would come with it, I think the ecosystem as a whole would be best served with this plan.

We could use 3.1 GA as the point in time to make this switch. That sounds super aggressive but it is important to talk about it because it would be a natural option. That said, I wouldn't do it for 2.2 images given that 2.2 is EOL in December. I'm thinking about 2.1.

[nguerrera]

Do those convenience options work already (like the dotnet new example)

No, we're talking long term here. To the extent that certain experiences are not as good when you use this combo, we can make investments like that to make them better. We would design based on the combo being a constraint, not try to make more SDKs as the design, which does not scale.

The size difference is the one thing I don't think we can solve, and we should just accept it, it's less bad than the alternatives.

There is more than just how many SDKs we produce if we wanted an image with say 2.1 runtime and the absolute latest tools. It means that all those tools are required to build for 2.1 until it goes out of support. But they also have to build for 3.0/3.1/5.0 depending where we're at. The cost spreads beyond just producing the SDKs to the tooling teams that make the tools. And indeed all the extra versions add more confusion.

[richlander]

Agreed and that's what I was expecting.

Ideally, we'll find a solution that works with the 3.1 SDK, as-is, using it to satisfy 2.1 SDK scenarios, and then it will be obvious which gaps to fill for 5.0 and beyond.

[MichaelSimons]

One thing that would help reduce the size impact, is if we would produce SDK only tarballs/zips that didn't include the runtimes, etc. When producing an SDK image that targets an older runtime, there is no value in including the latest aspnet and windows desktop runtimes.

[richlander]

That would be a very nice compromise. This is (kinda) the portable SDK I've always wanted. Ideally, there would be no native code in that SDK so that it would be usable for other scenarios, too.

[nguerrera]

Would docker scenarios be ok with build perf without R2R?

[mmitche]

One thing that would help reduce the size impact, is if we would produce SDK only tarballs/zips that didn't include the runtimes, etc. When producing an SDK image that targets an older runtime, there is no value in including the latest aspnet and windows desktop runtimes.

That would also dovetail very nicely with separated installer scenarios.

[mmitche]

So, just to recap for my understanding...

What we're saying is that the ideal scenario is that we have a single SDK (3.1) for all of .NET Core that cleanly supports older runtimes, sdk scenario, etc. Ideally this could mean a single SDK for any VS version too?

[richlander]

@mmitche I think you have it, yes.

@nguerrera I hate it when you take nice things away from me. ;) Ya, you are correct. R2R will continue to be required. As you know, I'm always trying to find ways to inject scenarios that I want that don't have enough motivation on their own.

[nguerrera]

Ideally this could mean a single SDK for any VS version too?

VS is already there for 16.3+, but we still have long term support for 15.9 and 16.0 to contend with.

[mmitche]

VS is already there for 16.3+, but we still have long term support for 15.9 and 16.0 to contend with.

I thought we still had one separate SDK band per VS LTS release...or am I remembering that wrong?

[nguerrera]

Per VS LTS, yes. We cannot put a new SDK full of VS non-LTS stuff into VS LTS servicing. So for example if 16.4 is LTS, then we will need to continue to produce 3.1.10x for a long time. We can't put 3.1.20x into VS 16.4 because then we'd be taking a 16.5 set of tools into VS 16.4. But for any given VS >= 16.3 there is at most one A.B.Cxx SDK that comes with it and that can potentially be serviced. This is contrasted with VS 16.0, 16.1, 16.2 that had two SDKs each, one for 2.1 and one for 2.2.

[gabrielsadaka]

I greatly appreciate the open discussion and although I have nothing to contribute regarding the long term solution to this issue I should let you all know that we have worked around this issue by updating the code that triggered the Roslyn bug to no longer use that pattern. We have plans within the next week or so to migrate to 3.0 as 2.2 is going out of service which should avoid this problem in the future for us.

[omajid]

There's some another consequence of not having further updates to the "2.2.4xx" series: https://stackoverflow.com/questions/58998538/how-does-the-net-installer-task-chooses-which-version-to-install

[richlander]

That SO post seems like an important question to answer (on SO) and to inform these decisions. The Azure DevOps task is another key scenario to inform our plans.

The Visual Studio Installer scenario is relatively easy because we're simply offering ".NET Core as a service" without requiring anyone to specify any specific version identity, other than runtime versions. With the Docker plan we discussed, we're proposing to adopt selecting a runtime version even when you want an SDK. We would need to do that for Azure DevOps, too, in order to have a coherent offering.

To be super concrete, with Docker, we are proposing that the "2.1" SDK tag will contain the 2.1 runtime and the 3.1 .NET Core SDK. I was just thinking about the versioning scheme for those tags. That's something we didn't discuss before. I'm not seeing a good option for that. Let's take a look at that problem.

Given:

• Runtime version: 2.1.21
• SDK version: 2.1.607
• Runtime and SDKs update and version on different schedules so need different version numbers.

I'm using the 2.1 SDK here to align with existing tags. In reality, we'd be using the 3.1 SDK. Everything is the same otherwise.

Let's look at how we version 2.1 container images today: 2.1.607-stretch, 2.1-stretch, 2.1.607, 2.1

How would that work given the new scheme? Ahhh! This is hard. This is what a runtime-oriented version would look like: 2.1.21-stretch, 2.1-stretch, 2.1.21, 2.1.

The problem with that is that it wouldn't allow for SDK updates between runtime updates. That's a problem. We could add more decoration to cover that: 2.1.21_1-stretch, 2.1-stretch, 2.1.21_1, 2.1.

That's super ugly and confusing. We could reserve the _n syntax only for cases where we have a second update of one of these images in a given month with only the first one including a runtime change. With that scheme, we'd only ever see _2 and _3 and never _1. That's a tradeoff. Another, even worse, idea is on the second image to show the SDK version, like _607. I am not proposing that, but wanted to through it out there as the obvious counter-proposal to the one I made.

Another challenging issue is distro version, which you can see by the presence of -stretch in the tags above. In Docker, we only support 3.0 on Buster. We could push these new SDK images to use Buster or accept supporting Stretch with 3.0 for this scenario. To keep the scenario whole, I'd strongly propose the second option, to keep the runtime and SDK images for a given .NET Core version symmetric w/rt distro version. Any other plans seems very questionstable.

The big remaining question then is what to do with Azure DevOps. It's not the exact same as Docker. With Docker, the tag name is a contract, and we only update images underneath a tag in narrow situations (only base image updates). I'd suggest that contract for DevOps is much weaker. I see two options available:

• Leave the experience unchanged, but guide users to adopt SDKs and runtimes separately.
• Change the experience to be runtime-first, and then provide a runtime, SDK, and any other assets that will satisfy the runtime version selected. Make SDK version selection an advanced option. This will make it more similar to what we're planning for Docker.

I think the first option won't work at all. Like Docker, we need to continue providing users with some form of functionally correct and fully serviced assets when we stop producing the 2.1 SDK while 2.1 is in support. The only good option is option two, IMO, and the fact that it aligns with the proposal for Docker is a good sign.

So, in short, we have two options:

• Continue producing SDKs for a runtime until the runtime is EOL.
• Switch SDK selection to be runtime version oriented in all the experiences where that makes sense.

Similarly, if we make this change, we have to figure out what to put on the 2.1 download page. The left-most cell is the SDK. What will we put there after we stop producing a 2.1 SDK?

This whole discussion is IMO our version of "courage". The Visual Studio Installer experience is by far the easiest case for us. All the other ones are hard with a variety of trade-offs. We just need to work all of the cases and make sure we can deliver an intuitive experience for each of them. It should be each to find people to test these ideas on with mockups to help validate our plans.

[MichaelSimons]

Ah tagging - it is the most difficult part of producing the SDK images! @richlander, do you remember this former tagging scheme? It is in essence equivalent to the one you mentioned with the _607 suffix.

- [`2.0.4-sdk-2.1.3-nanoserver-1709`, `2.0-sdk-nanoserver-1709`, `2.0.4-sdk-2.1.3`, `2.0-sdk`, `2-sdk`, `sdk`, `latest` (*2.0/sdk/nanoserver-1709/amd64/Dockerfile*)](https://github.com/dotnet/dotnet-docker/blob/master/2.0/sdk/nanoserver-1709/amd64/Dockerfile)

2.0.4 was the runtime version this SDK image was intended to be used for. It contained the 2.1.3 SDK.

From an engineering perspective, a challenge with the proposed incremental suffix which skips _1 is how to automate the tag generation. My initial thought is that we would have to keep historical information to know how to generate the tag.

I agree that we should keep distro version alignment between the SDK and Runtime images for each version. Without this I could see confusion related to running tests with the SDK images.

[MichaelSimons]

It is important to understand the tagging principle that is driving the complexity. There should always be a tag that allows users to isolate themselves from changes to the .NET content within the image. For example the 2.1.607 tag represents a specific snap shot of the SDK toolset. The image the tag references may get updated as the base OS is serviced but the SDK toolset is constant.

To satisfy this principle there are two high level tagging scheme proposals we are considering.

1. Include both the runtime and SDK versions. This tag is more specific in nature but may cause confusion as to what each version means.
2. Include the runtime version plus a toolset incrementor. See Rich's earlier description for additional details. This scheme is similar to what is done in the FluentD Docker repository which they provide a detailed explanation of.

Within each scheme there are numerous possibilities we could adopt. Here are two options to consider and discuss.

1. <RuntimeVersion>_<SdkVersion>-<DistroVersion> - e.g. 2.1.14_3.0.101-stretch

   An _ is intentionally used to separate the two versions to avoid conveying any type of span relationship within the versioning. The two version numbers are not intuitive to new users and will need an explanation. The advantage of this scheme is that once understood, it is descriptive in nature and you can tell exactly what is contained within the image.

Tags	Dockerfile	OS Version
2.1.14_3.0.101-stretch, 2.1-stretch, 2.1.14_3.0.101, 2.1	Dockerfile	Debian 9
2.1.14_3.0.101-alpine3.10, 2.1-alpine3.10, 2.1.14_3.0.101-alpine, 2.1-alpine	Dockerfile	Alpine 3.10
2.1.14_3.0.101-alpine3.9, 2.1-alpine3.9	Dockerfile	Alpine 3.9
2.1.14_3.0.101-bionic, 2.1-bionic	Dockerfile	Ubuntu 18.04

2. <RuntimeVersion>_<ToolsetIncrementor>-<DistroVersion> - e.g. 2.1.14_1-stretch

   The toolset incrementor is not intuitive to new users and will need an explanation. The advantage of this scheme is terseness. Does the user need to know the toolset version? Will they understand what it means? If not, keep the tag short and simple.

Tags	Dockerfile	OS Version
2.1.14_1-stretch, 2.1-stretch, 2.1.14_1, 2.1	Dockerfile	Debian 9
2.1.14_1-alpine3.10, 2.1-alpine3.10, 2.1.14_1-alpine, 2.1-alpine	Dockerfile	Alpine 3.10
2.1.14_1-alpine3.9, 2.1-alpine3.9	Dockerfile	Alpine 3.9
2.1.14_1-bionic, 2.1-bionic	Dockerfile	Ubuntu 18.04

In regards to the Azure DevOps task, a similar strategy would work. The Task's version should target the runtime be default. This will allow the user to express their intent and have a solution that will be durable over the lifetime of the runtime as the toolset versions change. For advanced special cases, it could be possible to support specifing both the runtime and toolset version. This proposal would of course be a breaking change.

[richlander]

We're getting closer! Of those two options, I suggest we start with the first one as the preferred option. Neither of the two is perfect, but I think the first one is better.

Stepping back, here are my principles on how we should proceed if we move to a tip-only SDK, specifically as it relates to the Docker experience:

• There is a single textual identity for a Runtime+SDK combination that I can use that will take me through the life-time of a .NET Core runtime release. This is like the 2.1 that exists for the SDK repo, today.
• This identity always pulls the latest tools and latest patched runtime (for that identity, like 2.1).
• The artifacts I pull are as lean as possible, minimizing wire and local disk costs and compute to decompress packages.

If both of those aspects of the experience are taken care of, I don't think we have any big concerns in Docker land. It's also fair to say that the Azure DevOps concerns are super similar.

[mthalman]

To be super concrete, with Docker, we are proposing that the "2.1" SDK tag will contain the 2.1 runtime and the 3.1 .NET Core SDK.

@richlander - I'm wondering what the story would be for dotnet new here. How would one use the SDK image to create a project using the 2.1 templates? I actually ran into this scenario this morning where I want to create a 2.1 project. Today, I can just use the 2.1 SDK image but if we change this so that that image has the 3.1 SDK installed, what's the alternative here?

[nguerrera]

It is actually already possible to install older templates and use them with a combination of dotnet new -i and dotnet new -f. I'm away from a computer where I could provide the exact steps. The experience has some rough edges and it's not super discoverable. We have talked about smoothing those over. It is very conceivable to imagine downlevel templates pre-installed in docker and even a global configuration in the image to make a downlevel TFM the default for new.

[richlander]

Ya, we'll need to do the things that @nguerrera suggests.

[mthalman]

Closing as this is stale. The current approach for .NET SDK container images is to provide the latest available SDK feature band version. This aligns with the version that is included in the latest VS version.