Establish a policy for tooling repos with arcade-powered source-build: how does one build work for 5.0 and 6.0?

[dagood]

With some tooling repos, there is only one branch:

• dotnet/sourcelink: one active main branch.
  • SDK 5.0.100-preview.6.20310.4
  • Arcade 5.0.0-beta.20453.7
• dotnet/xliff-tasks: one active main branch.
  • SDK 5.0.100-preview.6.20310.4
  • Arcade 5.0.0-beta.20459.8

(Roslyn, MSBuild, NuGet...? have multiple branches, but I believe they track Visual Studio versions or something else, rather than runtime major.minor.)

What happens in 6.0? Will these repos want to move to a 6.0 SDK/Arcade on the existing branch?

1. Should these repos be forced to always build on the oldest major/minor version of the SDK/Arcade that they feed into?
   • Then we just assume the 6.0 SDK/Arcade is backwards compatible and can also build them fine?
     • To validate this, we could have each repo have jobs for every active SDK version that uses that branch. If we don't validate, the dotnet/installer build will validate once it tries to build a tarball using all the sources.
   • This might already be our policy, but it isn't very clear to me.
2. Should the repos be forced to fork with the SDK major.minor, like the runtime-related repos?
   • What other restrictions do they have that may prevent this, or make it explode into an unmaintainable number of branches.
     • E.g. are Roslyn, MSBuild, and NuGet branches already tied Visual Studio, and would this be a cross product with runtime major.minor?

If we don't establish some sort of policy and let e.g. SourceLink master depend on a 6.0 SDK/Arcade without branching for 5.0, then if Arcade 5.0 takes a SourceLink fix (from master, the only branch), source-build likely gets broken. We would have to patch the new SourceLink commit back to a state where it can build with our N-1 copy of the 5.0 SDK/Arcade.

---

I think there's already an understanding around these problems, but I can't quite find docs around it, and I don't know if the source-build scenario here is widely known. I think this issue should track making the decision around this problem easily understood (doc'd) and into policy.

[dseefeld]

Triage: We're currently solving this issue by adding patches in source-build. When we move to arcade-powered source-build, the question is where should these patches live.

[dagood]

@mmitche, @crummel, @dseefeld and I met to talk over this. Here's a summary of what we came up with. (Correct me if you spot parts that seem wrong! 😄 I added a little bit at the end about incrementality that I didn't think of at the time.)

---

First, a note about this:

dotnet/sourcelink: one active main branch.

This is just how it is now, not a problem. When dotnet/arcade makes a release branch, the reference to sourcelink basically freezes to the current commit, so source-build is fine. If sourcelink ever does need to be serviced, it'll first fork a branch to match arcade's branch.

---

Here's a rough and very simplified diagram of what a build looks like in 5.0 to compare against:

On the left, we have an example repo that represents the kind of repo that gets inserted to each channel, and its branch. In the middle, we have Darc/Maestro++/BAR channels. On the right, the dotnet/installer product that these channels flow into.

(This is a very simplified graph. For example, in reality, Roslyn feeds into Arcade to set the default version of Roslyn that all repos can use. Arcade is a dependency that feeds into everywhere. There are also circular dependencies not shown. This is all handled by source-build as always: we choose a "correct" single commit of each repo and build the entire product with those "correct" ones, overriding defaults where necessary.)

---

The biggest concern here is what happens when we release a new SDK release/5.0.2xx in addition to a release/5.0.1xx currently in servicing. This kind of release happens when someone needs to take new tooling, in particular VS. For example, a new version of Roslyn, NuGet, MSBuild, .... Once 5.0.2xx releases, we get a scenario like this, with a hypothetical VS version: (changes highlighted)

The interesting thing here is that we need to be able to build runtime release/5.0 using 5.0.2xx for source-build bootstrapping. At the same time, since release/5.0.1xx is still in service, we need to continue being able to build runtime release/5.0 using the 5.0.1xx SDK. This means 5.0.2xx and 5.0.1xx need to be able to build the same commit.

In some cases, this is fine. Versioning implies that 5.0.2xx should have more capabilities than 5.0.1xx so they can both work. In practice, however, there are often breaking changes in the tooling, such as new compiler warnings, nuget pack features/validation, and analyzers. For these cases, we need to (priority order):

1. Make changes to the repo to try the best we can to have a codebase that works with both.
2. Make changes to the repo that might modify some MSBuild properties based on SDK version. (Polyfill-ish?)
3. Add source patches to the repo.

It's probably unavoidable that we need patches. To mitigate the effect on the build, we should validate them. The patches should live in the repo they apply to. Then, source-build validation jobs run, testing that both 5.0.1xx and 5.0.2xx SDKs can successfully build the repo (with patches if needed). We can decide on a case by case basis if the risk of breaks is high enough to warrant actually running these extra jobs on every single PR validation.

We may also need to temporarily include patches in dotnet/installer that apply to repos like dotnet/runtime, similar to how dotnet/source-build currently has patches for each repo. This could happen if 5.0.2xx has the exact same dotnet/runtime build as 5.0.1xx, because that old runtime build is built by an old commit we haven't had the chance to add the patches to. We should move the patches from dotnet/installer to dotnet/runtime at the earliest opportunity (the next 5.0.x runtime servicing build).

(I'm also not really getting into previews, here. (Roslyn previews, VS previews, etc.) After 5.0.1xx is released but before 5.0.2xx is officially released, the Microsoft build produces preview SDKs like 5.0.2xx-preview.x. Some dependencies still come from 5.0.1xx, and it may be extremely challenging to build that from source without large patches. For the sake of this post I'm focusing on post-release, where release/5.0.1xx and release/5.0.2xx are both in active Microsoft servicing mode.)

---

There's also the opposite case, a new Runtime release but no new VS channel:

I think this has basically the same problems/solutions.

(If we want to ship a 6.0 preview, then we hit issues, especially when the targetframework isn't created yet. I think there are situations here where bootstrapping can be impossible. It's a pain on the Microsoft build side of things, too. I'm not going to focus on it here.)

---

Something that will have major impact here is determining what we actually want to support with source-build.

So far, we've only source-built .NET Core for a single feature band version (1xx) per Major.Minor (3.1) version. It isn't practical for us to support more: it would take more time than we have to maintain. So, because feature band upgrades introduce breaking changes, we stay as compatible as possible by source-building the highest patch version of the lowest Microsoft-supported feature band. If we continue to do this, we will actually rarely (if ever) hit the difficult scenarios above.

However, this is a pain point for source-build users: they can't get new .NET SDK tooling. (E.g. we've been asked why source-build doesn't have NuGet improvements from 3.1.402--we only have 3.1.108.) This seems to me to be particularly bad when you're trying to work together on a project with someone who does have access to the new feature bands--you must agree not to use new versions, or else the source-build user won't be able to build. Arcade-powered source-build is an opportunity to fix this by making every single dotnet/installer branch source-buildable.

We can do this incrementally: get arcade-powered source-build working on the lowest supported feature band, then expand to all feature bands as a second step. I think incremental is fine. This is an existing problem, and the primary goal of arcade-powered source-build is to get infra sustainability (so we can hopefully free ourselves up to work on fixes like this).

[dagood]

We had a source-build champs meeting today and I wanted to capture some follow-up notes here. We talked more about feature bands, and some top level questions emerged that I think we need to look at before making decisions here:

• When do feature bands go out of support?
  • We know it does happen: 2.1's lowest SDK feature band is 2.1.5xx currently. Is there more specific planning we can be aware of about when this happens? (When users will need to pick up a new version or else be stuck on an EOL feature band?)
• How significant do we expect feature band upgrade breaking changes to be?
  • Can we guarantee that features will never be removed: there will always be a nondestructive migration path?
• Is the scenario real where 5.0.1xx and 6.0.1xx need to be bootstrappable with the same (e.g.) Roslyn commit?
• Can we split the runtime build from the SDK without being too complicated for distro maintanance?
  • Probably: it's mostly a matter of clarity. However, plan needs to account for this.
  • This lets us reuse the same lowest-feature-band runtime build and avoid multi-SDK build scenarios.

[tmds]

because feature band upgrades introduce breaking changes, we stay as compatible as possible by source-building the highest patch version of the lowest Microsoft-supported feature band.

Feature band upgrades shouldn't introduce breaking changes. Is this the case for 3.1.404 vs 3.1.110?
It's not so clear about what is the difference between these releases, besides one being for VS v16.4 and the other for v16.7.

From a user perspective I'd like to be on the latest version. And if this version is good for Visual Studio users, it is probably also good for source-build users.

[dagood]

[are there breaking changes] for 3.1.404 vs 3.1.110?

I believe one example here is some extra NuGet verification being added which may break existing builds. I might not have quite the right version diff for that, but it's at least an example of the kind of breaking change we're talking about. Another example is new compiler warnings from Roslyn that may be treated as errors depending on build infra. (Warn as error is very common in dotnet repos themselves, and although it's not the default for new user projects, I see it recommended broadly as a best practice.)

Feature band upgrades shouldn't introduce breaking changes.

I brought it up in the email thread for the last source-build champs meeting ("RE: Source-build 5.0 Champs Update", @omajid on CC). I was hoping that we could prove this, or at least find some level of support for it, but the only doc I was able to find on the topic was one that says there will be breaking changes:

[@dagood] The only doc I can find that talks about feature bands and compat is https://devblogs.microsoft.com/dotnet/net-core-releases-and-support/#feature-bands-sdk-only, which says bands may be incompatible.

Later in the thread, the SDK team confirms this. Breaking changes aren't intentional and have in their experience been reasonable to get around by e.g. setting MSBuild properties, but they do happen.

I'm curious @tmds why you say there shouldn't be breaking changes. Is there a doc I didn't find that should potentially be fixed? Or is it about semver, which SDK versions don't follow because feature bands are mushed into the patch version?

---

From a user perspective I'd like to be on the latest version.

All this stuff about breaking changes just means we do need to end up source-building every feature band, not only the latest. If there aren't any breaking changes, we could build just the latest feature band and all source-build users could use the tip 3.1.9xx to build all 3.1 repos.

In the thread, @leecow wrote up this scenario:

VS on Windows deploying to RHEL where the RHEL leg includes building the toolset (SDK) from source.

The feature gap is that source-build SDKs don’t support that if we source-build only some subset of the active SDK feature bands. Breaking changes are a practical cause of that gap. Devs wanting to exactly match SDK versions with VS is I suppose a "soft" cause of that gap.

For 3.1, we currently source-build only the oldest in-support band, so we have that gap. To fix it in 5.0(6.0+?), we need to source-build every band. This is coming up now because arcade-powered source-build will make it more feasible to fix the gap maintainably. 😊

Later in the thread, I raised another scneario: devs that want to delay migration to the newest feature band but still receive security updates to the SDK tooling.

• This is supported by the Microsoft builds because the old feature band is still supported.
• This is a feature gap with source-build currently. If we don't build the newest, there is no opportunity to try to move forward at your own pace. If we only build the newest, you aren't able to delay the migration.

Some messages in the thread from the SDK team back this up:

Wanting security update should not be the motivation [for a customer] to update SDK [feature band].

If someone wanted to stick with 5.0.2xx rather than upgrading [to e.g. 5.0.3xx], I’d be fine with that since we support that version for a while

Nothing came up in the thread invalidating these scenarios or suggesting these aren't real feature gaps, so it seems like to get parity with the Microsoft build, we definitely need to build all bands.

We plan to do this incrementally:

1. Initially only support the lowest feature band. This is what we do in 3.1, because it's all we can afford to do. Since our 5.0 infra is currently based on 3.1, we have to do it there too currently.
2. With arcade-powered source-build, initially continue to only support the lowest feature band.
3. Get more feature bands working with arcade-powered source-build.
   • This is where implementation-focused issues come up, like how to bootstrap, how to get N-1 flows working properly, etc.

---

The email thread is quiet since Oct 19th and I wanted to write this up earlier, but I've been busy with 5.0.0 stuff. There are some more implementation details from the champs meeting presentation that also aren't here yet, but starting at the "why" is valuable.

[dagood]

I'm going to describe more of the implementation details we went over the Oct 2020 meeting, then close, because the current plan is to not do any of this soon, only incrementally when we have time. I expect to come back here myself to read it again and want this info mainly in one place. 😄

---

This is the basic implementation idea to get the same commit building under multiple SDKs--patches specific to certain SDK versions that are applied if necessary. This example shows dotnet/runtime and dotnet/arcade having the same commits for both 5.0.1xx and 5.0.2xx because runtime and arcade only have one release/5.0 branch that feeds into both:

These should be stored in the repo if possible, for closer maintenance. We can add CI legs to validate that each SDK version builds correctly with the right patches. (Whether this is necessary is a value judgment and might be specific to each repo.)

Otherwise, the patches can go in dotnet/installer and only apply to the tarball build. This would be similar to how dotnet/source-build does it now.

Note that tooling can also share a commit. For example, something like dotnet/roslyn release/dev16.8-vs-deps might feed into both 5.0.3xx and 6.0.1xx. (There is a way that you could probably predict which versions this would be by examining the release schedules closely, but this is a totally fake example.)

---

We had this diagram to describe the basic feature band structure, e.g. with 5.0.1xx support ending at 5.0.103 and 5.0.202 being the new lowest feature band:

Currently we build the lowest feature band, so those are blue, with N-1 => N build links in bold green.

In the previous comment I went into why feature band support is important to source-build. There are multiple ways to implement it, though:

All bands with specific N-1 junctions

Support all feature bands, but only explicitly support building a new feature band from an N-1 feature band at specific junctions:

This makes it more feasible to test the junctions and make the build path more predictable for repos like Fedora where a continuous build train is important.

Split the runtime and SDK

Only build one copy of the runtime (5.0.0 => 5.0.1 => 5.0.2) and reuse that single runtime while building higher feature bands. This simplifies maintenance for source-build: there is only one build path, and you can't "miss" a junction. It makes things more difficult for distro package maintainers: the runtime and SDK still have to bootstrap together, there are more packages to keep track of, and the build process may be more intricate.

We could potentially simplify this: deliver a single source-build tarball that builds all bands. I haven't given a closer look at the implications, though:

---

Closing: this is out of scope for our imminent plans.

[hdost]

@dagood Have there been any new developments in this area?

[dagood]

Have there been any new developments in this area?

As far as I know, no, but I'm not actively working on source-build anymore. Based on @omajid's answer in #2480, I don't think this is coming particularly soon.

(In general, closed issues in .NET repos aren't monitored, so I'd suggest opening a new issue to get current maintainers' thoughts. The triage process makes sure the right eyes get on it--even if there is some initial conversation before that happens. Some repos have actually started to lock old inactive issues to make this workflow more obvious.)