[Release process] Verify changelog entries when making stable release

[parlough]

The changelog is the primary view into Dart SDK changes by developers. However, in the Dart 3.1 release, the changelog included entries that didn't make the cut-off and are instead part of 3.2. This has caused confusion for developers (#53285), tech writers (dart-lang/site-www#5115), and potentially content creators who share about Dart/Flutter releases.

There were at least two changes that were included in the 3.1.0 changelog but not part of the release. The following two CLs move them to the 3.2.0 header.

• utf8.encode return type change
• packageConfigSync and resolvePackageUriSync

I'm not exactly sure how it happened as usually the changelog entries would be moved with the source changes themselves. It could happen with a manual changelog update to add missing items, accidentally when merging changes to the beta branch, or if a change was (partially) reverted.

To avoid potential confusion in the future, as part of making the final beta cut-off and/or moving the changes to stable, the changelog should be verified as having up-to-date entries.

\cc @itsjustkevin @mit-mit (@MaryaBelanger)

[itsjustkevin]

Thanks for adding this issue @parlough, I think one quick change here that could benefit the team is adding the tech writing team to the reviews (sorry for not doing this already) when we build the beta and stable. This way there are more eyes to catch things that seem fishy.

[parlough]

Thanks for taking a look!

I think one quick change here that could benefit the team is adding the tech writing team to the reviews.

Good idea! That does seem like it would benefit the team overall, for this and potentially other concerns. However I'll defer to @MaryaBelanger and @atsansone on their desire to be included in that review process.

While I do think that could help in general, it might not help much with verifying accuracy/up-to-date status of the changelog, as they don't interact directly with the SDK much.

[itsjustkevin]

Another interesting idea here: What if we parsed the links in the stable changelog and verify that they landed on the branch that they are under for stable?

Commit abc123's latest tag is 3.1.0 meaning that it shouldn't be under the 3.2.0 section. Riffing here, but trying to speak out the idea.

[parlough]

I quite like the idea or some variation of it! I'm all for more view into how changes are made, since I think Dart developers are curious, and that would definitely enable more insight.

The challenge there would be adding the commit references/links, knowing when to do so, and who/what would do that.

---

I guess before any more expensive solution is explored, it should be determined why this happened this time. Normally the changelog entry wouldn't move to the stable channel without the change itself, and the last published beta didn't have the accidental entries. So they must have been accidentally added when releasing stable by copying the changelog for 3.1.0 from the main branch changelog.

So perhaps the primary source of the problem is those entries being added under 3.1.0 in main even though it was after the cut off but the CL authors not realizing. As a result, the solution could be to just make sure the next release is added to the changelog at the same time the cut off is made. Or maybe a check to make sure developers add their change to the correct version entry in the original CL. There's lots of choices to improve this, so that's just a few thoughts. For example, for breaking changes, the Flutter docs instead put "Not yet released to stable" and manually move breaking changes to a release.

Misunderstood landing versions can have other effects too, perhaps more important than the changelog, such as in https://dart-review.googlesource.com/c/sdk/+/322040 which incorrectly marks the added APIs as available in Dart 3.1 with Since annotations, potentially resulting in a lack of a diagnostic reporting the API as unavailable for package authors developing with a beta/dev SDK (3.2+) release, but wanting to support stable (3.1) releases.

[Cat-sushi]

Release date on the CHANGELOG.md should also be maintained.
https://github.com/dart-lang/sdk/blob/main/CHANGELOG.md#310

[parlough]

Another issue popped up, where changes to webdev relied on a certain commit and had a correct -dev SDK constraint, but since they landed after the cut off, causes issues since the -dev constraint allows a stable 3.1.0 SDK, which doesn't have the changes. This results in the version of webdev that pub resolves having errors for users of a 3.1.0 SDK.

It makes me think the core of the issue here was maybe more one-off accident(?) for 3.1.0 and the version of the main branch wasn't updated once the stable cut off was made, so developers thought their changes were landing in 3.1.0. A bunch of changes landed in what was 3.1.0-263.0-dev through 3.1.0-348.0-dev, but those weren't in the stable 3.1.0 release this time (without cherry picking). So anyone that landed a change during that period thought it was in 3.1.0 and any -dev SDK constraint that relies on changes in that period might actually not work in 3.1.0 stable despite resolving just fine.

\cc @athomas

[annagrin]

/cc @sigmundch

[annagrin]

Suggestion by @nshahan - mark all the dev versions in the release window (for example, 3.1.0-348.0-dev, which is past 3.1.x but but before the change to 3.2.0-dev.0) as 3.1.0-348.0-dev.unsafe and make pub aware of it (ie issue an error when we try to publish something including this constraint).

Another option is to make pub aware of out-of-order SDK versions and correct for it during constraint solving somehow (i.e. treat 3.1.0-348.0-dev as > 3.1.x. but < 3.2.0.-0

[annagrin]

/cc @elliette @nshahan FYI

[parlough]

I think the core issue here was due to the final beta cutoff for the discussed release being made but the main branch not updating to the next version in lock-step with that.

The final cut off for 3.3 was today and an updated release process resulted in the main branch being updated accordingly to 3.4 (8158275), including the changelog entry.

I think this process will sufficiently prevent confusion about what version a change is landing in. I'll monitor the next few releases to verify no issues pop up. Thanks everyone for your help and the discussion! Also thanks to the release and EngProd teams for the more consistent process :)