More frequent major releases for arrow-rs

[alamb]

TDLR: I propose doing major releases for arrow-rs more frequently (up to every other week) directly from master, breaking the correspondence with the main arrow release

For example, the release cadence might look like

• 2022-01-01: 7.0.0
• 2022-01-15: 7.1.0 (no backwards incompatible changes)
• 2022-02-01: 8.0.0 (new backwards incompatible change)
• 2022-02-15: 9.0.0 (new backwards incompatible change)
• 2022-03-01: 9.1.0 etc
• 2022-03-15: 9.2.0
• 2022-04-01: 9.3.0
• 2022-04-15: 10.0.0
  ...

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The rust ecosystem as a whole is "fast changing". This means:

1. Many pieces of the ecosystem change frequently that require downstream updates (e.g. new clippy lints released in stable versions of rust that require code fixes to pass CI)
2. Many crate authors don't spend effort maintaining stable releases, but instead fix issues by pushing new releases

This means the ability to quickly upgrade to new downstream libraries is critical to help fix issues like #1101

With a few notable exceptions (e.g. tokio) most of the rust ecosystem pushes new major releases frequently (monthly if not more so), often via 0.X releases.

The current release cadence and versioning scheme of arrow-rs inherits from the C/C++ and python ecosystems with a major release every 3 months while maintaining a backwards compatible branch.

I propose moving the rust implementation in closer alignment with the rest of the rust ecosystem with more frequent major version releases.

Describe the solution you'd like

Continue to release every 2 weeks; However, release all new versions directly from the master branch, picking a new version number based on the changes in the crate (a new major version if there were semver changes, minor if not)

Pros:

1. Same number of releases as today (1 every other week)
2. Less branch maintenance overhead (selfishly good for me)
3. Major changes to dependencies(e.g. upgrade to newer versions of prost) available sooner

Cons:

1. downstream crates would likely have to do major updates more frequently (cargo update doesn't pick up new major versions)
2. Rust versions would no longer necessarily align with C++/etc mono repo release version

Much of the rust ecosystem, as described above, is used to the "do major updates frequently" mindset, and furthermore tools such as dependabot reduce the effort required to do so, I believe the first con is manageable.

Describe alternatives you've considered

Option: 1 "major release on demand", where we released when there were "enough" changes built up to do a regular release;

Option 2: keep the same structure (a master and active_release branch); do a major release every month instead of every 3 months, and do a minor release every other week.

[tustvold]

often via 0.X releases

I think this touches on something key. In my opinion, the crates in this repository are still somewhat beta if not alpha software, with major APIs still getting fleshed out. As such it would perhaps be more expected for it to be 0.x software with the accompanying frequency of breaking releases. I accept the version ship has sailed, but I think it makes sense to pretend it hasn't and adopt a more rapid breaking release cadence as suggested 👍

FWIW similar arguments apply to many of the arrow-rs dependencies. It isn't so much that the ecosystem itself is "fast changing", just a lot of the ecosystem isn't yet v1 😆

[paddyhoran]

Also another pro is that we can hopefully bring arrow2 back into the fold (though that would need confirmation/additional work). As I remember it, @jorgecarleitao's main sticking point was that the Rust version of arrow was sticking with the C++/Python versioning scheme.

[alamb]

In conclusion, from my perspective maintaining an active release branch with stable patches and major releases every 3 months doesn't provide enough value for the cost

The cost is borne both by me actually creating the releases as well as users who are slowed down picking up updates for dependent libraries (the latest version of tonic, for example) or waiting for their changes to be released.

So unless there is significant pushback (ideally along with a volunteer to help maintain a stable branch) I will plan to start doing releases for arrow-rs directly from master as proposed on [1] , starting with arrow-rs 7.0 (will make a candidate later this week and hope to release early next)

I think the only downside to this will be the oft discussed "arrow will end up with versions 22.0.0 rather than version 0.22.0" which may lead to confusion about its relative stability. We can add some documentation about this and adjust if it causes too much confusion.

[alamb]

TLDR is I plan to release arrow-rs 7.0 as normally scheduled at the end of this week, and then after that only release from master every other week (rather than maintaining a more stable active_release branch)

So the first time something different will happen is in 2.5 weeks time

[liukun4515]

If the master has the API-change commit after the last release, how to handle this?
@alamb

[tustvold]

I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release. The major benefit would be this could be completely automated, potentially using existing tooling.

In the past I've used this approach for binaries, in particular using goreleaser, but I see no obvious reason something similar couldn't be done here...

Edit: a somewhat related question is must all the crates in this repo advance their version numbers in lockstep? Should a breaking release to parquet mandate a breaking semver bump to arrow?

[alamb]

If the master has the API-change commit after the last release, how to handle this?

@liukun4515 the proposal is that the following release would have a new major (rather than minor version)

So for example, if there was an API change on Jan 12 (after 7.0.0 was released) the next release after Jan 12 would become 8.0.0 (rather than 7.1.0)

[alamb]

I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release.

The major thing that prevents more frequent and/or automated releases is the release voting process. I often have to chase (via slack, etc) to get 3 PMC members to approve a release;

In terms of automation, sqlparser-rs uses cargo release -- https://github.com/sqlparser-rs/sqlparser-rs/blob/main/docs/releasing.md and it is pretty sweeet.

Edit: a somewhat related question is must all the crates in this repo advance their version numbers in lockstep? Should a breaking release to parquet mandate a breaking semver bump to arrow?

I don't know of any technical reason they not need to advance their numbers in lockstep -- it is a convenience so that we (mostly so I) don't have to scrutinize the changes to each crate and determine if/when a new version is warranted on each release

[liukun4515]

Waiting for the release of 7.0.0!

[liukun4515]

I wonder if we might further simplify matters by releasing weekly and incrementing the version based on if there are any new breaking changes on master since the last release.

The major thing that prevents more frequent and/or automated releases is the release voting process. I often have to chase (via slack, etc) to get 3 PMC members to approve a release;

Maybe we need more active PMC for arrow in the rust ecosystem.

[alamb]

Maybe we need more active PMC for arrow in the rust ecosystem.

Yeah -- I don't really know how to improve this. We do have several representatives of the Rust implementation on the PMC now (@jorgecarleitao @nevi-me @andygrove and @Dandandan) but their time is limited (as all of ours is)

I do wonder how much of the issue is the current process, which suggests running an automated script. I am not convinced this extra level of "quality assurance" is a real value add -- the real value add to me is having more than one person look at the release's content and say that the "content seems reasonable."

Are there any other thoughts on this thread?

[houqp]

I don't have anything to add other than that I fully support this move :) It will certain reduce @alamb 's maintenance burden and make it easier for downstream to get access to breaking changes 👍