Blessed Dask releases

[jrbourbeau]

For the past year plus we've been releasing Dask on a more-or-less regular two-week cadence. Recently we've been experimenting with weekly releases (xref #84). In the last monthly meeting, the idea of having a "blessed" version of Dask, which is released less often than the usual one or two-week cadence releases, was raised by folks from RAPIDS and Capital One as something that could be valuable to the Dask ecosystem of libraries.

I'm opening this issue to keep that discussion going and pin down how we want to go about this. A couple of logistical questions come to mind:

1. How often do we want to release a blessed version? Once a month? Once a quarter?
2. How should we indicate a version is blessed? For example, we could have a page in the docs with a table indicating which releases are blessed. We could also give blessed releases a special version number (xref Dask versioning to switch to CalVer #100)

cc @datametrician @kkraus14 @mmccarty for any additional thoughts or context

[kkraus14]

cc @jcrist since I imagine this has prefect implications as well

For one thing, unless enough people agree to support the "blessed" version, it doesn't matter or solve any problems. From the RAPIDS side we are ready to commit that we'll support always support a "blessed" version in addition to keeping up with the latest releases.

How often do we want to release a blessed version? Once a month? Once a quarter?

Once a month is still pretty frequently and makes it hard for projects with longer release cycles. I'd prefer to start with once a quarter and adjust from there.

How should we indicate a version is blessed? For example, we could have a page in the docs with a table indicating which releases are blessed. We could also give blessed releases a special version number (xref #100)

It would be nice to take the special version number approach, but it would still need to compare / sort nicely with other versions. Given the plan to move to CalVer I don't see how we'd do this nicely.

[mmccarty]

Thanks, James.

I'm also curious about how we determine that a version qualifies as "blessed"? Is this simply based on how many issues have been discovered by the community prior to a release date? Or is there a validation process to "bless" a release?

[jakirkham]

My guess is the former.

[jsignell]

Is there a validation process to "bless" a release?

I don't think there is much appetite on the maintainer side for having a formal validation process. I imagine that it'll be something like: when it's been 3 months since the last blessed release, we ask did the most recent one feel particularly buggy? If no, bless, otherwise wait for next release and ask again.

Maybe we should start by documenting the blessed releases in a table and go from there. I don't see another way to both let a version sit for a bit before declaring it blessed AND have the blessed version be part of a sortable list of versions where each version is in chronological order according to release date.

[jrbourbeau]

Maybe we should start by documenting the blessed releases in a table and go from there

That seems like a reasonable approach to me. We can start maintaining a table of blessed releases in the docs, publicize that we're doing this, and then see what level of adoption it gets

[mmccarty]

I don’t think it needs to be maintainer driven or a formalized process. I think if Julia’s suggestion is documented and the “blessed” versions are listed, that’s a great place to be!

[GenevieveBuckley]

@tomwhite and I were discussing the difficulty of identifying a stable enough version of Dask for sgkit to depend on.

Tom suggests maybe there should be better stability testing, or a better way to indicate to users what is likely to be stable. It's possible "blessed" releases might be one way of indicating stability, so I'd like to hear his thoughts here.

(They have set up a CI test that will test against upstream versions of Dask, to give more warning & a chance to file issues upstream. That's not a total solution to the problem though, there's still a burden)

[GenevieveBuckley]

cc @hammer - relevant to our slack conversation

[tomwhite]

Thanks @GenevieveBuckley. Having a list of released versions that are blessed by virtue of seeming to be fairly stable only partly solves the problem. Ideally there would be a stable release line, which would be a branch with select bug fixes backported from the main line. Obviously this is a lot of work, and not something that the community necessarily wants to take on. (It also doesn't fit well with the CalVer scheme, as noted above.)

The CI we have in sgkit running against upstream libraries is good for detecting incompatible API changes (that we can feed back and fix in Dask, as we have been doing), but not for finding changes in behaviour or stability of Dask distributed at scale, which is where we have been having problems recently.

[TomAugspurger]

Can someone how CalVar and backporting interact poorly? IIUC, the issue is being unable to do a "non-blessed" release in the same month as a blessed release? Is that the only issue?

Say we release dask 2021.2.0 on February 1st and declare it blessed. If we were to release a non-blessed 2021.2.1 on February 15th, we'd then be unable to backport fixes to the blessed 2021.2.x branch.

So, can we have a policy of "don't make non-blessed releases in the same month as a blessed release" and solve everyones problems? Users pin to a month like dask>=2021.2,<2021.3 and get the blessed release + conservative backports? Or rather, people should pin directly to dask==2021.2.0, and rely on a bot like dependabot + an unit test and performance test suite to catch any issues automatically, before merging the update :)

---

Versioning problems aside, I have a few scattered thoughts:

1. Backports aren't too much extra effort. With bots like pandas, matplotlib, a backport just requires adding a milestone with special description (assuming no conflicts, tests pass on the other branch, etc.). In the best case, backports are just the extra effort of releasing.
2. That said, I feel like the project's maintenance is stretched even more thinly right now? But I might just be projecting, since I've been pulled away for a while (though hoping to get back to it later this year).
3. We have microbenchmarks at https://pandas.pydata.org/speed/ using ASV. If people have thoughts (and more importantly time / money) to make those more useful (in particular, writing more benchmarks and reporting regressions) it'd be appreciated. I'm not sure how likely those microbenchmarks would be for catching scaling issues. Perhaps we'd need another approach (again, thoughts welcome here).

[GenevieveBuckley]

Hi @tomwhite. I also had a chat with Matt Rocklin today, and (1) sgkit absolutely got a lot of disruption from this, and (2) he thinks that level of disruption was a fluke and is unlikely to reoccur. I hope that is at least somewhat reassuring.

[mrocklin]

The stability issues in the 2021.12 release were serious. I don't want to trivialize them by calling them a fluke. They were also somewhat expected. We had changed *a lot* in that version change and somewhat expected it to be painful. We should have issued a patch release earlier, which was certainly a misstep. I don't foresee any change that large coming any time soon. There has not been a similarly sized release any time in the last few years. We can and should certainly improve stability and communication around releases, but I would caution folks against anchoring too strongly on the 2020.12 release.

…

On Wed, Feb 17, 2021 at 8:03 PM Genevieve Buckley ***@***.***> wrote: Hi @tomwhite <https://github.com/tomwhite>. I also had a chat with Matt Rocklin today, and (1) sgkit absolutely got a lot of disruption from this, and (2) he thinks that level of disruption was a fluke and is unlikely to reoccur. I hope that is at least somewhat reassuring. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#101 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGXNPH6V3W2NQCP2F3S7RYQTANCNFSM4SGR7BJA> .

[GenevieveBuckley]

I'm sorry for my wording, I had intended a meaning closer to "a one off, unusual" instead of any trivializing connotation. Hope that helps clear things up!

[tomwhite]

So, can we have a policy of "don't make non-blessed releases in the same month as a blessed release" and solve everyones problems?

You can't declare a release blessed or not at the time of release, since you don't know if it's stable until it has undergone some testing in the community, which I think happens after release - and could take weeks. So this would mean you could only do one release a month - which seems overly restrictive.

I don't foresee any change that large coming any time soon.

But even small changes can be destabilizing. There are multiple projects in play here (dask, distributed, dask-ml, etc) and changes to one can have unforeseen impacts on another. And the resulting problems can take several releases to surface and sort out.

It seems like the problem of choosing a suitable release is one that the commercial Dask vendors would be interested in solving for their users. Maybe something to feed back into which releases are blessed.