For 3.7.x, please use at least OTP 20.3.x

[michaelklishin]

As far as I can tell this image provisions Erlang 19.3 for RabbitMQ 3.7.x. I don't see any reason not to provision at least 20.3 or 21.x for a few reasons:

• Single digit % of installations (that we can track) use OTP 19.3. Only the most recent patch levels are safe (older have known bugs catastrophic to RabbitMQ)
• 19.x support will be dropped in November 2018

Team RabbitMQ now provides our own Debian Erlang package "remixes" that are published to Bintray for a variety of distributions, old and new. They should be entirely compatible with the official Debian packages as far as the user goes but you get the latest and greatest versions some 30 minutes after they are tagged in the erlang/otp repo.

I'd recommend using that repository and go with OTP 21.0.x.

[michaelklishin]

I'd be happy to contribute a PR that makes the switch in case @tianon and @yosifkit find this a good idea.

[gerhard]

+1 on using the RabbitMQ Debian repo for Erlang.

rabbitmq:3.7.7 docker image is using Erlang/OTP 20.3.8.3

docker run --rm rabbitmq:3.7.7 bash -c 'dpkg -l erlang-nox'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version           Architecture Description
+++-==============-=================-============-==========================================================
ii  erlang-nox     1:20.3.8.3+dfsg-1 all          Erlang/OTP applications that don't require X Window System

[tianon]

That sounds pretty awesome; how many architectures does it support? Our current architecture list is pretty long. 😬

[gerhard]

Only amd64 & i386 according to the build pipeline (private). @dumbbell can correct me if I am wrong.

Have you considered basing the rabbitmq image on erlang:21? I can see that it supports almost all architectures that the rabbitmq image supports.

[dumbbell]

The list is even more limited than what @gerhard said: we only build the Erlang packages for amd64.

It should be easy to build for i386 as well. I would love to add other architectures but that would require cross-compilation and I never did that so far with Debian packages. Thus, I have no idea about the amount of work.

[tianon]

Have you considered basing the rabbitmq image on erlang:21?

Yes! That image is significantly larger than the rabbitmq image, and I haven't had time to look into why. Even erlang:21-slim by itself is bigger than the current rabbitmq:latest image.

• rabbitmq:latest: ~125MB
• erlang:21: ~1.06GB
• erlang:21-slim: ~249MB

[gerhard]

@tianon Got it. Happy to help as soon as I come across this again, have too many things in flight to start a new thing. Leaving a reminder for next time 💡compile Erlang/OTP 21.0.x from source💡

@michaelklishin since rabbitmq:3.7.7 Docker image is already using Erlang/OTP 20.3.8.3, what are your thoughts on next steps?

[michaelklishin]

@gerhard I think that means we are all set for now. If our repo isn't suitable because we only support one architecture at the moment, that's fine. We can always revisit this in the future.

[tianon]

Looks like we get the privilege of revisiting this... 😬

We've been installing Erlang from buster into our stretch-based image, which has worked great, but now there's been a libssl transition which pulls in a newer libc (all via the erlang-crypto package), so that's impossible to continue.

The way I see it, we have a couple options:

1. base our image on buster officially

   • pros: trivially get plenty new enough Erlang, keep many architecture support
   • cons: it's "testing" and is not explicitly supported by the Debian security team, potentially too new of Erlang at some point

2. base our image on erlang:xxx-slim

   • pros: trivially pin to an explicit Erlang version, keep many architecture support
   • cons: our base image size will go up by 2x on both Debian and Alpine variants immediately (the current erlanx:xxx-slim and erlang:xxx-alpine are roughly 2x the size of the current rabbitmq:xxx and rabbitmq:xxx-alpine images)

3. keep stretch, use https://github.com/rabbitmq/erlang-debian-package

   • pros: explicitly officially supported Erlang version!
   • cons: lose almost all supported architectures

4. keep stretch, build Erlang from source ourselves? (possibly from the package source in https://github.com/rabbitmq/erlang-debian-package? appears to not have .dsc files on bintray, so reproducing the builds properly might be a little difficult)

   • pros: explicit Erlang version, keep many architecture support
   • cons: oh lordy the compile times 😱

If there are any options I've missed, I'm definitely open to further discussion -- the cons above really speak for themselves. 😞

[tianon]

(It would be great if whatever solution we end up with ends up aligning the Debian and Alpine variants on the exact same Erlang release, but I realize that's a bit of a "stretch" goal. 😅)

[michaelklishin]

https://github.com/rabbitmq/erlang-debian-package simply packages new releases because the Erlang community doesn't do a good job of getting builds of patch releases out quickly. It is orthogonal to what versions of Erlang RabbitMQ supports, officially or otherwise.

We can consider producing packages for more architectures. What architectures are you after @tianon?

Also, we can publish more things to make option 4 possible (even though I'd hate to see a yet another set of Erlang packages produced, the OTP team and Erlang Solutions should get their stuff together).

Lastly, using buster doesn't seem to be too aggressive to me as long as you control what Erlang version gets provisioned.

[michaelklishin]

@dumbbell FYI

[tianon]

The current arch list is amd64, arm32v5, arm32v6, arm32v7, arm64v8, i386, ppc64le, s390x (where arm32v6 is Alpine-only). It'd be awesome to have official Alpine releases of Erlang too, but I won't hold my breath (the Erlang packages included in the Alpine distribution have been pretty OK, I think, minus the issues we had with HiPE).

[michaelklishin]

@tianon that’s a pretty extensive list. Unless we can cross-compile on an amd64 host our Erlang package pipeline might grow substantially more complex. The same goes for architecture/distribution matrix :(

I am leaning towards options 1 and 2 above. If buster isn’t too new for new Erlang releases, would you consider it? As an extra data point, RabbitMQ 3.8 will require a cutting edge Erlang 21.x series, e.g. 21.2 or 21.3, when it ships.

[gerhard]

I am planning to pick option 4 come January (in a couple of weeks) and get involved first-hand.

I choose option 4 because there are specific flags that we should be using when compiling Erlang, such as the one that enables extra microstate accounting, the one that enables lock counting, as well as dropping all modules that we are unlikely to use (e.g. SCTP). Lastly, having access to the Erlang source code is very useful for debugging, so I really want us to have a dev & prod versions of Docker images. As I'm writing this, I am remembering that EPMD is less relevant in K8S & Swarm, so that's another thing which I would change in the Docker image.

[tianon]

I am leaning towards options 1 and 2 above. If buster isn’t too new for new Erlang releases, would you consider it? As an extra data point, RabbitMQ 3.8 will require a cutting edge Erlang 21.x series, e.g. 21.2 or 21.3, when it ships.

Buster is currently at 21.1 (https://packages.debian.org/buster/erlang), although that may or may not change before the actual Buster release (it's hard to say without talking to the package maintainer directly), and Buster does currently have a delay for security updates (by design) until it becomes Debian Stable which doesn't make it a great base for an image, but if that's what we have to do, so be it.

I am planning to pick option 4 come January (in a couple of weeks) and get involved first-hand.

🎉

Thanks for outlining so much detail -- it makes me feel a lot better about compiling Erlang from source knowing that there are strong justifications for doing so. 👍 🤘

Would it help you if one of us were to get started on a WIP, or would you rather this simply wait until you're ready in a few weeks? Should we do the buster solution to get the 3.7 and 3.6 updates in the meantime and drop back to stretch when you get time to work on it?

[gerhard]

Doing the simplest thing required to get the 3.7 & 3.6 updates is worth it - yes to buster from this perspective.

I can only encourage starting on a WIP, especially if you are keen to and have the time. If learnings are captured in good commits as you go along, this will be most useful for when I can join the effort.

My plan was to simply start with a clean slate (most probably alpine), and apply the learnings from rabbitmq/rabbitmq-server-boshrelease. As you can see, all Erlang/OTP packages are compiled from source (e.g. 21.0.9), and all things that I would do in a Docker image are already done in the BOSH equivalent.

There is one thing that I am curious about: why are you basing the rabbitmq image on 2 different distros, alpine & debian?

[tianon]

We typically don't default to Alpine because it uses musl and that typically causes bizarre bugs / quirks in our experience, so we prefer to have users explicitly choose that behaviour if they accept the risks. For example, most projects don't explicitly test their code against musl, so even if it's loosely "supported" by an upstream (in that they don't active discourage it) it often has regressions just due to lack of formal testing.

[gerhard]

Interesting - this fits with my anecdotal evidence.

My next suspect for a clean slate is archlinux. What's the archlinux state like in Docker? It looks a bit weird looking at the base/archlinux image - the tags look healthy, but nothing else does (repo info, dockerfile & build details).

[tianon]

There isn't an actively maintained Arch image within the official images program, so it's not really an option.

[yosifkit]

The Debian and Ubuntu images are kept extremely minimal with almost no unnecessary packages installed, so either would be a solid base for the new image that has to build erlang (we definitely favor Debian since it can be smaller).

[gerhard]

Ok, it's settled then, debian it is. I must say, Debian 9.6 is looking very good.

During #297, Ubuntu Bionic turned out to be the ideal choice:

The official Canonical Ubuntu Bionic image is ideal from a security perspective, especially for the enterprises that we, the RabbitMQ team, have to deal with.

[joekohlsdorf]

Please take into consideration that the image size for Erlang has significantly dropped with recent releases:

erlang:21: ~460MB
erlang:21-slim: ~107MB
erlang:21-alpine: ~47MB

[tianon]

@joekohlsdorf where are you getting those numbers? The compressed numbers you get from Docker Hub's UI? I've been comparing on-disk size, which IMO is a more important metric (since it correlates directly to download size and is the larger resource constraint to have to worry about for users).

Here's what I get for the Erlang 21 images currently:

$ docker pull erlang:21
21: Pulling from library/erlang
Digest: sha256:3b64e2ff86f427eb18c2a3eb3129820e2c3c5e9275ef821b7f540676c33da908
Status: Image is up to date for erlang:21
$ docker pull erlang:21-slim
21-slim: Pulling from library/erlang
Digest: sha256:a5e527239b1234afc89366aa81791a97f491ad9838cc081d03906284a81c2630
Status: Image is up to date for erlang:21-slim
$ docker pull erlang:21-alpine
21-alpine: Pulling from library/erlang
Digest: sha256:32bf4ab916ba33ed726da4da1c1f822740fdb2a72ef5cb4e8ad1f55c7034218a
Status: Image is up to date for erlang:21-alpine

$ docker images erlang:21*
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
erlang              21                  856940f9414c        7 days ago          1.06GB
erlang              21-alpine           952186d522b8        2 weeks ago         73.7MB
erlang              21-slim             d83b0f455b15        2 weeks ago         250MB

Those can also be found at:

• https://github.com/docker-library/repo-info/blob/master/repos/erlang/local/21.md
• https://github.com/docker-library/repo-info/blob/master/repos/erlang/local/21-slim.md
• https://github.com/docker-library/repo-info/blob/master/repos/erlang/local/21-alpine.md

[tianon]

Fixed by #297 🎉