Build the Observatory using Go 1.11

[christopher-henderson]

This fixes issues #271, #281, and #283.

[jvehent]

Ha, I had circleci configured to not build PRs from forks. Fixed now. Could you rebase and repush, which should trigger a build.

[jvehent]

ok, rebased through github, waiting on CI now

[jvehent]

Looks like circleci doesn't know Go 1.11 yet...
Error response from daemon: manifest for circleci/golang:1.11 not found

[christopher-henderson]

Yeah, I was afraid of that. Nuts. I'll see about rolling an image onto Docker Hub.

[jvehent]

errrr... not sure if I'm OK with using a 3rd party image instead of an official one here. This is going to run in our production infra, after all. How about building go1.11 ourselves?

[christopher-henderson]

Sure, yeah, that's what I did. This image is mine (chenderson was taken so I took an old internet alias of mine - Nvor). But it's not a Mozilla account, so fair enough.

I'd suggest either pulling my image and recomitting to your Docker Hub, or you can produce your own doing what I did:

$ docker run --name golang golang /bin/bash -c "git clone https://github.com/golang/go.git /tmp/go; cd /tmp/go/src/; ./all.bash; rm -rf /usr/local/go; mv /tmp/go /usr/local; go version"

$ docker run --name circleci_golang circleci/golang /bin/bash -c "git clone https://github.com/golang/go.git /tmp/go; cd /tmp/go/src/; ./all.bash; sudo rm -rf /usr/local/go; sudo mv /tmp/go /usr/local; go version"

[jvehent]

I think we should use a basic Alpine image and install go ourselves using the commands above. Want to give it a shot? Otherwise I'll try to get to it.

[christopher-henderson]

SO, building Go on Alpine has had know difficulties since last fall (build: get Alpine builders passing). I'm specifically get crit in the face by this SIGSEV in CGO. Seems like they use a specifically patched version of Go to build on Alpine. For the sake of reproducibility, here's the Dockerfile I have so far.

FROM golang:alpine
RUN apk add --no-cache git bash gcc build-base
RUN git clone https://github.com/golang/go.git /tmp/go
RUN cd /tmp/go/src/; git checkout 4458a357ab819a612c0c4cafae88a65287254fe9; ./all.bash
RUN rm -rf /usr/local/go
RUN mv /tmp/go /usr/local
RUN rm -rf /tmp/*
RUN apk del git bash gcc build-base

I think I'm going to continue to try to get the previous image smaller. It's based off of docker.io/golang:latest. I'm not a Docker wizard, although it's frustrating that even after cleaning up build artifacts and squashing the filesystem it ends over twice as big as the base image.

[jvehent]

Compress all of those RUN into a single one joined with && and it'll remove 6 layers from your image.

[christopher-henderson]

Thought I'd give that a whack, although it's getting the same result as before as I was using --squash anyways. Although still to good to know that.

[christopher-henderson]

Argh, I got alpine building by stealing the Go team's own Dockerfile which applies the necessary patches, but the Observatory is banking on a Debian-like environment for things like apt-get. Trying again...

[christopher-henderson]

Alright, I have the two images down to the following sizes:

Go: 850 MB, as compared to the official image's size of 780 MB
CircleCI: 1.17 GB, as a compared to the official image's size of 1.03 GB

Several interesting side notes here.

• When building these images I used git to pull the golang repo which is...a git repo. Git repos have .git directories. .git directories are big. Delete those .git directories. Duh.
• FROM lines do not have their layers squashed. The biggest gains I got in terms of size was to expand FROM directives into copy-pasting the entire original Dockerfile, coupled with the --squash flag.
• Use make.bash instead of all.bash. all.bash is nice since it runs all of the unit tests, but it also builds and packages hundreds of MB of unnecessary stuff. make.bash is what the Go team uses to deploy.

These gists are the dockerfiles I used to create these images, in case ya'll would rather build and vendor these under a Mozilla name.

• Go 1.11
• CircleCI, Go 1.11

[jvehent]

@christopher-henderson Should we close this PR for now?

[jcjones]

You'll need to make sure that this supersedes #333

[adamdecaf]

Are there known changes already in Go 1.11 that we need here? I see a couple linked issues, but don't know the context if Go 1.10 solves any of them.

Maybe these solve the linked issues? They're all from Go 1.11's milestone.

• crypto/x509: Go should be lenient in parsing CRL Distribution Points that do not conform to RFC 5280 4.2.1.13 golang/go#23403
• crypto/x509: Trailing data in the IssuerAlternativeName extension value does not return an error when parsing certificate. golang/go#23016
• encoding/asn1: Unmarshal rejects PrintableString containing ampersand golang/go#22970

Edit: Yep. We need to build Go off master / Go 1.11.

[jvehent]

Sorry but I still don't want to use an unofficial container hosted in your account as the base. I'm trying to import the needed code into the main docker and continue depending on golang:latest in #343

[christopher-henderson]

Yup, that's fine. It's exactly why I linked you the gists of how these were made. =)

[jvehent]

I'm going to close this in favor of #343 which builds go1.11 on top of the standard golang:latest container.