Add option to "go install" all dependencies for Docker build caching

[benweissmann]

Some context: I'm setting up a development workflow for Go-based services that will run on a Kubernetes cluster. I'm setting up a minikube-based dev environment using a setup like Draft (not just using Draft because of Azure/draft#432) where i watch for changes to the source files, and then re-build a docker container and run it on Minikube. I'm trying to minimize the time it takes to deploy a change.

I was able to cut the docker build time from about 15 seconds down to about 8 by go installing my dependencies, and then adding my source files and compiling those, because it means that the dependencies don't need to be re-compiled during each build.

My dockerfile looks something like this:

FROM golang:1.9.2-alpine3.6

RUN apk add --no-cache curl && \
    curl -Lo /bin/rq  https://s3-eu-west-1.amazonaws.com/record-query/record-query/x86_64-unknown-linux-musl/rq && \
    chmod +x /bin/rq

RUN mkdir -p /go/src/mycompany/myapp
ENV GOPATH=/go
WORKDIR /go/src/mycompany/myapp

ADD vendor /go/src/
ADD Gopkg.lock /go/src/mycompany/myapp
RUN cd /go/src && \
  cat mycompany/myapp/Gopkg.lock | /bin/rq -tJ 'map "projects" | spread | map "name"' | cat | tr -d '"' | xargs -I % go install %/...

ADD *.go ./

RUN go build -o app .

This works, but the bit where i'm using rq to parse Gopkg.lock is a pretty crazy hack. Would you be open to adding something like dep install-deps or dep ensure -vendor-only -install that would go install vendored dependencies? Or perhaps there's a simpler way of accomplishing what I'm trying to do, and we could update https://github.com/stephenafamo/dep/blob/master/docs/FAQ.md#how-do-i-use-dep-with-docker ?

[sdboyer]

hi, welcome!

i'll be clear up front and say that, sorry, but dep stays strictly away from doing any actual compilation. this is an important way that we keep our scope limited, which ultimately makes adapting dep into the toolchain a vastly simpler (read: feasible) process.

independent of that, though, i'm a bit confused about what your goal is, here - if i'm grokking the piped logic there correctly, then it appears you're trying to discover and go install every main package in vendor? ...where that vendor is actually a GOPATH/src...ick.

well, some thoughts:

1. it seems like you might be able to accomplish this goal with some nifty go list magic.
2. the only way to guarantee the dependencies for those main packages are present in vendor is if you include them in the required list in Gopkg.toml. you might have an easier time just parsing and installing that list.
3. github.com/GetStream/vg might be able to do what you want more directly.

[benweissmann]

hi, thanks for the quick response!

To explain myself a little better:

• My end goal is to speed up the docker build command for my Go program
• One way I can do that is by separating the "compile dependencies" step from the "compile app code" step, so that Docker can cache the compiled dependencies and not re-compile them each time it builds
• So I'd like to compile all the dependencies of my program (producing a .a for each package), and I'd like to be able to determine "all the dependencies" based solely on Gopkg.lock.

I looked through your three suggestion, and I don't think any of them will quite work:

1. I can't use go list for this because that operates on the .go source files, which I don't want to add to the Docker container before i do the install -- that would defeat the whole purpose, because then when I change app code, it would invalidate the build step that compiles dependencies, so I'd end up re-building my dependencies every time.
2. Adding all of my dependencies to required and using vg might work, but that removes a lot of the convenience of using dep -- I can't just use dep ensure, I have to then go add all of those dependencies to required. It also feels a bit hacky -- it seems like a mis-use of required.

I totally get that dep wants to stay away from actual compilation -- that makes a lot of sense. I also agree that the crazy piped logic is pretty gross -- that's what I'm trying to get rid of. What the piped logic is doing is determining all of the dependencies based on Gopkg.lock, and then sending those over to xargs to go install them.

What if we added a command to dep that just printed out dependencies based on Gopkg.lock (ignoring the go source)? Basically a version of dep ensure -vendor-only, but it prints out what it would be vendoring instead of putting them in vendor. Then that crazy piped logic could just be dep ensure -vendor-only -print | xargs go install (here i'm using -print as a straw-man for my proposed option to just print the dependencies rather than vendoring them). So dep would just be responsible for enumerating the dependencies (a subset of its current responsibilities) and we would send that over to go install for the actual compilation.

Does that seem more reasonable?

[sdboyer]

oooook, i think i understand better now 😄 🎉

this still isn't quite the right thing for dep ensure to be doing, but it is something that could readily be within the purview of dep status (/cc @darkowlzz). have a look at the (still slightly evolving) command spec - the formatted output options could likely produce the information you want.

however, for your immediate purposes at least, i'd really just recommend writing a little custom Go binary to do what you need. github.com/golang/dep contains helper functions to load and parse a Gopkg.lock, which you can then use to not only more easily and accurately parse the data therein, but possibly also produce a more exact list of the dependencies to go install than the ./... syntax would produce for you. while we'd like to ultimately have dep status be flexible enough to serve all these needs, i can't fully promise it will be, or when we'll get there. doing it yourself, though - i imagine you could bang it out in an afternoon or so.

[darkowlzz]

I've created a PR implementing the template output flag -f in dep status, which should help with this issue.
@benweissmann it would be great if you could try #1389 and see if it helps and maybe suggest any improvements.

[benweissmann]

@darkowlzz That flag does almost what I need, but dep status won't work without the go source code -- so in order do use dep status in my Dockerfile, i'd need to add all the sources first which is what i'm trying to avoid. I want to use docker to cache the compiled dependencies before I add my sources so I don't need to re-compile dependencies when my source changes.

Could there be a -vendor-only flag to dep status like there is to dep ensure to only examine Gopkg.lock and vendor/ and ignore sources?