Dockerize everything

[jupp0r]

It would be nice if the whole playground would be encapsulated into a docker image. If that image was published on dockerhub, installation would boild down to a single docker run command.

[shepmaster]

Thanks for the suggestion! Could you expand a bit on the ultimate goal that you have in mind here?

[jupp0r]

Sorry for being so unspecific. Ideally I'd like to docker run integer32llc/rust-playground as the only installation step.

[shepmaster]

I understood the command you want to run 😸 What I don't understand is why you want to do that. For example, if we literally did as you asked, the child containers would never be updated — those are built via a separate cronjob.

[jupp0r]

My rationale is easy installation. The child containers could also be pushed to docker hub regularly and the parent container could pull their latest tag.

[shepmaster]

The child containers could also be pushed to docker hub regularly

So we'd need to set up some third-party service to build and push these containers on a schedule.

and the parent container could pull their latest tag.

And update something, I'm not sure what, to perform a docker pull, which doesn't happen currently.

Additionally, you couldn't run docker run integer32llc/rust-playground as we need to start more containers from inside. This means it would be something like docker run -v /var/run/docker.sock:/var/run/docker.sock ...

[wimh]

This means it would be something like docker run -v /var/run/docker.sock:/var/run/docker.sock ...

When you do that, you can also build new images from inside the container. This means that if you run cron inside a container, that can take care of rebuilding the containers on the host.

I don't think it is a good idea to combine the cron job and the UI in a single container. The cron container could also create and start the UI container. That way you could use a single docker run .... But Docker Compose is probably the better way to start multiple containers.

If you want, I can work on a POC or PR.

[shepmaster]

@wimh

If you want, I can work on a POC or PR.

That would be wonderful. Let me provide a bit more detail about the "day-to-day" running of the playground that I do now:

• A tmux session is running on an EC2 instance.
• Inside tmux, we run the Iron-based UI server. This isn't ideal as after a system restart (e.g. kernel updates), nothing comes back automatically. Iron has not crashed on us yet, so that hasn't been a problem.
• A cronjob rebuilds the stable/beta/nightly/rustfmt/clippy containers once a day. Generally, this only triggers a rebuild of nightly and clippy; the others haven't changed.
• Manually, I update the list of crates for the playground. I then "deploy" by doing a git fetch on the server, then usually wait for the next cronjob to run, but sometimes I rebuild right away.
• Manually, I update the source code for the Rust backend or JS frontend. I then "deploy" by doing a git fetch on the server, rebuild the appropriate component (inside a container for meta-levels of fun), then kill and restart the running server.

To the trained eye, you can see that this process is pretty amateur hour. I'm all about creating an improved process, but as far as I can see, we need to be able to accomplish the same goals.

I'd be more than happy to discuss any details you are interested in. I'd feel terrible if you spend time on something that ultimately is not a good fit!

I don't think it is a good idea to combine the cron job and the UI in a single container.

I agree; it's my understanding and experience that basically each container should be restricted and very focused on a single task.

[wimh]

I'll make some changes step by step, to try things out, to look how it works, and at the end I can squash the commits and create a pull request. I would not mind if you won't use anything in the end. By doing this, I'll also learn from it.

I probably need some guidance for style, naming convention, directory structure, etc. First I want to make it work, and possibly discuss security implications. Changing names later is easy.

I created a branch dockerize-everything in my fork on github. I concentrated first on building the compiler containers in a container. I did not update the documentation, so I'll explain here what I did.

docker-compose.yml references compiler/Dockerfile, and for testing I created a crontab which updates every 15 minutes.

To start (detached), use docker-compose up -d
To just build use docker-compose build (or docker-compose --project-name rustplayground build).
To initially create all compiler containers, use:

docker run -i -t -v /var/run/docker.sock:/var/run/docker.sock rustplayground_rebuild /root/compiler/build.sh

This means if compiler/base/Cargo.toml is updated, the container running cron has to be rebuild. It would also be possible to automatically fetch it from github.

It should not be difficult to automatically run the docker-compose up on EC2.

[wimh]

There is currently not a lot of progress on my side. I got a (new) job and found it difficult to find time for this. I found a technical limitation of docker-compose, so I started looking into dobi and s2i, but I am not sure yet if that will solve my problem.

If someone else want to pick this issue up, I am fine with that. Otherwise I hope I have more spare time in a few months.

[shepmaster]

In a semi-related piece of info, I'm now building the 5 primary containers in Travis CI and pushing them to Docker Hub. They are thus available for anyone to use locally.