Use emsdk on travis for CI #5087
Conversation
|
Maybe this can't be done, we are hitting Perhaps the 16.04-built SDK can't be run on the 12.04 or 14.04 that is on travis? |
|
Yeah this looks like a symbol versioning issue of the sort that is likely to arise if you build on a newer system than the one you want to run on. There are a couple of ways you might try to get around that. You can link with |
Dockerfile
Outdated
| RUN ls | ||
| RUN ./emcc # first use | ||
| RUN ./emcc -v # check s tuff | ||
| RUN python tests/runner.py test_hello_world |
There was a problem hiding this comment.
Right now this Docker file will consume 11 layers, it will be better to make it look like:
RUN apt-get update \
&& apt-get install -y python wget cmake \
&& wget https://s3.amazonaws.com/mozilla-games/emscripten/releases/emsdk-portable.tar.gz \
&& tar -xvf emsdk-portable.tar.gz \
&& cd emsdk-portable \
&& ./emsdk update \
&& ./emsdk install latest \
&& ./emsdk activate latest \
&& ls \
&& ./emcc \
&& ./emcc -v \
&& \
&& python tests/runner.py test_hello_world
It's important, because each RUN creates a snapshot. Especially step where you install latest will expand size of the image
There was a problem hiding this comment.
Thanks! I am just starting to learn about Docker now ;)
There was a problem hiding this comment.
Please note that I wrote this comment, before you pushed some change with 27a227f
There was a problem hiding this comment.
Yeah, I've pushed a bunch of commits like that just for debugging, but I saw which commit you meant with the comment. Should be fixed now.
|
@dschuff thanks, meanwhile I am trying to run 16.04 using docker. so far no luck there either. |
|
Ok, I think I'm stuck at the point of docker starting to work, but the emsdk doesn't seem to just download a binary, it wants to run cmake and I guess install from source - which fails but shouldn't be necessary. I filed emscripten-core/emsdk#69 , hopefully @juj can help. |
|
Disabling travis for now, as @juj is on vacation and travis gives emails on each branch as failing. |
|
I have setup Travis CI successfully to build Emscripten and Binaryen from source using emsdk. The build artifacts works correctly on Travis CI (Ubuntu 12.04). If you are interested, this is how we do it. https://github.com/urho3d/emscripten-sdk/tree/sdk-update-trigger It has the "extra" bit (like building the clang-tool-extra and storing of build artifacts for later use by other CI build jobs) that you may not need and can be safely removed for your case. Also I set it up to build on demand (manual trigger) instead of as continuous integration build, but there should be no problem to make it so. The build is relatively fast (after the initial build) with the help from ccache. Let me know if you need more information. |
|
Thanks @weitjong. I was hoping to use the emsdk to avoid building from source, as emsdk now has binary downloads. But sounds like with ccache it's fast? But does the ccache work across different travis instantiations? I thought they were isolated from each other, but I understand little here... |
|
The cache objects are automatically uploaded to S3 at the end of the CI job before the VM ends and restored from S3 at the beginning of the next CI job after a new VM starts. When the cache hits is higher than the cache miss, the build from source just whizzes by regardless of how complex the source code is. And this is usually true for CI build job where the code changes incrementally. For the initial build when there is still no cache objects available then it may take longer than 50 minutes (time limit for free Travis-CI account). We handle that situation by "continuation". That is, we time-out the build process gracefully instead of being killed by Travis. This way we have the chance to save whatever cache objects we have so far in the initial build, then the next CI build can restore them and "continue" from there. Eventually when cache is warmed up, all the subsequent CI jobs will be relatively fast (the build time is a function of the code delta). |
|
Oh wow, nice use of travis + S3. Maybe we'll need to do something similar, but I'm hoping we can figure out the issues and be able to use the binaries, avoiding building entirely. |
|
Just want to clearify. The restoring/saving from/to S3 is part of Travis cache service and it is absolutely free. We just need to learn how to take advantage of it. As for reusing the the prebuilt binary from emsdk on Travis worker VM, I think it would be tough to pull off unless juj also prebuild one specifically for Ubuntu 12.04 (or 14.04 if you plan just use "trusty" VM image for your CI). |
|
Ubuntu 12.04 LTS is nearing EOL soon and 14.04 LTS in a few years (source: https://en.wikipedia.org/wiki/Ubuntu_version_history#Table_of_versions), is it possible to update your CI system to the latest LTS version, 16.04? Don't know how to do it on Travis CI (if it is possible), but if it is helpful here's what I am doing on a different continuous integration service, CircleCI, to run the downloaded binary emsdk on Ubuntu 16.04 LTS (Xenial Xerus): satoshinm/NetCraft@1066b1b |
|
I think this is rather off topic now. But still, this is my two cents. I suppose when that time comes, Travis-CI will phase out their "precise" VM image and set the "trusty" as the new default and probably the "xenial" will be made available. For those that could not wait, of course they can always use roll-your-own docker approach. The topic of the discussion here is about how to get a "working" binary for the Travis-CI now today, whether the binary is prebuilt or or build from source just before it being used in the following CI tests. To me, build from source has the advantage. The CI tests are always tested against the latest binary. The binary itself is always guaranteed to be compatible with the Travis-CI VM. It is future proof because as the LTS version is changed, so does the binary. |
|
Emsdk does have precompiled SDK builds for Linux, which are made using Ubuntu 16.04. I wonder if the |
|
I believe |
|
You can get the precompiled binaries from emsdk to work on Trusty by grabbing a copy of libstdc++ from the First note that emcc fails to run on Trusty (this is Emscripten 1.37.16): I then grab libstdc++ from |
|
@cosinusoidally since the deb is just being extracted does that mean your approach runs on container (sudo: false) builds? |
|
@rajsite should do. I extract the deb and use LD_LIBRARY_PATH to avoid needing to install the deb (it would also be a very bad idea to install that deb since it would replace the system copy of libstdc++, which can break your system). Having said that, I have not tested the above workaround on Travis. I've only tested on my Desktop installation. I assume you would just need to tweak your .travis.yml to download the deb, check the sha256sum, extract it somewhere, and then add it to your LD_LIBRARY_PATH . |
|
As mentioned before, our approach is to prebuild the EMSDK on Travis CI itself and store the prebuilt artifact in a GitHub repo to be quickly restored and activated in our CI build. We have updated all our CI builds to use Trusty. No issue so far. |
|
What's the main blocker on this PR? It seems the last error is |
|
Probably the main thing here is that this needs someone with time to look at it. I started this PR but I don't have time myself currently, would be great if someone else wants to. |
|
Should we close this PR? |
|
Thanks, yes, this was an earlier attempt that failed. |
See #4794