Travis build flakiness

[ittaiz]

I thought the travis build flakiness was just because of the "is_identical" test (#217) but I'm not that sure now.
I've seen a lot of failing builds just from bazel build //test/... (example).
Pretty confident solving #76 would solve our problem since we wouldn't need Travis for our e2e.
Not too sure what to do but we currently have 3 PRs which don't pass on seemingly unrelated issues
( #215 #219 #220 ).
We can decide to lax the criteria and merge PRs even if travis isn't green but that's an awfully slippery slope.
WDYT?

[ittaiz]

There are two interesting questions I can think about:

1. Why is bazel build //test... taking 10 minutes many of the times? (not sure- maybe downloads? maybe bazel installer init?)
2. Why is it not outputting anything to the console (which makes travis timeout)?

re the second bullet I'm thinking of using experimental_ui or using a travis keep-alive hack.

it almost feels like our slogan should be "slow, usually incorrect" and we've picked two...

[johnynek]

the download and install is slow, then the machines are very slow. Next, we don't have any caching that is working as far as I remember. Travis does have caching which we could potentially use.

We aren't the only ones using travis + bazel. We should look and see what other things people are doing.

I think we should be able to get the cache to save us, but it will probably take some work.

[johnynek]

I don't want to merge without green travis in the mean time. Sooner or later, we are going to get burned doing that.

If we did get everything running in bazel, then we could use the bazel.io CI, which I think would be fine, but I don't want to switch until we don't lose coverage (we should probably be working to get test coverage higher even, since all our builds are very simple now).

[or-shachar]

Seems like rules_go had made changes in their travis CI so it won't run any slow tests:
bazel-contrib/rules_go@cf0aecd

[or-shachar]

Also - look what they done here: bazel-contrib/rules_go@85d118f#diff-7ba31f31d9e5684e82d5cc51a2c1fbf9
They used experimental_repository_cache... That also might help. I'm checking all of these in my fork and will report back