provide frozen config for JVM torture testing

[SethTisue]

the goal is to provide a frozen, known-good config for JVM developers (thinking mainly of Graal, but perhaps others as well) to torture test VM changes against a ton of Scala code

the main thing we can do to make this reproducible and reliable is:

• put a frozen config file on a branch that uses a fixed Scala version and fixed SHAs of all projects

ideally, this could be run offline, but there are two barriers:

• hitting GitHub: even for fixed SHAs, dbuild always looks up every project on GitHub every time
  • workaround: none that I know of. if this a show-stopper perhaps we could interest @cunei in tackling can I skip re-fetching of the repos? lightbend-labs/dbuild#210
• binary dependencies: binary dependencies (Scala and sbt and dbuild themselves, plus all the binary dependencies of all the build definitions) must be retrieved at least once
  • possible workaround Switch to mainline of SBT 0.13 #1: run the build once; capture the resulting ~/.ivy2/cache directory
  • possible workaround Remove references to now-removed akka sub projects. #2: cache everything on a local Artifactory instance; we already have a suitable config at https://github.com/scala/community-builds/blob/2.12.x/artifactory/artifactory-lightbend.xml. this is how I do all my community build work, so I know the config works.

other cautions and things to be aware of:

• Scala's own tests: the community build does not run Scala's own test suite for its compiler and standard library. anyone wanting to torture-test their VM ought to start there, and run the community build second.
• flaky tests some projects have flaky tests. when a project has a test that fails often, we disable it (we can disable per-source-file, per-subproject, or for an entire project). but that still leaves a long tail of occasional failures. with 160+ projects in the build, it's not uncommon to get at least one spurious failure
  • workaround: to determine whether a failure is real or spurious, you can simply re-run the build. the overall community build takes about 13 hours, but dbuild is smart about caching the results of projects that have already run, so even if something fails and you have to try again, the next run will be much faster. (in the best case, as little as 5 minutes or so.)
• Java version matters: on Java 8, the default config is expected to be all-green. on Java 9, many projects fail; about 2/3 of the build is green
  • solution: I would happily provide, on request, an all-green config for Java 9 by commenting out failing projects.

[SethTisue]

I've pushed a stab at this to the freeze-2.12-20180616 branch

let's try it: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3016/

[SethTisue]

that attempt needed a few tweaks. trying again: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3019/

[SethTisue]

nearly green. cats and sbt failed; these are projects known to have intermittent test failures, so let's run one more time

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3023/ (404 til Jenkins queue clears)

[SethTisue]

@milessabin once this is done you might like to use it for PR testing (e.g. scala/scala#6582)

[SethTisue]

run 3023 is green, so this is good to go

I'm going to leave the ticket open while I think a bit more about whether to set up some kind of formal process for doing this once a month, something like that.

[SethTisue]

dear users of the frozen branch, if you have any trouble with pesky intermittent failures in particular projects, consider letting me know (via https://github.com/scala/community-builds/issues/new — include the failure log). I can disable certain tests, and/or gently nag the upstream maintainers a bit about either fixing or at least segregating flaky tests.

[SethTisue]

current status:

• I'm not currently actively pursuing further improvements in this area
• I'm happy to provide new frozen configs on request
  • (you can also try using the freeze script in this repo yourself, but you might prefer to just ask me and not fumble with it yourself, and that's totally fine)

[SethTisue]

we recently froze the 2.12 build for the Scala 2.12.9 release. and then since then we have unfrozen again, but the frozen state is here: 56eae0f ; a Jenkins log for a green run at that commit is https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/4794/consoleFull

[SethTisue]

I'm going to close this ticket; I remain happy to provide a link to a frozen-and-fully-green state any time, to anyone who could use it

[SethTisue]

note that we always use fixed project SHAs now, and we only advance those SHAs after verifying that the new SHAs pass. so this whole ticket is much less relevant than it used to be

[SethTisue]

also there's much less of a problem with intermittent test failures than before. it still happens, but I excluded a ton of tests in the last month or so and really reduced the problem a lot.

[SethTisue]

Java version matters: on Java 8, the default config is expected to be all-green. on Java 9, many projects fail; about 2/3 of the build is green solution: I would happily provide, on request, an all-green config for Java 9 by commenting out failing projects

Note that this has since progressed. Java 8 and 11 are 100% green. A few projects are known to fail on JDK 14, but the report/Report.scala script knows which ones they are and exempts them from failing the overall build status.