scalaparseJVM/test fails on master with OutOfMemoryError

[SethTisue]

noticed in the context of the Scala community build:

• https://scala-ci.typesafe.com/job/scala-2.11.x-jdk8-integrate-community-build/148/consoleFull
• https://scala-ci.typesafe.com/job/scala-2.11.x-jdk8-integrate-community-build/149/consoleFull

[fastparse] Checking Dir target/repos/scala
[fastparse:error] <console>:2: warning: Detected apparent refinement of Unit; are you missing an '=' sign?
[fastparse:error]     def f1(a: T): Unit { }
[fastparse:error]                        ^
[fastparse:error] java.lang.OutOfMemoryError: GC overhead limit exceeded
[fastparse:error]   at java.util.zip.ZipFile.getInflater(ZipFile.java:455)
[fastparse:error]   at java.util.zip.ZipFile.getInputStream(ZipFile.java:374)
[fastparse:error]   at java.util.jar.JarFile.getInputStream(JarFile.java:447)
[fastparse:error]   at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:940)
[fastparse:error]   at sun.misc.Resource.cachedInputStream(Resource.java:77)
[fastparse:error]   at sun.misc.Resource.getByteBuffer(Resource.java:160)
[fastparse:error]   at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)
[fastparse:error]   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
[fastparse:error]   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
[fastparse:error]   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
[fastparse:error]   at java.security.AccessController.doPrivileged(Native Method)
[fastparse:error]   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
[fastparse:error]   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[fastparse:error]   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[fastparse:error]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
[fastparse:error]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
[fastparse:error]   at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
[fastparse:error]   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
[fastparse:error]   at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[fastparse:error]   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[fastparse:error]   at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[fastparse:error] java.lang.OutOfMemoryError: GC overhead limit exceeded
[fastparse:error] java.lang.OutOfMemoryError: GC overhead limit exceeded
[fastparse:error] java.lang.OutOfMemoryError: GC overhead limit exceeded

should I just crank up the heap size, or you think there's a real regression here you want to investigate?

[lihaoyi]

I'd say bump up the heap size? I haven't touched anything so maybe the source trees of the various cloned Scala projects got bigger...

[SethTisue]

trial run: https://scala-ci.typesafe.com/job/scala-2.11.x-jdk8-integrate-community-build/157/consoleFull https://scala-ci.typesafe.com/job/scala-2.11.x-jdk8-integrate-community-build/158/consoleFull

[SethTisue]

same failure. I get it locally also (not inside dbuild, just running the test in a regular clone of the repo), even if I modify .sbtopts and double the heap from 2G to 4G. it looks to me like there is a real regression here. are you able to reproduce it locally as well? if I take a thread dump while it's thrashing but before it actually halts, I see some pretty long hairy stack traces.

I don't have time to look it into this any further, so the test will stay disabled in the community build unless you decide (or somebody out there decides) to take it on.

[lihaoyi]

Let's take the test out for now, I have an idea about how to fix it (reduce the level of parallelism so we stop instantiating so many scala compilers at the same time) but don't have time to investigate/repro/fix right now.

Anyway I'm relatively confident in my unit test suite so leaving this out shouldn't impact stability too much

[lihaoyi]

This is a bit of a pain; I can't reproduce it locally on my macbook even with lots of runs, and on travis-ci it seems to pass intermittently. It's probably either variation in the available memory on the CI machine, or different scheduling for either SBT's parallel compilation or my own parallel-testcase-parsing that's resulting in more simultaneous memory usage than the system can handle.

Next step would be to see if killing the parallelism makes the problem go away...

[SethTisue]

disabling parallelism sounds like it could be a smart move — I have done that on several other projects, with success in fixing intermittent heap exhaustion in CI runs

[lihaoyi]

Not sure if this is still happening on your side but one our side the travis build seems pretty solid green these days

[SethTisue]

Not sure if this is still happening on your side

we disabled the test on our side and it remains disabled

[SethTisue]

if you think there's reason to believe the problem has gone away, I could certainly try re-enabling it...