memory and jar file handler leak

[scabug]

I have spotted this bug in the context of the presentation compiler but it also affects the compiler: https://github.com/ensime/ensime-server/issues/1322

The compiler's classloader is not releasing references to jars and they get kind of lost. It's actually a well known JVM bug and also affects sbt as I noted here sbt/sbt#2496

The impact is that jar files (which could be targets of the build tool, or scalac itself) on Windows become immutable and cannot be deleted, changed, or moved. This is really very bad. In the best case scenario (Linux), it's just a memory leak.

Some more reading

• http://management-platform.blogspot.co.uk/2009/01/classloaders-keeping-jar-files-open.html
• https://github.com/ancoron/apache-cxf/blob/master/common/common/src/main/java/org/apache/cxf/common/logging/JDKBugHacks.java
• http://stackoverflow.com/questions/7371277

with the http://www.docjar.com/docs/api/com/sun/tools/javac/file/CloseableURLClassLoader.html being proposed as a workaround (I've not personally used it, but it sounds sensible). I got IllegalArgumentExceptions when I tried to reflectively access the internals of the classloader, so I hope the openjdk class is somehow able to workaround that.

Note that this is independent of #9632 but they have a similar effect. But this one is far more serious as when the classloader is thrown away, the JVM seems to be unable to ever run the finalizers of the JarFiles.

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-9682?orig=1
Reporter: Samuel Halliday (fommil)

[scabug]

@retronym said:
More specifically, Global has a lazy val macroClassLoader which is used to load macro implementations. This class loader is neither closed, nor is a reference to it released, as the end of a run.

Here's a sketch of how we could improve the situation:

https://github.com/scala/scala/compare/2.12.x...retronym:ticket/9682?expand=1

The diff gets large as I started applying the same idea to the compilation classpath. That is tricker as we need to propagate the close() calls all the way down the line to the places to all the places that opened file handles.

[scabug]

Samuel Halliday (fommil) said:
Sbt does something similar, I have a failing appveyor reproduction and I'm hunting for a workaround no matter how evil sbt/sbt#2496

[scabug]

Samuel Halliday (fommil) said:
I have a workaround for this, see https://github.com/fommil/class-monkey

It would be much better if any use of URLClassLoader was replaced with a custom implementation, aimed at cleaning up afterwards and only handling local files. Happy to contribute my code if it could form the basis of a better solution, but the tests must remain GPLv2 as they are from OpenJDK.

[scabug]

@adriaanm said:
Pushed to 2.12, as 2.11 development has wound down.