Runtime reflection of classes in the empty package broken under Java 9

[retronym]

⚡ cat /tmp/Test.java; javac -d /tmp/out /tmp/Test.java && (java_use 9; scala -nobootcp -J-Dscala.ext.dirs=/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/scala-ext)
public class Test {
}

Welcome to Scala 2.12.1 (Java HotSpot(TM) 64-Bit Server VM, Java 9-ea).
Type in expressions for evaluation. Or try :help.

scala> val cl = new java.net.URLClassLoader(Array(new java.io.File("/tmp/out").toURI.toURL), getClass.getClassLoader); val cls = cl.loadClass("Test"); val instance = cls.getConstructor().newInstance(); val mirror = reflect.runtime.universe.runtimeMirror(cl); val sym = mirror.reflect(instance)(reflect.ClassTag(cls)).symbol
java.lang.AssertionError: assertion failed: no symbol could be loaded from class Test in package with name Test and classloader java.net.URLClassLoader@199806aa
  at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala1(JavaMirrors.scala:1022)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$classToScala$1(JavaMirrors.scala:980)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$toScala$1(JavaMirrors.scala:97)
  at scala.reflect.runtime.TwoWayCaches$TwoWayCache.$anonfun$toScala$1(TwoWayCaches.scala:38)
  at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toScala(TwoWayCaches.scala:34)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.toScala(JavaMirrors.scala:95)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala(JavaMirrors.scala:980)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.classSymbol(JavaMirrors.scala:196)
  at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.symbol(JavaMirrors.scala:234)
  at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.symbol(JavaMirrors.scala:233)
  ... 29 elided

By contrast, if we place Test in some package, p1:

⚡ cat /tmp/Test.java; javac -d /tmp/out /tmp/Test.java && (java_use 9; scala -nobootcp -J-Dscala.ext.dirs=/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/scala-ext)
package p1;

public class Test {
}

Welcome to Scala 2.12.1 (Java HotSpot(TM) 64-Bit Server VM, Java 9-ea).
Type in expressions for evaluation. Or try :help.


scala> val cl = new java.net.URLClassLoader(Array(new java.io.File("/tmp/out").toURI.toURL), getClass.getClassLoader); val cls = cl.loadClass("p1.Test"); val instance = cls.getConstructor().newInstance(); val mirror = reflect.runtime.universe.runtimeMirror(cl); val sym = mirror.reflect(instance)(reflect.ClassTag(cls)).symbol
cl: java.net.URLClassLoader = java.net.URLClassLoader@1a8e44fe
cls: Class[_] = class p1.Test
instance: Any = p1.Test@5d8e4fa8
mirror: reflect.runtime.universe.Mirror = JavaMirror with java.net.URLClassLoader@1a8e44fe of type class java.net.URLClassLoader with classpath [file:/tmp/out/] and parent being scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@7b9088f2 of type class scala.tools.nsc.interpreter.IMain$TranslatingClassLoader with classpath [(memory)] and parent being scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@3acc3ee of type class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader with classpath [jrt:/modules/java.activation,jrt:/modules/java.annotations.common,jrt:/modules/java.base,jrt:/modules/java.compact1,jrt:/modules/java.compact2,jrt:/modules...

The same problem exists in a more vanilla classloader setup:

cat /tmp/Test.java; javac -d /tmp/out /tmp/Test.java && (java_use 9; scala -classpath /tmp/out -nobootcp -J-Dscala.ext.dirs=/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/scala-ext)
public class Test {
}

Welcome to Scala 2.12.1 (Java HotSpot(TM) 64-Bit Server VM, Java 9-ea).
Type in expressions for evaluation. Or try :help.

scala> val test = new Test()
test: Test = Test@6aad919c

scala> val mirror = reflect.runtime.universe.runtimeMirror(test.getClass.getClassLoader)
mirror: reflect.runtime.universe.Mirror = JavaMirror with scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@3b57f915 of type class scala.reflect.internal.util.ScalaClassLoader$URLClassLoader with classpath [jrt:/modules/java.activation,jrt:/modules/java.annotations.common,jrt:/modules/java.base,jrt:/modules/java.compact1,jrt:/modules/java.compact2,jrt:/modules/java.compact3,jrt:/modules/java.compiler,jrt:/modules/java.corba,jrt:/modules/java.datatransfer,jrt:/modules/java.desktop,jrt:/modules/java.httpclient,jrt:/modules/java.instrument,jrt:/modules/java.jnlp,jrt:/modules/java.logging,jrt:/modules/java.management,jrt:/modules/java.naming,jrt:/modules/java.prefs,jrt:/modules/java.rmi,jrt:/modules/java.scripting,jrt:/modules/java.se,jrt:/modules/java.se.ee,jrt:/modules/java.secu...
scala> val sym = mirror.reflect(test).symbol
java.lang.AssertionError: assertion failed: no symbol could be loaded from class Test in package with name Test and classloader scala.reflect.internal.util.ScalaClassLoader$URLClassLoader@3b57f915
  at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala1(JavaMirrors.scala:1022)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$classToScala$1(JavaMirrors.scala:980)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$toScala$1(JavaMirrors.scala:97)
  at scala.reflect.runtime.TwoWayCaches$TwoWayCache.$anonfun$toScala$1(TwoWayCaches.scala:38)
  at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toScala(TwoWayCaches.scala:34)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.toScala(JavaMirrors.scala:95)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala(JavaMirrors.scala:980)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.classSymbol(JavaMirrors.scala:196)
  at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.symbol(JavaMirrors.scala:234)
  at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.symbol(JavaMirrors.scala:233)
  ... 29 elided

This issue prevents startup of SBT under Java 9 (even after using the patched SBT in sbt/sbt#2951 with the instructions) if the build definitition contains an AutoPlugin with an autoimport module, like this one in scala/scala.

[retronym]

The workaround is to move .scala files in the build definition out of the empty package.

[dwijnand]

Generally isn't using the root/empty package there-be-dragons territory? Akin to using dollars in type names?

[dwijnand]

sbt doesn't help by setting a bad example

[retronym]

REPL free reproduction:

⚡ tail sandbox/test.java sandbox/Test.scala
==> sandbox/test.java <==
public class Test {
    public static void main(String[] args) {
        new p1.ReflectTest().test(new Test());
    }
}

==> sandbox/Test.scala <==
package p1

class ReflectTest {
  def test(a: AnyRef): Unit = {
    val mirror = reflect.runtime.universe.runtimeMirror(a.getClass.getClassLoader)
    println(mirror.reflect(a).symbol)
  }
}

% scalac -d /tmp sandbox/test.scala; javac -cp /tmp -d /tmp sandbox/Test.java 

% (java_use 1.8; java -classpath $HOME'/scala/2.12.1/lib/*:/tmp' Test)
class Test

% (java_use 9; java -classpath $HOME'/scala/2.12.1/lib/*:/tmp' Test)
Exception in thread "main" java.lang.AssertionError: assertion failed: no symbol could be loaded from class Test in package with name Test and classloader jdk.internal.loader.ClassLoaders$AppClassLoader@484b61fc
	at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala1(JavaMirrors.scala:1022)
	at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$classToScala$1(JavaMirrors.scala:980)
	at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$toScala$1(JavaMirrors.scala:97)
	at scala.reflect.runtime.TwoWayCaches$TwoWayCache.$anonfun$toScala$1(TwoWayCaches.scala:38)
	at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toScala(TwoWayCaches.scala:34)
	at scala.reflect.runtime.JavaMirrors$JavaMirror.toScala(JavaMirrors.scala:95)
	at scala.reflect.runtime.JavaMirrors$JavaMirror.classToScala(JavaMirrors.scala:980)
	at scala.reflect.runtime.JavaMirrors$JavaMirror.classSymbol(JavaMirrors.scala:196)
	at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.symbol(JavaMirrors.scala:234)
	at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.symbol(JavaMirrors.scala:233)
	at p1.ReflectTest.test(test.scala:6)
	at Test.main(Test.java:3)

[retronym]

Okay, finally took another look at this. Here's the behaviour difference in Java reflection that we're stumbling into:

⚡ for V in 1.8 9; do (java_use $V; scalac $(f "class C; object C extends App { println(classOf[C].getPackage) }") && scala C ); done
null
package

The null return value in 1.8 might have been a bug. The relevant JDK code has been overhauled with the implementation JPMS. Should be easy enough to adapt on our side, I'm going to try that now.

[SethTisue]

@eed3si9n @dwijnand is there any possibility of an sbt 0.13.17 release that has some kind of workaround for this?

this is already turning into a big problem for the Scala community build on JDK 9 — nearly every project I've tried to add has hit it, because practically any build uses some sbt plugin that has code in the empty package

[eed3si9n]

I'd be happy to ship 0.13.x if we have some workaround for better JDK 9 compatibility.

[dwijnand]

Will scala/scala#6097 be backported to 2.10 and a release of 2.10 cut with it? Otherwise we can only fix this in sbt 1.

[retronym]

We're considering that in scala/scala#6113