add a Java 9 community build

[SethTisue]

starting, I suppose, with just a stripped-down thing that only attempts to build some basic stuff like the Scala standard modules, then expanding from there

Scala 2.12 only is fine for now, I think. the behemoths are already somewhat pressed for space (and the space can't be increased without rebuilding the nodes), and even just adding the 2.12/jdk9 combo will make the problem, one might say, worse enough

plan:

• get it running locally on my Mac
• get it running on Jenkins

re: Jenkins, @retronym did some work, prior to JDK 9 going final/gold/GA, on making it available on our Jenkins nodes; see scala/scala-jenkins-infra#174 for details

[SethTisue]

my experience attempting this locally was that all extractions failed with [scalacheck] [error] scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.

but then I made an empty sbt project and ran it with Java 9, and sbt/sbt-launcher-package#143 kicked in and showed the "Copying runtime jar." message. I thought then running dbuild again might work, but same error

maybe I also need to do something to make sure -Dscala.ext.dirs=... gets set?

[SethTisue]

maybe I also need to do something to make sure -Dscala.ext.dirs=... gets set?

helps, but is not sufficient. adding extra.options: ["-Dscala.ext.dirs=/Users/tisue/.sbt/0.13/java9-rt-ext-oracle_corporation_9"] to every projects allows extraction to proceed farther, but then eventually every projects gets an error like:

[scala-parser-combinators] java.lang.AssertionError: assertion failed: not a type: symbol
[scala-parser-combinators]  loaded from class ScalaModulePlugin$ in package with name ScalaModulePlugin$ and classloader sbt.PluginManagement$PluginClassLoader@9b8d3db
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$classToScala1(JavaMirrors.scala:979)
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:938)
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:938)

[SethTisue]

an old (2014) ticket on Java 9 support dbuild is lightbend-labs/dbuild#155

[retronym]

That might be scala/scala-dev#304. The only workaround I know it to refactor the .scala files in the build to avoid using the empty package.

[SethTisue]

java.lang.AssertionError: assertion failed: not a type: symbol

this is reproducible outside dbuild simply by switching to Java 9 and then firing up sbt in e.g. the scala-xml or scala-parser-combinators repo

two repos that don't have this problem: utest, scalacheck

as an experiment, I went into scala-parser-combinators and ripped out sbt-scala-module, after which scala-parser-combinatorsJVM/test passed. so the problem is with that plugin somehow

[SethTisue]

the problem is with that plugin somehow

okay, so let's work on that over at scala/sbt-scala-module#33, then come back here later

[SethTisue]

a "community build" consisting only of scalacheck (with scala and scala-xml binaries retrieved via aether), at https://github.com/SethTisue/community-builds/blob/wip-2.12.x-jdk9/configs/community.dbuild, is passing locally with the extra.options: ["-Dscala.ext.dirs=... workaround 🎉

so it looks like dbuild works on Java 9, which is good news, and we'll just need to expand the build a step at a time from here (and work around the scala.ext.dirs thing in a more elegant way)

[SethTisue]

we'll just need to expand the build a step at a time from here

questioning, all the while, how ambitious it's appropriate to get with this how soon! I could see this staying pretty minimal for a long time

until at least:

• community projects have added Java 9 to their own CI matrices
  • which will probably be happening pretty rapidly over the next year or so... but in the meantime we don't need to get out ahead of the community on this

and probably not even until either or both of these stronger conditions are met:

• we start work on a version of Scala which requires Java 9
• significant community projects start requiring Java 9
  • this is what prompted the creation of the scala-2.11.x-jdk8 build — significant libraries started dropping support for Java 6 and/or 7

[retronym]

Are we able to set the Java version individually per-project? That would allow us more flexibility: we could build problematic projects on Java 8 and run compliant ones on 9.

[cunei]

Not at the moment; dbuild uses whatever version of Java is in the environment, and makes no attempt to control it.

[SethTisue]

Akka has announced Java 9 support (as of Akka 2.5.6). I'll make Akka a priority next time I do some work on this

[SethTisue]

the new build has grown from one to five projects.

they are: scala-xml, plus

[info] Project scala-js----------------: SUCCESS (project rebuilt ok)
[info] Project scala-parser-combinators: SUCCESS (unchanged, not rebuilt)
[info] Project scalacheck--------------: SUCCESS (unchanged, not rebuilt)
[info] Project utest-------------------: SUCCESS (project rebuilt ok)

how I got there:

I rebased Jason's backport PR onto 2.10.x (which has other JDK9 stuff merged on it that Jason's PR doesn't include yet)

after much trial and error, my efforts to get dbuild+sbt to use a Scala version with "SNAPSHOT" and/or a SHA in the version number failed, so I publish a 2.10.7 snapshot as 2.10.7 to ~/.m2 locally (no -SNAPSHOT or SHA in the version):

brew install ant@1.9
PATH=/usr/local/Cellar/ant@1.9/1.9.9/bin:$PATH
export JAVA_HOME=$(/usr/libexec/java_home -v 1.6)
ANT_OPTS="-Xmx2048M -Xss4M -XX:MaxPermSize=256M" ant -Dbuild.release=true fastdist distpack
cd dists/maven/2.10.7
ant deploy.local

and then in my repositories file put:

  maven-local-alt: file:///Users/tisue/.m2/repository/

since as @dwijnand discovered, sbt is hardwired to ignore a normal maven-local entry when resolving Scala snapshots specifically (?!)

the config file changes necessary are in the wip branch in my fork

I'm also not going to bother trying to run this on Jenkins until Scala 2.10.7 and sbt 0.13.17 are published — I think it would be wasted work, it's enough if I just run it locally for now.