2.12 build: retrieve pre-built Scala instead of building it ourselves

[SethTisue]

build's "scala" build system is ant-based and isn't really worth updating to use sbt, because there's no need anymore to "address the case in which a cycle exists between the core (library/compiler) and the modules" within dbuild — we can just let scripts/jobs/validate/publish-core do it for us and consume the artifacts it produces

once this is done, scala/scala#5311 can be merged, bye forever Ant!

[SethTisue]

Adriaan's WIP commits on this:

• adriaanm@bc9909a ("resolve scala locker using aether")
  • a much older version: adriaanm@2b1ce1b
• adriaanm@de77bb8 ("sample jenkins.properties")
  • Adriaan: "my experiment with sucking in the jenkins.properties into the community build didn't work. let's just hardcode it all for now, as you suggested, Seth". eventually we would like the community build to be part of a jenkins workflow, so that was the experiment, but for now it's fine to just check the properties into version control directly

[SethTisue]

in Adriaan's WIP the publish-core-built Scala was used everywhere as the extraction version, but I think it's better to use STARR so dbuild doesn't think it needs to re-extract every project every time

[SethTisue]

if we don't have jenkins.properties we need to get the Scala version from somewhere. I suggest we use scala_ref, but instead of a Git ref it will be e.g. 2.12.0-abcd123-SNAPSHOT

[SethTisue]

in Adriaan's WIP we are still using the "assemble" build system to pull JARs (for scala-library, scala-compiler, scala-reflect) even though we aren't using it to build anything. perhaps that's not actually necessary? I'm not sure. leaving it that way for now anyway

[SethTisue]

my WIP branch: https://github.com/SethTisue/community-builds/commits/anticide . test run https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/539/consoleFull failed with a bunch of these errors: sbt.ResolveException: unresolved dependency: org.scala-lang#scala-reflect;2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c: not found

here's the Scala JARs being retrieved and republished:

[scala-library] Requested:
[scala-library]   org.scala-lang#scala-library;2.12.0-170373f-SNAPSHOT
[scala-library] Not provided by dbuild, will try to fetch from an external repository.
[scala-library] It will be republished as:
[scala-library]   org.scala-lang#scala-library;2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c

[scala-reflect] Requested:
[scala-reflect]   org.scala-lang#scala-reflect;2.12.0-170373f-SNAPSHOT
[scala-reflect] Not provided by dbuild, will try to fetch from an external repository.
[scala-reflect] It will be republished as:
[scala-reflect]   org.scala-lang#scala-reflect;2.12.0-dbuildx65a318b3584e9bfa4afbe242817396807dea8385

[scala-compiler] Requested:
[scala-compiler]   org.scala-lang#scala-compiler;2.12.0-170373f-SNAPSHOT
[scala-compiler] Not provided by dbuild, will try to fetch from an external repository.
[scala-compiler] It will be republished as:
[scala-compiler]   org.scala-lang#scala-compiler;2.12.0-dbuildxf35436caba20bb5785d6e9ab11ccd6c1aa32de3c

and then it tries to build the first project:

[async] [info] Preparing Scala binaries: scala-library version 2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c
[async] [info] Setting Scala version to: 2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c in 2 scopes
[async] All Dependencies for subproject default-sbt-project:
[async]    org.scala-lang:scala-compiler:2.12.0-dbuildxf35436caba20bb5785d6e9ab11ccd6c1aa32de3c:scala-tool->default,optional(default)
[async]    org.scala-lang:scala-library:2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c:scala-tool->default,optional(default)
[async]    org.scala-lang:scala-library:2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c
[async]    org.scala-lang:scala-reflect:2.12.0-dbuildx65a318b3584e9bfa4afbe242817396807dea8385:provided
[async]    org.scala-lang:scala-compiler:2.12.0-dbuildxf35436caba20bb5785d6e9ab11ccd6c1aa32de3c:test

so far so good, the versions match

but then:

[async] [info] Updating {file:/home/jenkins/workspace/scala-2.12.x-integrate-community-build/target-0.9.5/project-builds/async-d089459ccceebd0e36b131474b374827fa9767af/}async-d089459ccceebd0e36b131474b374827fa9767af...
[async] [warn]  module not found: org.scala-lang#scala-reflect;2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c

it's looking for xeed9f5 of scala-reflect, but that's only right for scala-library,
not for scala-reflect :-(

is there a clue here?

[async] [warn]  Note: Unresolved dependencies path:
[async] [warn]      org.scala-lang:scala-reflect:2.12.0-dbuildxeed9f5dad952ad70ecb8e07ca56059866ae0bd3c (/home/jenkins/workspace/scala-2.12.x-integrate-community-build/target-0.9.5/project-builds/async-d089459ccceebd0e36b131474b374827fa9767af/build.sbt#L13-14)
[async] [warn]        +- org.scala-lang:scala-compiler:2.12.0-dbuildxf35436caba20bb5785d6e9ab11ccd6c1aa32de3c
[async] [warn]        +- org.scala-lang.modules:scala-async:0.9.6-dbuildxd089459ccceebd0e36b131474b374827fa9767af

[SethTisue]

oh, it's not normal that compiler, library, and reflect all have different dbuild-rewritten Scala version numbers. if you look at any recent green community build run you see e.g.

[utest] All Dependencies for subproject utestJVM:
[utest]    org.scala-lang:scala-compiler:2.12.0-dbuildx0148c1c541133faa1b3e4436a07f80e2cbe1fa99:scala-tool->default,optional(default)
[utest]    org.scala-lang:scala-library:2.12.0-dbuildx0148c1c541133faa1b3e4436a07f80e2cbe1fa99:scala-tool->default,optional(default)
[utest]    org.scala-lang:scala-library:2.12.0-dbuildx0148c1c541133faa1b3e4436a07f80e2cbe1fa99
[utest]    org.scala-lang:scala-reflect:2.12.0-dbuildx0148c1c541133faa1b3e4436a07f80e2cbe1fa99

where it's dbuildx0148c1c54... every time. (and for further confirmation dbuild's CHANGELOG file says "all Scala artifacts will get the same dbuild-specific default version number as scala-library")

[SethTisue]

hmm, maybe getting rid of the whole "system: assemble" thing would fix it. we don't need dbuild to rewrite Scala version numbers at all, do we? just leave the Scala version as e.g. 2.12.0-170373f-SNAPSHOT?

[SethTisue]

http://typesafehub.github.io/dbuild/0.9.5/dbuild.html#assemble-specific-options recommends (in bright yellow!) not using "assemble" at all: "It is therefore advisable to adopt a regular (non-cyclic) build as soon as that is feasible". so yeah, I'll try just getting rid of it

[SethTisue]

looking at the RP build, it looks like another approach could be to use set-version to force the aether-retrieved JARs to have a shared version number

[SethTisue]

getting somewhere with set-version. next error (from log) is

[async] java.lang.NoClassDefFoundError: scala/xml/NodeSeq
[async]     at scala.tools.nsc.doc.html.HtmlFactory.writeTemplates(HtmlFactory.scala:119)
[async]     at scala.tools.nsc.doc.html.HtmlFactory.generate(HtmlFactory.scala:107)
[async]     at scala.tools.nsc.doc.html.Doclet.generateImpl(Doclet.scala:23)
[async]     at scala.tools.nsc.doc.doclet.Generator.generate(Generator.scala:23)

taking a cue from the RP build again, it looks like we need deps.inject: [ "org.scala-lang.modules#scala-xml"]

[SethTisue]

holy circularity Batman!

[scala-xml] java.lang.NoClassDefFoundError: scala/xml/NodeSeq
[scala-xml]     at scala.tools.nsc.doc.html.HtmlFactory.writeTemplates(HtmlFactory.scala:119)
[scala-xml]     at scala.tools.nsc.doc.html.HtmlFactory.generate(HtmlFactory.scala:107)
[scala-xml]     at scala.tools.nsc.doc.html.Doclet.generateImpl(Doclet.scala:23)
[scala-xml]     at scala.tools.nsc.doc.doclet.Generator.generate(Generator.scala:23)

[SethTisue]

Adriaan took a stab at it in 2.12.x...adriaanm:scala-resolve — looks like the key difference is to get scala-xml (and other modules? maybe only scala-xml is necessary?) via Aether. test run

@adriaanm: "We'll need to run it against a build that publishes sources of the core for scala-js to work, otherwise it seems to be working"

[adriaanm]

Running that build here: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/544/console

[adriaanm]

I confused myself a bit, but that approach works. scala-js just needed a patch or two (see scala-js/scala-js#2580)

I'm hopeful this one will be green: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/553

[adriaanm]

@SethTisue could you clean up my scala-resolve branch and make the necessary changes in the Jenkins jobs to run this downstream from release-main? @szeiger is working on the bootstrap script / sbt support to pass down version numbers

[adriaanm]

We should also be able to trigger the community build job to consume the artifacts produced by publish-core (it will have to publish sources for scala-js to succeed, but otherwise good to go) when we don't need to bootstrap.

[SethTisue]

Adriaan's changes for cleaning-up are at #275; his green test run was https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-community-build/556/

[adriaanm]

As I said on Slack, I'd like you to adopt & polish #275 along with the needed changes to the jenkins jobs (variable renames, incorporate into release-main flow). Please coordinate with Stefan for the sbt changes to propagate version from the release/nightly/publish-core jobs to the community build job.

[SethTisue]

#309 means we'll still be using the "assemble" build system, to resolve the scala/scala-xml circularity.

ultimately we'd like to cut that circularity altogether (by having scala-xml implement an API), then we won't need "assemble" anymore. someday.

[SethTisue]

the basic thing is done. the Jenkins/Chef part is now scala/scala-jenkins-infra#189