Make SBT based benchmarking more flexible

[retronym]

Followup to #30

• Allow injection of extra settings in to the build
• Parameterize the command name used to compile.
• Create a custom "compileRaw" task in the build (for SBT 0.13.x, the
  API I'm using must have shifted in 1.0.x)

Testing a local build of SBT that disables its metadata collection compiler phases
when antStyle = true.

sbt> set scalaVersion in compilation := "2.12.3-bin-d1ec01a-SNAPSHOT"

sbt> compilation/jmh:run HotSbtBenchmark -psource=scalap -psbtVersion=0.13.16-bin-c2e2daf -pextraBuild=incOptions:=incOptions.value.withNameHashing(false).withAntStyle(true)
...
[info] Benchmark                                (compileCmd)  (corpusVersion)  (extraArgs)  (extraBuild)  (sbtVersion)  (source)    Mode  Cnt     Score   Error  Units
[info] HotSbtBenchmark.compile                          a8c43dc               incOptions:=incOptions.value.withNameHashing(false).withAntStyle(true)  0.13.16-bin-7b5d818    scalap  sample  337   922.433 ± 9.060  ms/op

Bypassing the compile task altogether in favour of user of SBT's RawCompiler.

sbt> compilation/jmh:run HotSbtBenchmark -psource=scalap -psbtVersion=0.13.15 -pcompileCmd=compileRaw

[info] Benchmark                                (compileCmd)  (corpusVersion)  (extraArgs)  (extraBuild)  (sbtVersion)  (source)    Mode  Cnt     Score   Error  Units
[info] HotSbtBenchmark.compile                    compileRaw          a8c43dc                                  0.13.15    scalap  sample  357   888.775 ± 7.243  ms/op

In context of the base and top-lines that we're comparing against:

Scenario	Time
HotScalacBenchmark for the same Scala revision	765ms
HotSbtBenchmark using compileRaw	888ms
HotSbtBenchmark without AnalysisReport collection	922ms
HotSbtBenchmark using default config and compile task from 0.13.15	1063ms

Discussion:

• The overhead of the terminal scraping between the JMH benchmark process and the child SBT process might be a factor here. Try a larger codebases to reduce that effect.
• The SBT compiler phases appear to levy only ~12% overhead, which seems quite reasonable.
• We need to find out what other factors are preventing the results above from coming in closer to the direct compiler usage.

[retronym]

The custom version of SBT that disables the AnalysisReport collection under antStyle = true is based on sbt/sbt@0.13...retronym:topic/rawCompile

[retronym]

Changing the compileRaw task to run compilation 10 times:

diff --git a/compilation/src/main/scala/scala/tools/nsc/HotSbtBenchmark.scala b/compilation/src/main/scala/scala/tools/nsc/HotSbtBenchmark.scala
index 5f2ecf8..5f0acbf 100644
--- a/compilation/src/main/scala/scala/tools/nsc/HotSbtBenchmark.scala
+++ b/compilation/src/main/scala/scala/tools/nsc/HotSbtBenchmark.scala
@@ -31,9 +31,6 @@ class HotSbtBenchmark {
   @Param(value = Array(""))
   var extraBuild: String = _
 
-  @Param(value= Array("compile"))
-  var compileCmd: String = _
-
   // This parameter is set by ScalacBenchmarkRunner / UploadingRunner based on the Scala version.
   // When running the benchmark directly the "latest" symlink is used.
   @Param(value = Array("latest"))
@@ -55,7 +52,8 @@ class HotSbtBenchmark {
       |def addCompileRaw = compileRaw := {
       |    val compiler = new sbt.compiler.RawCompiler(scalaInstance.value, sbt.ClasspathOptions.auto, streams.value.log)
       |    classDirectory.value.mkdirs()
-      |    compiler.apply(sources.value, classDirectory.value +: dependencyClasspath.value.map(_.data), classDirectory.value, scalacOptions.value)
+      |    for (i <- 0 until 10)
+      |      compiler.apply(sources.value, classDirectory.value +: dependencyClasspath.value.map(_.data), classDirectory.value, scalacOptions.value)
       |  }
       |
       |inConfig(Compile)(List(addCompileRaw))
@@ -104,7 +102,14 @@ class HotSbtBenchmark {
 
   @Benchmark
   def compile(): Unit = {
-    issue(s";cleanClasses;$compileCmd")
+    issue(s";cleanClasses;compile")
+    awaitPrompt()
+  }
+
+  @Benchmark()
+  @OperationsPerInvocation(10)
+  def compileRaw(): Unit = {
+    issue(s";cleanClasses;compileRaw")
     awaitPrompt()
   }

% sbt 'set scalaVersion in compilation := "2.12.3-bin-d1ec01a-SNAPSHOT"' 'compilation/jmh:run HotSbtBenchmark.compileRaw -psource=scalap -psbtVersion=0.13.15'
[warn] Executing in batch mode.

Gives time/operation of ~800ms, closer to the raw performance we expect.