java.util.NoSuchElementException: None.get when using Compiler.newRun() during ContextBase.initialize()

[GavinRay97]

Heya, sorry for the troubles, but I've been spending a good chunk of time attempting to PR support for Scala 3 to Quarkus as a newbie to both the JVM and to Scala.

• [WIP] Untested impl for Scala 3 dynamic compile quarkusio/quarkus#17907

I think the feature is pretty much finished, but on the actual invocation of the compiler I've hit a roadblock and I'd be grateful for any advice or direction -- googling doesn't seem to turn up much. This is the error that I get:

java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:627)
	at scala.None$.get(Option.scala:626)
	at dotty.tools.dotc.core.Definitions.add$1(Definitions.scala:1776)
	at dotty.tools.dotc.core.Definitions.addSyntheticSymbolsComments(Definitions.scala:1779)
	at dotty.tools.dotc.core.Definitions.init(Definitions.scala:1772)
	at dotty.tools.dotc.core.Contexts$ContextBase.initialize(Contexts.scala:855)
	at dotty.tools.dotc.core.Contexts$Context.initialize(Contexts.scala:560)
	at dotty.tools.dotc.Run.rootContext(Run.scala:69)
	at dotty.tools.dotc.Run.<init>(Run.scala:89)
	at dotty.tools.dotc.Compiler.newRun(Compiler.scala:158)
	at org.acme.quarkus.scala.three.deployment.Scala3CompilationProvider.compile(Scala3CompilationProvider.scala:70)
	at io.quarkus.deployment.dev.QuarkusCompiler.compile(QuarkusCompiler.java:191)

And the code for it this:

• https://github.com/GavinRay97/quarkus-scala-three-extension/blob/master/quarkus-scala-three/deployment/src/main/scala/org/acme/quarkus/scala/three/deployment/Scala3CompilationProvider.scala#L32-L71
• (This is using scala3-library_3:3.0.0 and scala-library-2.13.5 via Maven)

class Scala3CompilationProvider extends CompilationProvider:
  override def handledExtensions: util.Set[String]  = Collections.singleton(".scala")
  override def handledSourcePaths: util.Set[String] = super.handledSourcePaths

  override def compile(files: util.Set[File], context: CompilationProvider.Context): Unit =
    val log = Scala3CompilationProvider.log

    val callback = CustomCompilerCallback()
    val classPath =
      context.getClasspath.stream
        .map(_.getAbsolutePath)
        .collect(Collectors.joining(File.pathSeparator))

    // https://github.com/lampepfl/dotty/blob/b7d2a122555a6aa44cc7590852a80f12512c535e/compiler/test/dotty/tools/DottyTest.scala
    // https://github.com/scalameta/mdoc/blob/b9437b7d3d0c9e04a9c8e86de96f10db0be5de91/mdoc/src/main/scala-3/mdoc/internal/markdown/MarkdownCompiler.scala
    val base = new ContextBase {}
    import base.settings._
    val ctx = base.initialCtx.fresh
    ctx.setSetting(ctx.settings.classpath, classPath)
    ctx.setSetting(ctx.settings.outputDir, new PlainDirectory(Directory(context.getOutputDirectory.getAbsolutePath)))
    base.initialize()(using ctx)

    val dottyFiles: List[AbstractFile] = files.stream().map(it =>
      new PlainFile(dotty.tools.io.Path(it.toPath))
    ).collect(Collectors.toList)

    log.info("dottyFiles")
    log.info(dottyFiles)

    log.info("dottyFiles.iterator().asScala.toList")
    log.info(dottyFiles.iterator().asScala.toList)

    val compiler = new Compiler
    val run      = compiler.newRun(using ctx)
    run.compile(dottyFiles.iterator().asScala.toList)

From the logs, I can see that it picks up on the 1 changed file and everything looks okay.
I'm not really sure what is going on here 🤔

2021-07-01 22:19:39,878 INFO dottyFiles
2021-07-01 22:19:39,878 INFO [C:\Users\rayga\Projects\tmp\quarkus-scala-three-container\quarkus-scala-three-app\src\main\scala\org\acme\GreetingResource.scala]
2021-07-01 22:19:39,878 INFO dottyFiles.iterator().asScala.toList
2021-07-01 22:19:39,880 INFO List(C:\Users\rayga\Projects\tmp\quarkus-scala-three-container\quarkus-scala-three-app\src\main\scala\org\acme\GreetingResource.scala)

For reference in case it's useful, the current code that handles compiling Scala 2 is:

• https://github.com/quarkusio/quarkus/blob/main/extensions/scala/deployment/src/main/java/io/quarkus/scala/deployment/ScalaCompilationProvider.java#L22-L36
• (I wrote this also in Java first, and then I thought maybe there was some problem because of this so I rewrote it in Scala. You can find the Java source in the same directory structure in my project)

    public void compile(Set<File> files, Context context) {
        Settings settings = new Settings();
        context.getClasspath().stream()
                .map(File::getAbsolutePath)
                .forEach(f -> settings.classpath().append(f));
        settings.outputDirs().add(context.getSourceDirectory().getAbsolutePath(),
                context.getOutputDirectory().getAbsolutePath());
        try (Global g = new Global(settings)) {
            Global.Run run = g.new Run();
            Set<String> fileSet = files.stream()
                    .map(File::getAbsolutePath)
                    .collect(Collectors.toSet());
            run.compile(JavaConverters.asScalaSet(fileSet).toList());
        }
    }

[som-snytt]

For reference, the line in question is a ctx without a docCtx at https://github.com/lampepfl/dotty/blob/3.0.0/compiler/src/dotty/tools/dotc/core/Definitions.scala#L1776

[smarter]

Yep, that .get should be removed and replaced by a foreach or pattern match, can you submit a PR for that @GavinRay97 ?

[GavinRay97]

@smarter Done =)

I did a codebase search for all usages of docCtx and checked how this was handled in other places.

There's one case where the scenario looked nearly identical, so I used the same pattern from there:
https://github.com/lampepfl/dotty/blob/79fae194b80e136a243e7c7f949268166d169e1e/compiler/src/dotty/tools/dotc/typer/Namer.scala#L449

PR submitted at #13013

---

Unrelated: I actually managed to get this working, and in a better way than originally written!

Using the scala3-interfaces for the Dotty compiler so that it's version-agnostic to the Scala 3 version that the user would like to pull in.

Now the only problem left is that Quarkus' JUnit5 implementation doesn't seem to detect the presence of tests, in either their official Scala 2 extension or this new one for Scala 3 I've written 😦

But I'm going to try to figure that out as well, and get full Scala 3 support pushed there =)

• No tests were found when trying to run JUnit tests with Scala 2 or WIP Scala 3 extension quarkusio/quarkus#18373

// Main.process() documentation for "dotty-interface" overload:

/** Entry point to the compiler that can be conveniently used with Java reflection.
 *
 *  This entry point can easily be used without depending on the `dotty` package,
 *  you only need to depend on `dotty-interfaces` and call this method using
 *  reflection. This allows you to write code that will work against multiple
 *  versions of dotty without recompilation.
 *
 *  The trade-off is that you can only pass a SimpleReporter to this method
 *  and not a normal Reporter which is more powerful.
 *
 *  Usage example: [[https://github.com/lampepfl/dotty/tree/master/compiler/test/dotty/tools/dotc/InterfaceEntryPointTest.scala]]
 */
class Scala3CompilationProvider extends CompilationProvider:
  override def handledExtensions: util.Set[String]  = Collections.singleton(".scala")
  override def handledSourcePaths: util.Set[String] = super.handledSourcePaths

  override def compile(files: util.Set[File], context: CompilationProvider.Context): Unit =
    val sources = files.asScala.map(it => it.getAbsolutePath.toString).toList
    val classpath = context.getClasspath.asScala.map(_.getAbsolutePath).mkString(File.pathSeparator)

    val args = sources ++ scala.collection.immutable.List(
      "-d", context.getOutputDirectory.getAbsolutePath.toString,
      "-classpath", classpath,
    )

    val mainClass = Class.forName("dotty.tools.dotc.Main")
    val process = mainClass.getMethod("process",
      classOf[Array[String]], classOf[SimpleReporter], classOf[CompilerCallback])

    val reporter = new CustomSimpleReporter
    val callback = new CustomCompilerCallback

    // Run the compiler by calling dotty.tools.dotc.Main.process
    process.invoke(null, args.toArray, reporter, callback)

quarkus-scala3-dev-reload.mp4