Programmable Classpath to support Pickle caching, Build Pipelining

[retronym]

Provide an extension point for plugins to implement classpath caching and build pipelining.

Part of scala/scala-dev#416

[jvican]

You can see my implementation of this in sbt/zinc, supported in 2.11, 2.12 and 2.13: https://github.com/scalacenter/zinc/blob/topic/pipelined-parallel-compilation-rebase/internal/compiler-bridge/src/main/scala_2.12-13-only/scala/tools/nsc/ZincPicklePath.scala and https://github.com/scalacenter/zinc/blob/topic/pipelined-parallel-compilation-rebase/internal/compiler-bridge/src/main/scala_2.12-13-only/xsbt/ZincSymbolLoaders.scala (note: I don't implement symbol invalidation)

The beauty of having this implemented in Zinc instead of the Scala compiler is that Zinc is compiler agnostic. For example, I can support pipelining in Dotty as soon as the bridge supports this capability.

I like the changes in this PR but I wonder if we need this abstraction. At a higher level, all we need to implement pipelining and classpath caching is a way to communicate to the compiler which classpath entries have changed, whether they are class files in a directory or jars. If we add such an interface to the compiler, Zinc could tell the compiler directly which things need to be invalidated, and the compiler will invalidate the symbol table accordingly.

I feel this is a better approach because what changes or not is already known by the tools using the compiler (build tools, incremental compiler, IDEs/editors).

[jvican]

For example, how do you feel about adding the following two compiler settings instead:

1. -hashpath: contains a classpath-formatted list of hashes, one per classpath entry. This interface would be a generic mechanism to invalidate caches of classpath entries (no need to add different strategies to detect changes in the compiler).
2. --invalidate-class-files: a list of class files that are present in the classpath and changed. With this information, the compiler can invalidate the symbol table entries associated with the symbols that created the class files.

[retronym]

Using https://github.com/guardian/frontend as a test case:

Traditional:

Pipeliining (via Outline Typing) + Parallel compilation of large sub-projects:

The latter has better utilization, but the throwaway work of outline typing has too high a cost in this case.

I'm about to implement the variation of build pipelining that grabs the pickles from a regular compilation run instead. (UPDATE: Done)

Pipelining (via extracting pickles from regular compilation)

[retronym]

I've moved the pattern matcher phase after pickler which should slightly improve pipelining efficiency.

Build traces for the three strategies above: https://www.dropbox.com/sh/42bzdwag94ywlpo/AACxjoumoDZh0Oolz8t0dymwa?dl=0

• Save locally.
• Open chrome://tracing in Chrome
• Load the file