scala 3.6.3 -Yprofile-enabled is buggy

[lenguyenthanh]

Compiler version

3.6.3

Minimized code

Not possible, it only happen with lila (maybe other large projects too), steps to reproduce are:

• add "-Yprofile-enabled", "-Yprofile-trace:compiler.trace" to scalac options (for example: lenguyenthanh/lila@6f269e9)
• run sbt 'clean;compile',
• I did try increase max heap (26GB) but it didn't help

Output

It failed most of the times and the output are invalid according to https://ui.perfetto.dev.
Errors are not deterministic, I got one success run after dozen attempts. Here are some errors:

java.lang.IllegalStateException: Wrong context

[error] ## Exception when compiling 13 sources to
{redacted}/lila/modules/db/target/scala-3.6.3/classes[error]
java.lang.IllegalStateException: Wrong context:
ArrayContext(false)[error]
dotty.tools.dotc.profile.ChromeTrace.fld(ChromeTrace.scala:184)[error]
dotty.tools.dotc.profile.ChromeTrace.str(ChromeTrace.scala:136)[error]
dotty.tools.dotc.profile.ChromeTrace.traceCounterEvent(ChromeTrace.scala:93)[error]
dotty.tools.dotc.profile.RealProfiler.traceThreadSnapshotCounters(Profiler.scala:305)[error]
dotty.tools.dotc.profile.RealProfiler.afterUnit(Profiler.scala:298)[error]
dotty.tools.dotc.profile.Profiler.dotty$tools$dotc$profile$Profiler$$inline$afterUnit(Profiler.scala:104)[error]
dotty.tools.dotc.profile.Profiler.dotty$tools$dotc$profile$Profiler$$inline$afterUnit$(Profiler.scala:87)[error]
dotty.tools.dotc.profile.RealProfiler.dotty$tools$dotc$profile$Profiler$$inline$afterUnit(Profiler.scala:171)[error]
dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:374)[error]
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)[error]
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)[error]
scala.collection.immutable.List.foreach(List.scala:334)[error]
dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:373)[error]
dotty.tools.backend.jvm.GenBCode.runOn(GenBCode.scala:98)[error]
dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)[error]
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)[error]
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)[error]
scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)[error]
dotty.tools.dotc.Run.runPhases$1(Run.scala:336)[error]
dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:383)[error]
dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:395)[error]
dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)[error]
dotty.tools.dotc.Run.compileUnits(Run.scala:395)[error]
dotty.tools.dotc.Run.compileSources(Run.scala:282)[error]
dotty.tools.dotc.Run.compile(Run.scala:267)

BufferOverflowException

[error] java.nio.BufferOverflowException
[error] 	at java.base/java.nio.HeapCharBuffer.put(HeapCharBuffer.java:346)
[error] 	at dotty.tools.dotc.profile.FileUtils$AsyncBufferedWriter.write(FileUtils.scala:121)
[error] 	at java.base/java.io.Writer.write(Writer.java:278)
[error] 	at dotty.tools.dotc.profile.ChromeTrace.objEnd(ChromeTrace.scala:165)
[error] 	at dotty.tools.dotc.profile.ChromeTrace.traceCounterEvent(ChromeTrace.scala:100)
[error] 	at dotty.tools.dotc.profile.RealProfiler.traceThreadSnapshotCounters(Profiler.scala:304)
[error] 	at dotty.tools.dotc.profile.RealProfiler.afterUnit(Profiler.scala:298)
[error] 	at dotty.tools.dotc.profile.Profiler.dotty$tools$dotc$profile$Profiler$$inline$afterUnit(Profiler.scala:104)
[error] 	at dotty.tools.dotc.profile.Profiler.dotty$tools$dotc$profile$Profiler$$inline$afterUnit$(Profiler.scala:87)
[error] 	at dotty.tools.dotc.profile.RealProfiler.dotty$tools$dotc$profile$Profiler$$inline$afterUnit(Profiler.scala:171)
[error] 	at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:374)
[error] 	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
[error] 	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
[error] 	at scala.collection.immutable.List.foreach(List.scala:334)
[error] 	at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:373)
[error] 	at dotty.tools.backend.jvm.GenBCode.runOn(GenBCode.scala:98)
[error] 	at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)
[error] 	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
[error] 	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
[error] 	at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
[error] 	at dotty.tools.dotc.Run.runPhases$1(Run.scala:336)
[error] 	at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:383)
[error] 	at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:395)
[error] 	at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
[error] 	at dotty.tools.dotc.Run.compileUnits(Run.scala:395)
[error] 	at dotty.tools.dotc.Run.compileSources(Run.scala:282)
[error] 	at dotty.tools.dotc.Run.compile(Run.scala:267)
[error] 	at dotty.tools.dotc.Driver.doCompile(Driver.scala:37)
[error] 	at dotty.tools.xsbt.CompilerBridgeDriver.run(CompilerBridgeDriver.java:141)
[error] 	at dotty.tools.xsbt.CompilerBridge.run(CompilerBridge.java:22)
[error] 	at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:91)
[error] 	at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$7(MixedAnalyzingCompiler.scala:196)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] 	at sbt.internal.inc.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:252)
[error] 	at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4(MixedAnalyzingCompiler.scala:186)
[error] 	at sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4$adapted(MixedAnalyzingCompiler.scala:166)
[error] 	at sbt.internal.inc.JarUtils$.withPreviousJar(JarUtils.scala:241)
[error] 	at sbt.internal.inc.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:166)
[error] 	at sbt.internal.inc.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:214)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1(IncrementalCompilerImpl.scala:542)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1$adapted(IncrementalCompilerImpl.scala:542)
[error] 	at sbt.internal.inc.Incremental$.$anonfun$apply$3(Incremental.scala:182)
[error] 	at sbt.internal.inc.Incremental$.$anonfun$apply$3$adapted(Incremental.scala:180)
[error] 	at sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:458)
[error] 	at sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:117)
[error] 	at sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
[error] 	at sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
[error] 	at sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:263)
[error] 	at sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:413)
[error] 	at sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:500)
[error] 	at sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:400)
[error] 	at sbt.internal.inc.Incremental$.apply(Incremental.scala:208)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:542)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:496)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:433)
[error] 	at sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
[error] 	at sbt.Defaults$.compileIncrementalTaskImpl(Defaults.scala:2443)
[error] 	at sbt.Defaults$.$anonfun$compileIncrementalTask$2(Defaults.scala:2393)
[error] 	at sbt.internal.server.BspCompileTask$.$anonfun$compute$1(BspCompileTask.scala:41)
[error] 	at sbt.internal.io.Retry$.apply(Retry.scala:47)
[error] 	at sbt.internal.io.Retry$.apply(Retry.scala:29)
[error] 	at sbt.internal.io.Retry$.apply(Retry.scala:24)
[error] 	at sbt.internal.server.BspCompileTask$.compute(BspCompileTask.scala:41)
[error] 	at sbt.Defaults$.$anonfun$compileIncrementalTask$1(Defaults.scala:2391)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 	at java.base/java.lang.Thread.run(Thread.java:1583)
[error] (room / Compile / compileIncremental) java.nio.BufferOverflowException
[error] Total time: 50 s, completed 16 Feb 2025, 11:15:02 am

Compilation failed

[info] done compiling
[error] (clas / Compile / compileIncremental) Compilation failed
[error] (setup / Compile / compileIncremental) Compilation failed
[error] Total time: 54 s, completed 16 Feb 2025, 11:22:46 am

Expectation

Should compile fine and write valid trace output

[WojciechMazur]

One of the possible issues might be caused by parallel collection of traces, at least the IO specific one. The collection is performed once per compilation run. Multiple projects compiled in parallel (class and setup from the sbt logs) means we're having 2 compiler runs/threads writing to the same output file.
AFAIR we're creating an additional trace files (with natural numbers suffix to the original path) in case of additional runs triggered by compilation of macros / staged compilation, but at the level of the compiler settings we have no context wether the build tool compiles a single or multi module project using the same output file, so we cannot decouple it, unless the -Yprofile-trace value would only contain a prefix/base path where traces should be stored.

IllegalStateException is another problem not related directly to the IO. The ChromeTrace code itself was ported directly from Scala 2. We create 1 instance per RealProfiler instance, and this one is create once were initialising Context for the compier Run. My initial thought is that maybe the instance of the profiler is shared by multiple Contexts by inheritance (altough I'm not sure about this one)

[ornicar]

Just to confirm that lila compiles a lot of modules in parallel.

Here's the modules dependency graph:

These are all lila compilation units; external dependencies are not included.

We can see that in the last compilation stage, about 40 modules are compiled in parallel.