Issues using Ammonite 2.5 in SBT Project #1241

OOPMan · 2021-12-22T15:32:25Z

Hi, I'm having some issues trying to use Ammonite 2.5 in my project.

My SBT file can be viewed at https://github.com/OOPMan/collectioneer/blob/master/build.sbt

When I attempt to sbt update I get the following:

[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/E:/Development/Scala/collectioneer/"), "collectioneer"):
[error]    com.lihaoyi:sourcecode _3, _2.13
[error]    com.lihaoyi:fansi _3, _2.13
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/E:/Development/Scala/collectioneer/"), "core"):
[error]    org.scala-lang.modules:scala-collection-compat _3, _2.13
[error]    com.lihaoyi:sourcecode _3, _2.13
[error]    com.lihaoyi:fansi _3, _2.13
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/E:/Development/Scala/collectioneer/"), "repl"):
[error]    com.lihaoyi:sourcecode _3, _2.13
[error]    com.lihaoyi:fansi _3, _2.13
[error] stack trace is suppressed; run 'last core / update' for the full output
[error] stack trace is suppressed; run 'last repl / update' for the full output
[error] stack trace is suppressed; run 'last update' for the full output
[error] (core / update) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-collection-compat, com.lihaoyi:sourcecode, com.lihaoyi:fansi
[error] (repl / update) Conflicting cross-version suffixes in: com.lihaoyi:sourcecode, com.lihaoyi:fansi
[error] (update) Conflicting cross-version suffixes in: com.lihaoyi:sourcecode, com.lihaoyi:fansi
[error] Total time: 0 s, completed Dec 22, 2021, 10:31:51 AM
[IJ]```

I'm not quite sure what's happening here. Can anyone assist?

The text was updated successfully, but these errors were encountered:

pme123 · 2022-01-29T10:39:49Z

I have the same issue (same exception) - I could not figure out how to use Ammonite in a Scala 3 project.

The sbt dependencyTree shows this:

Strange that com.lihaoyi:ammonite-cross-3.1.1-util_2.13 depends on both.

OOPMan · 2022-01-29T13:48:51Z

@pme123 The frustrating thing is that I was using an earlier version with no issues.

re com-lihaoyi/Ammonite#1241

mpollmeier · 2022-08-19T09:38:56Z

tl;dr; I tried to work around the duplicate dependencies but haven't found a way forward yet.

Full version: I tried to exclude the _2.13 sibling of the duplicated dependencies:

excludeDependencies ++= Seq(
  "fansi",
  "os-lib",
  "geny",
  "upack",
  "ujson",
  "upickle",
  "upickle-core",
  "upickle-implicits",
  "sourcecode",
  "pprint"
).map(artifactName =>
  ExclusionRule("com.lihaoyi", s"${artifactName}_2.13")
) :+ ExclusionRule("org.scala-lang.modules", "scala-xml_2.13")

Some ammonite artifacts don't exist for scala 3 yet, e.g. ammonite-storage and ammonite-util. I guess because they rely on things like TypeTag which got dropped in Scala3.

Apart from that scalac is happy. However, at runtime there's a variety of linker issues, e.g.

scala> ammonite.runtime.Storage.toString
java.lang.NoSuchMethodError: 'void upickle.core.Types$Writer.$init$(upickle.core.Types$Writer)'
  at ammonite.runtime.Storage$$anon$27.<init>(Storage.scala:110)
  at ammonite.runtime.Storage$.<clinit>(Storage.scala:110)
  ... 66 elided

I also tried to selectively pick and mix _2.13 and _3 from the exclusionlist above, but just ended up with different compiler and/or linker issues.

mpollmeier · 2022-08-19T15:38:16Z

fwiw a workaround is to turn it around: rather than adding ammonite as a dependency to your build, you can start the regular amm repl and load up all your project's jars.

mpollmeier · 2022-09-28T14:45:57Z

fwiw tried something else, still no success though:

Selectively add dependencies in util: only add fansi for scala2 and only add fansi_3 for scala3: mpollmeier@97b5931

mill "amm[3.2.0].compile"
...
[error] amm/repl/src/main/scala/ammonite/repl/AmmoniteFrontEnd.scala:50:9: Unsupported Scala 3 inline macro method generate; found in trait sourcecode.EnclosingMacros.
[error]     val autocompleteFilter: Filter = Filter.action(SpecialKeys.Tab){
[error]         ^
[error] amm/repl/src/main/scala/ammonite/repl/AmmoniteFrontEnd.scala:104:6: could not find implicit value for parameter i: sourcecode.Enclosing
[error]     ){
[error]      ^
[error] amm/repl/src/main/scala/ammonite/repl/AmmoniteFrontEnd.scala:113:34: could not find implicit value for parameter i: sourcecode.Enclosing
[error]     val allFilters = Filter.merge(
[error]                                  ^
[error] three errors found
1 targets failed
amm.repl[3.2.0].compile Compilation failed

re com-lihaoyi/Ammonite#1241

Fixes #1241 Having a sane classpath using the Scala 2.13 compiler to compile a Scala 3 project is _hard_. This uses some tricks to compile using Scala 2.13 dependencies (since we need the Scala 2.13 macros) and then change the artifacts with Scala 3 artifacts so the Ammonite runtime can compile the macros using the Scala 3 compiler. Pull request: #1301

re com-lihaoyi/Ammonite#1241

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

@main

upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg

OOPMan mentioned this issue Feb 4, 2022

Dependency problem in ammonite-cross-3.1.1-util #1252

Closed

mpollmeier added a commit to joernio/joern that referenced this issue Aug 17, 2022

try to exclude fansi_2.13 when using scala3

5b28776

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Aug 17, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

e9d9871

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Oct 4, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

0c3d04f

re com-lihaoyi/Ammonite#1241

mpollmeier mentioned this issue Oct 10, 2022

add @main entrypoint support for Scala 3 #1298

Closed

lolgab mentioned this issue Oct 12, 2022

Fix issues with _3 and _2.13 dependencies #1301

Merged

mpollmeier added a commit to joernio/joern that referenced this issue Oct 13, 2022

try to exclude fansi_2.13 when using scala3

73e287d

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Oct 13, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

1862bbd

re com-lihaoyi/Ammonite#1241

lefou closed this as completed in #1301 Oct 16, 2022

lefou added this to the after 2.5.5 milestone Oct 16, 2022

mpollmeier added a commit to joernio/joern that referenced this issue Oct 26, 2022

try to exclude fansi_2.13 when using scala3

d1277e3

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Oct 26, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

2e302e4

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 1, 2022

try to exclude fansi_2.13 when using scala3

3903276

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 1, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

f12ee49

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 5, 2022

try to exclude fansi_2.13 when using scala3

43406ee

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 5, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

4d8d6de

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 7, 2022

try to exclude fansi_2.13 when using scala3

5bb07e0

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 7, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

4a44dcc

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 7, 2022

try to exclude fansi_2.13 when using scala3

6a82ada

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 7, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

a8b7165

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 9, 2022

try to exclude fansi_2.13 when using scala3

4e60334

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 9, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

0632ccb

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 10, 2022

try to exclude fansi_2.13 when using scala3

3100a1e

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 10, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

282bf3d

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 17, 2022

try to exclude fansi_2.13 when using scala3

d55926e

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 17, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

c4ea68c

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 22, 2022

try to exclude fansi_2.13 when using scala3

f449baf

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 22, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

68d15a7

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 23, 2022

try to exclude fansi_2.13 when using scala3

b501de6

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 23, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

e6789a6

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 28, 2022

try to exclude fansi_2.13 when using scala3

1c36673

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Nov 28, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

3ca5e4f

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Dec 1, 2022

try to exclude fansi_2.13 when using scala3

4c10ea1

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Dec 1, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

6c41594

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Dec 2, 2022

try to exclude fansi_2.13 when using scala3

f062cd3

re com-lihaoyi/Ammonite#1241

mpollmeier added a commit to joernio/joern that referenced this issue Dec 2, 2022

drop scala2 for now, use workaround for ammonite dependency mixup

16ebce0

re com-lihaoyi/Ammonite#1241

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Issues using Ammonite 2.5 in SBT Project #1241

Issues using Ammonite 2.5 in SBT Project #1241

OOPMan commented Dec 22, 2021

pme123 commented Jan 29, 2022

Uh oh!

OOPMan commented Jan 29, 2022

Uh oh!

mpollmeier commented Aug 19, 2022

Uh oh!

mpollmeier commented Aug 19, 2022

Uh oh!

mpollmeier commented Sep 28, 2022

Uh oh!

Uh oh!

Issues using Ammonite 2.5 in SBT Project #1241

Issues using Ammonite 2.5 in SBT Project #1241

Comments

OOPMan commented Dec 22, 2021

pme123 commented Jan 29, 2022

Uh oh!

OOPMan commented Jan 29, 2022

Uh oh!

mpollmeier commented Aug 19, 2022

Uh oh!

mpollmeier commented Aug 19, 2022

Uh oh!

mpollmeier commented Sep 28, 2022

Uh oh!