Skip to content

Commit 6bb78ee

Browse files
committed
squashed commit of michael/scala3 - rebased
upgrade ammonite drop better files dep - no scala 3 port available yet... try to exclude fansi_2.13 when using scala3 re com-lihaoyi/Ammonite#1241 Revert "try to exclude fansi_2.13 when using scala3" This reverts commit 018515ac949f45d330ae3833379eed348cd7e6a7. drop scala2 for now, use workaround for ammonite dependency mixup re com-lihaoyi/Ammonite#1241 console/compile and console/Test/compile are green now TypeTags were dropped in scala3, replaced with ??? for now, scripts won't work WIP take out ammonite from console take out some subprojects that we don't need for prototype better-files_2.13 comes in via cpg, that's ok querydb/kotlin2cpg still have some issues - that's ok for now WIP - console2 with scala-repl semi-working, minus some classpath issues wip - `io` namespace is shadowed by `scala.io` poor man's predef multiple statements multiline predef works too old school syntax bring back console dependencies fmt one more import to test things back to `io` package - use `_root_` as a workaround trying to extend standard dotty ReplDriver - it's too restricted though upgrade to scala 3.2.1-RC1 - temporarily disable scaladoc bring our predef back in extend ReplDriver - need yet another private -> protected change catch C-c, handle C-d bring in code from `console` - compiles, but not wired up yet use scala3-style main take out pysrc2cpg for now - relies on some compiler sources.. joern compiles now invoke dotty repl fix syntax wire up basics `App` doesn't really work well in scala3 - use `main` instead remove odb.traversal implicits which break autocompletion `joern> ` prompt refactor rip shutdownHooks console2 -> console RIP storagebackend backport to scala 3.1.3, in order to ease transition for the entire codebase drop unused slproduct usage allow to pass in onExit shutdown hooks onExit hook: save all projects debug shutdown hook fix shutdown hook less noise predef: runQuietly - not cleaned up yet cleanup simplify `slproduct` usage and use in scopt helptext before: ``` ./joern --help Usage: (joern|ocular) [options] [<cpg.bin>] ``` now: ``` ./joern --help Usage: joern [options] [<cpg.bin>] ``` `./joern --verbose` flag fix all warnings in generated predef --nocolors: use standard scala compiler arg WIP WIP WIP: use reflection does the trick! start cleanup more cleanup WIP try to handle productName fix - new elementname works for all cases refactor: drop `object pprinter` - integrate into PPrinter cleanup/refactor docs first working version with `-XreplMaxPrintElements` - todo cleanup WIP doc depend on scala3 base version, cleanup wip experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good... Revert "experimenting with using existing XreplDisableDisplay rather than `runQuietly` - no good..." This reverts commit b756206. trying with replDriver - doesn't detect @main for obvious reasons WIP - scriptingdriver works - but need to add our predef etc. WIP - scripting with multiple input files works! wip - doesn't quite work yet - order is incorrect predef support for scripting: extend ScriptingDriver imports are still problematic... part of a different context... other attempt: compile predef before script - doesn't work ``` joern(michael/scala3-repl ✗) ./joern --script joernscript2.sc executing /home/mp/Projects/shiftleft/joern/joernscript2.sc with params=Map() exception occurred while compiling /home/mp/Projects/shiftleft/joern/joernscript2.sc Exception in thread "main" java.lang.AssertionError: assertion failed: phase parser has already been used once; cannot be reused at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:383) at dotty.tools.dotc.core.Phases$Phase.init(Phases.scala:397) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases(Phases.scala:166) at dotty.tools.dotc.core.Phases$PhasesBase.usePhases$(Phases.scala:35) at dotty.tools.dotc.core.Contexts$ContextBase.usePhases(Contexts.scala:845) at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:214) at dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:253) at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68) at dotty.tools.dotc.Run.compileUnits(Run.scala:253) at dotty.tools.dotc.Run.compileSources(Run.scala:186) at dotty.tools.dotc.Run.compile(Run.scala:170) at io.joern.console.ScriptingDriver.doCompile(ScriptingDriver.scala:44) at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22) at io.joern.console.ScriptExecution.runScript(BridgeBase.scala:286) at io.joern.console.ScriptExecution.runScript$(BridgeBase.scala:208) at io.joern.joerncli.console.AmmoniteBridge$.runScript(AmmoniteBridge.scala:7) at io.joern.console.BridgeBase.runAmmonite(BridgeBase.scala:175) at io.joern.console.BridgeBase.runAmmonite$(BridgeBase.scala:41) at io.joern.joerncli.console.AmmoniteBridge$.runAmmonite(AmmoniteBridge.scala:7) at io.joern.joerncli.console.AmmoniteBridge$.main(AmmoniteBridge.scala:12) at io.joern.joerncli.console.AmmoniteBridge.main(AmmoniteBridge.scala) ``` next failed attempt: using replDriver doesn't work because I can't make it detect the @main first version that sorta works cleanup cleanup refactor: deduplicate change to use our own predef - test with script cleanup script parameters are no longer key/value pairs unfortunately ScriptingDriver only supports simple Seq[String] unfortunately. Bummer, bit not a showstopper... WIP initial coursier/dependency functionality, test partly working --dependencies for user-defined additional deps ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 versionsort.VersionHelper.compare("1.0", "0.9") gremlin.scala.GremlinScala ``` scripted mode works as well: ``` ./joern --script ~/joernscript2.sc --dependency com.michaelpollmeier:versionsort:1.0.7,com.michaelpollmeier:gremlin-scala_2.13:3.5.1.5 ``` fiddly twiddly looks like got it working... WIP WIP - no workie idea: save state, create a new repldriver - no workie... ``` Exception in thread "main" java.util.NoSuchElementException: key not found: 1 at scala.collection.MapOps.default(Map.scala:274) at scala.collection.MapOps.default$(Map.scala:273) at scala.collection.AbstractMap.default(Map.scala:405) at scala.collection.mutable.HashMap.apply(HashMap.scala:425) at dotty.tools.repl.ReplCompiler$$anon$1.importPreviousRun$1(ReplCompiler.scala:49) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$1(ReplCompiler.scala:65) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext$$anonfun$adapted$1(ReplCompiler.scala:65) at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646) at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642) at scala.collection.AbstractIterable.foldLeft(Iterable.scala:926) at dotty.tools.repl.ReplCompiler$$anon$1.rootContext(ReplCompiler.scala:65) at dotty.tools.dotc.Run.<init>(Run.scala:370) ``` debug java classloader - no results - need dotty classloader cleanup WIP WIP WIP import works now! but renderer needs update as well... now also works for rendering... however previous state is lost drilling in - not there yet WIP wip WIP nothing works, really back to something that works, but still loses history WIP WIP WIP update context classpath setting - no change either last try with ReplDriver wip back to workaround: dynamic `using` works, but resets the session cleanup cleanup test starting repl again cleanup WIP back to trying to find a proper solution cleanup workaround works again, but still loses history cleanup cleanup wip ensure state.ctx and ReplDriver.rootCtx are identical - still doesn't find old state that's because the old state is in the context somewhere... setSource - no difference; lot's of debugging... reusing `initCtx`: less code, yet no difference... WIP latest get back to working state, incl. `using` switch bring joern predef back in add `-deprecation` compiler flag for better warning messages more context in error msg save workspace on exit automcatically improve user info on exit/save --dependency supports multiple --dependency: better user feedback disable `using` for now, because it resets the repl state anyway also: cleanup experiments reenable and update pprinter add mainargs dependency resolve rebase error fix scala3 compilation issues --script --verbose temporarily disable predef WIP sample scripts if given script file doesn't exist: don't print the entire stacktrace wrap code for mainargs - works for simple scripts and @main without args scripting: use mainargs for @main support with args... this works now: ``` ./joern --script test-main-withargs.sc --params name=Michael ``` passing multiple args bring in predef code - disable compiler warnings rename main method in generated code to avoid name collisions works with regular test from docs now add test with additional deps currently only works if passing deps when running joern: ``` ./joern --dependency com.michaelpollmeier:versionsort:1.0.7 --script test-dependencies.sc ``` dependencies in scripts: introduce `using` syntax, preprocess script this works now: ``` ./joern --script test-dependencies.sc ``` tests for repl and script mode: allow to add additional predef code from files WIP joern server: simple initial version joern server: simplistic version working, almost WIP - among others, allow to exit server WIP: almost works, currently hanging on stderr exhaustion server: works :tada: cleanup repl prints everything to one output stream only... no need to throw an exception extract ReplDriver to separate file --verbose: print all incoming and outgoing stuff fix shutdown ensure cpg queries work as well: need predef don't really have preparsing - leave out for now drop ScriptManager|AmmoniteExecutor etc - outdated WIP use scala-replpp (extracted most of the console code there) fix compiler errors WIP: use scala-repl-pp: extracted code from this branch into that repo WIP use replpp.ScriptRunner cleanup WIP server - initial version fix generated predef code indentation to avoid warnings on startup fix predef and exit code bring back `import odb.traversal._` - autocompletion issues were due to ammonite... fix predef code for scripts single test.sc script works now, as of https://docs.joern.io/interpreter refactor, use repl-pp's builtin functionality for additional file import upgrade repl-pp - truncation problem is fixed drop old dependencies cleanup: drop unused code fix script with additional script with cpg dsl usage this works now: ``` ./joern --script test-cpg-callotherscript.sc --import test-additionalfuncs.sc --params cpgFile=workspace/src/cpg.bin,outFile=a.txt ``` server: always verbose drop better files DSL drop unused classes (got extracted into repl-pp) fix `help` command - was shadowed by import cleanup: drop code that was extracted to repl-pp target 3.2.1 reenable py2cpg - needs full clean! bring querydb back in bring in kotlin2cpg fix warnings re 'match may not be exhaustive' cleanup bring in overflowdb.traversal implicits for scala3 implicit scope resolution was changed in scala3, hence extracting implicits for later reuse https://docs.scala-lang.org/scala3/reference/changed-features/implicit-resolution.html depends on ShiftLeftSecurity/overflowdb#336 latest cpg querydb/Test: change test archetype in preparation for scala3 context: Scala3 is stricter on deriving the type of an inherited method, specifically: ``` trait A object A1 extends A { def onlyInA1 = 42 } trait B { def a: A } class B1 extends B { override def a = A1 def fun = a.onlyInA1 } ``` fails with ``` [error] 31 | def fun = a.onlyInA1 [error] | ^^^^^^^^^^ [error] | value onlyInA1 is not a member of io.joern.scanners.android.Foo.A ``` One fix is to annotate `def a` with the specific type in each subclass, but in our use case I found it more natural to pass the QueryBundle to the parent and use a type parameter. drop unused and broken import fix warnings move tests over to scala-repl-pp extracted tests to repl-pp fix warnings, adapt signature kotline2cpg/pysrc2cpg reenabled bring back joern-cli -> console%Test dependency refactor bring scripted tests back ConsoleTests WIP debug dataflow tracker outcome: `Engine.deduplicate` performed a `++` in groupBy, which, in scala2 would trigger an automatic implicit conversion `Traversal -> Iterable` and therefor execute the traversal. In Scala3, that logic works differently, and this deduplicate didn't work any longer... 🤦 Revert "WIP debug dataflow tracker" This reverts commit 78e9e80. Keeping it in the history for later reference fix test JoernExport: `def main` instead of `App` `App` trait doesn't work in Scala3 `joerncli/test` now depends on c2cpg/stage AmmoniteBridge -> ReplBridge readme drop unused code refactor: InteractiveShell deserves it's own trait anything else is just confusing... latest cpg, which uses latest codegen, which automatically brings in odb.traversal.Implicits minify diff to master readme github actions runner configure scalafmt scalafmt fix rebase: slProduct wasn't defined any more whitespace readme latest odb (without chained implicits which break autocompletion) configure "-Xtarget:8" globally latest odb drop scala2 version of QueryMacros latest cpg
1 parent 1080e7e commit 6bb78ee

File tree

87 files changed

+526
-2125
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

87 files changed

+526
-2125
lines changed

.github/workflows/pr.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ jobs:
3030
distribution: 'temurin'
3131
java-version: '19'
3232
- name: Check formatting
33-
run: sbt ++2.13.8 scalafmtCheck test:scalafmtCheck
33+
run: sbt scalafmtCheck Test/scalafmtCheck
3434
- run: echo "Previous step failed because code is not formatted. Run 'sbt scalafmt Test/scalafmt'"
3535
if: ${{ failure() }}
3636
test-scripts:

.scalafmt.conf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
version = 3.7.3
2-
runner.dialect = scala213
2+
runner.dialect = scala3
33
preset = IntelliJ
44
maxColumn = 120
55
align.preset = true

README.md

Lines changed: 51 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ Documentation: https://docs.joern.io/
2121

2222
Specification: https://cpg.joern.io
2323

24+
<!-- drop in a few months, e.g. march 2023 -->
25+
## Announcement: upgrading from Joern 1.1.x to 1.2.x: see notes [below](#12x-upgrade-to-scala-3)
26+
2427
## Requirements
2528

2629
- JDK 19 (other versions _might_ work, but have not been properly tested)
@@ -38,20 +41,16 @@ chmod +x ./joern-install.sh
3841
sudo ./joern-install.sh
3942
joern
4043
41-
Compiling (synthetic)/ammonite/predef/interpBridge.sc
42-
Compiling (synthetic)/ammonite/predef/replBridge.sc
43-
Compiling (synthetic)/ammonite/predef/DefaultPredef.sc
44-
Compiling /home/tmp/shiftleft/joern/(console)
45-
4644
██╗ ██████╗ ███████╗██████╗ ███╗ ██╗
4745
██║██╔═══██╗██╔════╝██╔══██╗████╗ ██║
4846
██║██║ ██║█████╗ ██████╔╝██╔██╗ ██║
4947
██ ██║██║ ██║██╔══╝ ██╔══██╗██║╚██╗██║
5048
╚█████╔╝╚██████╔╝███████╗██║ ██║██║ ╚████║
5149
╚════╝ ╚═════╝ ╚══════╝╚═╝ ╚═╝╚═╝ ╚═══╝
50+
Version: 1.2.0
51+
Type `help` to begin
5252
5353
joern>
54-
5554
```
5655

5756
If the installation script fails for any reason, try
@@ -102,3 +101,49 @@ are:
102101
For more instructions on how to run benchmarks individually head over to the `benchmarks` subproject. If you would
103102
like the benchmark results to be written to a file instead of printed to STDOUT, set the path to the environment
104103
variable `JOERN_BENCHMARK_RESULT_FILE`.
104+
105+
## Upgrade notes
106+
107+
### 1.2.x: Upgrade to Scala 3
108+
Joern is based on Scala. As of Joern 1.2.x we upgraded from Scala 2 to Scala 3.
109+
This is a major version upgrade and Scala 3 is essentially a new language with a new REPL implementation, so this may sound scary.
110+
111+
That being said, both the Scala as well as Joern maintainers have made an effort to minimize changes to the API, in order to ease the transition for users. Most importantly, the Joern workspace DSL (`importCode(...)` etc.) and the CPG Traversal DSL (e.g. `cpg.method.name("foo").l`) are unchanged. The latter is based on Scala collections API, which is actually identical (a shared library) between Scala 2 and Scala 3.
112+
113+
Depending on your level of integration with Joern you most likely won't notice anything. If you do, please check the list below, and if that doesn't help: open a [github issue](https://github.com/joernio/joern/issues/new) or hit us up on [discord](https://discord.gg/vv4MH284Hc).
114+
115+
Common issues when upgrading Scala 2 to Scala 3:
116+
117+
1. anonymous functions need an extra parenthesis around their parameter list:
118+
```scala
119+
Seq(1,2,3).map { i: Int => i + 1 }
120+
// error: parentheses are required around the parameter of a lambda
121+
122+
// option 1: add parentheses, as suggested by compiler:
123+
Seq(1,2,3).map { (i: Int) => i + 1 }
124+
125+
// option 2: drop type annotation (if possible):
126+
Seq(1,2,3).map { i => i + 1 }
127+
```
128+
129+
2. `main` entrypoint: `def main` instead of `extends App`
130+
See https://docs.scala-lang.org/scala3/book/methods-main-methods.html
131+
```scala
132+
object Main extends App {
133+
println("hello world")
134+
}
135+
136+
// depending on usage, may lead to NullPointerExceptions
137+
// context: Scala3 doesn't support the 'magic' DelayedInit trait
138+
139+
// rewrite to:
140+
object Main {
141+
def main(args: Array[String]) = {
142+
println("hello world")
143+
}
144+
}
145+
```
146+
147+
148+
149+

benchmarks/build.sbt

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,5 @@
11
name := "benchmarks"
22

3-
crossScalaVersions := Seq("2.13.8", "3.2.2")
4-
53
dependsOn(Projects.dataflowengineoss)
64
dependsOn(Projects.semanticcpg)
75
dependsOn(Projects.console)

build.sbt

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name := "joern"
22
ThisBuild / organization := "io.joern"
3-
ThisBuild / scalaVersion := "2.13.8"
3+
ThisBuild / scalaVersion := "3.2.1"
44

55
val cpgVersion = "1.3.598"
66

@@ -63,6 +63,10 @@ ThisBuild / scalacOptions ++= Seq(
6363
"11"
6464
)
6565

66+
// we want to consume this from a java8 build
67+
compile / javacOptions ++= Seq("--release", "8")
68+
scalacOptions += "-Xtarget:8"
69+
6670
lazy val createDistribution = taskKey[File]("Create a complete Joern distribution")
6771
createDistribution := {
6872
val distributionFile = file("target/joern-cli.zip")

c2cpg.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
joern-cli/frontends/c2cpg/c2cpg.sh

console/build.sbt

Lines changed: 12 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ enablePlugins(JavaAppPackaging)
55
val ScoptVersion = "4.0.1"
66
val CaskVersion = "0.8.3"
77
val CirceVersion = "0.14.5"
8-
val AmmoniteVersion = "2.5.8"
98
val ZeroturnaroundVersion = "1.15"
109

1110
dependsOn(
@@ -17,58 +16,17 @@ dependsOn(
1716
)
1817

1918
libraryDependencies ++= Seq(
20-
"io.shiftleft" %% "codepropertygraph" % Versions.cpg,
21-
"com.github.scopt" %% "scopt" % ScoptVersion,
22-
"org.typelevel" %% "cats-effect" % Versions.cats,
23-
"io.circe" %% "circe-generic" % CirceVersion,
24-
"io.circe" %% "circe-parser" % CirceVersion,
25-
"org.zeroturnaround" % "zt-zip" % ZeroturnaroundVersion,
26-
"com.lihaoyi" %% "ammonite" % AmmoniteVersion cross CrossVersion.full,
27-
"com.lihaoyi" %% "os-lib" % "0.8.1",
28-
"com.lihaoyi" %% "cask" % CaskVersion,
29-
"org.scalatest" %% "scalatest" % Versions.scalatest % Test
19+
"com.michaelpollmeier" %% "scala-repl-pp" % "0.0.7",
20+
"io.shiftleft" %% "codepropertygraph" % Versions.cpg,
21+
"com.github.scopt" %% "scopt" % ScoptVersion,
22+
"org.typelevel" %% "cats-effect" % Versions.cats,
23+
"io.circe" %% "circe-generic" % CirceVersion,
24+
"io.circe" %% "circe-parser" % CirceVersion,
25+
"org.zeroturnaround" % "zt-zip" % ZeroturnaroundVersion,
26+
"com.lihaoyi" %% "os-lib" % "0.8.1",
27+
"com.lihaoyi" %% "pprint" % "0.7.3",
28+
"com.lihaoyi" %% "cask" % CaskVersion,
29+
"org.scalatest" %% "scalatest" % Versions.scalatest % Test
3030
)
3131

32-
Test / compile := (Test / compile).dependsOn((Projects.c2cpg / stage)).value
33-
34-
scalacOptions ++= Seq(
35-
"-deprecation", // Emit warning and location for usages of deprecated APIs.
36-
"-encoding",
37-
"utf-8", // Specify character encoding used by source files.
38-
"-explaintypes", // Explain type errors in more detail.
39-
"-feature", // Emit warning and location for usages of features that should be imported explicitly.
40-
"-language:existentials", // Existential types (besides wildcard types) can be written and inferred
41-
"-language:experimental.macros", // Allow macro definition (besides implementation and application)
42-
"-language:higherKinds", // Allow higher-kinded types
43-
"-language:implicitConversions", // Allow definition of implicit functions called views
44-
"-unchecked", // Enable additional warnings where generated code depends on assumptions.
45-
"-Xcheckinit", // Wrap field accessors to throw an exception on uninitialized access.
46-
"-Xlint:adapted-args", // Warn if an argument list is modified to match the receiver.
47-
"-Xlint:constant", // Evaluation of a constant arithmetic expression results in an error.
48-
"-Xlint:delayedinit-select", // Selecting member of DelayedInit.
49-
"-Xlint:doc-detached", // A Scaladoc comment appears to be detached from its element.
50-
"-Xlint:inaccessible", // Warn about inaccessible types in method signatures.
51-
"-Xlint:infer-any", // Warn when a type argument is inferred to be `Any`.
52-
"-Xlint:missing-interpolator", // A string literal appears to be missing an interpolator id.
53-
"-Xlint:option-implicit", // Option.apply used implicit view.
54-
"-Xlint:package-object-classes", // Class or object defined in package object.
55-
"-Xlint:poly-implicit-overload", // Parameterized overloaded implicit methods are not visible as view bounds.
56-
"-Xlint:private-shadow", // A private field (or class parameter) shadows a superclass field.
57-
"-Xlint:stars-align", // Pattern sequence wildcard must align with sequence component.
58-
"-Xlint:type-parameter-shadow", // A local type parameter shadows a type already in scope.
59-
"-Ywarn-dead-code", // Warn when dead code is identified.
60-
"-Ywarn-extra-implicit", // Warn when more than one implicit parameter section is defined.
61-
"-Ywarn-numeric-widen", // Warn when numerics are widened.
62-
"-Ywarn-unused:implicits", // Warn if an implicit parameter is unused.
63-
"-Ywarn-unused:imports", // Warn if an import selector is not referenced.
64-
"-Ywarn-unused:locals", // Warn if a local definition is unused.
65-
"-Ywarn-unused:params", // Warn if a value parameter is unused.
66-
"-Ywarn-unused:patvars", // Warn if a variable bound in a pattern is unused.
67-
"-Ywarn-unused:privates", // Warn if a private member is unused.
68-
"-Yrangepos"
69-
)
70-
71-
// would love to reenable, but somehow StorageBackend.scala triggers a strange `[warn] method with a single empty parameter list overrides method without any parameter list` that doesn't make sense to me...
72-
scalacOptions -= "-Xfatal-warnings"
73-
74-
Test / fork := false
32+
Test/compile := (Test/compile).dependsOn((Projects.c2cpg/stage)).value

0 commit comments

Comments
 (0)