Skip to content

Commit 9eb99c1

Browse files
prabhuprabhu
authored
Make chen 2 the default (#68)
* Switch to java 21 (#59) Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Switch to cpg2 and overflowdb2 (#60) * Use cpg2 Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Use odb2 1.0.0 Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> --------- Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Use nop logger (#61) Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Fix/issue 62 (#63) * Fix importCode predef. Default to reachables slicing by default Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * native image fixes for java Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> --------- Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Update packages Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Automatically invoke cdxgen while importing code (#65) * Automatically invoke cdxgen while importing code Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Use dynamic source Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> --------- Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Feature/js cli source (#66) * Added cli source for js Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Added cli source for js Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Improves method full name construction for js Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Revert back changes to get tests working back Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Better way to resolve a method full name Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Fix tests Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> --------- Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Update java version Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> * Tags for crypto libs and methods (#67) Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com> --------- Signed-off-by: Prabhu Subramanian <prabhu@appthreat.com>
1 parent 0e9d1e5 commit 9eb99c1

File tree

49 files changed

+718
-608
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+718
-608
lines changed

‎.github/workflows/containers.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ jobs:
4141
- name: Use Node.js
4242
uses: actions/setup-node@v3
4343
with:
44-
node-version: '20.x'
44+
node-version: '21.x'
4545
- name: Delete `.rustup` directory
4646
run: rm -rf /home/runner/.rustup # to save disk space
4747
if: runner.os == 'Linux'

‎.github/workflows/master.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929
- name: Use Node.js
3030
uses: actions/setup-node@v3
3131
with:
32-
node-version: '20.x'
32+
node-version: '21.x'
3333
- name: Delete `.rustup` directory
3434
run: rm -rf /home/runner/.rustup # to save disk space
3535
if: runner.os == 'Linux'

‎.github/workflows/pr.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ jobs:
66
strategy:
77
matrix:
88
os: [ubuntu-latest, windows-latest, macos-latest]
9-
jvm: ['17', '19', '20', '21']
9+
jvm: ['21']
1010
steps:
1111
- uses: actions/checkout@v3
1212
with:
@@ -29,7 +29,7 @@ jobs:
2929
- name: Use Node.js
3030
uses: actions/setup-node@v3
3131
with:
32-
node-version: '20.x'
32+
node-version: '21.x'
3333
- name: Delete `.rustup` directory
3434
run: rm -rf /home/runner/.rustup # to save disk space
3535
if: runner.os == 'Linux'

‎.github/workflows/release.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
- name: Use Node.js
3434
uses: actions/setup-node@v3
3535
with:
36-
node-version: '20.x'
36+
node-version: '21.x'
3737
- name: Delete `.rustup` directory
3838
run: rm -rf /home/runner/.rustup # to save disk space
3939
if: runner.os == 'Linux'

‎.github/workflows/win_compat.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,4 @@ jobs:
2121
run: |
2222
python -m pip install poetry
2323
python -m poetry install
24-
python -m poetry run chen ${{ matrix.with-science }}
24+
python -m poetry run chen ${{ matrix.with-science }}

‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,5 @@ chen.zip
5858
project/metals.sbt
5959
conda-out/
6060
notebooks/.ipynb_checkpoints/
61+
*.slices.json
62+

‎README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -144,7 +144,7 @@ Refer to the documentation site to learn more about the commands.
144144
145145
## Languages supported
146146
147-
- C/C++ (Requires Java 17 or above)
147+
- C/C++ (Requires Java 21 or above)
148148
- H (C/C++ Header files alone)
149149
- Java (Requires compilation) - 8 to 17
150150
- Jar

‎build.sbt

+6-8
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
name := "chen"
22
ThisBuild / organization := "io.appthreat"
3-
ThisBuild / version := "1.1.6"
3+
ThisBuild / version := "2.0.6"
44
ThisBuild / scalaVersion := "3.3.1"
55

6-
val cpgVersion = "1.4.22"
6+
val cpgVersion = "1.0.0"
77

88
lazy val platform = Projects.platform
99
lazy val console = Projects.console
@@ -35,26 +35,24 @@ lazy val aggregatedProjects: Seq[ProjectReference] = Seq(
3535

3636
ThisBuild / libraryDependencies ++= Seq(
3737
"org.slf4j" % "slf4j-api" % "2.0.11",
38-
"org.apache.logging.log4j" % "log4j-slf4j2-impl" % "2.22.1" % Optional,
39-
"org.apache.logging.log4j" % "log4j-core" % "2.22.1" % Optional
40-
// `Optional` means "not transitive", but still included in "stage/lib"
38+
"org.slf4j" % "slf4j-nop" % "2.0.11" % Optional,
4139
)
4240

4341
ThisBuild / compile / javacOptions ++= Seq(
4442
"-g", // debug symbols
4543
"-Xlint",
46-
"--release=17"
44+
"--release=21"
4745
) ++ {
4846
// fail early if users with JDK8 try to run this
4947
val javaVersion = sys.props("java.specification.version").toFloat
50-
assert(javaVersion.toInt >= 17, s"this build requires JDK17+ - you're using $javaVersion")
48+
assert(javaVersion.toInt >= 21, s"this build requires JDK21+ - you're using $javaVersion")
5149
Nil
5250
}
5351

5452
ThisBuild / scalacOptions ++= Seq(
5553
"-deprecation", // Emit warning and location for usages of deprecated APIs.
5654
"--release",
57-
"17",
55+
"21"
5856
)
5957

6058

‎chenpy/cli.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ def fix_envs():
9393
)
9494
if not os.getenv("JAVA_HOME"):
9595
LOG.info(
96-
"Ensure Java >= 17 up to 20 is installed. Set the environment variable JAVA_HOME to point the correct "
96+
"Ensure Java >= 21 is installed. Set the environment variable JAVA_HOME to point the correct "
9797
"java directory."
9898
)
9999
LOG.info(

‎ci/Dockerfile

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ LABEL maintainer="appthreat" \
44
org.opencontainers.image.authors="Team AppThreat <cloud@appthreat.com>" \
55
org.opencontainers.image.source="https://github.com/appthreat/chen" \
66
org.opencontainers.image.url="https://github.com/appthreat/chen" \
7-
org.opencontainers.image.version="1.1.x" \
7+
org.opencontainers.image.version="2.0.x" \
88
org.opencontainers.image.vendor="appthreat" \
99
org.opencontainers.image.licenses="Apache-2.0" \
1010
org.opencontainers.image.title="chen" \

‎codemeta.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
"downloadUrl": "https://github.com/AppThreat/chen",
88
"issueTracker": "https://github.com/AppThreat/chen/issues",
99
"name": "chen",
10-
"version": "1.1.6",
10+
"version": "2.0.6",
1111
"description": "Code Hierarchy Exploration Net (chen) is an advanced exploration toolkit for your application source code and its dependency hierarchy.",
1212
"applicationCategory": "code-analysis",
1313
"keywords": [

‎console/build.sbt

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ dependsOn(
1717
)
1818

1919
libraryDependencies ++= Seq(
20-
"io.shiftleft" %% "codepropertygraph" % Versions.cpg,
20+
"io.appthreat" %% "cpg2" % Versions.cpg,
2121
"com.michaelpollmeier" %% "scala-repl-pp-server" % Versions.scalaReplPP,
2222
"com.github.scopt" %% "scopt" % ScoptVersion,
2323
"org.typelevel" %% "cats-effect" % Versions.cats,

‎console/src/main/scala/io/appthreat/console/BridgeBase.scala

+3-4
Original file line numberDiff line numberDiff line change
@@ -290,7 +290,7 @@ trait ScriptExecution:
290290
*/
291291
private def importCpgCode(config: Config): List[String] =
292292
config.cpgToLoad.map { cpgFile =>
293-
"importCpg(\"" + cpgFile + "\")"
293+
"importAtom(\"" + cpgFile + "\")"
294294
}.toList ++ config.forInputPath.map { name =>
295295
s"""
296296
|openForInputPath(\"$name\")
@@ -340,9 +340,8 @@ trait PluginHandling:
340340

341341
val storeCode = if config.store then "save"
342342
else ""
343-
val runDataflow = if productName == "ocular" then "run.dataflow"
344-
else "run.ossdataflow"
345-
val argsString = argsStringFromConfig(config)
343+
val runDataflow = "run.ossdataflow"
344+
val argsString = argsStringFromConfig(config)
346345

347346
s"""
348347
| if (${config.overwrite} || !workspace.projectExists("$src")) {

‎console/src/main/scala/io/appthreat/console/Console.scala

+3
Original file line numberDiff line numberDiff line change
@@ -522,6 +522,8 @@ class Console[T <: Project](
522522
.add(
523523
c.methodFullName + (if c.callee(
524524
NoResolve
525+
).nonEmpty && c.callee(
526+
NoResolve
525527
).head.nonEmpty && c.callee(
526528
NoResolve
527529
).head.isExternal
@@ -530,6 +532,7 @@ class Console[T <: Project](
530532
)
531533
addedMethods += c.methodFullName -> true
532534
)
535+
end if
533536
)
534537
rootTree.add(childTree)
535538
}

‎console/src/main/scala/io/appthreat/console/ConsoleConfig.scala

+4-4
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,12 @@ import scala.collection.mutable
1212
*/
1313
class InstallConfig(environment: Map[String, String] = sys.env):
1414

15-
/** determining the root path of the joern/ocular installation is rather complex unfortunately,
16-
* because we support a variety of use cases:
15+
/** determining the root path of the installation is rather complex unfortunately, because we
16+
* support a variety of use cases:
1717
* - running the installed distribution from the install dir
1818
* - running the installed distribution anywhere else on the system
19-
* - running a locally staged ocular/joern build (via `sbt stage` and then either `./joern`
20-
* or `cd platform/target/universal/stage; ./joern`)
19+
* - running a locally staged build (via `sbt stage` and then either `./chennai` or `cd
20+
* platform/target/universal/stage; ./chennai`)
2121
* - running a unit/integration test (note: the jars would be in the local cache, e.g. in
2222
* ~/.coursier/cache)
2323
*/

‎console/src/main/scala/io/appthreat/console/Run.scala

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@ package io.appthreat.console
33
import io.shiftleft.passes.CpgPass
44
import io.shiftleft.semanticcpg.language.HasStoreMethod
55
import io.shiftleft.semanticcpg.layers.{LayerCreator, LayerCreatorContext}
6-
import org.reflections8.Reflections
7-
import org.reflections8.util.{ClasspathHelper, ConfigurationBuilder}
6+
import org.reflections.Reflections
7+
import org.reflections.util.{ClasspathHelper, ConfigurationBuilder}
88

99
import scala.jdk.CollectionConverters.*
1010

‎console/src/main/scala/io/appthreat/console/cpgcreation/AtomGenerator.scala

+22-6
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.appthreat.console.cpgcreation
22

3+
import better.files.File
34
import io.appthreat.console.FrontendConfig
45
import io.shiftleft.codepropertygraph.Cpg
56

@@ -10,26 +11,41 @@ case class AtomGenerator(
1011
config: FrontendConfig,
1112
rootPath: Path,
1213
language: String,
13-
sliceMode: String = "usages",
14-
slicesFile: String = "usages.json"
14+
sliceMode: String = "reachables",
15+
slicesFile: String = "reachables.slices.json"
1516
) extends CpgGenerator:
16-
private lazy val command: String = "atom"
17+
private lazy val command: String = sys.env.getOrElse("ATOM_CMD", "atom")
18+
private lazy val cdxgenCommand: String = sys.env.getOrElse("CDXGEN_CMD", "cdxgen")
1719

1820
/** Generate an atom for the given input path. Returns the output path, or None, if no CPG was
1921
* generated.
2022
*/
2123
override def generate(inputPath: String, outputPath: String = "app.atom"): Try[String] =
24+
// If there is no bom.json file in the root directory, attempt to automatically invoke cdxgen
25+
val bomPath = File(inputPath) / "bom.json"
26+
if !bomPath.exists then
27+
val cdxLanguage = language.toLowerCase().replace("src", "")
28+
val arguments = Seq(
29+
"-t",
30+
cdxLanguage,
31+
"--deep",
32+
"-o",
33+
(File(inputPath) / "bom.json").pathAsString,
34+
inputPath
35+
)
36+
runShellCommand(cdxgenCommand, arguments)
2237
val arguments = Seq(
2338
sliceMode,
2439
"-s",
25-
slicesFile,
40+
(File(inputPath) / slicesFile).pathAsString,
2641
"--output",
27-
outputPath,
42+
(File(inputPath) / outputPath).pathAsString,
2843
"--language",
2944
language,
3045
inputPath
3146
) ++ config.cmdLineParams
32-
runShellCommand(command, arguments).map(_ => outputPath)
47+
runShellCommand(command, arguments).map(_ => (File(inputPath) / outputPath).pathAsString)
48+
end generate
3349

3450
override def isAvailable: Boolean = true
3551

‎console/src/main/scala/io/appthreat/console/cpgcreation/ImportCode.scala

+2-6
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,12 @@
11
package io.appthreat.console.cpgcreation
22

33
import better.files.File
4-
import io.appthreat.console.{Console, FrontendConfig, Reporting}
54
import io.appthreat.console.workspacehandling.Project
6-
import io.appthreat.console.{ConsoleException, FrontendConfig, Reporting}
5+
import io.appthreat.console.{Console, ConsoleException, FrontendConfig, Reporting}
76
import io.shiftleft.codepropertygraph.Cpg
87
import io.shiftleft.codepropertygraph.generated.Languages
9-
import overflowdb.traversal.help.Table
108
import me.shadaj.scalapy.py
11-
import me.shadaj.scalapy.py.SeqConverters
12-
import py.PyQuote
13-
import me.shadaj.scalapy.interpreter.CPythonInterpreter
9+
import overflowdb.traversal.help.Table
1410

1511
import java.nio.file.Path
1612
import scala.util.{Failure, Success, Try}

‎console/src/test/scala/io/appthreat/console/LanguageHelperTests.scala

+7-7
Original file line numberDiff line numberDiff line change
@@ -26,47 +26,47 @@ class LanguageHelperTests extends AnyWordSpec with Matchers {
2626
}
2727

2828
"guess `JavaSrc` for a directory containing `.java`" in {
29-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
29+
File.usingTemporaryDirectory("chentests") { tmpDir =>
3030
val subdir = mkdir(tmpDir / "subdir")
3131
touch(subdir / "ServiceIdentifierComposerVisitorBasedStrategy.java")
3232
guessLanguage(tmpDir.pathAsString) shouldBe Some(Languages.JAVASRC)
3333
}
3434
}
3535

3636
"guess `Go` for a directory containing `Gopkg.lock`" in {
37-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
37+
File.usingTemporaryDirectory("chentests") { tmpDir =>
3838
val subdir = mkdir(tmpDir / "subdir")
3939
touch(subdir / "Gopkg.lock")
4040
guessLanguage(tmpDir.pathAsString) shouldBe Some(Languages.GOLANG)
4141
}
4242
}
4343

4444
"guess `Go` for a directory containing `Gopkg.toml`" in {
45-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
45+
File.usingTemporaryDirectory("chentests") { tmpDir =>
4646
val subdir = mkdir(tmpDir / "subdir")
4747
touch(subdir / "Gopkg.toml")
4848
guessLanguage(tmpDir.pathAsString) shouldBe Some(Languages.GOLANG)
4949
}
5050
}
5151

5252
"guess `Javascript` for a directory containing `package.json`" in {
53-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
53+
File.usingTemporaryDirectory("chentests") { tmpDir =>
5454
val subdir = mkdir(tmpDir / "subdir")
5555
touch(subdir / "package.json")
5656
guessLanguage(tmpDir.pathAsString) shouldBe Some(Languages.JSSRC)
5757
}
5858
}
5959

6060
"guess `C` for a directory containing .ll (LLVM) file" in {
61-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
61+
File.usingTemporaryDirectory("chentests") { tmpDir =>
6262
val subdir = mkdir(tmpDir / "subdir")
6363
touch(subdir / "foobar.ll")
6464
guessLanguage(tmpDir.pathAsString) shouldBe Some(Languages.LLVM)
6565
}
6666
}
6767

6868
"guess the language with the largest number of files" in {
69-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
69+
File.usingTemporaryDirectory("chentests") { tmpDir =>
7070
val subdir = mkdir(tmpDir / "subdir")
7171
touch(subdir / "source.c")
7272
touch(subdir / "source.java")
@@ -79,7 +79,7 @@ class LanguageHelperTests extends AnyWordSpec with Matchers {
7979
}
8080

8181
"not find anything for an empty directory" in {
82-
File.usingTemporaryDirectory("oculartests") { tmpDir =>
82+
File.usingTemporaryDirectory("chentests") { tmpDir =>
8383
guessLanguage(tmpDir.pathAsString) shouldBe None
8484
}
8585
}

‎dataflowengineoss/src/main/scala/io/appthreat/dataflowengineoss/queryengine/Engine.scala

+1-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ class Engine(context: EngineContext):
2828

2929
private val logger: Logger = LoggerFactory.getLogger(this.getClass)
3030
private val executorService: ExecutorService =
31-
Executors.newWorkStealingPool(2)
31+
Executors.newVirtualThreadPerTaskExecutor()
3232
private val completionService =
3333
new ExecutorCompletionService[TaskSummary](executorService)
3434

‎macros/build.sbt

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name := "macros"
33
dependsOn(Projects.semanticcpg % Test)
44

55
libraryDependencies ++= Seq(
6-
"io.shiftleft" %% "codepropertygraph" % Versions.cpg,
6+
"io.appthreat" %% "cpg2" % Versions.cpg,
77
"org.scalatest" %% "scalatest" % Versions.scalatest % Test
88
)
99

‎macros/src/main/scala/io/appthreat/console/QueryDatabase.scala

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
package io.appthreat.console
22

3-
import org.reflections8.Reflections
4-
import org.reflections8.util.{ClasspathHelper, ConfigurationBuilder}
3+
import org.reflections.Reflections
4+
import org.reflections.util.{ClasspathHelper, ConfigurationBuilder}
55

66
import java.lang.reflect.{Method, Parameter}
77
import scala.annotation.unused
@@ -44,7 +44,7 @@ class QueryDatabase(
4444
// the namespace currently looks like `io.appthreat.scanners.c.CopyLoops`
4545
val namespaceParts = bundleNamespace.split('.')
4646
val language =
47-
if bundleNamespace.startsWith("io.appthreat.ocular.scanners") then
47+
if bundleNamespace.startsWith("io.appthreat.chen.scanners") then
4848
namespaceParts(4)
4949
else if namespaceParts.length > 3 then
5050
namespaceParts(3)

‎meta.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{% set version = "1.1.6" %}
1+
{% set version = "2.0.6" %}
22

33
package:
44
name: chen

0 commit comments

Comments
 (0)