Skip to content

Commit 06f59c8

Browse files
Feature/157 improve playbook execution speed by embedding the json schemas (#199)
1 parent c1217ec commit 06f59c8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+3356
-18
lines changed

docs/content/en/docs/core-components/log.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,9 +72,6 @@ To change logging for your SOARCA instance you can use the following environment
7272
|LOG_MODE |development \| production |If production is chosen the `LOG_GLOBAL_LEVEL` is used for all modules defaults to `production`
7373
|LOG_FILE_PATH |filepath |Path to the logfile you want to use for all logging. Defaults to `""` (empty string)
7474
|LOG_FORMAT |text \| json |The logging can be in plain text format or in JSON format. Defaults to `json`
75-
|MQTT_BROKER | dns name or ip | The broker address for SOARCA to connect to, for communication with fins default is `localhost`
76-
|MQTT_PORT | port | The broker address for SOARCA to connect to, for communication with fins default is `1883`
77-
|ENABLE_FINS| true \| false | Enable fins in SOARCA defaults to `false`
7875

7976

8077

docs/content/en/docs/getting-started/_index.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ LOG_FORMAT: "json"
9898
ENABLE_FINS: false
9999
MQTT_BROKER: "localhost"
100100
MQTT_PORT: 1883
101+
VALIDATION_SCHEMA_URL: ""
101102
{{< /tab >}}
102103
{{< /tabpane >}}
103104

@@ -113,3 +114,23 @@ make build
113114
cp .env.example .env
114115
./build/soarca
115116
```
117+
118+
### Configuring SOARCA
119+
120+
|variable |content |description
121+
|---|---|---|
122+
|PORT |port |Set the exposed port of SOARCA the default is `8080`
123+
|DATABASE |true \| false | Set if you want to run with external database default is `false`
124+
|MONGODB_URI |uri |Set the Mongo DB uri default is `mongodb://localhost:27017`
125+
|DATABASE_NAME |name |Set the Mongo DB database name when using docker default is `soarca`
126+
|DB_USERNAME |user |Set the Mongo DB database user when using docker default is `root`
127+
|DB_PASSWORD |password |Set the Mongo DB database users password when using docker default is `rootpassword`. IT IS RECOMMENDED TO CHANGE THIS IN PRODUCTION!
128+
|MAX_REPORTERS |number |Set the maximum number of downstream reporters default is `5`
129+
|LOG_GLOBAL_LEVEL |[Log levels] |One of the specified log levels. Defaults to `info`
130+
|LOG_MODE |development \| production |If production is chosen the `LOG_GLOBAL_LEVEL` is used for all modules defaults to `production`
131+
|LOG_FILE_PATH |filepath |Path to the logfile you want to use for all logging. Defaults to `""` (empty string)
132+
|LOG_FORMAT |text \| json |The logging can be in plain text format or in JSON format. Defaults to `json`
133+
|MQTT_BROKER | dns name or ip | The broker address for SOARCA to connect to, for communication with fins default is `localhost`
134+
|MQTT_PORT | port | The broker address for SOARCA to connect to, for communication with fins default is `1883`
135+
|ENABLE_FINS| true \| false | Enable fins in SOARCA defaults to `false`
136+
|VALIDATION_SCHEMA_URL|url| Set a custom validation schema to be used to validate playbooks defaul is `""` to use internal. NOTE: changing this heavily impacts performance.

go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ require (
1111
github.com/google/uuid v1.3.1
1212
github.com/joho/godotenv v1.5.1
1313
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1
14+
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1
1415
github.com/sirupsen/logrus v1.9.3
1516
github.com/stretchr/testify v1.9.0
1617
github.com/swaggo/files v1.0.1

go.sum

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3
1919
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
2020
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
2121
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
22+
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
23+
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
2224
github.com/eclipse/paho.mqtt.golang v1.4.3 h1:2kwcUGn8seMUfWndX0hGbvH8r7crgcJguQNCyp70xik=
2325
github.com/eclipse/paho.mqtt.golang v1.4.3/go.mod h1:CSYvoAlsMkhYOXh/oKyxa8EcBci6dVkLCbo5tTC1RIE=
2426
github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
@@ -104,6 +106,8 @@ github.com/rogpeppe/go-internal v1.8.0 h1:FCbCCtXNOY3UtUuHUYaghJg4y7Fd14rXifAYUA
104106
github.com/rogpeppe/go-internal v1.8.0/go.mod h1:WmiCO8CzOY8rg0OYDC4/i/2WRWAB6poM+XZ2dLUbcbE=
105107
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1 h1:lZUw3E0/J3roVtGQ+SCrUrg3ON6NgVqpn3+iol9aGu4=
106108
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1/go.mod h1:uToXkOrWAZ6/Oc07xWQrPOhJotwFIyu2bBVN41fcDUY=
109+
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1 h1:PKK9DyHxif4LZo+uQSgXNqs0jj5+xZwwfKHgph2lxBw=
110+
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
107111
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
108112
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
109113
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=

models/validator/schema.go

Lines changed: 62 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,32 @@
11
package validator
22

33
import (
4+
"embed"
45
"encoding/json"
56
"errors"
7+
"io/fs"
68
"reflect"
79
"soarca/logger"
810
"soarca/models/cacao"
11+
"soarca/utils"
12+
"strings"
913

1014
"github.com/go-playground/validator/v10"
11-
jsonschema "github.com/santhosh-tekuri/jsonschema/v5"
1215
_ "github.com/santhosh-tekuri/jsonschema/v5/httploader"
16+
jsonschema "github.com/santhosh-tekuri/jsonschema/v6"
1317
)
1418

1519
type Empty struct{}
1620

1721
var component = reflect.TypeOf(Empty{}).PkgPath()
1822
var log *logger.Log
1923

20-
var oca_cacao_schemas string = "https://raw.githubusercontent.com/opencybersecurityalliance/cacao-roaster/main/lib/cacao-json-schemas/schemas/playbook.json"
24+
const (
25+
oca_cacao_schemas string = "./schemas/playbook.json"
26+
)
27+
28+
//go:embed schemas/*
29+
var schemas embed.FS
2130

2231
//var oasis_cacao_schemas string = "https://raw.githubusercontent.com/oasis-open/cacao-json-schemas/main/schemas/playbook.json"
2332
//var cyentific_cacao_schemas string = "https://raw.githubusercontent.com/cyentific-rni/cacao-json-schemas/cacao-v2.0-csd03/schemas/playbook.json"
@@ -45,6 +54,52 @@ func UnmarshalJson[BodyType any](b *[]byte) (any, error) {
4554
return body, nil
4655
}
4756

57+
func validateWithLocalSchema(playbookToValidate map[string]interface{}) error {
58+
59+
compiler := jsonschema.NewCompiler()
60+
61+
err := fs.WalkDir(schemas, ".", func(path string, d fs.DirEntry, err error) error {
62+
isFile := d.Type().IsRegular()
63+
64+
if isFile {
65+
content, _ := fs.ReadFile(schemas, path)
66+
data, err := jsonschema.UnmarshalJSON(strings.NewReader(string(content)))
67+
if err != nil {
68+
return err
69+
}
70+
if err := compiler.AddResource(path, data); err != nil {
71+
return err
72+
}
73+
}
74+
return nil
75+
})
76+
if err != nil {
77+
log.Error(err)
78+
return err
79+
}
80+
81+
sch, err := compiler.Compile(oca_cacao_schemas)
82+
if err != nil {
83+
return err
84+
}
85+
86+
err = sch.Validate(playbookToValidate)
87+
return err
88+
}
89+
90+
func validateWithRemoteSchema(data map[string]interface{}, url string) error {
91+
compiler := jsonschema.NewCompiler()
92+
93+
sch, err := compiler.Compile(url)
94+
if err != nil {
95+
return err
96+
}
97+
if err := sch.Validate(data); err != nil {
98+
return err
99+
}
100+
return nil
101+
}
102+
48103
func IsValidCacaoJson(data []byte) error {
49104

50105
var rawJson map[string]interface{}
@@ -54,25 +109,18 @@ func IsValidCacaoJson(data []byte) error {
54109

55110
version := rawJson["spec_version"]
56111

57-
compiler := jsonschema.NewCompiler()
58-
compiler.Draft = jsonschema.Draft7
59-
60-
var sch *jsonschema.Schema
61-
var err error
62112
switch version {
63113
case cacao.CACAO_VERSION_1:
64114
return errors.New("you submitted a cacao v1 playbook. at the moment, soarca only supports cacao v2 playbooks")
65115
case cacao.CACAO_VERSION_2:
66-
sch, err = compiler.Compile(oca_cacao_schemas)
67-
if err != nil {
68-
return err
116+
schemaUrl := utils.GetEnv("VALIDATION_SCHEMA_URL", "")
117+
if schemaUrl != "" {
118+
return validateWithRemoteSchema(rawJson, schemaUrl)
119+
} else {
120+
return validateWithLocalSchema(rawJson)
69121
}
70122
default:
71123
return errors.New("unsupported cacao version")
72124
}
73125

74-
if err := sch.Validate(rawJson); err != nil {
75-
return err
76-
}
77-
return nil
78126
}
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
{
2+
"$id": "agent-target",
3+
"$schema": "http://json-schema.org/draft-07/schema#",
4+
"title": "agent-target",
5+
"description": "In a CACAO playbook, agents are the entities that execute commands (see section 5) on or against targets. Agents are stored in a dictionary where the ID is the key and the value is an 'agent-target' object (see section 10.1). Targets are stored in a dictionary where the ID is the key and the value is an 'agent-target' object (see section 10.1). Common properties for agents and targets are defined in section 7.1. \n\nAgents can involve either manual or automated processing. For example, an individual may process a command manually, while a firewall may process a command automatically. An agent and target type vocabulary is defined in section 7.2, and each agent and target type is further defined in the rest of the sections. Types include security infrastructure such as firewalls, routers, and threat intelligence platforms, as well as specific network addressable agents like URLs and IPv4/IPv6/MAC addresses. \n\nAgents and targets can use and refer to variables just like other parts of the playbook. For any agent or target property value, the producer may define a variable substitution such that the actual property value is determined at runtime based on the variable assigned to the agent or target. In Example 7.1, an agent is referenced within a workflow step, but the agent's actual values are based on variables (e.g., name, email, phone, location) instead of being hard-coded by the agent itself. \n\nEach object (agent or target) contains base properties that are common across all objects. These properties are defined in the following table. The ID for each object is stored as the key in the agent_definitions dictionary or the target_definitions dictionary.",
6+
"type": "object",
7+
"properties": {
8+
"type": {
9+
"$ref": "#/$defs/agent-target-type-ov",
10+
"description": "The type of object being used. The value of this property SHOULD come from the 'agent-target-type-ov' vocabulary."
11+
},
12+
"name": {
13+
"type": "string",
14+
"description": "The name that represents this object that is meant to be displayed in a user interface or captured in a log message. This property MUST be populated."
15+
},
16+
"description": {
17+
"type": "string",
18+
"description": "More details, context, and possibly an explanation about this object. This property SHOULD be populated."
19+
},
20+
"location": {
21+
"$ref": "../data-types/civic-location.json",
22+
"description": "Physical address information for this object."
23+
},
24+
"agent_target_extensions": {
25+
"minProperties": 1,
26+
"type": "object",
27+
"patternProperties": {
28+
"^extension-definition--[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$": {
29+
"type": "object"
30+
}
31+
},
32+
"description": "This property declares the extensions that are in use on this action or target and contains any of the properties and values that are to be used by that extension. \n\nThe key for each entry in the dictionary MUST be an 'identifier' (see section 10.10 for more information on identifiers) that uniquely identifies the extension. The value for each key is a JSON object that contains the structure as defined in the extension definition's schema property. The actual step extension definition is located in the 'extension_definitions' property found at the Playbook level."
33+
}
34+
},
35+
"required": [
36+
"type",
37+
"name"
38+
],
39+
"$defs": {
40+
"agent-target-type-ov": {
41+
"anyOf": [
42+
{
43+
"type": "string"
44+
},
45+
{
46+
"type": "string",
47+
"enum": [
48+
"group",
49+
"individual",
50+
"location",
51+
"organization",
52+
"sector",
53+
"http-api",
54+
"linux",
55+
"net-address",
56+
"security-category",
57+
"ssh"
58+
]
59+
}
60+
]
61+
}
62+
}
63+
}
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"$id": "agent-target",
3+
"$schema": "http://json-schema.org/draft-07/schema#",
4+
"title": "group",
5+
"description": "This type defines a group object and is used for commands that need to be processed or executed by a group. This object inherits the common agent properties. In addition to the inherited properties, this section defines the following additional property that is valid for this type.",
6+
"type": "object",
7+
"allOf": [
8+
{
9+
"$ref": "agent-target.json"
10+
},
11+
{
12+
"properties": {
13+
"type": {
14+
"type": "string",
15+
"description": "The value of this property MUST be 'group'.",
16+
"enum": [
17+
"group"
18+
]
19+
},
20+
"contact": {
21+
"$ref": "../data-types/contact.json",
22+
"description": "Contact information for this agent."
23+
}
24+
}
25+
}
26+
]
27+
}

0 commit comments

Comments
 (0)