ModSecurity-nginx support SecAuditLogFormat JSON ?

[truman-test]

ModSecurity-nginx support SecAuditLogFormat JSON ?

ModSecurity configure

./configure --with-yajl=../lloyd-yajl-66cb08c/build/yajl-2.1.0/

ModSecurity - v3.0.0+b58f713 for Linux

Mandatory dependencies

• libInjection ....v2.9.0-799-gb58f713
• SecLang tests ....b58f713

Optional dependencies

• GeoIP ....not found
• LibCURL ....not found
• YAJL ....found
  -lyajl, -DWITH_YAJL -I../lloyd-yajl-66cb08c/build/yajl-2.1.0//include
• LMDB ....not found
• LibXML2 ....found v2.9.1
  -lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2

Other Options

• Test Utilities ....enabled
• SecDebugLog ....enabled
• afl fuzzer ....disabled
• library examples ....enabled
• Building parser ....disabled

nginx.conf

modsecurity on;
modsecurity_rules_file /usr/local/nginx/conf/modsecurity.conf

modsecurity.conf

SecAuditLogFormat JSON

nginx start error

nginx: [emerg] "modsecurity_rules_file" directive Rules error. File: /usr/local/nginx/conf/modsecurity.conf. Line: 256. Column: 23. Invalid input: SecAuditLogFormat JSON in /usr/local/nginx/conf/nginx.conf:118

[johnmckinght]

same here i just following compiling and fail
2017/05/02 12:14:20 [emerg] 20990#20990: "modsecurity_rules_file" directive Rules error. File: /etc/modsecurity/modsecurity.conf. Line: 215. Column: 23. Invalid input: SecAuditLogFormat JSON in /etc/nginx/conf.d/default.conf:83
i think its still not supported yet

[zimmerle]

Hi @johnmckinght and @truman-test,

You don't need to use the SecAuditLogFormat any longer. In version 3 JSON is the default output.

[truman-test]

@zimmerle
thanks for your answer. but output isn't json format

I installed the following installation guide
https://github.com/SpiderLabs/ModSecurity/wiki/Compilation-recipes

What installation guide should I look for if Json is the default output format as you said?

[zimmerle]

Hi @truman-test,

Make sure that you have the yajl library installed in your server. In the configuration summary you should have a entry that saying if the yajl was found or not.

[truman-test]

Hi @zimmerle
I installed yajl library
Found yajl library when modsecurity configure the following

ModSecurity configure

./configure --with-yajl=../lloyd-yajl-66cb08c/build/yajl-2.1.0/

ModSecurity - v3.0.0+b58f713 for Linux

Mandatory dependencies

libInjection ....v2.9.0-799-gb58f713
SecLang tests ....b58f713
Optional dependencies

GeoIP ....not found
LibCURL ....not found
YAJL ....found
-lyajl, -DWITH_YAJL -I../lloyd-yajl-66cb08c/build/yajl-2.1.0//include
LMDB ....not found
LibXML2 ....found v2.9.1
-lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2
Other Options

Test Utilities ....enabled
SecDebugLog ....enabled
afl fuzzer ....disabled
library examples ....enabled
Building parser ....disabled

[zimmerle]

@truman-test great!! :)

[truman-test]

@zimmerle
but not json format :'(

[Pjack]

Hi
I find SecAuditLogFormat is no longer valid in v3.
It's not a problem to me since the default output format is json format.

However, in the audit log, it seems not json format. Let me paste one example to you.
Is it the content we should expected?
(I confirm YAJL is enabled in libmodsecurity)

---x44Gha1w---H--
ModSecurity: Warning. Matched "Operator PmFromFile' with parameter unix-shell.data' against variable ARGS:cmd' (Value: /bin/sh' ) [file "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "404"] [id "932160"] [rev "1"] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/sh found within ARGS:cmd: /bin/sh"] [severity "2"] [ver "OWASP_CRS/3.0.0"] [maturity "1"] [accuracy "8"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [ref "o1,6v10,7t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]

[truman-test]

@zimmerle
The same issue as Pjack

cc @Pjack

[lukeber4]

Same as @Pjack. @zimmerle, where should we expect to find the JSON-formatted log?

Thanks!

[Pjack]

Finally, I find that the JSON format is only supported in parallel mode, not in serial mode.
But the default behavior is serial mode.
That's why we cannot see the JSON format.
We have to change the SecAuditLogType to Parallel in the configuration file.
After that, you can find the JSON format under /var/log/modsec.
Please remember to create the folder by yourself and change the owner to www-data:www-data

Source code:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/parser/seclang-scanner.ll#L301
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/audit_log/writer/parallel.cc#L106

Wiki:
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#secauditlogtype

Here is my settings in the configuration file.

SecDebugLog /var/log/modsec_debug.log
SecDebugLogLevel 9
SecAuditEngine RelevantOnly
SecAuditLogType Parallel
SecAuditLogParts ABCFHZ
SecAuditLog2 /var/log/modsec_audit2.log
SecAuditLogStorageDir /var/log/modsec
SecAuditLogRelevantStatus "^(?:5|4(?!04))"

Is there any reason that JSON format is not supported in serial mode?
Thanks!

[johnmckinght]

@Pjack you would likely find your setup fill with error like Audit log: Failed to lock global mutex: Permission denied me too have the same issue like you and i decide to just use apache/2.4.10 with proxypass module so far i can live with that

[Pjack]

@johnmckinght
It works very well at my side, just like I said, did you change the mode of directory /var/log/modsec or /var/log/modsec_audit2.log ?

You may find the root cause by this command.
$ strace service nginx start

Good luck to you.

[truman-test]

@zimmerle @Pjack (cc @johnmckinght )

Thanks to your help, i can log the json format.
But, I want to json log in one file and record response data.
Logs are accumulated by folder as below and response data can not be recorded.
Parallel configuration is probably the same as setting up ConcAuditLog with the same settings as Concurrent.

configuration file:
SecAuditLogType Parallel
SecAuditLog /var/log/modsec_audit.log
SecAuditLogStorageDir /var/log/

cat /var/log/modsec_audit.log

nginx-modsecurity.test.com 172.26.137.64 - - [20/Jun/2017:22:03:13 +0900] "GET /tt_test.html HTTP/1.1" 304 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36" 149796379390.877722 - /var/log//20170620/20170620-2203/20170620-220313-149796379390.877722 0 1571.000000 md5:199ba06aae67f381497c7664eafe407b

cat /var/log/20170620/20170620-2203/20170620-220313-149796379390.877722

{
"transaction": {
"client_ip": "172.26.137.64",
"time_stamp": "Tue Jun 20 22:03:13 2017",
"server_id": "e9d3b50535bbfa3381fcd195344437344ea38e46",
"client_port": 56638,
"host_ip": "172.26.137.64",
"host_port": 80,
"id": "149796379390.877722",
"request": {
"method": "GET",
"http_version": 1.1,
"uri": "/tt_test.html",
"body": "",
"headers": {
"Host": "nginx-modsecurity.test.com",
"Connection": "keep-alive",
"Cache-Control": "max-age=0",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4",
"If-None-Match": ""59491b17-548"",
"If-Modified-Since": "Tue, 20 Jun 2017 12:54:47 GMT"
}
},
"response": {
"http_code": 304,
"headers": {
"Server": "nginx/1.11.13",
"Date": "Tue, 20 Jun 2017 13:03:13 GMT",
"Last-Modified": "Tue, 20 Jun 2017 12:54:47 GMT",
"Connection": "keep-alive",
"ETag": ""59491b17-548""
}
}
}
}