How to use persistent collections in ModSecurty v3.0.x?

[LeeShan87]

Hi!

I'm trying to create some rules which use IP persistent collection.
First i tried with Nginx v1.14.0, ModSecurity-nginx master, ModSecurity v3.0.0 without lua or lmdb.

The collection initialized and I could make a counter. Like count every page load on the given uri.
But expirevar is not working. Which is very bad, if I try something like this on a production server.

After this I tried ModSecurity v3.0.2, with the same setup and configurations.
But in this case the rules previously worked are not working at all.

Operator like gt eq not working with that counter. I can't even get it value logged out.

I saw in the commits there were many changes related to collection naming since v3.0.0, but haven't found any example how should I use after these changes.

So my question is can you provide some example how to use IP collection in v3.0.2?
How can we change backend from in memory collection to other, like lmdb( or later to redist)?

Some example rules:

SecAction "phase:1,\
  log,\
  msg:'Collection intited for %{REMOTE_ADDR}',\
  logdata:'Collection intited for %{REMOTE_ADDR}',\
  pass,initcol:ip=%{REMOTE_ADDR},\
  setvar:ip.counter=0,\
  expirevar:ip.counter=30"

SecRule &IP:counter "eq 0" "id:4000,\
msg:'init filter',\
  logdata:'initfilter',\
setvar:ip.counter=0,\
expirevar:ip.counter=30"


SecRule ARGS "attack" "id:401131,\
phase:2,\
setvar:ip.counter=+1,\
expirevar:ip.counter=15,\
  log,\
  msg:'Counter %{IP.counter} remote %{REMOTE_ADDR}',\
  logdata:'Counter %{IP.counter} remote %{REMOTE_ADDR}'"

SecRule IP:counter "@eq 0" \
  "id:401114,\
  msg:'counter is 0',\
  logdata:'counter is 0'"

SecRule IP:counter "@eq 1" \
  "id:401115,\
  msg:'counter equals to 1',\
  logdata:'counter equals to 1'"

SecRule IP:counter "@gt 1" \
  "id:401116,\
  msg:'counter greater than 1',\
  logdata:'counter greater than 1'"


SecRule IP:counter "@eq 6" \
  "id:401117,\
  msg:'this should be exipred',\
  logdata:'expired',\
setvar:ip.counter=0,\
expirevar:ip.counter=1"

[LeeShan87]

Well regression tests are passing for collections.

I ran some rules collected from the regression test with debuglog 9 with v3.0.0 and v3/master.
I have attached the the log files and the rule file.

debug.zip

The difference of the to logs:

 diff debug-v302.log debug-v300.log
20a21
> [6] Resolving: matched_var to: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
30a32
> [6] Resolving: remote_addr to: 1.2.3.1
36a39,40
> [6] Resolving: remote_addr to: 1.2.3.1
> [6] Resolving: tx.ua_hash to: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
48a53
> [6] Resolving: REMOTE_ADDR to: 1.2.3.1
65c70,71
< [4] (Rule: 200005) Executing operator "StrEq" with param "0" against TX:regex(^MSC_).
---
> [4] (Rule: 200005) Executing operator "StrEq" with param "0" against TX.
> [6] Resolving: MATCHED_VAR_NAME to: NULL
82c88
< [8] Saving variable: IP:auth_attempt with value: 1
---
> [8] Saving variable: IP:auth_attempt with value: 2
92c98
< [8] Saving variable: IP:auth_attempt with value: 1
---
> [8] Saving variable: IP:auth_attempt with value: 3

I think this maybe a bug :(

[zimmerle]

Hi @LeeShan87,

Within version 3 we have opted to make a plug-able architecture in terms of collections storage. By default we support the in-memory collection, which is an independent storage per process. The sdbm support (compilation flag) allow the communication via shared memory between the different process. There are also tickets opened to support the collections in Memcahe (#1140) and Redis (#1139).

Keep in mind that even in v2 this math won't match as the collection where sync with the main process from time to time depending on various factors including a dice. With means that an increment may or may not work.

[p0pr0ck5]

@zimmerle @victorhora how does one move back and forth between storage engines when SDBM support is compiled in?

[LeeShan87]

Hi @zimmerle and @victorhora!

Thank you for your reply. I have found that compilation flag. And I actively follow the progression of the mentioned to tickets.

My issue is since commit 6f7fdd9

I can even use collections.
I use the following rules for test if collection handling works:

SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "id:'900018',phase:1,t:none,t:sha1,t:hexEncode,setvar:tx.ua_hash=%{matched_var},log,pass"
SecRule &TX:REAL_IP "@eq 0" "id:'900021',phase:1,t:none,initcol:global=global,initcol:ip=%{remote_addr}_%{tx.ua_hash},setvar:tx.real_ip=%{remote_addr},log,pass"
# reset the counter to start at 0
SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "id:'900019',phase:2,t:none,setvar:ip.auth_attempt=0,log,pass"
SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "id:'900020',phase:2,t:none,setvar:ip.auth_attempt=+1,log,pass"
SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "id:'900022',phase:2,t:none,setvar:ip.auth_attempt=+1,log,pass"
SecRule REQUEST_HEADERS:User-Agent "^(.*)$" "id:'900023',phase:2,t:none,setvar:ip.auth_attempt=+1,log,pass"
# if collection is working request should be droped
SecRule IP:AUTH_ATTEMPT "@eq 3" "id:'900024',phase:2,t:none,log,msg:'%{MATCHED_VAR_NAME}', logdata:'%{MATCHED_VAR_NAME}: %{MATCHED_VAR}',drop"

My current setup is:
Nginx 1.14.0
ModSecurity nginx connector master
ModSecurity v3 (tried many commits :()
With lmdb and without lmdb.

[victorhora]

We are investigating this one. Reopening...

Also seems related with #1778

[zimmerle]

fixed by 892beb5

[LeeShan87]

Hi,
It seems IP collection is working again. Thank you for your help @zimmerle and @victorhora.

A good to know notice:
Collection names are case insensitive. ip,Ip,IP means the same collection.
If you create a variable in a collection, it's case sensitive.
So if we create a ip.auth_attempt, than we need to use it like: ip:auth_attempt. Not like previously IP:AUTH_ATTEMPT.

[zimmerle]

Humm... good catch @LeeShan87 i am going to have a look on ti. Thanks.

[revilzs]

Hi,

Hi,
It seems IP collection is working again. Thank you for your help @zimmerle and @victorhora.

A good to know notice:
Collection names are case insensitive. ip,Ip,IP means the same collection.
If you create a variable in a collection, it's case sensitive.
So if we create a ip.auth_attempt, than we need to use it like: ip:auth_attempt. Not like previously IP:AUTH_ATTEMPT.

This could lead to problems with owasp-modsecurity-crs rules: REQUEST-912-DOS-PROTECTION and REQUEST-910-IP-REPUTATION.

For example the vars are set in lowercase: setvar:'ip.dos_counter=+1' and accessed in uppercase: SecRule IP:DOS_COUNTER

[victorhora]

Hey @revilzs / @LeeShan87

Can you please check if this still happens in the current master of libModSecurity?

As far as I can recall, PR #1810 was targeted to fix this inconsistency, but then #1820 covered this fix and other improvements which was merged at d810de9

[LeeShan87]

Hi @victorhora @revilzs

It's looks really fixed to me in v3.0.3.

nginx.conf.txt

modsec_debug.txt

Nginx: 1.15.6
ModSecurity: v3.0.3
Nginx-connector: git commit 4b50399a4499c1e97e118bea3df0fd2801824e45

It' not related to this issue, but in can be a pain in the a...
ModSecurity use in memory collection by default.
Which means, in case of Nginx workers, every worker has its own collection.

So if a threshold exceeded in one worker, it's not means, that another will not serve the HTTP request.

This other issue could be solved, if it were implemented yet, with redis or memcached collection backends. Or with lmdb, but it has other bugs (last time I've checked).

[plamer]

I'm having trouble with lmdb and enabling it so processes can communicate with one another. I've compiled modsecurity (3.0.3) with --with-lmdb - both with the package manager provided one (v0.9.18-1) and with manually compiled one - v0.9.23. But neither works - for example if I limit DOS tries to 5 in a minute (for testing) I can issue 10+ requests before getting denied. If I lower the workers to 1 it works as expected and blocks me after 5 requests.

Also @zimmerle commented about sdbm:

The sdbm support (compilation flag) allow the communication via shared memory between the different process

But I can't find such flag when compiling modsecurity?

[victorhora]

@plamer SDBM is the default collection backend. It means that if you don't specify other collection backends such as LMDB, SDBM will be used by default. See here the collections backends here:

https://github.com/SpiderLabs/ModSecurity/tree/v3/master/src/collection/backend

Please notice that LMDB support is still experimental, so I would advise trying to reproduce the same scenario without LMDB. If the issue still persists, let us know and we can investigate.

Thanks.

[plamer]

@victorhora I did compile libmodsecurity without --with-lmdb flag, but the result is the same:

HTTP/1.1 403 Forbidden
Server: nginx/1.15.11
Date: Tue, 07 May 2019 14:03:03 GMT
Content-Type: text/html
Content-Length: 154
Connection: keep-alive

HTTP/1.1 200 OK
Server: nginx/1.15.11
Date: Tue, 07 May 2019 14:03:03 GMT
Content-Type: text/html
Connection: keep-alive
X-Powered-By: PHP/5.3.3

HTTP/1.1 403 Forbidden
Server: nginx/1.15.11
Date: Tue, 07 May 2019 14:03:03 GMT
Content-Type: text/html
Content-Length: 154
Connection: keep-alive

HTTP/1.1 403 Forbidden
Server: nginx/1.15.11
Date: Tue, 07 May 2019 14:03:03 GMT
Content-Type: text/html
Content-Length: 154
Connection: keep-alive

I'm using v3 from master (cloned and compiled today):

libmodsecurity compiled with:

./configure --with-yajl=/opt/lloyd-yajl-66cb08c/build/yajl-2.1.0/ --with-curl=/opt/curl/

nginx compiled with:

./configure --add-module=/opt/ModSecurity-nginx --with-pcre=/opt/pcre-8.43 --with-http_geoip_module --with-http_ssl_module --with-http_v2_module --with-http_realip_module

[Elyasnz]

Hi @plamer
any news on how to use shared memory between processes?

[martinhsv]

Hello @Elyasnz ,

The LMDB functionality does work.

(Note that the default directory for the files may not be writeable. See #2835 for probably the simplest way to address this.)