Huge memory consumption and slow nginx configtest/start/restart with ModSecurity v3 and ModSecurity-nginx

[vaLski]

Hellow everybody.

I am trying to make modsecurity 3 + modsecurity-nginx connector working on my setup with ~2000 vhosts and ~3000+ rules.

Compilation configuration etc. are all fine but when I try to run nginx configtest/start/restart it it is taking ages (5,8,10 minutes) for each of the operations to complete.

Also during the configtest and after the restart nginx is consuming huge amount of memory talking GBs here.

The problem(s):

nginx -t is taking ages
nginx restart/reload is taking even more time
nginx is consuming huge amount of memory (talking GBs here)

At first it sounds very similar to #1546 but it is not. The the patches mentioned there are already applied in my build.

Also I am not really sure if this is a memory leak bug at all or something that is not very well optimized for use cases like mine.

I am also using apache 2.4.x with modsecurity 2.x with the very same rule-set and configtest/start/restart operations are completed in matter of (sub)seconds with the same amount of virtual hosts.

Here is a quick look at the ltrace of nginx -t which I think might be useful for you. It is showing excessive amount of calls to msc_rules_merge which is taking most of the configtest time.

[email protected]:/etc/nginx# ltrace -f -c nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
"% time seconds usecs/call calls function

---

32.87 1783.733281 157 11349307 strncmp
28.18 1529.493827 1529493827 1 __libc_start_main
18.44 1000.688939 23966 41753 msc_rules_merge
5.63 305.725415 162 1882658 strcmp
4.99 270.902396 158 1711981 memset
2.35 127.432618 171 743696 memcpy
1.77 95.819104 160 598387 malloc

If you need more information please let me know. I will be happy to assist.

The setup:

• number of modsecurity rules: 3355
• number of nginx virtual hosts: 1779
• ModSecurity v3 VERSION - c1cd668
• ModSecurity v3 build options
  ./configure \ --prefix=/usr/local \ --enable-shared=yes \ --with-geoip=no \ --with-ssdeep=no \ --with-lua=no \ --with-libxml=/usr \ --with-pcre=/usr/local/pcre-8.38 \ --with-yajl=/usr/local/yajl-2.1.0 \ --with-curl=/usr/local/curl-7.46.0
• ModSecurity-nginx connector v3 - c0ae166cc30c01b96147b6bc3d0cda708f5cdfb7
• nginx version: nginx/1.13.7
• nginx build options
  --with-cc=/opt/rh/devtoolset-2/root/usr/bin/gcc --prefix=/usr/local/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-pcre --with-http_addition_module --with-http_degradation_module --with-http_gzip_static_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --without-http_ssi_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/tmp/client_body --http-proxy-temp-path=/var/cache/nginx/tmp/proxy --http-fastcgi-temp-path=/var/cache/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/cache/nginx/tmp/uwsgi --http-scgi-temp-path=/var/cache/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E --with-openssl=submodules/openssl --add-module=submodules/ngx_pagespeed-1.12.34.3-stable --add-module=submodules/cache_purge --add-module=submodules/testcookie --add-module=submodules/nginx-module-vts --add-module=submodules/ModSecurity-nginx
• modsecurity is enabled in the nginx configuration directly in the http block (tried that in each vhost separately which seems to make the things and memory consumption even worse)
  http { modsecurity on; modsecurity_rules_file /etc/nginx/mod_security2.conf; ..... }

[p0pr0ck5]

@vaLski is it possible for you to share a full example config? I'd be curious to see if it's possible to reproduce with the standalone libmodsecurity env, or if the Nginx connector is somehow involved.

[victorhora]

@vaLski

There is a number of recent changes pushed into libModSecurity master concerning memory leaks and fixes including this one 8285a97 which did not got into 3.0.2 release.

The same applies for ModSecurity-nginx connector. Commit owasp-modsecurity/ModSecurity-nginx@bcfe69a should fix some memory leaks there.

Can you check if these changes makes any difference on the issue that you're facing?

Thanks

[p0pr0ck5]

@victorhora I don't think those changes are related, at the very least not the Nginx connector changes. I wrote a simple mock harness to stress loading many rules: https://gist.github.com/p0pr0ck5/3b58a98b653ef0bde2bd6da3c68240f7. This just tries to load many rulesets, all with simple rules as generated from a simple script:

#!/usr/bin/perl

use strict;
use warnings;

my $x = 12345;
my $buf;
my ($i, $j, $n);
$i = $j = $n = 0;

for ($n; $n < 2000; ++$n) {
    $buf = "";
    $i = $j = 0;
    for ($i; $i < 6; $i++) {
        for ($j; $j < 200; $j++) {
            $x++;
            $buf = $buf . "SecRule ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES " .
                "\"\@streq foo\" \"id:${n}${x},phase:$i,deny,auditlog,t:none\"\n";
        }
    }
    open (my $fh, '>', "mock-$n.conf") or die $!;
    print $fh $buf;
    close $fh;
    print "did $n\n";
}

This took almost 1G of memory and several minutes to execute, when building from current v3/master. When applying the patch noted in #1667 the execution time drops dramatically:

poprocks@mini-vm:~/src/ModSecurity/examples/simple_example_using_c$ time ./test
real	0m7.965s
user	0m3.421s
sys	0m4.545s

This was also noted by @vaLski. As noted, this is an incomplete/incorrect fix as it does break libmodsec behaviors, but it does indicate that the exponential loop in https://github.com/SpiderLabs/ModSecurity/blob/v3/master/headers/modsecurity/rules_properties.h#L438-L450 is the culprit. I suspect such functionality could be replaced with a simpler set search, which could improve performance significantly in cases like this.

[p0pr0ck5]

@vaLski would you consider testing the patch above? ^

[vaLski]

@victorhora Alas the fixes you referenced are not related to the problem I am experiencing. The merge request I sent initially for this report is fixing my problem (in a very dirty way) and it pinpoints where is the exact bottleneck.

@p0pr0ck5 I will check this at my first convenience and will let you know the results. Alas I am unable to share the ruleset but I think that the generator you used is pretty much represent my case exactly. Looking into the code of your pull request, it touches the very same part that according to my opinion is causing the bottle neck. 7 seconds sounds much more reasonable for check and reload so the optimization you added appears to be working fine. In my case I simply deleted the entire check so it is running a bit faster.

Thank you for your time and for looking into this! I appreciate that.

[Stono]

With mod security:

Without mod security:

:(

[zimmerle]

Hi @Stono

The way that your configure your rules play a big hole in the memory usage. For a larger rule set our recommendation is to having the rules loaded on the server context, so that every vhost got the benefit of using it without loading the rules in to memory again. Share your configuration file on gists. We may be able to help you.

Also make sure that you have at very least: eed6b5f

[zimmerle]

@vaLski did you had the chance to test it?

[spstsdev]

@zimmerle Thanks so much for your suggestion... I was having big problems while reloading nginx configuration: huge CPU, time and memory needed. Finally it was because of loading large ruleset inside servers and locations, instead of loading it simply once in main nginx.conf http section.

I think this problem/solution may be pointed out in main modsecurity documentation for Nginx.

Thanks!!!