Huge memory consumption and slow nginx configtest/start/restart with ModSecurity v3 and ModSecurity-nginx

[vaLski]

Hellow everybody.

I am trying to make modsecurity 3 + modsecurity-nginx connector working on my setup with ~2000 vhosts and ~3000+ rules.

Compilation configuration etc. are all fine but when I try to run nginx configtest/start/restart it it is taking ages (5,8,10 minutes) for each of the operations to complete.

Also during the configtest and after the restart nginx is consuming huge amount of memory talking GBs here.

The problem(s):

nginx -t is taking ages
nginx restart/reload is taking even more time
nginx is consuming huge amount of memory (talking GBs here)

At first it sounds very similar to #1546 but it is not. The the patches mentioned there are already applied in my build.

Also I am not really sure if this is a memory leak bug at all or something that is not very well optimized for use cases like mine.

I am also using apache 2.4.x with modsecurity 2.x with the very same rule-set and configtest/start/restart operations are completed in matter of (sub)seconds with the same amount of virtual hosts.

Here is a quick look at the ltrace of nginx -t which I think might be useful for you. It is showing excessive amount of calls to msc_rules_merge which is taking most of the configtest time.

root@server.com:/etc/nginx# ltrace -f -c nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
"% time seconds usecs/call calls function

---

32.87 1783.733281 157 11349307 strncmp
28.18 1529.493827 1529493827 1 __libc_start_main
18.44 1000.688939 23966 41753 msc_rules_merge
5.63 305.725415 162 1882658 strcmp
4.99 270.902396 158 1711981 memset
2.35 127.432618 171 743696 memcpy
1.77 95.819104 160 598387 malloc

If you need more information please let me know. I will be happy to assist.

The setup:

• number of modsecurity rules: 3355
• number of nginx virtual hosts: 1779
• ModSecurity v3 VERSION - c1cd668
• ModSecurity v3 build options
  ./configure \ --prefix=/usr/local \ --enable-shared=yes \ --with-geoip=no \ --with-ssdeep=no \ --with-lua=no \ --with-libxml=/usr \ --with-pcre=/usr/local/pcre-8.38 \ --with-yajl=/usr/local/yajl-2.1.0 \ --with-curl=/usr/local/curl-7.46.0
• ModSecurity-nginx connector v3 - c0ae166cc30c01b96147b6bc3d0cda708f5cdfb7
• nginx version: nginx/1.13.7
• nginx build options
  --with-cc=/opt/rh/devtoolset-2/root/usr/bin/gcc --prefix=/usr/local/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-pcre --with-http_addition_module --with-http_degradation_module --with-http_gzip_static_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --without-http_ssi_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/cache/nginx/tmp/client_body --http-proxy-temp-path=/var/cache/nginx/tmp/proxy --http-fastcgi-temp-path=/var/cache/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/cache/nginx/tmp/uwsgi --http-scgi-temp-path=/var/cache/nginx/tmp/scgi --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=-Wl,-E --with-openssl=submodules/openssl --add-module=submodules/ngx_pagespeed-1.12.34.3-stable --add-module=submodules/cache_purge --add-module=submodules/testcookie --add-module=submodules/nginx-module-vts --add-module=submodules/ModSecurity-nginx
• modsecurity is enabled in the nginx configuration directly in the http block (tried that in each vhost separately which seems to make the things and memory consumption even worse)
  http { modsecurity on; modsecurity_rules_file /etc/nginx/mod_security2.conf; ..... }

[p0pr0ck5]

@vaLski is it possible for you to share a full example config? I'd be curious to see if it's possible to reproduce with the standalone libmodsecurity env, or if the Nginx connector is somehow involved.

[victorhora]

@vaLski

There is a number of recent changes pushed into libModSecurity master concerning memory leaks and fixes including this one 8285a97 which did not got into 3.0.2 release.

The same applies for ModSecurity-nginx connector. Commit owasp-modsecurity/ModSecurity-nginx@bcfe69a should fix some memory leaks there.

Can you check if these changes makes any difference on the issue that you're facing?

Thanks

[p0pr0ck5]

@victorhora I don't think those changes are related, at the very least not the Nginx connector changes. I wrote a simple mock harness to stress loading many rules: https://gist.github.com/p0pr0ck5/3b58a98b653ef0bde2bd6da3c68240f7. This just tries to load many rulesets, all with simple rules as generated from a simple script:

#!/usr/bin/perl

use strict;
use warnings;

my $x = 12345;
my $buf;
my ($i, $j, $n);
$i = $j = $n = 0;

for ($n; $n < 2000; ++$n) {
    $buf = "";
    $i = $j = 0;
    for ($i; $i < 6; $i++) {
        for ($j; $j < 200; $j++) {
            $x++;
            $buf = $buf . "SecRule ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES " .
                "\"\@streq foo\" \"id:${n}${x},phase:$i,deny,auditlog,t:none\"\n";
        }
    }
    open (my $fh, '>', "mock-$n.conf") or die $!;
    print $fh $buf;
    close $fh;
    print "did $n\n";
}

This took almost 1G of memory and several minutes to execute, when building from current v3/master. When applying the patch noted in #1667 the execution time drops dramatically:

poprocks@mini-vm:~/src/ModSecurity/examples/simple_example_using_c$ time ./test
real	0m7.965s
user	0m3.421s
sys	0m4.545s

This was also noted by @vaLski. As noted, this is an incomplete/incorrect fix as it does break libmodsec behaviors, but it does indicate that the exponential loop in https://github.com/SpiderLabs/ModSecurity/blob/v3/master/headers/modsecurity/rules_properties.h#L438-L450 is the culprit. I suspect such functionality could be replaced with a simpler set search, which could improve performance significantly in cases like this.