modsecurity_rules_file directive failed to attach shared memory

[Roger-Man]

Hi
using v3.0.2 with nginx-1.14.0 we are getting "modsecurity_rules_file directive failed to attach shared memory" after tolerating heavy traffic for some hours.

Is there any workaround?

P.S. Is there any relation with #38?
P.S. We are collecting nginx access logs every minute by cutting the access log file to a different location and issuing USR1 signal as documented in Rotating Log-files. This causes the mentioned error to be issued. Then nginx goes away which results in refusing connections.

[victorhora]

Hi @Roger-Man ,

We had many improvements and fixes (including memory leaks) since 3.0.2 came out. The CHANGES file summarizes them.

Can you maybe try with the latest libModSecurity code from master and let us know if the issue still happens: https://github.com/SpiderLabs/ModSecurity/commits/v3/master?

[Roger-Man]

Hi @victorhora
I'll try it.
But I will be grateful if you answer following questions?

1. Do you think this is somehow related to memory leaks?
2. Should I disable pcre-jit like reommended in SELinux & modsecurity-nginx #38?

[victorhora]

@Roger-Man

I'm not sure as I've never stumbled in this error before. By glancing at the code related with this error message, it seems related with a failure to perform an shmat() while handling debug logs.

By any chance you have SecDebugLog enabled for your production server? We don't recommend enabling debug logs for production as it incurs a performance penalty. I'm wondering if the "heavy load" that you mention (along with debug_logs) could be leading to this error.

If debug_log is indeed disabled, you can consider "removing" debugging functionality altogether to make sure that the issue is not related with that and also for increased performance. This can be accomplished by appending the --disable-debug-logs option to the ./configure script/step.

Regardless, I still recommend updating to the latest libModSecurity code from master until the next release.

As for pcre-JIT, there's a history of some small memory leaks from that functionality in v2 but I think it's not the case for libModSecurity (v3) today. But I think that the pcre_jit offdirective would only affect Nginx PCRE processing (assuming it's enabled on your Nginx build), and not libModSecurity regex processing.

Are you running with SELinux? PCRE-JIT was optional on v2 but it is enabled by default on libModSecurity (v3). Currently there's no flag to disable it cleanly, but if you still would like to disable it you could do it through a minor modification of the pcre.m4 file or by manually setting the PCRE_CFLAGS (env) variable.

[Roger-Man]

@victorhora
Hi
Yes, SecDebugLog was enabled. But the file is always empty unless I increase the SecDebugLogLevel which is not the case that I am reporting.
However, I will do both upgrade to the latest version and disable the SecDebugLog and I will report back.

Thanks a lot.

[zimmerle]

Hi @Roger-Man,

Did you had the chance to test it? By any chance, did you had gdbm enabled?

[victor]

Hi @victor Hora ...

@Roger-Man please be careful with your mentions

thanks

_{Sent with GitHawk}

[Roger-Man]

@victorhora
Hi all,
Doing both below actions solved the problem.

1. Removing SecDebugLog from the configs.
2. Updating to 68398a51f3a98170e192974708351079a5c6d85c

Now it works smoothly as expected.
Thanks.

[Roger-Man]

gdbm

@zimmerle
Hi All,

I have tested and reported back.
I don't know if I had gdbm enable or not. How can I determine it?

P.S. Modsecurity configure script shows + LMDB ....disabled
It's the most relevant thing I could find. If you will please tell me what is it I'll find the correct answer.

Thanks

[Roger-Man]

Hi @victor Hora ...

@Roger-Man please be careful with your mentions

thanks

Sent with GitHawk

Fixed it.

[zimmerle]

@victorhora
Hi all,
Doing both below actions solved the problem.

1. Removing SecDebugLog from the configs.
2. Updating to 68398a51f3a98170e192974708351079a5c6d85c

Now it works smoothly as expected.
Thanks.

Based on that comment I am closing the issue.

[Roger-Man]

gdbm

@zimmerle
Hi All,

I have tested and reported back.
I don't know if I had gdbm enable or not. How can I determine it?

P.S. Modsecurity configure script shows + LMDB ....disabled
It's the most relevant thing I could find. If you will please tell me what is it I'll find the correct answer.

Thanks

@zimmerle
Hi
Am going to use lmdb,
Do you think I might run into problem again?

[Roger-Man]

Yes, I ran into the problem as I mentioned in #1141

[zimmerle]

Hi @Roger-Man,

If you want to use shared memory you need to change your SELinux profile to allow it. If SELinux is explicit set to not allow shared memory, the process won't be able to use it.

[Roger-Man]

Hi @zimmerle
Am not using SELINUX.

[zimmerle]

Hi @Roger-Man,

You said that you are getting "modsecurity_rules_file directive failed to attach shared memory" is that right?

[Roger-Man]

@zimmerle
Yes that's right. And I Also said I wasn't using SElinux.

[zimmerle]

Plase run ipcs -m and paste the output here.

[zimmerle]

In owasp-modsecurity/ModSecurity#1141 you said that you have a double free corruption, how you managed to get it, if your nignx refuses to start?

[Roger-Man]

Plase run ipcs -m and paste the output here.

I had reported the issue about 50 days ago. The system condition has changed since then.
I updated the system packages using apt-get dist-upgrage and also upgraded nginx and also libmodsecurity together with connector.
But as you wish. Here is the output:
------ Shared Memory Segments --------

key shmid owner perms bytes nattch status

0x0052e2c1 589824 postgres 600 56 5

P.S. No changes on SElinux since am not using it.

[Roger-Man]

In SpiderLabs/ModSecurity#1141 you said that you have a double free corruption, how you managed to get it, if your nignx refuses to start?

It starts and works, but as I said in SpiderLabs/ModSecurity#1141 arbitrary worker processes (not the whole process tree) crash and exit.
Please be aware that this problem persists even after I overcame the issue of current thread

And if it is worth, my current issue is the memory leaks while issuing nginx -s reload

[zimmerle]

Hi @Roger-Man,

So, after the upgrade this particular issue is no longer a problem?

[Roger-Man]

Hi @Roger-Man,

So, after the upgrade this particular issue is no longer a problem?

As I mentioned in last comment both updating and removing SecDebugLog from the configs at the same time resolved the issue.
As I didn't test these measures separately I can't confirm which was the remedy.

[Kage]

I'm now having the same issue whenever trying to load any rules file. Nginx 1.14.2, libmodsecurity v3.0.3, FreeBSD 10.3. The problem presents itself whenever trying to set anything for SecAuditLog (if I comment out all SecAudit* lines in modsecurity.conf, everything works), which presents the following error:

# service nginx restart
Performing sanity check on nginx configuration:
nginx: [emerg] "modsecurity_rules_file" directive Failed to allocate shared memory (1): Function not implemented in /usr/local/etc/nginx/conf.d/hackthissite.org.conf:36
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

Environment:

# uname -a
FreeBSD waf 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 02:10:02 UTC 2016     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64

# ipcs -m
Shared Memory:
T           ID          KEY MODE        OWNER    GROUP   

# nginx -V
nginx version: nginx/1.14.2
built with OpenSSL 1.0.1s-freebsd  1 Mar 2016
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --modules-path=/usr/local/libexec/nginx --with-file-aio --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_v2_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-pcre --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-mail_ssl_module --add-module=/var/ports/basejail/usr/ports/www/nginx/work/ModSecurity-nginx-71ede63 --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-http_perl_module=dynamic --with-mail=dynamic --with-stream=dynamic --add-dynamic-module=/var/ports/basejail/usr/ports/www/nginx/work/ngx_devel_kit-0.3.0 --add-dynamic-module=/var/ports/basejail/usr/ports/www/nginx/work/headers-more-nginx-module-085fbbc --add-dynamic-module=/var/ports/basejail/usr/ports/www/nginx/work/ngx_http_geoip2_module-3.2 --add-dynamic-module=/var/ports/basejail/usr/ports/www/nginx/work/lua-nginx-module-0.10.14

[victorhora]

@Kage, I would suggest creating a new issue since it could be something specific with the FreeBSD build that needs to be investigated. Thanks.

[Kage]

Duly noted, will do. Thanks!