Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself) #29

md2k · 2017-01-13T16:08:29Z

Hi Devs, i noticed that latest master of connector (maybe issue not in connector exactly) causing nginx segfaults during its restart.
Reload or Stop then Start do not cause such segfaults.
Also by default files for shared collection dropped to root folder '/'

ls -l / | grep shared
-rw-r--r--   1 root root 1048576 Jan 13 17:03 modsec-shared-collections
-rw-r--r--   1 root root    8192 Jan 13 17:05 modsec-shared-collections-lock

But, during restart Nginx with init/init.d scripts those files created in directory where cli command was used.

Jan 13 17:03:07 abyss kernel: [15515838.077384] nginx[7190]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077395] nginx[7192]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077398] nginx[7186]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077400] nginx[7191]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077401]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077402] nginx[7188]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077403]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077404]
Jan 13 17:03:07 abyss kernel: [15515838.077405]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077405]
Jan 13 17:03:07 abyss kernel: [15515838.077406]
Jan 13 17:03:07 abyss kernel: [15515838.077407]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077414]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.080151] nginx[7185]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b92e8 error 4 in libc-2.19.so[7fbe74bd3000+1ba000]




Jan 13 17:03:24 abyss kernel: [15515854.689932] nginx[7689]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4
Jan 13 17:03:24 abyss kernel: [15515854.689934] nginx[7687]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689936]  in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689960] nginx[7686]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689978] nginx[7690]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689983] nginx[7691]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.690068] nginx[7688]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.692350] nginx[7685]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954eda8 error 4 in libc-2.19.so[7fea863e5000+1ba000]

The text was updated successfully, but these errors were encountered:

defanator · 2017-01-31T15:37:10Z

Hello @md2k, could you please provide full nginx configuration (in recent versions it can be obtained by running nginx -T), full modsecurity configuration, and backtraces?

Thanks.

mimugmail · 2017-02-02T15:36:03Z

I can semi-confirm this bug! Have already posted it in the modsec_dev list.

And also after updating the source from 12.12.16 - 27.01.17 the shared-collections are in /

root@waf-1-a-02:~# ls -la /etc/nginx/modsec/
-rw-r--r-- 1 root root 1048576 Jan 27 13:59 modsec-shared-collections
-rw-r--r-- 1 root root 8192 Jan 27 14:00 modsec-shared-collections-lock

root@waf-1-a-02:~# ls -la /
-rw-r--r-- 1 root root 1048576 Jan 27 14:34 modsec-shared-collections
-rw-r--r-- 1 root root 8192 Jan 27 14:34 modsec-shared-collections-lock

I'm running latest N+ and build MS3 from source, not the N+ module.

mimugmail · 2017-02-03T12:10:58Z

I could also send you a nginx -T directly if you like cause it's very huge and sensible

md2k · 2017-02-03T12:59:24Z

@defanator i can send it, but it also big for me and and obfuscation will take some time (for nginx).

defanator · 2017-02-03T13:08:15Z

@md2k we can start from backtraces then.

defanator · 2017-02-03T13:09:25Z

@mimugmail if you're using N+, I would suggest to address this via N+ support channel. You can share backtraces here though.

md2k · 2017-02-03T13:29:59Z

It will be helpful if you can tell me how i can get backtrace from modsec/nginx

mimugmail · 2017-02-03T13:47:41Z

@defanator Since I'm not using the commercial WAF module I don't think that we can expect much support from N+, also I don't want to keep these guys rotating (was a promise to OwenGarret@Nginx) :)

I'll try to make a detailed bug report

defanator · 2017-02-03T15:58:31Z

@md2k you need to obtain core file and then use GDB to get output of the full bt command. These links could be useful for further details:

https://www.nginx.com/resources/admin-guide/debug/
https://github.com/spiderlabs/modsecurity/tree/v3/master#debugging

md2k · 2017-02-03T16:37:50Z

Thank, will check them later, going to prepare virtual box for this to not mess with my production server

mimugmail · 2017-02-04T12:05:16Z

OS: Debian 8
Nginx: 1.11.9
MS3 source: 04.02.2017
Nginx connector: 04.02.2017

Backtrace full:
root@nginx:~# gdb /opt/nginx/sbin/nginx /tmp/core
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/nginx/sbin/nginx...done.
[New LWP 16984]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
106 ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) backtrace full
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
No locals.
#1 0x00007fe95e437c28 in std::string::compare(char const*) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#2 0x00007fe95f5d36cc in operator==<char, std::char_traits, std::allocator > (__rhs="/var/log/modsec_audit.log", __lhs=)
at /usr/include/c++/4.9/bits/basic_string.h:2528
No locals.
#3 modsecurity::utils::SharedFiles::find_handler (this=this@entry=0x7fe95f857ce8 modsecurity::utils::SharedFiles::getInstance()::instance,
fileName="/var/log/modsec_audit.log") at utils/shared_files.cc:42
current = 0x7fe960912000
#4 0x00007fe95f5d3710 in modsecurity::utils::SharedFiles::close (this=0x7fe95f857ce8 modsecurity::utils::SharedFiles::getInstance()::instance, fileName=...)
at utils/shared_files.cc:181
a =
#5 0x00007fe95f56d119 in modsecurity::audit_log::writer::Serial::~Serial (this=0x2443220, __in_chrg=) at audit_log/writer/serial.cc:28
No locals.
#6 0x00007fe95f56b148 in modsecurity::audit_log::AuditLog::~AuditLog (this=0x2443100, __in_chrg=) at audit_log/audit_log.cc:69
No locals.
#7 0x00007fe95f56fc16 in ~RulesProperties (this=0x24471a0, __in_chrg=) at ../headers/modsecurity/rules_properties.h:106
No locals.
#8 modsecurity::Rules::~Rules (this=0x24471a0, __in_chrg=) at rules.cc:80
No locals.
#9 0x00007fe95f56ff0e in modsecurity::msc_rules_cleanup (rules=0x24471a0) at rules.cc:335
No locals.
#10 0x00007fe95f85a4d4 in ngx_http_modsecurity_config_cleanup (data=0x2454298) at /opt/ModSecurity-nginx/src/ngx_http_modsecurity_module.c:595
t = 0x2454298
#11 0x000000000040df31 in ngx_destroy_pool (pool=0x23e6d30) at src/core/ngx_palloc.c:57
p =
n =
l =
c = 0x24542f8
#12 0x00000000004330a0 in ngx_worker_process_exit (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:1001
i =
c =
#13 0x00000000004331af in ngx_worker_process_cycle (cycle=cycle@entry=0x23e6d80, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:758
worker = 0
#14 0x000000000043185a in ngx_spawn_process (cycle=cycle@entry=0x23e6d80, proc=proc@entry=0x4330d8 <ngx_worker_process_cycle>, data=data@entry=0x0,
---Type to continue, or q to quit---
name=name@entry=0x48ad7e "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:198
on = 1
pid = 0
s = 0
#15 0x000000000043241d in ngx_start_worker_processes (cycle=cycle@entry=0x23e6d80, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:358
i = 0
ch = {command = 1, pid = 0, slot = 0, fd = 0}
#16 0x0000000000433a7b in ngx_master_process_cycle (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:130
title = 0x2458ab4 "master process /opt/nginx/sbin/nginx"
p =
size =
i =
n =
sigio =
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 38111800, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
live =
delay =
ls =
ccf = 0x23e8878
#17 0x000000000040c7b7 in main (argc=, argv=) at src/core/nginx.c:368
b =
log = 0x6b9880 <ngx_log>
i =
cycle = 0x23e6d80
init_cycle = {conf_ctx = 0x0, pool = 0x23e6920, log = 0x6b9880 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0,
handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0,
free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0,
listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {
elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {
key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0},
size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 27, data = 0x23e6970 ""},
conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 17, data = 0x23e6970 ""}, prefix = {len = 12, data = 0x485bb0 "/opt/nginx//"}, lock_file = {
len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
cd =
---Type to continue, or q to quit---
ccf =

nginx -T:

root@nginx:~# /opt/nginx/sbin/nginx -T
nginx: the configuration file /opt/nginx//conf/nginx.conf syntax is ok
nginx: configuration file /opt/nginx//conf/nginx.conf test is successful

configuration file /opt/nginx//conf/nginx.conf:

#user nobody;
worker_processes 1;

worker_rlimit_core 500M;
working_directory /tmp/;

#error_log logs/error.log;
#error_log logs/error.log notice;
error_log logs/error.log debug;

pid logs/nginx.pid;

load_module "modules/ngx_http_modsecurity_module.so";

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  logs/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
    listen       80;
    server_name  localhost;

modsecurity on;
modsecurity_rules_file /opt/nginx/modsec/main.conf;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

    location / {
        root   html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}


# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
#    listen       8000;
#    listen       somename:8080;
#    server_name  somename  alias  another.alias;

#    location / {
#        root   html;
#        index  index.html index.htm;
#    }
#}


# HTTPS server
#
#server {
#    listen       443 ssl;
#    server_name  localhost;

#    ssl_certificate      cert.pem;
#    ssl_certificate_key  cert.key;

#    ssl_session_cache    shared:SSL:1m;
#    ssl_session_timeout  5m;

#    ssl_ciphers  HIGH:!aNULL:!MD5;
#    ssl_prefer_server_ciphers  on;

#    location / {
#        root   html;
#        index  index.html index.htm;
#    }
#}

}

configuration file /opt/nginx//conf/mime.types:

types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;

text/mathml                           mml;
text/plain                            txt;
text/vnd.sun.j2me.app-descriptor      jad;
text/vnd.wap.wml                      wml;
text/x-component                      htc;

image/png                             png;
image/tiff                            tif tiff;
image/vnd.wap.wbmp                    wbmp;
image/x-icon                          ico;
image/x-jng                           jng;
image/x-ms-bmp                        bmp;
image/svg+xml                         svg svgz;
image/webp                            webp;

application/font-woff                 woff;
application/java-archive              jar war ear;
application/json                      json;
application/mac-binhex40              hqx;
application/msword                    doc;
application/pdf                       pdf;
application/postscript                ps eps ai;
application/rtf                       rtf;
application/vnd.apple.mpegurl         m3u8;
application/vnd.ms-excel              xls;
application/vnd.ms-fontobject         eot;
application/vnd.ms-powerpoint         ppt;
application/vnd.wap.wmlc              wmlc;
application/vnd.google-earth.kml+xml  kml;
application/vnd.google-earth.kmz      kmz;
application/x-7z-compressed           7z;
application/x-cocoa                   cco;
application/x-java-archive-diff       jardiff;
application/x-java-jnlp-file          jnlp;
application/x-makeself                run;
application/x-perl                    pl pm;
application/x-pilot                   prc pdb;
application/x-rar-compressed          rar;
application/x-redhat-package-manager  rpm;
application/x-sea                     sea;
application/x-shockwave-flash         swf;
application/x-stuffit                 sit;
application/x-tcl                     tcl tk;
application/x-x509-ca-cert            der pem crt;
application/x-xpinstall               xpi;
application/xhtml+xml                 xhtml;
application/xspf+xml                  xspf;
application/zip                       zip;

application/octet-stream              bin exe dll;
application/octet-stream              deb;
application/octet-stream              dmg;
application/octet-stream              iso img;
application/octet-stream              msi msp msm;

application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

audio/midi                            mid midi kar;
audio/mpeg                            mp3;
audio/ogg                             ogg;
audio/x-m4a                           m4a;
audio/x-realaudio                     ra;

video/3gpp                            3gpp 3gp;
video/mp2t                            ts;
video/mp4                             mp4;
video/mpeg                            mpeg mpg;
video/quicktime                       mov;
video/webm                            webm;
video/x-flv                           flv;
video/x-m4v                           m4v;
video/x-mng                           mng;
video/x-ms-asf                        asx asf;
video/x-ms-wmv                        wmv;
video/x-msvideo                       avi;

}

This is just a development machine, non-productive!

Installed like in:
http://www.routerperformance.net/howtos/setup-modsecurity-3-and-nginx-in-debian-8/

mimugmail · 2017-02-04T21:07:14Z

This is what nginx error logs throws after reload:

2017/02/04 22:03:11 [notice] 576#0: signal process started
2017/02/04 22:03:11 [notice] 574#0: signal 1 (SIGHUP) received, reconfiguring
2017/02/04 22:03:11 [debug] 574#0: wake up, sigio 0
2017/02/04 22:03:11 [notice] 574#0: reconfiguring
2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 00000000022EAD40:16384 @16
2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CBF6A0:16384 @16
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002343720:4096
2017/02/04 22:03:11 [debug] 574#0: read: 21, 0000000002343720, 2833, 0
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022ECC08
2017/02/04 22:03:11 [debug] 574#0: module: ngx_http_modsecurity_module before ngx_http_range_header_filter_module:38
2017/02/04 22:03:11 [debug] 574#0: module: ngx_http_modsecurity_module i:48
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022EEA08
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022EEA80
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CC8330:4280
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CC93F0:4280
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CCA4B0:4280
2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CCB570:16384 @16
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CCF580:4280
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD0640:4280
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD1700:4280
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD27C0:4096
2017/02/04 22:03:11 [debug] 574#0: include mime.types
2017/02/04 22:03:11 [debug] 574#0: include /opt/nginx//conf/mime.types
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD37D0:4096
2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CD47E0:16384 @16
2017/02/04 22:03:11 [debug] 574#0: read: 32, 0000000002CD37D0, 3957, 0
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD87F0:4096
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CD7C30
2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CD9800:16384 @16
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CD8760
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CDB6E0
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:2048
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDE020:4352
2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CDBBB8
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD4550:512
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD4760:96
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024
2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024
2017/02/04 22:03:12 [debug] 575#0: epoll: fd:20 ev:2011 d:00007FF6D89140F0
2017/02/04 22:03:12 [debug] 575#0: epoll_wait() error on fd:20 ev:2011
2017/02/04 22:03:12 [debug] 575#0: channel handler
2017/02/04 22:03:12 [debug] 575#0: recvmsg() returned zero
2017/02/04 22:03:12 [debug] 575#0: channel: -1
2017/02/04 22:03:12 [debug] 575#0: epoll del connection: fd:20
2017/02/04 22:03:12 [debug] 575#0: reusable connection: 0
2017/02/04 22:03:12 [debug] 575#0: timer delta: 6870
2017/02/04 22:03:12 [debug] 575#0: worker cycle
2017/02/04 22:03:12 [debug] 575#0: epoll timer: -1
2017/02/04 22:03:43 [notice] 575#0: signal 15 (SIGTERM) received, exiting
2017/02/04 22:03:43 [info] 575#0: epoll_wait() failed (4: Interrupted system call)
2017/02/04 22:03:43 [debug] 575#0: timer delta: 31224
2017/02/04 22:03:43 [notice] 575#0: exiting
2017/02/04 22:03:43 [debug] 575#0: flush files
2017/02/04 22:03:43 [debug] 575#0: run cleanup: 0000000002358AE0
2017/02/04 22:03:43 [debug] 575#0: run cleanup: 00000000023547D0
2017/02/04 22:03:43 [debug] 575#0: cleanup resolver
2017/02/04 22:03:43 [debug] 575#0: run cleanup: 00000000023542F8

@md2k What distro do you use?

mimugmail · 2017-02-04T21:57:57Z

This is my last one ... also installed it with CentOS7 and also got the segfault.

defanator · 2017-02-06T11:14:50Z

@mimugmail, thanks for the backtrace!

It seems like the issue you're observing is related to a number of other issues we've been also facing. Latest attempt to provide a fix was made here: owasp-modsecurity/ModSecurity#1306 (though there's a number of questions to that PR).

Tagging @zimmerle here - this is related to the libmodsecurity itself, not just the connector.

@md2k, were you able to grab a core and obtain a backtrace from it?

mimugmail · 2017-02-06T12:30:52Z

@defanator @zimmerle
This error must be added to the source some time after 12.12.16, because my productive machine is running this code base and I can reload without any issues.

Hope you will find this one :)

@md2k install gdb packet, compile nginx --with-debug and add the stuff in the links @defanator added here. Then start nginx, reload it, then the segfault comes but there's no core, after that type a killall nginx and now there's the coredump int /tmp. Then type "gdb /opt/nginx/sbin/nginx /tmp/core" and "backtrace full"

md2k · 2017-02-07T11:09:34Z

@mimugmail thanks.
I use Ubuntu 14.04.5 LTS with Nginx 1.10.2-1~trusty (from nginx repository)

@defanator most probably i will have time to deal with it next week due my main project workload.

mimugmail · 2017-02-08T13:53:54Z

As @zimmerle told me to disable lmdb in another issue I tried it on this one, but --without-lmdb I also get a segfault here.

md2k · 2017-02-15T15:17:04Z

Hi @defanator , as you asked, in Gist output of gdb from core file after process crashed during nginx restart operation. (also configuration parameters which is used to compile nginx, its config (absolutely default, except modsec entries )), and log file from nginx with debug level
https://gist.github.com/md2k/4e18cc10649601bb93eed6a17bffc106

defanator · 2017-02-15T15:19:10Z

Thanks @md2k.

md2k · 2017-02-15T15:36:02Z

@defanator i added 2 more backtraces to same gist as Another GDB (modsec compiled there with CFLAGS=-g -O0)

zimmerle · 2017-02-24T01:38:25Z

Hi,

The problem was related to shared memory. The first design was meant to be used by forked processes. In the forked process, the address space and file descriptor will be common in every forked-process (worker). The reload (in this case ./nginx -t) creates a new process, leading to different file descriptors and consequently a segfault. The segfault seems to be resolved, although I have faced `zombie' processes while stressing the implementation.

Do you guys mind to test? Those are the commits:

SharedFile class:

To use the SharedFile class within the DebugLogs:

owasp-modsecurity/ModSecurity@1225076

The are available on the branch dev/parser:

https://github.com/SpiderLabs/ModSecurity/tree/v3/dev/parser

Thank you guys, and sorry for the huge delay.

mimugmail · 2017-02-24T09:31:49Z

Thanks @zimmerle for fixing this! I can confirm that with dev parser branch I can cleanly compile and reload the process. BUT, now when I start nginx it looks like this:

root@nginx:# /opt/nginx/sbin/nginx
**********************************************************************************root@nginx:#

Also when reloading there are these asterisks.
When I disable ModSec they are gone.

The performance like in the other issue is also not better than before, will post my results there.

Thank you!

Menahem1 · 2017-02-24T14:20:53Z

Hello,

Same that @mimugmail with the last release (thanks for the update @zimmerle ) on check of status of nginx

Feb 24 12:29:59 ip-10-65-3-219 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 systemd[1]: Started A high performance web server and a reverse proxy server.
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

defanator · 2017-02-26T15:48:31Z

@zimmerle I've just tested libmodsecurity from the current head of v3/dev/parser, and can confirm that nginx is not segfaulting anymore.

Then I realized that I've been running nginx with older connector module, built with libmodsecurity v3/master. I tried to rebuild it with v3/dev/parser code and got the following:

cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include -I objs -I src/http -I src/http/modules \
    -o objs/addon/src/ngx_http_modsecurity_module.o \
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:473:34: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror=incompatible-pointer-types]
     msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
                                  ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
                 from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/usr/include/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb {aka void (*)(void *, const void *)}' but argument is of type 'void (*)(void *, const char *)'
 void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
      ^~~~~~~~~~~~~~
cc1: all warnings being treated as errors

Are there any plans to merge v3/dev/parser to v3/master, or cherry pick a set of changes affecting the logging part, so we could adjust nginx connector code here?

zimmerle · 2017-03-07T13:06:03Z

v3/dev/parser is now part of v3/master ;)

Thank you for the reports ;)

hernandanielg · 2017-03-07T18:31:05Z

I got this error too

../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:426:5: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror]
     msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
     ^
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
                 from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/tmp/ModSecurity/headers/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb' but argument is of type 'void (*)(void *, const char *)'
 void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
      ^
cc1: all warnings being treated as errors

What I am doing wrong or how do I fix it? Thanks.

zimmerle · 2017-03-07T18:52:08Z

Hi @hernandanielg,

Please upgrade both: ModSecurity and ModSecurity-nginx connector

hernandanielg · 2017-03-07T19:14:05Z

Worked like a charm :) thanks!

jurgenweber · 2017-04-20T03:37:21Z

I am having this issue now; details here:

owasp-modsecurity/ModSecurity#1318 (comment)

any thoughts?

mimugmail mentioned this issue Feb 12, 2017

ModSecurity 3.0 Memory Leak ? owasp-modsecurity/ModSecurity#1318

Closed

zimmerle self-assigned this Feb 13, 2017

zimmerle closed this as completed Mar 7, 2017

TaiSHiNet mentioned this issue Oct 2, 2017

High memory consumption while using ModSecurity AnoopAlias/AUTOM8N#284

Closed

KfirManSnunit mentioned this issue Nov 18, 2018

segfault at 0 ip 00007fdb9bdfa746 sp 00007fff7c0475f8 error 4 in libc-2.23.so #136

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself) #29

Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself) #29

md2k commented Jan 13, 2017

defanator commented Jan 31, 2017

mimugmail commented Feb 2, 2017

mimugmail commented Feb 3, 2017

md2k commented Feb 3, 2017

defanator commented Feb 3, 2017

defanator commented Feb 3, 2017

md2k commented Feb 3, 2017 •

edited

Loading

mimugmail commented Feb 3, 2017

defanator commented Feb 3, 2017

md2k commented Feb 3, 2017

mimugmail commented Feb 4, 2017

mimugmail commented Feb 4, 2017

mimugmail commented Feb 4, 2017

defanator commented Feb 6, 2017

mimugmail commented Feb 6, 2017

md2k commented Feb 7, 2017 •

edited

Loading

mimugmail commented Feb 8, 2017

md2k commented Feb 15, 2017 •

edited

Loading

defanator commented Feb 15, 2017

md2k commented Feb 15, 2017

zimmerle commented Feb 24, 2017 •

edited

Loading

mimugmail commented Feb 24, 2017

Menahem1 commented Feb 24, 2017 •

edited

Loading

defanator commented Feb 26, 2017

zimmerle commented Mar 7, 2017

hernandanielg commented Mar 7, 2017 •

edited

Loading

zimmerle commented Mar 7, 2017

hernandanielg commented Mar 7, 2017

jurgenweber commented Apr 20, 2017

Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself) #29

Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself) #29

Comments

md2k commented Jan 13, 2017

defanator commented Jan 31, 2017

mimugmail commented Feb 2, 2017

mimugmail commented Feb 3, 2017

md2k commented Feb 3, 2017

defanator commented Feb 3, 2017

defanator commented Feb 3, 2017

md2k commented Feb 3, 2017 • edited Loading

mimugmail commented Feb 3, 2017

defanator commented Feb 3, 2017

md2k commented Feb 3, 2017

mimugmail commented Feb 4, 2017

configuration file /opt/nginx//conf/nginx.conf:

configuration file /opt/nginx//conf/mime.types:

mimugmail commented Feb 4, 2017

mimugmail commented Feb 4, 2017

defanator commented Feb 6, 2017

mimugmail commented Feb 6, 2017

md2k commented Feb 7, 2017 • edited Loading

mimugmail commented Feb 8, 2017

md2k commented Feb 15, 2017 • edited Loading

defanator commented Feb 15, 2017

md2k commented Feb 15, 2017

zimmerle commented Feb 24, 2017 • edited Loading

mimugmail commented Feb 24, 2017

Menahem1 commented Feb 24, 2017 • edited Loading

defanator commented Feb 26, 2017

zimmerle commented Mar 7, 2017

hernandanielg commented Mar 7, 2017 • edited Loading

zimmerle commented Mar 7, 2017

hernandanielg commented Mar 7, 2017

jurgenweber commented Apr 20, 2017

md2k commented Feb 3, 2017 •

edited

Loading

md2k commented Feb 7, 2017 •

edited

Loading

md2k commented Feb 15, 2017 •

edited

Loading

zimmerle commented Feb 24, 2017 •

edited

Loading

Menahem1 commented Feb 24, 2017 •

edited

Loading

hernandanielg commented Mar 7, 2017 •

edited

Loading