Nginx with Modsecurity: POST request gives 500 error

[pijnack]

Nginx with Modsecurity gives 500 errors after a POST request.

In error.log:

2013/10/23 13:21:07 [alert] 29432#0: *4 no upstream configuration, client: 88.159.16.136, server: test.huygenscollege.nl, request: "POST /proefwerk/prog/index.php?day=28&month=10&year=2013&klas=2 HTTP/1.1", host: "test.huygenscollege.nl", referrer: "http://test.huygenscollege.nl/proefwerk/prog/index.php?day=4&month=11&year=2013&klas=2"

In nginx.conf:

upstream smoothrac {
server 88.159.13.153;
}
server {
listen 80;
server_name test.huygenscollege.nl;
location / {
ModSecurityEnabled on;
ModSecurityConfig modsecurity.conf;
proxy_pass http://smoothrac/;
proxy_set_header Host test.huygenscollege.nl;
proxy_read_timeout 150s;
}
}

After changing ModSecurityEnabled on to off, the problem is gone.
I guess something is missing in my upstream configuration?

Piet Pijnacker,
Eindhoven, Netherlands

[zimmerle]

Hi @pijnack, I am reopening this issue, I'm investigating it.

[zimmerle]

Wordpress set multiple cookies in the very same request, which is a issue in our Nginx implementation as report at #154. It is not directly related to this issue, but it is somehow related in the sense that it does affect the behavior of Wordpress.

[yumahil]

I also had two issues regarding "POST request 500 error" and only one set-cookie.
The Fix #154 solved only one set-cookie problem. But "POST request 500 error" still remains.

[jijojv]

Same issue here , no upstream or proxy setting used. just nginx php-fpm.

Even more weird is this only triggers when 'Host: ' header is set - using browser or curl etc
So using this http://wiki.nginx.org/HttpHeadersMoreModule - "more_clear_input_headers 'Host';" is one workaround for some folks where the application doesn't need the host headers.

[shiguti]

I've got the same issue, any ideas?

[infinitnet]

Same issue for me with IPB. Latest NGINX and ModSecurity version and ASL rules. If SecRequestBodyAccess is enabled, it will give a 500 error with POST requests while writing nothing useful on the ModSecurity logs, even with debug level 9 and in the NGINX logs it only produces the lines mention in OP. This issue was reported in #115 10 months ago and nobody took care of it? I can provide server access to one of the devs if no one is able to reproduce it - it's a non-production environment.

[zimmerle]

Hi @infinitnet, there are some problems with the way that ModSecurity is handling the post requests on nginx. If you look at #142 you will be able to see that contents over 8k are causing problems as well. Our version 2.8.0-RC1 contain a fix related to the issue #645. But we still have problems while the request body is bigger than a certain amount of data. The main reason for the problem is that ModSecrurity is not assembling Nginx chunks right, the function is here:

https://github.com/SpiderLabs/ModSecurity/blob/master/nginx/modsecurity/ngx_http_modsecurity.c#L1011-L1020

In certain cases (e.g. when the request body content is not complete) this function should return NGX_AGAIN, so that nginx will call it again with the rest of the body data.

[infinitnet]

@zimmerle I'm actually using 2.8.0-RC1 already. I didn't check the POST data size, but the issue appears even with simple login forms, rendering ModSecurity pretty much unusable with NGINX.

[10/Apr/2014:16:19:03 +0200] "POST /index.php?app=core&module=global&section=login&do=process HTTP/1.1" 500

[alert] 19480#0: *4 no upstream configuration, client: 1.2.3.4, server: domain.com, request: "POST /index.php?app=core&module=global&section=login&do=process HTTP/1.1"

And nothing useful in the audit or debug log.

[infinitnet]

@zimmerle I should have checked dmesg earlier, actually, it's indeed somehow related to APR (my version of libapr1(-dev) is 1.4.6-3+deb7u1), as I'm getting segfaults for each NGINX worker when starting NGINX:

Apr 10 17:01:13 a143 kernel: [787456.369365] nginx[19482]: segfault at 7fe879c61098 ip 00007fe88484792d sp 00007fff04fc4b90 error 4 in libapr-1.so.0.4.6[7fe88482d000+30000]
Apr 10 17:01:13 a143 kernel: [787456.374651] nginx[19480]: segfault at 7fe879c61098 ip 00007fe88484792d sp 00007fff04fc4b90 error 4 in libapr-1.so.0.4.6[7fe88482d000+30000]
Apr 10 17:01:13 a143 kernel: [787456.375342] nginx[19481]: segfault at 7fe879c61098 ip 00007fe88484792d sp 00007fff04fc4b90 error 4 in libapr-1.so.0.4.6[7fe88482d000+30000]
Apr 10 17:01:13 a143 kernel: [787456.377620] nginx[19483]: segfault at 7fe879c61098 ip 00007fe88484792d sp 00007fff04fc4b90 error 4 in libapr-1.so.0.4.6[7fe88482d000+30000]

[zimmerle]

@infinitnet This happens when you start nginx or whenever you receive a new request?

[infinitnet]

@zimmerle Updated my comment, it happens for each NGINX worker process when starting NGINX, not with each request and also not when I get the 500 error, only when starting it.

[zimmerle]

it just happens if SecRequestBodyAccess is marked as on ?

Can you run into inside gdb and share a stack trace?
https://github.com/SpiderLabs/ModSecurity/wiki/Debugging-ModSecurity

Thanks.

[infinitnet]

@zimmerle Actually it happens also with SecRequestBodyAccess Off, so I'm not sure if the segfault issue is even related. Even though I compiled NGINX --with-debug and CFlags -g -O0 for both NGINX and mod_sec and both daemon and master_process off in my NGINX config, I can't get any useful info (stack trace) with GDB. If you could add me on Skype (infinitnet) or let me know an IRC channel, I can provide you access. Here's a normal strace if that helps: http://pastebin.com/3DcEni6h

[brennantom]

helpful

[infinitnet]

Update: We figured that the 500 error with POST requests is related to the missing NGX_AGAIN and a fix is in progress apparently. Maybe this statement should be put in all the bug reports related to this, like #115 #142 #645 etc.

Edit: The segfault is not related to ModSecurity.

[infinitnet]

Is there any ETA on the error 500 issue when processing POST requests yet (NGX_AGAIN)?

[infinitnet]

This was reported 6 months ago and makes a huge part of ModSecurity useless with NGINX and yet it doesn't have anyone assigned to fix it or anything. I'd very much like to know about the progress on this.

[zimmerle]

Hi @pijnack large POSTs with SecRequestBodyAccess On, should be functional in the following branch:

https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

Can you test and confirm it?

Thanks,
F.

[badcrc]

Hello,

I'm having this problem too, my scenario:

• FreeBSD 9.1
• nginx 1.5.6 & nginx 1.7.0
• modsecurity: 2.7.7 & 2.8.0

I've tried the nginx_refactoring tree (I just switched the "nginx" folder within modsecurity) with nginx 1.7.0 and it doesn't even give error 500, it core dumped, here is a gdb run of a core:

http://pastebin.com/winjWMY7

[badcrc]

Hi again,

I'm trying a clean, hand-compiled (i was using ports before) version of nginx 1.7.1 against nginx_refactoring and I'm still getting "no upstream configuration".

 # ./nginx -V 
nginx version: nginx/1.7.1
built by gcc 4.2.1 20070831 patched [FreeBSD]
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx-error.log --user=www --group=www --with-ipv6 --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx-access.log --add-module=/expert/software/agentzh-headers-more-nginx-module-0c6e05d --with-http_geoip_module --with-http_stub_status_module --add-module=/expert/software/ModSecurity-nginx_refactoring/nginx/modsecurity --with-pcre --with-http_ssl_module

And I've compiled ModSecurity with:

--enable-standalone-module --disable-apache2-module"

I get the error with and without an Upstream configured:

location {
        ModSecurityEnabled on;
        ModSecurityConfig modsecurity/web-modsecurity.conf;

        proxy_pass        http://AAA.BBB.CCC.DDD:80;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Protocol $scheme;

        proxy_set_header Host $http_host;

    } 

upstream my_server{
    server AAA.BBB.CCC.DDD:80;
}

location {
        ModSecurityEnabled on;
        ModSecurityConfig modsecurity/web-modsecurity.conf;

        proxy_pass        http://myserver;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Protocol $scheme;

        proxy_set_header Host $http_host;

    } 

[infinitnet]

Same for me, issue still exists in nginx_refactoring branch.

[arianf]

@zimmerle
Is there a target date for version 3, libmodsec?

[arianf]

One of the most frustrating things about this... is the websites states that the quality of this release is stable.

ModSecurity for Nginx (Stable Release Quality)

With an issue as big as this, I wouldn't consider this a "Stable Release Quality"

[csanders-git]

@arianf I have updated the website as both @zimmerle and I thought that your complaint was valid. Please see the updated site, we apologize for the confusion: http://modsecurity.org/download.html

[chewi]

@ton31337, since it isn't clear when this whole issue will be properly resolved, would it be possible for you to rebase your branch on the latest nginx_refactoring so that we can benefit from other fixes that have been added in the meantime? This would include the other >8K fix. Thanks.

[zimmerle]

@chewi did you tested the latest version of the nginx_refactoring branch ?

[chewi]

@zimmerle, I did but the issue remains. I can only assume it is @ton31337's fix that is needed in this case.

[zimmerle]

@chewi that is interesting because @ton31337's commit was partially merged in the nginx_refactoring branch.

[chewi]

Hmmm. I don't think I went wrong anywhere but I'll double check.

[cukal]

Hi, I'm running into the same issue.Used latest nginx_refactoring and nginx 1.9.14
Which branch should I use to stop this issue from occurring?

Thanks!

2016/04/25 12:46:19 [alert] 25240#0: *1 no upstream configuration, client: xx.xx.xx.xx, server: xxxx.xxx, request: "POST /owncloud/index.php HTTP/1.1", host: "xxx.xx.xx"

[chewi]

I never got around to double checking though I am due to look at this again soon. I'm now quite confident that I didn't mess up if you're still seeing it.

[cukal]

Okay & Thanks!
Branch ton31337_x-headers still works btw.

[chewi]

Okay, I have now verified this in a different environment. The nginx_refactoring branch as of 207c85d does not fix the issue. I did not want to use @ton31337's branch as it stands because it's very old now. I rebased nginx_refactoring on v2.9.1 but skipped d135f88 so that I could apply 9abda55 instead. This fixed the issue. In short, @zimmerle's version of the fix does not work. You can see the result of my merge at yakara-ltd/ModSecurity@f067515. If it helps, we are seeing this triggered by Apache commons-httpclient 3.0.1 against nginx 1.9.10. I know the former is really ancient but this still shouldn't happen.

[chewi]

Unfortunately we had to disable ModSec again after deploying this because requests were randomly having their headers mangled. I don't know the details yet but Rails was occasionally picking up the requested domain as localhost instead of the one that was really requested.

[LoadLow]

Hi,
I know, it's too late ... but I've (perhaps?) found a fix.

According to: http://lxr.nginx.org/source/src/http/ngx_http_request.c#1611
server and host should be set, but on the module's context, some properties are not correctly set and it sends an internal error (500 err occurs on the "Host" header's handler).

We can easily set server and host properties in header_in because they're already checked by the core module.
A workaround could be : https://gist.github.com/LoadLow/1532ade09e673b69e2dbf8535fae6cc3/revisions

I've tested it with OWASP rules and the current stable version of Nginx and it works properly.

Thanks for your consideration,
LoadLow.

[zimmerle]

Hi @LoadLow,

This fix is somewhat implemented in the nginx_refactoring branch, however we advice to use the ModSecurity-nginx connector instead of this implementation.

For further information on the ModSecurity-nginx connector, check here:
https://github.com/SpiderLabs/ModSecurity-nginx

[jinri]

@ton31337 l use this branch: ton31337_x-headers but get 404 ..
nginx-1.8.1 nginx-1.10.3 ModSecurity_ton31337 nginx_refactoring .. all had try but no effect
I use the ubuntu 12.04 . I have been concerned about this problem for a long time and
Try all of the above.Crashing in ...

nginx version: nginx/1.8.1
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --prefix=/usr/local/bluedon/bdwaf --add-module=/root/new/nginx-module-vts --add-module=/root/new/modsecurity/nginx/modsecurity --add-module=/root/new/substi --add-module=/root/new/ngx_cache_purge-2.3 --add-module=/root/new/mod_strip --with-http_ssl_module --with-http_perl_module --with-http_sub_module --with-http_geoip_module

[zimmerle]

@jinri Did you ever tried the ModSecurity v3 with nginx? -
https://github.com/SpiderLabs/ModSecurity-nginx

[jinri]

@zimmerle
I have tried V3.0 master version, it solved the post 500 server error, but there are some configurations can not be achieved (or I do not know how to configure the V3.0).
eg: in modsecurity_rules' SecRule TX:test_block "@ge" log, deny, phase:2, id:1211111, msg:'testing, block_rule=%{tx.test_id}', redirect:http://www.baidu.com, ruleid=%{tx.test_id} " '
Moreover, in the production environment, we do not dare to use V3.0 for a while until it is stable.
ModSecurity_ton31337, nginx_refactoring,, nginx_refactoring does not work, ModSecurity_ton31337 returns 404,there are several other versions I have tried too but no effect ether.
Are there any other versions available?
Look forward to your reply, thank you very much！

[zimmerle]

No longer a concern in libModSecurity. Marking it as won't fix for 2.x. Further information about libModSecurity available here:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master