nginx reverse proxy strips response headers from backend

[pwyliu]

In a set up like this:

(client) <--> (nginx+modsec reverse proxy) <--> (backend web server)

If the backend server adds a new response header, modsec doesn't pass it to the client.

This happens even if all the rules are disabled and SecRuleEngine Off is set. It seems anything new that is added by the backend will get dropped.

We noticed this because we were trying to byte serve a PDF from the backend. Chrome uses range requests, but modsec drops the Content-Range header from the response, so it makes the client download the file forever.

I set up a super tiny test environment with nginx 1.6.0 and modsec 2.8.0 and could recreate the issue there. Here is a gist with some logs and the configs. We also saw this in another similar set up which was nginx 1.4.4 and modsec 2.7.7.

I hope I didn't miss anything, please let me know if any more info is needed.

[rmongia]

I am facing the same issue. The culprit commit seems to be this one: 177b5b9.

[scaarup]

I am also facing this problem. We are adding a custom header to the response on the backend-servers. But this never returns to the client.

[scaarup]

And one important header "Content-Type" is not passed - this breaks our applications...

[scaarup]

@zimmerle I have tried your fix, but with no luck...

[rmongia]

@scaarup, can you tell me the response headers with and without ModSecurity in play.

[scaarup]

@zimmerle, yes here they are, first with modsecurity and then without. It is actually only content-type which is missing. X-Server-Name and Jokum are custom header fields provided by the apache upstream. So something is working.

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 06 Aug 2014 06:06:42 GMT
Connection: keep-alive
Set-Cookie: PMLCID=192%2E168%2E11%2E125%2E30211140730519; expires=Fri, 05-Sep-14 06:06:33 GMT;path=/; secure; HttpOnly
Set-Cookie: PMLSID=192%2E168%2E11%2E125%2E30211140730519; path=/; secure; HttpOnly
X-Server-Name: server11
Jokum: server11
Content-Length: 559148

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 06 Aug 2014 06:07:19 GMT
Content-Type: text/html
Connection: keep-alive
Set-Cookie: PMLCID=192%2E168%2E11%2E125%2E302531407305239; expires=Fri, 05-Sep-14 06:07:19 GMT;path=/; secure; HttpOnly
Set-Cookie: PMLSID=192%2E168%2E11%2E125%2E302531407305239; path=/; secure; HttpOnly
X-Server-Name: server11
Jokum: server11
Content-Length: 559148

[ZaleskiR]

I found inside of ngx_http_modsecurity.c, ngx_http_clean_header(r) is called, which deletes all of the output headers. Not sure why this is done, but commenting it out fixes the problem.

static ngx_inline ngx_int_t
ngx_http_modsecurity_save_headers_out(ngx_http_request_t *r)
{
ngx_http_modsecurity_ctx_t *ctx;
ngx_http_upstream_t *upstream;

ctx = ngx_http_get_module_ctx(r, ngx_http_modsecurity);

/* r->chunked = ctx->req->chunked; */

// WHY ARE HEADERS OUT BEING DELETED?
//// ngx_http_clean_header(r);

upstream = r->upstream;
r->upstream = &ngx_http_modsecurity_upstream;

[rmongia]

@scaarup, the fix was for ModSecurity + nginx. It won't impact your setup.

@ZaleskiR, you can try the patch for ModSecurity Nginx module provided here to solve response headers problem. Commenting the ngx_http_clean_header() will work fine with DetectionOnly mode but with ModSecurity Engine ON, it would require this code to be there to sanitize the response.

[scaarup]

@rmongia I am using ModSecurity + nginx...

[rmongia]

that's interesting. i have tested the scenario with my setup and it seems to work. my setup is a simple recommended configuration with a few custom rules. i am not using OWASP rule set here.

My guess is there is some rule which is doing this. You can investigate further by disabling all rules and running the same scenario again.

[zimmerle]

Fixed by: #749.

[davidgenn]

Hi @zimmerle Has this been included in a release?

I've just compiled the latest version of Nginx (1.11.6) with Modsecurity (2.9.1). Nginx is acting as a reverse proxy for a Jetty server. I'm using the default OWASP rules (3.0.0).

The Java app is adding headers to the response which are being stripped out by Modsecurity. It was fine before Modsecurity was added.

Thanks in advance!

[zimmerle]

Hi @davidgenn,

Please use the `nginx_refactoring' branch. Or the ModSecurity-nginx connector:
https://github.com/SpiderLabs/ModSecurity-nginx