ModSecurity: collections_remove_stale: Failed deleting collection

[rcbarnett-zz]

MODSEC-428: When server is under load of 1500 concurrent users Modsecurity starts to show the below error in apache's error log and the server CPU usage goes up to dangerous levels.

[Wed Oct 02 21:21:52 2013] [error] [client 109.127.81.10] ModSecurity: collections_remove_stale: Failed deleting collection (name "ip", key "109.127.81.138_9e4f93d096e2ca3744251c41dde47a1a7b26fa75"): Internal error [hostname "www.regayzanko.com"] [uri "/php5-fcgi/setting/student_collage_list.php"] [unique_id "UkxkMMaIMEoAADg9zw0AAAgr"]

I can see that this issue was patched see: https://www.modsecurity.org/tracker/browse/MODSEC-97 but i am using the latest modsecurity version "2.7.5" so i do not know why it is happening.

[rcbarnett-zz]

Original reporter: zertux

[rcbarnett-zz]

zertux: Can this process be disabled ? or is it a necessity for mod_Security's operation ?

[kurktchiev]

I am having the exact same issue here on 2.7.5

[wneessen]

Seeing similar behaviour on FreeBSD 8.4, Apache/2.2.25, mod_sec 2.7.5.
Machine is running fine for a day or two (even just in DetectionOnly mode). Then all of a sudden, the CPU load increases extremely and Apache-Server status shows lots of "L"s. At some point all slots are exhaused and Apache will be unresponsive. Can be fixed by restarting apache, but it will happen again after some time.

mod_sec debug.log shows a couple of these:
[15/Nov/2013:21:10:30 +0000] [www.somehost.com/sid#805894478][rid#809e3a0a0][/some/uri/excel_2010_advanced_popup.jpg][1] collection_retrieve_ex: Failed deleting collection (name "ip", key "123.123.123.123_6e1c38356769095ad76267f9edb43ddae2af1491"): Internal error
[15/Nov/2013:21:13:10 +0000] [www.somehost.com/sid#80178fdb0][rid#80924e0a0][/some/uri/cookie.htm][1] collection_retrieve_ex: Failed deleting collection (name "ip", key "122.122.122.122_81548c7486a53bd388be273db20199283dfac871"): Internal error

[wneessen]

Hi, is there any update on this issue? Let me know if I can provide any further information.

[marcstern]

Same problem with 2.7.6 :-(
Can we expect a fix

[klydel]

same issue. is there a particular way to disable this behavior altogether?

[ekenberg]

Same issue 2.7.7, any news?

[oparoz]

Soon after that error message, we get a few "Failed reading DBM file" and Apache is unreachable until the process is killed.

[oparoz]

We're using ZFS on FreeBSD. How about the other people affected by this bug?

[marcstern]

Same problem with Redhat, CentOS & Windows

[oparoz]

Thanks!

[lifeforms]

I think we are having the same issue on ModSecurity 2.7.7 on FreeBSD 9.2 UFS with a reasonably busy server (but not SUPER busy, e.g. 100 httpd clients but lots of heavy database work).

A few times a week, "something" happens that makes all Apache processes seem to spin on disk activity. Disk I/O by Apache shoots up to crazy levels (100MB/s) with CPU time spent mostly in system. In Apache status, children get stuck in "Logging" (L) state. It stays that way for 10-20 minutes, then calms down.

I narrowed it down to the user of collections. The problem completely goes away when I disable an anti-bruteforce rule which stores a block flag/timer in a collection. When I disable this rule, the problem is immediately gone. When I kill Apache and re-enable the rule, the problem is back.

When looking in modsec_audit.log, I see all kinds of "Internal errors" such as the following:

Message: collections_remove_stale: Failed deleting collection (name "ip", key "198.167.x.x_x"): Internal error

I now think that this condition probably happens when some clients hit the collection concurrently. The issue has become more problematic since switching to a twin-site SAN (I imagine that it takes longer before finishing a sync write because of the speed of light), so it might depend on certain I/O latency to happen.

The fact that the issue always disappears by itself after a number of minutes (but NOT by restarting Apache!), leads me to think that probably it ends by some collection item being deprecated. I have no reliable way to reproduce the problem unfortunately, besides using collections in production and waiting until the site is crippled.

[antoxxa]

stil no answer ?