"Database is not open. Use: SecGeoLookupDb directive." after reload

@Within

Describe the bug

I'm using mmdb to block countries from my website and it works fine until I reload my nginx server and I start getting the message "Database is not open. Use: SecGeoLookupDb directive." in the debug log and no country would be blocked anymore.

Logs and dumps

Before reload:

[...] [/] [4] (Rule: 20) Executing operator "GeoLookup against REMOTE_ADDR.
[...] [/] [9] Target value: "111.111.111.111" (Variable: REMOTE_ADDR)
[...] [/] [9] Matched vars updated.
[...] [/] [9] Saving msg: Block China IP address
[...] [/] [4] Rule returned 1.
[...] [/] [4] Executing chained rule.
[...] [/] [4] (Rule: 0) Executing operator "Within" with param "CN CL" Was: "" against GEO:COUNTRY_CODE.
[...] [/] [9] Target value: "CN" (Variable: GEO:COUNTRY_CODE)
[...] [/] [9] Matched vars updated.
[...] [/] [4] Rule returned 1.

After reload:

[...] [...] [4] (Rule: 20) Executing operator "GeoLookup against REMOTE_ADDR.
[...] [...] [9] Target value: "111.111.111.111" (Variable: REMOTE_ADDR)
[...] [...] [4] Database is not open. Use: SecGeoLookupDb directive.
[...] [...] [4] Rule returned 0.

ModSecurity config log:

ModSecurity - v3.0.3-56-g6d5198b1 for Linux
Mandatory dependencies

libInjection ....v3.0.3-56-g6d5198b1

SecLang tests ....6d5198b

Optional dependencies

GeoIP/MaxMind ....found

(MaxMind) v1.3.2
-lmaxminddb, -DWITH_MAXMIND -I/usr/local/include

(GeoIP) v1.6.12
-lGeoIP, -I/usr/include/

LibCURL ....found v7.58.0
-lcurl, -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL

YAJL ....found v2.1.0
-lyajl, -DWITH_YAJL -I/usr/include/yajl

LMDB ....not found

LibXML2 ....found v2.9.4
-lxml2, -I/usr/include/libxml2 -DWITH_LIBXML2

SSDEEP ....not found

LUA ....found v501
-llua5.1 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_1 -I/usr/include/lua5.1

Other Options

Test Utilities ....enabled

SecDebugLog ....enabled

afl fuzzer ....disabled

library examples ....enabled

Building parser ....disabled

Treating pm operations as critical section ....disabled

To Reproduce

Use the following configuration with your own country code and restart nginx.

SecGeoLookupDb /mnt/nginx/defaults/geoip/geoip.mmdb

SecAction
"id:19,
phase:1,
nolog,
pass,
t:none,
setvar:'tx.high_risk_country_codes=CN CL'"
SecRule REMOTE_ADDR "@geoLookup" "chain,id:20,drop,msg:'Block China IP address',phase:2"
SecRule GEO:COUNTRY_CODE "@Within %{tx.high_risk_country_codes}"

Test with your own country code and you should get a 403 forbidden error code, then remove your country from the collection, reload and try again, you should receive 200. Add your country code again to the collection and you should get a 200 status and the previous reported message in the logs as the geoip database was unloaded.
Or you can just create a disruptive rule with geoip and after testing it works, you can check if it works after a reload.

Expected behavior

Allow me to reload the geoip database without restarting the server.

Server:

ModSecurity version (and connector): ModSecurity master (6d5198b), Connector master (d7101e1)
WebServer: 1.15.19
OS: 4.15.0-1021-aws Fixed: nginx INCS config on fedora #21-Ubuntu x86_64
libmaxminddb: 1.3.2 (from source)
Nginx flags: ./configure
--with-http_ssl_module
--add-module=/tmp/ModSecurity-nginx/
--with-http_ssl_module
--with-http_sub_module
--with-http_realip_module
--with-http_sub_module
--with-http_gunzip_module
--with-http_gzip_static_module

More info

It seems like Utils::GeoLookup::getInstance().cleanUp(); is being called and Utils::GeoLookup::setDataBase is not being called again, I'm trying to find out the problem over there.

Small workaround

You can "reload" nginx by renewing the process:

create new master and workers

pid=$(cat /usr/local/nginx/logs/nginx.pid)
sudo kill -USR2 $pid

gracefully kill the old workers

sudo kill -WINCH $pid

gracefully kill the old master

sudo kill -QUIT $pid

That would reload the whole engine and nginx without any downtime but using many resources.
Benchmarks for 1000 reloads and winchs in a 8 core 16gb memory server with just 100 rules:

#https://gist.github.com/jptosso/aa72ff932d3fad39614923bccb2d175a
ubuntu@server:~$ sudo bash /etc/test.sh
winch: 17 seconds
reload: 35 seconds

I believe the issue is probably due to the fact that the call to GeoLookup::setDataBase is made only at the parser here: https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/parser/seclang-parser.cc#L2706-L2723

The leads me to believe that on a reload, when GeoLookup::evaluate eventually is called, GeoLookup::lookup is called directly without any prior call to GeoLookup::setDataBase. We might be missing an init() at the geo_lookup.cc operator.

That being said, I could only reproduce this bug when I had BOTH MaxMind and the legacy GeoIP libraries/db installed.
So I believe that a valid temporary workaround is having only one of the libraries.

Tested your workaround and it does work:

ModSecurity - v3.0.3-56-g6d5198b1 for Linux

Mandatory dependencies

libInjection ....v3.0.3-56-g6d5198b1

SecLang tests ....6d5198b

Optional dependencies

GeoIP/MaxMind ....found

(MaxMind) v1.3.2
-lmaxminddb, -DWITH_MAXMIND -I/usr/local/include

LibCURL ....found v7.58.0
-lcurl, -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL

YAJL ....found v2.1.0
-lyajl, -DWITH_YAJL -I/usr/include/yajl

LMDB ....not found

LibXML2 ....found v2.9.4
-lxml2, -I/usr/include/libxml2 -DWITH_LIBXML2

SSDEEP ....not found

LUA ....found v502
-lluajit-5.1 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_2 -I/usr/include/lua5.2

Thanks for the feedback @asdfuken. Good to know that it works as intended this way :)

Regardless, we'll try to get it fixed prior to 3.0.4 release. Thanks for reporting!

@jptosso

This is still an issue for us, we're on e9dce44.

Sorry what exactly is the workaround @jptosso ?

Avoids to cleanup GeoIp on ModSecurity destructor

Fix under QA here - https://travis-ci.org/github/SpiderLabs/ModSecurity/builds/743954613?utm_source=github_status&utm_medium=notification

Avoids to cleanup GeoIp on ModSecurity destructor

From the looks of this ticket, this fix has not yet made it to libmodsecurity? It's still planned for 3.1.0? Is there an estimate when it will become part of a release?

@davand01

Hello @davand01 ,

The SecGeoLookupDb configuration item does work in ModSecurity v3. I have used it myself moderately recently with:

SecGeoLookupDB /usr/share/GeoIP/GeoLite2-Country.mmdb

@martinhsv

@martinhsv - it works for me too, however if there is a reload (not restart) of nginx this exact error persists.
This means that for any potential change in the configuration, we need to restart our pods running nginx-ingress w/ModSecurity enabled. With this fix, it seems as if we don't have the need for that workaround.

@davand01

Hello @davand01 ,

I do not see this effect when performing an nginx reload either. However, reading through this closed ticket, I'll note that I only have MaxMind installed.

Per the comments from 2019, it looks like this issue was only reproducible if "BOTH MaxMind and the legacy GeoIP libraries/db installed".

Is that the case for you?

Hello, I am running Nginx with modsecurity which has both "MaxMind and the legacy GeoIP libraries/db installed". I can still notice the bug that when I perform an nginx -s "reload", it no longer works. I would have to restart the container (docker) running the engine for it to work!

Has this bug been resolved?

Hello, I am running Nginx with modsecurity which has both "MaxMind and the legacy GeoIP libraries/db installed". I can still notice the bug that when I perform an nginx -s "reload", it no longer works. I would have to restart the container (docker) running the engine for it to work!

Has this bug been resolved?

I'm afraid it hasn't. But I think it's rather a connector issue than libmodsecurity3 - we have to investigate it.

Thanks for this reminder!

Would it be okay for me to raise this as a separate issue, or will the team be reopening the existing issue?

I reopen this issue - thanks!

Hi all!
Decided to write in an existing thread. I've gone back and forth a few times trying to get geoLookup running on ModSecurity and each time I've encountered some issues. This time I decided to describe in detail the course of my installation, perhaps you can help me figure out the problem.

My environment: Debian, nginx/1.26.3, modsecurity: ModSecurity v3.0.14 (Linux), connector: ModSecurity-nginx v1.0.3.

Output from configure:

ModSecurity -  for Linux
 
 Mandatory dependencies
   + libInjection                                  ....
   + Mbed TLS                                      ....
   + SecLang tests                                 ....
 
 Optional dependencies
   + GeoIP/MaxMind                                 ....found 
      * (GeoIP) v1.6.12
         -lGeoIP , -I/usr/include/ 
   + LibCURL                                       ....found v8.13.0 
      -lcurl,  -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
   + YAJL                                          ....found v2.1.0
      -lyajl , -DWITH_YAJL  -I/usr/include
   + LMDB                                          ....not found
   + LibXML2                                       ....found v2.9.14
      -lxml2 , -I/usr/include/libxml2  -DWITH_LIBXML2
   + SSDEEP                                        ....found 
      -lfuzzy -L/usr/lib/x86_64-linux-gnu/, -DWITH_SSDEEP -I/usr/include
   + LUA                                           ....found v503
      -llua5.3 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_3 -I/usr/include/lua5.3
   + PCRE                                          ....found 
     using pcre v
      , 
   + PCRE2                                          ....found v10.45
      -lpcre2-8 , 
 
 Other Options
   + Test Utilities                                ....enabled
   + Assertions                                    ....disabled
   + SecDebugLog                                   ....enabled
   + afl fuzzer                                    ....disabled
   + library examples                              ....enabled
   + Building parser                               ....disabled
   + Treating pm operations as critical section    ....disabled

I know about the problems:

That being said, I could only reproduce this bug when I had BOTH MaxMind and the legacy GeoIP libraries/db installed.
So I believe that a valid temporary workaround is having only one of the libraries.

Therefore, to avoid conflicts between MaxMind and GeoIP, I did not install MaxMind, leaving only GeoIP.
Next, I added the required configuration with SecGeoLookupDb and geoLookup (xanadu suggested it to me at the time):

SecGeoLookupDb /etc/somestuff/GeoLite2-City.mmdb

SecRule REMOTE_ADDR "@geoLookup" \
    "id:1000,\
    phase:1,\
    pass,\
    t:none,\
    nolog"

# Check for failed geo lookup
SecRule &GEO "@eq 0" \
    "id:1010,\
    phase:1,\
    pass,\
    t:none,\
    log,\
    msg:'geoLookup operator was unable to return a result'"

SecAction \
    "id:1020,\
    phase:1,\
    pass,\
    log,\
    msg:'REMOTE_ADDR: %{REMOTE_ADDR}, COUNTRY_NAME: %{GEO.COUNTRY_NAME}, COUNTRY_CODE: %{GEO.COUNTRY_CODE}, COUNTRY_CONTINENT: %{GEO.COUNTRY_CONTINENT}'"

I restart Nginx via restart and try to send requests. I get the output in the log:

"ruleId": "1010" 
"match": "Matched \"Operator `Eq' with parameter `0' against variable `GEO' (Value: `0' )",
"message": "geoLookup operator was unable to return a result"

"ruleId": "1020",
"message": "REMOTE_ADDR: *.*.*.*, COUNTRY_NAME: , COUNTRY_CODE: , COUNTRY_CONTINENT: "

I tried with different IP addresses, checking the actual presence of the IP address in GeoLite2-City.mmdb via mmdblookup:

mmdblookup --file GeoLite2-City.mmdb --ip *.*.*.*
*make sure that the IP address is actually present in GeoLite2-City.mmdb*

During the experiments I decided to install MaxMind and try to check with MaxMind and GeoIP together:

ModSecurity -  for Linux
 
 Mandatory dependencies
   + libInjection                                  ....
   + Mbed TLS                                      ....
   + SecLang tests                                 ....
 
 Optional dependencies
   + GeoIP/MaxMind                                 ....found 
      * (MaxMind) v1.12.2
         -lmaxminddb , -DWITH_MAXMIND -I/usr/include/x86_64-linux-gnu 
      * (GeoIP) v1.6.12
         -lGeoIP , -I/usr/include/ 
   + LibCURL                                       ....found v8.13.0 
      -lcurl,  -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
   + YAJL                                          ....found v2.1.0
      -lyajl , -DWITH_YAJL  -I/usr/include
   + LMDB                                          ....not found
   + LibXML2                                       ....found v2.9.14
      -lxml2 , -I/usr/include/libxml2  -DWITH_LIBXML2
   + SSDEEP                                        ....found 
      -lfuzzy -L/usr/lib/x86_64-linux-gnu/, -DWITH_SSDEEP -I/usr/include
   + LUA                                           ....found v503
      -llua5.3 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_3 -I/usr/include/lua5.3
   + PCRE                                          ....found 
     using pcre v
      , 
   + PCRE2                                          ....found v10.45
      -lpcre2-8 , 
 
 Other Options
   + Test Utilities                                ....enabled
   + Assertions                                    ....disabled
   + SecDebugLog                                   ....enabled
   + afl fuzzer                                    ....disabled
   + library examples                              ....enabled
   + Building parser                               ....disabled
   + Treating pm operations as critical section    ....disabled

The result of checking the log is also negative.

If I configure ModSecurity without geoip (./configure --with-geoip=no)

ModSecurity -  for Linux
 
 Mandatory dependencies
   + libInjection                                  ....
   + Mbed TLS                                      ....
   + SecLang tests                                 ....
 
 Optional dependencies
   + GeoIP/MaxMind                                 ....found 
      * (MaxMind) v1.12.2
         -lmaxminddb , -DWITH_MAXMIND -I/usr/include/x86_64-linux-gnu 
   + LibCURL                                       ....found v8.13.0 
      -lcurl,  -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
   + YAJL                                          ....found v2.1.0
      -lyajl , -DWITH_YAJL  -I/usr/include
   + LMDB                                          ....not found
   + LibXML2                                       ....found v2.9.14
      -lxml2 , -I/usr/include/libxml2  -DWITH_LIBXML2
   + SSDEEP                                        ....found 
      -lfuzzy -L/usr/lib/x86_64-linux-gnu/, -DWITH_SSDEEP -I/usr/include
   + LUA                                           ....found v503
      -llua5.3 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_3 -I/usr/include/lua5.3
   + PCRE                                          ....found 
     using pcre v
      , 
   + PCRE2                                          ....found v10.45
      -lpcre2-8 , 
 
 Other Options
   + Test Utilities                                ....enabled
   + Assertions                                    ....disabled
   + SecDebugLog                                   ....enabled
   + afl fuzzer                                    ....disabled
   + library examples                              ....enabled
   + Building parser                               ....disabled
   + Treating pm operations as critical section    ....disabled

I get errors at the make stage:

make[3]: Leaving directory '/usr/local/src/ModSecurity/src'
make[2]: Leaving directory '/usr/local/src/ModSecurity/src'
make[1]: Leaving directory '/usr/local/src/ModSecurity/src'
Making all in doc
make[1]: Entering directory '/usr/local/src/ModSecurity/doc'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/usr/local/src/ModSecurity/doc'
Making all in tools
make[1]: Entering directory '/usr/local/src/ModSecurity/tools'
Making all in rules-check
make[2]: Entering directory '/usr/local/src/ModSecurity/tools/rules-check'
/bin/bash ../../libtool  --tag=CXX   --mode=link g++ -std=c++17  -g -O2     -L/usr/lib/x86_64-linux-gnu/ -L/usr/lib/x86_64-linux-gnu/   -o modsec-rules-check modsec_rules_check-rules-check.o ../../src/.libs/libmodsecurity.la -lcurl  -lmaxminddb  -lrt   -lxml2   -llua5.3  -lpcre2-8  -lfuzzy -lyajl  
libtool: link: g++ -std=c++17 -g -O2 -o .libs/modsec-rules-check modsec_rules_check-rules-check.o  -L/usr/lib/x86_64-linux-gnu/ ../../src/.libs/libmodsecurity.so -lcurl -lmaxminddb -lrt -lxml2 -llua5.3 -lpcre2-8 -lfuzzy -lyajl -Wl,-rpath -Wl,/usr/local/modsecurity/lib
/usr/bin/ld: ../../src/.libs/libmodsecurity.so: undefined reference to `GeoIP_region_name_by_code'
/usr/bin/ld: ../../src/.libs/libmodsecurity.so: undefined reference to `GeoIP_record_by_name'
/usr/bin/ld: ../../src/.libs/libmodsecurity.so: undefined reference to `GeoIP_delete'
/usr/bin/ld: ../../src/.libs/libmodsecurity.so: undefined reference to `GeoIP_open'
/usr/bin/ld: ../../src/.libs/libmodsecurity.so: undefined reference to `GeoIPRecord_delete'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:567: modsec-rules-check] Error 1
make[2]: Leaving directory '/usr/local/src/ModSecurity/tools/rules-check'
make[1]: *** [Makefile:502: all-recursive] Error 1
make[1]: Leaving directory '/usr/local/src/ModSecurity/tools'
make: *** [Makefile:1068: all-recursive] Error 1

Am I doing something wrong?

Revisiting this one, with the root cause from the code. GeoLookup is a process-wide singleton (src/utils/geo_lookup.h). SecGeoLookupDb opens the database into it at parse time, and ModSecurity::~ModSecurity() calls GeoLookup::getInstance().cleanUp(), which closes whatever is open and resets the state to not loaded. On a reload the connector parses the new configuration (opening the database again) and then destroys the old ModSecurity instance, whose destructor closes the database the new configuration just opened. Every lookup afterwards reports "Database is not open".

Note the destructor call is inside #ifdef WITH_GEOIP, so builds with only libmaxminddb do not hit it; the reports here come from builds with both backends enabled. Fix: tie the database to the RulesSet that declared it (open and close with the rule set) instead of a singleton cleaned up by any instance, or at least reference-count opens. Marking as a confirmed bug.

Follow-up: open PR #3637 ("Fix memory leak when SecGeoLookupDb loads a database more than once") changes the same GeoLookup singleton; the reload fix described above should be coordinated with it so the two do not conflict.

fix: do not close the GeoLookup database from ~ModSecurity

Navigation Menu

Repository navigation

"Database is not open. Use: SecGeoLookupDb directive." after reload #2041

create new master and workers

gracefully kill the old workers

gracefully kill the old master

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

"Database is not open. Use: SecGeoLookupDb directive." after reload #2041

Description

create new master and workers

gracefully kill the old workers

gracefully kill the old master

Activity

victorhora commented on Mar 10, 2019

jptosso commented on Mar 10, 2019

victorhora commented on Mar 12, 2019

welljsjs commented on Sep 30, 2020

zimmerle commented on Nov 16, 2020

8 remaining items

davand01 commented on Dec 28, 2023

martinhsv commented on Dec 28, 2023

davand01 commented on Dec 28, 2023

martinhsv commented on Jan 2, 2024

Dr-Lazarus-V2 commented on Sep 23, 2024

airween commented on Sep 23, 2024

Dr-Lazarus-V2 commented on Sep 26, 2024

airween commented on Sep 26, 2024

brbrmaker commented on May 5, 2025

fzipi commented on Oct 12, 2026

fzipi commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions