V3/collection re fix

[airween]

Looks like the Regex::searchAll() method is not case insensitive (InMemoryPerProcess::resolveSingleMatch() needs that). This bug detected with help of CRS regression test 920450.

Also fix (temporary) the 'ovector' limit bug, which triggered when the regex contains too much brackets, eg. in CRS rule 942130.

[airween]

Could somebody re-run the CI build? Looks like it was failed (by Travis):

InstalledDir: /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
before_script.1
0.06s$ echo $TRAVIS_OS_NAME
osx
$ [ "$TRAVIS_OS_NAME" != osx ] || brew update
==> Downloading https://homebrew.bintray.com/bottles-portable-ruby/portable-ruby-2.3.7.mavericks.bottle.tar.gz
######################################################################## 100.0%
==> Pouring portable-ruby-2.3.7.mavericks.bottle.tar.gz
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations


No output has been received in the last 10m0s

Thanks :)

[victorhora]

hey @airween, I've triggered a rebuild on the failed build. We should have the results in a moment.

Thanks for your contribution, me or @zimmerle will have a chance to review it soon :)

[zimmerle]

Merged ea937cf and 640daa4. I do have something that I want to discuss about 3334d2a and 4e1b99a.

The regular expression implementation is self-contained in the regex class. It means that this is the only class which effectively depends on libpcre*. Within this patch, the definition PCRE_CASELESS will be spread among other classes making them also dependent on pcre. As of now, it won't be a problem, but, as we aiming to support different regular expression backends, we better avoid this scenario. I would rather prefer to have a new signature to the method -- considering the PCRE_CASELESS flag only inside the regex class or even a modificator on the regular expression string.

• VerifyCC it is an exception and considered to be a bug to be fixed.

[WGH-]

You can make a regular expression case insensitive by prefixing it with (?i). It seems to be supported almost everywhere (I checked libpcre, Python re, RE2, Hyperscan).

[airween]

Hi @WGH-, thanks, I know it - but the problem is not that.

The problem is, the comparing variables on collections (in-memory and LMDB) must be case insensitive.

This condition is not met.
See this PR, and the documentation.

BTW, the ModSec2 follows this way... so I think we have to align the ModSec3 too.

[zimmerle]

You may not need this to be explicit anylonger.

[zimmerle]

@airween as commented by @WGH- the regular expression could be handled this with the ignore case "modificator". In any case, your pull request is currently holding four commits:

• 8040904 - Removed pcre dependency from outside of regex class
• 3334d2a - Regex search in collection data case insensitive fix
• 4e1b99a - Fixed regex case insensitive search bug in LMDB
• 640daa4 - Added test cases

In 8040904 you have fixed a problem added in 3334d2a and 3334d2a. Do you mind to change 3334d2a and 3334d2a to actually incorporate 8040904 ?

[airween]

The 8040904 should ignore. Do I revoke it?

I know what @WGH- said and your idea, but the CRS contains regex's without modifications, and modsec2 works with it. I'ld like to follow that way, tha's why I sent this PR.

The other solution would be that all occurrences of necessary regex's need to add the modifier, and works of libmodsec3 should differ from ModSec2.

[airween]

Closed this PR, and re-opened a new one here: #2107