pcre2grep does not set PCRE2_UCP

[klpn]

When pcre2grep is called with -u, the flag PCRE2_UCP is not set, see:

pcre2/src/pcre2grep.c

Line 3607 in 0746b3d

case 'u': options |= PCRE2_UTF; utf = TRUE; break;

This leads to incorrect behavior with boundary matches and non-ASCII letters, such as:

$  echo 'Öm' | pcre2grep -u '\bm'
Öm

The output should be nothing. Adding the PCRE2_UCPflag fixes the problem:

  case 'u': options |= PCRE2_UTF|PCRE2_UCP; utf = TRUE; break;

GNU grep with the -P option also behaves incorrectly in this case, unlike perl (see this Twitter thread for more examples and discussion).

[zherczeg]

I think this change is reasonable, However, we don't know who depends on the current behaviour. Anyway pcre2grep is just an example code, not intended for use in production.

[klpn]

Yes. And if the flag added for the -u option, the -U option should probably also be updated accordingly.

[klpn]

This issue was noticed by GNU grep team, and has been patched there.

[zherczeg]

I would not call it an issue, more of a feature request. The proper change probably needs an option to go back to the old mode as well.

[PhilipHazel]

Yes indeed. I am always wary about making incompatible changes, however "obvious" they may be. In this case, I agree that it's probably best to make it the default - especially as that's what others do - but I will not be surprised if it catches out at least one person. I will look at this in due course and include something like --utf-no-ucp.

[carenas]

Looking at the performance impact of this change, was wondering if we would need to at least introduce an equivalent to the /aa modifier as well as suggested in #11.

I would suspect that most people would still prefer to have \d matching only [0-9] even when PCRE2_UCP is enabled and so a flag to avoid UTF/ASCII mixing would be needed to allow signaling that preference.

[PhilipHazel]

There are now (in HEAD) some new options for restricting various things to ASCII and disabling Unicode/ASCII case folding. I will now consider how best to use these in pcre2grep.

[PhilipHazel]

I have now updated pcre2grep in HEAD. Both -u and -U set PCRE2_UCP by default, but there is a --no-ucp option to disable this. There is also a --case-restrict option which sets PCRE2_EXTRA_CASELESS_RESTRICT. Finer control is now available via the new options settings within patterns, e.g. (?aD) to force \d to ASCII while leaving \s and \w as Unicode.