avoid inconsistency between \d and [:digit:] when using /a

Since a608946 (Additional PCRE2_EXTRA_ASCII_xxx code, 2023-02-01)
PCRE2_EXTRA_ASCII_BSD could be used to restrict \d to ASCII causing
the following inconsistent behaviour in UCP mode.

  PCRE2 version 10.43-DEV 2023-01-15
    re> /\d/utf,ucp,ascii_bsd
  data> ٣
  No match
  data>
  re> /[[:digit:]]/utf,ucp,ascii_bsd
  data> ٣
    0: \x{663}

It has been suggested[1] that the change to match \p{Nd} when UTF
is enabled for [:digit:] might had been a bug, so allow avoiding it
while making the behaviour consistent again by making the new flag
to also affect [:digit:] and for clarity rename it to
PCRE2_EXTRA_ASCII_DIGIT.

The bug in perl could be avoided by the use of /a, but the equivalent
in PCRE2 would also affect the other POSIX classes and cause further
inconsistences.

Additional adjustments might be needed to other classes but that was punted
for now.

[1] https://lore.kernel.org/git/CANgJU+U+xXsh9psd0z5Xjr+Se5QgdKkjQ7LUQ-PdUULSN3n4+g@mail.gmail.com/

Author: carenas

ChangeLog

@@ -55,21 +55,21 @@ change needed for 9(a) above; (b) fix bugs in ucptest,
 
 12. Integer overflow testing is now centralized in a new function.
 
-13. Made PCRE2_UCP the default in UTF mode in pcre2grep, and added new options 
+13. Made PCRE2_UCP the default in UTF mode in pcre2grep, and added new options
 --case-restrict and --no-ucp.
 
-14. In the debugging printint module (which is normally only linked into 
-pcre2test), avoid the use of a variable called "not" because that's deprecated 
-in C and forbidden in C++. Also rewrite some code to avoid a goto into a block 
+14. In the debugging printint module (which is normally only linked into
+pcre2test), avoid the use of a variable called "not" because that's deprecated
+in C and forbidden in C++. Also rewrite some code to avoid a goto into a block
 that bypassed its initialization (though it didn't actually matter).
 
-15. More minor code adjustments to avoid using reserved C++ words as variable 
-names ("new" and "typename") and another jump that bypassed an (irrelevant) 
+15. More minor code adjustments to avoid using reserved C++ words as variable
+names ("new" and "typename") and another jump that bypassed an (irrelevant)
 initialization.
 
-16. Merged a pull request that removed pcre2_ucptables.c from the list of files 
-to compile in NON-AUTOTOOLS-BUILD because it is #included in pcre2_tables.c. 
-Also adjusted the BUILD.bazel and build.zig files, which had the same issue. At 
+16. Merged a pull request that removed pcre2_ucptables.c from the list of files
+to compile in NON-AUTOTOOLS-BUILD because it is #included in pcre2_tables.c.
+Also adjusted the BUILD.bazel and build.zig files, which had the same issue. At
 the same time, fixed a typo in the Bazel file.
 
 

doc/html/pcre2_match_data_free.html

@@ -28,7 +28,7 @@ <h1>pcre2_match_data_free man page</h1>
 If <i>match_data</i> is NULL, this function does nothing. Otherwise,
 <i>match_data</i> must point to a match data block, which this function frees,
 using the memory freeing function from the general context or compiled pattern
-with which it was created, or <b>free()</b> if that was not set. If the match 
+with which it was created, or <b>free()</b> if that was not set. If the match
 data block was previously passed to <b>pcre2_match()</b>, it will have an
 attached heapframe vector; this is also freed.
 </P>

doc/html/pcre2_set_compile_extra_options.html

@@ -33,12 +33,12 @@ <h1>pcre2_set_compile_extra_options man page</h1>
   PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK     Allow \K in lookarounds
   PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES  Allow \x{d800} to \x{dfff} in UTF-8 and UTF-32 modes
   PCRE2_EXTRA_ALT_BSUX                 Extended alternate \u, \U, and \x handling
-  PCRE2_EXTRA_ASCII_BSD                \d remains ASCII in UCP mode
   PCRE2_EXTRA_ASCII_BSS                \s remains ASCII in UCP mode
   PCRE2_EXTRA_ASCII_BSW                \w remains ASFII in UCP mode
-  PCRE2_EXTRA_ASCII_POSIX              POSIX classes remain ASCII in UCP mode 
+  PCRE2_EXTRA_ASCII_DIGIT              [:digit:] POSIX class and \d remain ASCII in UCP mode
+  PCRE2_EXTRA_ASCII_POSIX              POSIX classes remain ASCII in UCP mode
   PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL    Treat all invalid escapes as a literal following character
-  PCRE2_EXTRA_CASELESS_RESTRICT        Disable mixed ASCII/non-ASCII  case folding
+  PCRE2_EXTRA_CASELESS_RESTRICT        Disable mixed ASCII/non-ASCII case folding
   PCRE2_EXTRA_ESCAPED_CR_IS_LF         Interpret \r as \n
   PCRE2_EXTRA_MATCH_LINE               Pattern matches whole lines
   PCRE2_EXTRA_MATCH_WORD               Pattern matches "words"

doc/html/pcre2api.html

@@ -1540,7 +1540,7 @@ <h1>pcre2api man page</h1>
 one other case, and for all characters whose code points are greater than
 U+007F. Note that there are two ASCII characters, K and S, that, in addition to
 their lower case ASCII equivalents, are case-equivalent with U+212A (Kelvin
-sign) and U+017F (long S) respectively. If you do not want this case 
+sign) and U+017F (long S) respectively. If you do not want this case
 equivalence, you can suppress it by setting PCRE2_EXTRA_CASELESS_RESTRICT.
 </P>
 <P>
@@ -1887,7 +1887,7 @@ <h1>pcre2api man page</h1>
 This option has two effects. Firstly, it change the way PCRE2 processes \B,
 \b, \D, \d, \S, \s, \W, \w, and some of the POSIX character classes. By
 default, only ASCII characters are recognized, but if PCRE2_UCP is set, Unicode
-properties are used to classify characters. There are some PCRE2_EXTRA 
+properties are used to classify characters. There are some PCRE2_EXTRA
 options (see below) that add finer control to this behaviour. More details are
 given in the section on
 <a href="pcre2pattern.html#genericchartypes">generic character types</a>
@@ -1977,11 +1977,6 @@ <h1>pcre2api man page</h1>
 defined by ECMAscript 6; setting PCRE2_EXTRA_ALT_BSUX has the effect of
 PCRE2_ALT_BSUX, but in addition it recognizes \u{hhh..} as a hexadecimal
 character code, where hhh.. is any number of hexadecimal digits.
-<pre>
-  PCRE2_EXTRA_ASCII_BSD
-</pre>
-This option forces \d to match only ASCII digits, even when PCRE2_UCP is set.
-It can be changed within a pattern by means of the (?aD) option setting.
 <pre>
   PCRE2_EXTRA_ASCII_BSS
 </pre>
@@ -1994,6 +1989,12 @@ <h1>pcre2api man page</h1>
 This option forces \w to match only ASCII word characters, even when PCRE2_UCP
 is set. It can be changed within a pattern by means of the (?aW) option
 setting.
+<pre>
+  PCRE2_EXTRA_ASCII_DIGIT
+</pre>
+This option forces the POSIX character class [:digit:] and \d to match only
+ASCII characters, even when PCRE2_UCP is set. As in previous options, It can
+be changed within a pattern by means of the (?aD) option setting.
 <pre>
   PCRE2_EXTRA_ASCII_POSIX
 </pre>
@@ -2029,8 +2030,8 @@ <h1>pcre2api man page</h1>
 case-equivalent character sets that contain both ASCII and non-ASCII
 characters. The ASCII letter S is case-equivalent to U+017f (long S) and the
 ASCII letter K is case-equivalent to U+212a (Kelvin sign). This option disables
-recognition of case-equivalences that cross the ASCII/non-ASCII boundary. In a 
-caseless match, both characters must either be ASCII or non-ASCII. The option 
+recognition of case-equivalences that cross the ASCII/non-ASCII boundary. In a
+caseless match, both characters must either be ASCII or non-ASCII. The option
 can be changed with a pattern by the (?r) option setting.
 <pre>
   PCRE2_EXTRA_ESCAPED_CR_IS_LF

doc/html/pcre2pattern.html

@@ -676,9 +676,9 @@ <h1>pcre2pattern man page</h1>
 is set, the behaviour is changed so that Unicode properties are used to
 determine character types, as follows:
 <pre>
-  \d  any character that matches \p{Nd} (decimal digit)
   \s  any character that matches \p{Z} or \h or \v
   \w  any character that matches \p{L} or \p{N}, plus underscore
+  \d  any character that matches \p{Nd} (decimal digit)
 </pre>
 The upper case escapes match the inverse sets of characters. Note that \d
 matches only decimal digits, whereas \w matches any Unicode digit, as well as
@@ -688,9 +688,9 @@ <h1>pcre2pattern man page</h1>
 </P>
 <P>
 The effect of PCRE2_UCP on any one of these escape sequences can be negated by
-the options PCRE2_EXTRA_ASCII_BSD, PCRE2_EXTRA_ASCII_BSS, and
-PCRE2_EXTRA_ASCII_BSW, respectively. These options can be set and reset within
-a pattern by means of an internal option setting
+the options PCRE2_EXTRA_ASCII_BSS, PCRE2_EXTRA_ASCII_BSW, and
+PCRE2_EXTRA_ASCII_DIGIT, respectively. These options can be set and reset
+within a pattern by means of an internal option setting
 <a href="#internaloptions">(see below).</a>
 </P>
 <P>
@@ -1526,7 +1526,7 @@ <h1>pcre2pattern man page</h1>
   [:alpha:]  becomes  \p{L}
   [:blank:]  becomes  \h
   [:cntrl:]  becomes  \p{Cc}
-  [:digit:]  becomes  \p{Nd}
+  [:digit:]  becomes  \p{Nd}  unless PCRE2_EXTRA_ASCII_DIGIT is set
   [:lower:]  becomes  \p{Ll}
   [:space:]  becomes  \p{Xps}
   [:upper:]  becomes  \p{Lu}
@@ -1634,7 +1634,7 @@ <h1>pcre2pattern man page</h1>
 Some PCRE2-specific options can be changed by the same mechanism using these
 pairs or individual letters:
 <pre>
-  aD for PCRE2_EXTRA_ASCII_BSD
+  aD for PCRE2_EXTRA_ASCII_DIGIT
   aS for PCRE2_EXTRA_ASCII_BSS
   aW for PCRE2_EXTRA_ASCII_BSW
   aP for PCRE2_EXTRA_ASCII_POSIX

doc/html/pcre2test.html

@@ -628,9 +628,9 @@ <h1>pcre2test man page</h1>
       alt_verbnames             set PCRE2_ALT_VERBNAMES
       anchored                  set PCRE2_ANCHORED
   /a  ascii_all                 set all ASCII options
-      ascii_bsd                 set PCRE2_EXTRA_ASCII_BSD
       ascii_bss                 set PCRE2_EXTRA_ASCII_BSS
       ascii_bsw                 set PCRE2_EXTRA_ASCII_BSW
+      ascii_digit               set PCRE2_EXTRA_ASCII_DIGIT
       ascii_posix               set PCRE2_EXTRA_ASCII_POSIX
       auto_callout              set PCRE2_AUTO_CALLOUT
       bad_escape_is_literal     set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL

doc/pcre2.txt

@@ -1935,12 +1935,6 @@ COMPILING A PATTERN
        as a hexadecimal character code, where hhh.. is any number of hexadeci-
        mal digits.
 
-         PCRE2_EXTRA_ASCII_BSD
-
-       This option forces \d to match only ASCII digits, even  when  PCRE2_UCP
-       is  set.   It can be changed within a pattern by means of the (?aD) op-
-       tion setting.
-
          PCRE2_EXTRA_ASCII_BSS
 
        This option forces \s to match only ASCII space characters,  even  when
@@ -1953,6 +1947,13 @@ COMPILING A PATTERN
        PCRE2_UCP  is  set.  It can be changed within a pattern by means of the
        (?aW) option setting.
 
+         PCRE2_EXTRA_ASCII_DIGIT
+
+       This option forces the POSIX character class [:digit:] and \d to  match
+       only  ASCII  characters, even when PCRE2_UCP is set. As in previous op-
+       tions, It can be changed within a pattern by means of the (?aD)  option
+       setting.
+
          PCRE2_EXTRA_ASCII_POSIX
 
        This option forces the POSIX character  classes  to  match  only  ASCII
@@ -6895,9 +6896,9 @@ BACKSLASH
        changed so that Unicode properties  are  used  to  determine  character
        types, as follows:
 
-         \d  any character that matches \p{Nd} (decimal digit)
          \s  any character that matches \p{Z} or \h or \v
          \w  any character that matches \p{L} or \p{N}, plus underscore
+         \d  any character that matches \p{Nd} (decimal digit)
 
        The  upper case escapes match the inverse sets of characters. Note that
        \d matches only decimal digits, whereas \w matches any  Unicode  digit,
@@ -6906,8 +6907,8 @@ BACKSLASH
        Matching these sequences is noticeably slower when PCRE2_UCP is set.
 
        The  effect  of  PCRE2_UCP  on any one of these escape sequences can be
-       negated by the  options  PCRE2_EXTRA_ASCII_BSD,  PCRE2_EXTRA_ASCII_BSS,
-       and  PCRE2_EXTRA_ASCII_BSW,  respectively. These options can be set and
+       negated by the  options  PCRE2_EXTRA_ASCII_BSS,  PCRE2_EXTRA_ASCII_BSW,
+       and PCRE2_EXTRA_ASCII_DIGIT, respectively. These options can be set and
        reset within a pattern by means of an internal option setting (see  be-
        low).
 
@@ -7688,7 +7689,7 @@ POSIX CHARACTER CLASSES
          [:alpha:]  becomes  \p{L}
          [:blank:]  becomes  \h
          [:cntrl:]  becomes  \p{Cc}
-         [:digit:]  becomes  \p{Nd}
+         [:digit:]  becomes  \p{Nd}  unless PCRE2_EXTRA_ASCII_DIGIT is set
          [:lower:]  becomes  \p{Ll}
          [:space:]  becomes  \p{Xps}
          [:upper:]  becomes  \p{Lu}
@@ -7789,7 +7790,7 @@ INTERNAL OPTION SETTING
        Some  PCRE2-specific options can be changed by the same mechanism using
        these pairs or individual letters:
 
-         aD for PCRE2_EXTRA_ASCII_BSD
+         aD for PCRE2_EXTRA_ASCII_DIGIT
          aS for PCRE2_EXTRA_ASCII_BSS
          aW for PCRE2_EXTRA_ASCII_BSW
          aP for PCRE2_EXTRA_ASCII_POSIX

doc/pcre2_match_data_free.3

@@ -16,7 +16,7 @@ PCRE2 - Perl-compatible regular expressions (revised API)
 If \fImatch_data\fP is NULL, this function does nothing. Otherwise,
 \fImatch_data\fP must point to a match data block, which this function frees,
 using the memory freeing function from the general context or compiled pattern
-with which it was created, or \fBfree()\fP if that was not set. If the match 
+with which it was created, or \fBfree()\fP if that was not set. If the match
 data block was previously passed to \fBpcre2_match()\fP, it will have an
 attached heapframe vector; this is also freed.
 .P

doc/pcre2_set_compile_extra_options.3

@@ -25,17 +25,19 @@ options are:
 .\" JOIN
   PCRE2_EXTRA_ALT_BSUX                 Extended alternate \eu, \eU, and
                                          \ex handling
-  PCRE2_EXTRA_ASCII_BSD                \ed remains ASCII in UCP mode
   PCRE2_EXTRA_ASCII_BSS                \es remains ASCII in UCP mode
   PCRE2_EXTRA_ASCII_BSW                \ew remains ASFII in UCP mode
+.\" JOIN
+  PCRE2_EXTRA_ASCII_DIGIT              [:digit:] POSIX class and \ed remain
+                                         ASCII in UCP mode
 .\" JOIN
   PCRE2_EXTRA_ASCII_POSIX              POSIX classes remain ASCII in
-                                         UCP mode 
+                                         UCP mode
 .\" JOIN
   PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL    Treat all invalid escapes as
                                          a literal following character
 .\" JOIN
-  PCRE2_EXTRA_CASELESS_RESTRICT        Disable mixed ASCII/non-ASCII 
+  PCRE2_EXTRA_CASELESS_RESTRICT        Disable mixed ASCII/non-ASCII
                                          case folding
   PCRE2_EXTRA_ESCAPED_CR_IS_LF         Interpret \er as \en
   PCRE2_EXTRA_MATCH_LINE               Pattern matches whole lines

doc/pcre2api.3

@@ -1482,7 +1482,7 @@ PCRE2_UCP is set, Unicode properties are used for all characters with more than
 one other case, and for all characters whose code points are greater than
 U+007F. Note that there are two ASCII characters, K and S, that, in addition to
 their lower case ASCII equivalents, are case-equivalent with U+212A (Kelvin
-sign) and U+017F (long S) respectively. If you do not want this case 
+sign) and U+017F (long S) respectively. If you do not want this case
 equivalence, you can suppress it by setting PCRE2_EXTRA_CASELESS_RESTRICT.
 .P
 For lower valued characters with only one other case, a lookup table is used
@@ -1838,7 +1838,7 @@ are not representable in UTF-16.
 This option has two effects. Firstly, it change the way PCRE2 processes \eB,
 \eb, \eD, \ed, \eS, \es, \eW, \ew, and some of the POSIX character classes. By
 default, only ASCII characters are recognized, but if PCRE2_UCP is set, Unicode
-properties are used to classify characters. There are some PCRE2_EXTRA 
+properties are used to classify characters. There are some PCRE2_EXTRA
 options (see below) that add finer control to this behaviour. More details are
 given in the section on
 .\" HTML <a href="pcre2pattern.html#genericchartypes">
@@ -1936,11 +1936,6 @@ the way that ECMAscript (aka JavaScript) does. Additional functionality was
 defined by ECMAscript 6; setting PCRE2_EXTRA_ALT_BSUX has the effect of
 PCRE2_ALT_BSUX, but in addition it recognizes \eu{hhh..} as a hexadecimal
 character code, where hhh.. is any number of hexadecimal digits.
-.sp
-  PCRE2_EXTRA_ASCII_BSD
-.sp
-This option forces \ed to match only ASCII digits, even when PCRE2_UCP is set.
-It can be changed within a pattern by means of the (?aD) option setting.
 .sp
   PCRE2_EXTRA_ASCII_BSS
 .sp
@@ -1953,6 +1948,12 @@ option setting.
 This option forces \ew to match only ASCII word characters, even when PCRE2_UCP
 is set. It can be changed within a pattern by means of the (?aW) option
 setting.
+.sp
+  PCRE2_EXTRA_ASCII_DIGIT
+.sp
+This option forces the POSIX character class [:digit:] and \ed to match only
+ASCII characters, even when PCRE2_UCP is set. As in previous options, It can
+be changed within a pattern by means of the (?aD) option setting.
 .sp
   PCRE2_EXTRA_ASCII_POSIX
 .sp
@@ -1987,8 +1988,8 @@ rules, which allow for more than two cases per character. There are two
 case-equivalent character sets that contain both ASCII and non-ASCII
 characters. The ASCII letter S is case-equivalent to U+017f (long S) and the
 ASCII letter K is case-equivalent to U+212a (Kelvin sign). This option disables
-recognition of case-equivalences that cross the ASCII/non-ASCII boundary. In a 
-caseless match, both characters must either be ASCII or non-ASCII. The option 
+recognition of case-equivalences that cross the ASCII/non-ASCII boundary. In a
+caseless match, both characters must either be ASCII or non-ASCII. The option
 can be changed with a pattern by the (?r) option setting.
 .sp
   PCRE2_EXTRA_ESCAPED_CR_IS_LF

doc/pcre2pattern.3

@@ -673,9 +673,9 @@ support was available, mainly for efficiency reasons. If the PCRE2_UCP option
 is set, the behaviour is changed so that Unicode properties are used to
 determine character types, as follows:
 .sp
-  \ed  any character that matches \ep{Nd} (decimal digit)
   \es  any character that matches \ep{Z} or \eh or \ev
   \ew  any character that matches \ep{L} or \ep{N}, plus underscore
+  \ed  any character that matches \ep{Nd} (decimal digit)
 .sp
 The upper case escapes match the inverse sets of characters. Note that \ed
 matches only decimal digits, whereas \ew matches any Unicode digit, as well as
@@ -684,9 +684,9 @@ any Unicode letter, and underscore. Note also that PCRE2_UCP affects \eb, and
 is noticeably slower when PCRE2_UCP is set.
 .P
 The effect of PCRE2_UCP on any one of these escape sequences can be negated by
-the options PCRE2_EXTRA_ASCII_BSD, PCRE2_EXTRA_ASCII_BSS, and
-PCRE2_EXTRA_ASCII_BSW, respectively. These options can be set and reset within
-a pattern by means of an internal option setting
+the options PCRE2_EXTRA_ASCII_BSS, PCRE2_EXTRA_ASCII_BSW, and
+PCRE2_EXTRA_ASCII_DIGIT, respectively. These options can be set and reset
+within a pattern by means of an internal option setting
 .\" HTML <a href="#internaloptions">
 .\" </a>
 (see below).
@@ -1522,7 +1522,7 @@ classes with other sequences, as follows:
   [:alpha:]  becomes  \ep{L}
   [:blank:]  becomes  \eh
   [:cntrl:]  becomes  \ep{Cc}
-  [:digit:]  becomes  \ep{Nd}
+  [:digit:]  becomes  \ep{Nd}  unless PCRE2_EXTRA_ASCII_DIGIT is set
   [:lower:]  becomes  \ep{Ll}
   [:space:]  becomes  \ep{Xps}
   [:upper:]  becomes  \ep{Lu}
@@ -1637,7 +1637,7 @@ be re-instated, but a hyphen may not appear.
 Some PCRE2-specific options can be changed by the same mechanism using these
 pairs or individual letters:
 .sp
-  aD for PCRE2_EXTRA_ASCII_BSD
+  aD for PCRE2_EXTRA_ASCII_DIGIT
   aS for PCRE2_EXTRA_ASCII_BSS
   aW for PCRE2_EXTRA_ASCII_BSW
   aP for PCRE2_EXTRA_ASCII_POSIX

doc/pcre2test.1

@@ -583,9 +583,9 @@ for a description of the effects of these options.
       alt_verbnames             set PCRE2_ALT_VERBNAMES
       anchored                  set PCRE2_ANCHORED
   /a  ascii_all                 set all ASCII options
-      ascii_bsd                 set PCRE2_EXTRA_ASCII_BSD
       ascii_bss                 set PCRE2_EXTRA_ASCII_BSS
       ascii_bsw                 set PCRE2_EXTRA_ASCII_BSW
+      ascii_digit               set PCRE2_EXTRA_ASCII_DIGIT
       ascii_posix               set PCRE2_EXTRA_ASCII_POSIX
       auto_callout              set PCRE2_AUTO_CALLOUT
       bad_escape_is_literal     set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL

doc/pcre2test.txt

@@ -563,9 +563,9 @@ PATTERN MODIFIERS
              alt_verbnames             set PCRE2_ALT_VERBNAMES
              anchored                  set PCRE2_ANCHORED
          /a  ascii_all                 set all ASCII options
-             ascii_bsd                 set PCRE2_EXTRA_ASCII_BSD
              ascii_bss                 set PCRE2_EXTRA_ASCII_BSS
              ascii_bsw                 set PCRE2_EXTRA_ASCII_BSW
+             ascii_digit               set PCRE2_EXTRA_ASCII_DIGIT
              ascii_posix               set PCRE2_EXTRA_ASCII_POSIX
              auto_callout              set PCRE2_AUTO_CALLOUT
              bad_escape_is_literal     set PCRE2_EXTRA_BAD_ESCAPE_IS_LITERAL