Replace empty matches adjacent to a previous non-empty match in re.sub()

[serhiy-storchaka]

BPO	32308
Nosy	@ezio-melotti, @serhiy-storchaka, @boxed
PRs	bpo-32308: Replace empty matches adjacent to a previous non-empty match in re.sub(). #4846

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2018-01-04.09:06:40.130>
created_at = <Date 2017-12-13.18:28:37.987>
labels = ['expert-regex', 'type-feature', 'library', '3.7']
title = 'Replace empty matches adjacent to a previous non-empty match in re.sub()'
updated_at = <Date 2020-04-16.16:28:26.175>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-04-16.16:28:26.175>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2018-01-04.09:06:40.130>
closer = 'serhiy.storchaka'
components = ['Library (Lib)', 'Regular Expressions']
creation = <Date 2017-12-13.18:28:37.987>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 32308
keywords = ['patch']
message_count = 14.0
messages = ['308229', '309055', '309458', '339949', '339950', '339989', '340040', '340102', '360352', '360355', '366595', '366602', '366604', '366606']
nosy_count = 6.0
nosy_names = ['ezio.melotti', 'mrabarnett', 'mu_mind', 'serhiy.storchaka', 'Anders.Hovm\xc3\xb6ller', 'Mark Borgerding']
pr_nums = ['4846']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32308'
versions = ['Python 3.7']

[serhiy-storchaka]

Currently re.sub() replaces empty matches only when not adjacent to a previous match. This makes it inconsistent with re.findall() and re.finditer() which finds empty matches adjacent to a previous non-empty match and with other RE engines.

Proposed PR makes all functions that makes repeated searching (re.split(), re.sub(), re.findall(), re.finditer()) mutually consistent.

The PR change the behavior of re.split() too, but this doesn't matter, since it already is different from the 3.6 behavior.

BDFL have approved this change.

This change doesn't break any stdlib code. It is expected that it will not break much third-party code, and even if it will break some code, it can be easily rewritten. For example replacing re.sub('(.*)', ...) (which now matches an empty string at the end of the string) with re.sub('(.+)', ...) is an obvious fix.

[serhiy-storchaka]

Could anybody please make a review of at least the documentation part?

[serhiy-storchaka]

New changeset fbb490f by Serhiy Storchaka in branch 'master':
bpo-32308: Replace empty matches adjacent to a previous non-empty match in re.sub(). (bpo-4846)
fbb490f

[boxed]

This was a really bad idea in my opinion. We just found this and we have no way to know how this will impact production. It's really absurd that

re.sub('(.*)', r'foo', 'asd')

is "foo" in python 1 to 3.6 but 'foofoo' in python 3.7.

[boxed]

Just as a comparison, sed does the 3.6 thing:

echo foo | sed 's/\(.*\)/x\1y/g'
xfooy

[mrabarnett]

It's now consistent with Perl, PCRE and .Net (C#), as well as re.split(), re.sub(), re.findall() and re.finditer().

[boxed]

That might be true, but that seems like a weak argument. If anything, it means those others are broken. What is the logic behind "(.*)" returning the entire string (which is what you asked for) and exactly one empty string? Why not two empty strings? 3? 4? 5? Why not an empty string at the beginning? It makes no practical sense.

We will have to spend considerable effort to work around this change and adapt our code to 3.7. The lack of a discussion about backwards compatibility in this, and the other, thread before making this change is also a problem I think.

[mrabarnett]

Consider re.findall(r'.{0,2}', 'abcde').

It finds 'ab', then continues where it left off to find 'cd', then 'e'.

It can also find ''; re.match(r'.*', '') does match, after all.

It could, in fact, an infinite number of ''.

And what about re.match(r'()*', '')?

What should it do? Run forever? Raise an exception?

At some point you have to make a decision as to what should happen, and the general consensus has been to match once.

[mumind]

We were also bitten by this behavior change in google/vroom#110. I'm kinda baffled by the new behavior and assumed it had to be an accidental regression, but I guess not. If you have any other context on the BDFL conversation and reasoning for calling this behavior correct, I'd love to see additional info.