Deprecate invalid escape sequences in str/bytes

[Vgr255]

BPO	27364
Nosy	@gvanrossum, @terryjreedy, @jaraco, @vstinner, @ezio-melotti, @bitdancer, @vadmium, @serhiy-storchaka, @ztane, @jayvdb, @yan12125, @Vgr255
Files	deprecate_invalid_unicode_escapes.patch deprecate_invalid_unicode_escapes_2.patch deprecate_invalid_escapes_only_1.patch invalid_stdlib_escapes_1.patch deprecate_invalid_escapes_only_2.patch deprecate_invalid_escapes_only_2.patch: Error handling deprecate_invalid_escapes_only_3.patch deprecate_invalid_escapes_both_1.patch invalid_stdlib_escapes_2.patch deprecate_invalid_escapes_both_2.patch deprecate_invalid_escapes_both_3.patch invalid_stdlib_escapes_3.patch deprecate_invalid_escapes_both_4.patch invalid_stdlib_escapes_3_regenerated.patch invalid_stdlib_escapes_3_rebased_2.patch invalid_stdlib_escapes_4.patch invalid_stdlib_escapes_5.patch deprecate_invalid_escapes_both_5.patch verbose-deprecation.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-08.19:35:58.631>
created_at = <Date 2016-06-21.20:34:19.971>
labels = ['interpreter-core', 'type-bug', 'library', 'expert-unicode']
title = 'Deprecate invalid escape sequences in str/bytes'
updated_at = <Date 2018-02-22.18:36:32.149>
user = 'https://github.com/Vgr255'

bugs.python.org fields:

activity = <Date 2018-02-22.18:36:32.149>
actor = 'abarry'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-08.19:35:58.631>
closer = 'abarry'
components = ['Interpreter Core', 'Library (Lib)', 'Unicode']
creation = <Date 2016-06-21.20:34:19.971>
creator = 'abarry'
dependencies = []
files = ['43499', '43527', '43549', '43550', '43552', '43553', '43569', '43777', '44107', '44108', '44322', '44382', '44432', '44435', '44457', '44463', '44465', '44475', '44550']
hgrepos = []
issue_num = 27364
keywords = ['patch']
message_count = 56.0
messages = ['269022', '269114', '269119', '269122', '269152', '269155', '269156', '269158', '269322', '269323', '269326', '269329', '269332', '269333', '269334', '269335', '269340', '269358', '269368', '269372', '269373', '269376', '269382', '269388', '269413', '269416', '270765', '272439', '272441', '272696', '274119', '274120', '274126', '274332', '274475', '274806', '274837', '274999', '275009', '275010', '275084', '275111', '275123', '275124', '275125', '275219', '275237', '275298', '275757', '276016', '276287', '298112', '298114', '298115', '298170', '312576']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'terry.reedy', 'jaraco', 'vstinner', 'ezio.melotti', 'r.david.murray', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'ztane', 'jayvdb', 'yan12125', 'abarry']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27364'
versions = ['Python 3.6']

[Vgr255]

Attached patch deprecates invalid escape sequences in unicode strings. The point of this is to prevent issues such as bpo-27356 (and possibly other similar ones) in the future.

Without the patch:

>>> "hello \world"
'hello \\world'

With the patch:

>>> "hello \world"
DeprecationWarning: invalid escape sequence 'w'

I'll need some help (patch isn't mergeable yet):

test_doctest fails on my machine with the patch (and -W), and I don't know how to fix it. test_ast fails an assertion (!PyErr_Occurred() in PyObject_Call in abstract.c) when -W is on, and I also don't know how to fix it (I don't even know what causes it).

Of course, I went ahead and fixed all instances of invalid escape sequences in the stdlib (that I could find) so that no DeprecationWarning is encountered.

Lastly, I thought about also doing this to bytes, but I ran into some issues with some invalid escapes such as \u, and _codecs.escape_decode would trigger the warning when passed br"\8" (for example). Ultimately, I decided to leave bytes alone for now, since it's mostly on the lower-level side of things. If there's interest I can add it back.

[bitdancer]

Have you searched the python-dev and python-ideas archives for the previous discussions of this issue? I don't remember for sure, but I think Guido might have made a ruling (not that the discussion couldn't be reopened if he has, but, well...)

[Vgr255]

Now I have! I found nothing on Python-Dev, but apparently it's been discussed on Python-ideas before: https://mail.python.org/pipermail/python-ideas/2015-August/035031.html Guido hasn't participated in that discussion, and most of it was "This will break people's code", with people both for and against the idea, without an apparent consensus.

Should I try a second round on Python-ideas, to try and get a consensus (or a BDFL ruling)?

[ztane]

it is handy to be able to use \w and \d in non-raw-string regular expressions, without too much backslashitis. Seems to be in use in Python standard library as well, for example in csv.py

[Vgr255]

Yes, it's in use in an awful lot of places (see my patch). The proper fix is to use raw strings, or, if you need actual escapes in the same string, manually escape them. However, as you'll see by looking at the patch, the vast majority of cases are fixed by prepending a single 'r' to the front of the string. In fact, only csv.py and html/parser.py needed more finer-grained escaping.

I think that the argument "It works in non-raw strings" is weak. I've always used raw strings for regular expressions, and this patch would simply move this from being a style issue to being a syntax one (and I think it's fine :).

[serhiy-storchaka]

There was a long discussion on Python-Dev. [1] Guido taken part in it.

[1] http://comments.gmane.org/gmane.comp.python.devel/151612

[Vgr255]

Thanks, didn't find that one. Apparently Guido's stance is "Make this a silent warning, then we can discuss about preventing it later", which happens to be what I'm doing here.

[Vgr255]

I found the cause of the failed assertion, an invalid escape sequence slipped through in a file. Patch attached (also with Serhiy's comments).

It worries me a little though that pure Python code can cause a hard crash. Ok, it worries me a lot. Please don't merge this until it's fixed. I'm guessing this is a combination of unittest catching warnings and compiling the faulty source file. As to why a malformed node (i.e. one that raised a DeprecationWarning) managed to pass through unharmed is beyond me.

[gvanrossum]

I am okay with making it a silent warning.

Can we do it in two stages though? It doesn't have to be two releases, I just mean two separate commits: (1) fix all places in the stdlib that violate this principle; (2) separately commit the code that causes the silent deprecation (and tests for it).

What exactly was the hard crash you got? Do you think it was a bug in your own C code or in existing C code?

[Vgr255]

I originally considered making two different patches, so there you go. deprecate_invalid_escapes_only_1.patch has the deprecation plus a test, and invalid_stdlib_escapes_1.patch fixes all invalid escapes in the stdlib.

My code was the cause, although no directly; it was 'assert(!PyErr_Occurred())' at the beginning of PyObject_Call in Objects/abstract.c which failed.

This happened when I ran the whole test suite (although just running test_ast was fine to reproduce it) with the '-W error' command line switch. One stdlib module (I don't remember which one) had one single invalid escape sequence in it, and then test_ast.ASTValidatorTests.test_stdlib_validates triggered the failed assertion. Fixing the invalid escape removes the failure and all tests pass.

One can reliably reproduce the crash with the patch by adding a string with an invalid escape in any of the stdlib files (and running with '-W error'):

No invalid sequence:

>>> import unittest, test.test_ast
>>> unittest.main(test.test_ast)
..............................................................................

---

Ran 78 tests in 5.538s

OK

With an invalid sequence in a file:

>>> import unittest, test.test_ast
>>> unittest.main(test.test_ast)
............................................Fatal Python error: a function returned a result with an error set
DeprecationWarning: invalid escape sequence 'w'

During handling of the above exception, another exception occurred:

SystemError: <built-in function compile> returned a result with an error set

Current thread 0x00001ba0 (most recent call first):
File "E:\GitHub\cpython\lib\ast.py", line 35 in parse
File "E:\GitHub\cpython\lib\test\test_ast.py", line 944 in test_stdlib_validates
File "E:\GitHub\cpython\lib\unittest\case.py", line 600 in run
File "E:\GitHub\cpython\lib\unittest\case.py", line 648 in __call__
File "E:\GitHub\cpython\lib\unittest\suite.py", line 122 in run
File "E:\GitHub\cpython\lib\unittest\suite.py", line 84 in __call__
File "E:\GitHub\cpython\lib\unittest\suite.py", line 122 in run
File "E:\GitHub\cpython\lib\unittest\suite.py", line 84 in __call__
File "E:\GitHub\cpython\lib\unittest\runner.py", line 176 in run
File "E:\GitHub\cpython\lib\unittest\main.py", line 255 in runTests
File "E:\GitHub\cpython\lib\unittest\main.py", line 94 in __init__
File "<stdin>", line 1 in <module>

Then I get the usual "Python has stopped working" Windows prompt (strangely enough, before I'd get a prompt saying "Assertion failed" with the line, but not this time).

I'm not sure where the error lies exactly. Should I open another issue for that?

[gvanrossum]

Hm, if you manage to trigger an assert() in the C code by writing some evil
Python code, the C code is considered broken (unless it was using ctypes or
one or two other explicit "void-the-warranty" exceptions).

Maybe someone who has worked more with the C code recently could help you
dig into this more; my memory is unreliable when it comes to these details.
Maybe assert() calls are disabled by default? In general the error "...
returned a result with an error set" means there's a problem at the C level
where a function should have either returned an object or returned NULL
with the per-thread exception state set, but it was found to return an
object *and* set the exception state. IIRC only debug mode checks for that,
so such a bug occasionally creeps into the code. But you shouldn't assume
everything is fine until you've tracked down the cause.