gh-98401: Invalid escape sequences emits SyntaxWarning #99011

vstinner · 2022-11-02T16:21:32Z

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning.

The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions.
Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.

Issue: Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Parser/string_parser.c

vstinner · 2022-11-02T21:50:47Z

Previous attempt in 2018:

Commit: 6543912
Serhiy only changed the Python parser, not codecs: rationale
The new SyntaxWarning was seen in multiple projects: docutils (fixed), bottle, SymPy (bug in a docstring)
SyntaxWarning are only emitted once, and only when a Python file is parsed: when a cached PYC file is used, not warning is emitted
Serhiy reverted his change

vstinner · 2022-11-03T11:06:06Z

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

$ PYTHONWARNINGS=error::SyntaxWarning ./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui

vstinner · 2022-11-03T11:19:33Z

This issue mostly hit code defining regular expressions. Example in BeautifulSoup 3.2.2:

$ pycodestyle --select W605 $(find -name "*.py")
./BeautifulSoup.py:455:52: W605 invalid escape sequence '\d'
./BeautifulSoup.py:455:73: W605 invalid escape sequence '\w'

./BeautifulSoup.py:574:47: W605 invalid escape sequence '\d'
./BeautifulSoup.py:574:66: W605 invalid escape sequence '\w'

./BeautifulSoup.py:1080:38: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1589:36: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1933:16: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1933:46: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1935:35: W605 invalid escape sequence '\s'

Examples of code, re.compile() calls:

    BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
                                           + "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
                                           + ")")
(...)
                      (re.compile('<!\s+([^<>]*)>'),
(...)
        xml_encoding_match = re.compile(
            '^<\?.*encoding=[\'"](.*?)[\'"].*\?>').match(xml_data)

Doc/reference/lexical_analysis.rst

serhiy-storchaka · 2022-11-03T11:25:58Z

Lib/test/test_codecs.py

        for i in range(97, 123):
            b = bytes([i])
            if b not in b'abfnrtvx':
-                with self.assertWarns(DeprecationWarning):
+                with self.assertWarns(SyntaxWarning):


SyntaxWarning is not related to codecs. It only should be emitted by the compiler.

Which warning should be emitted if SyntaxWarning is not the best choice? UnicodeWarning?

Does UnicodeWarning make sense for codecs.escape_decode()?

DeprecationWarning, as for all other deprecated features.

Parser/string_parser.c

serhiy-storchaka · 2022-11-03T11:31:54Z

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

vstinner · 2022-11-03T12:08:27Z

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

I ran PYTHONWARNINGS=error::SyntaxWarning make regen-all PYTHON_FOR_REGEN=./python: it works as expected.

vstinner · 2022-11-03T12:09:10Z

@serhiy-storchaka:

Perhaps SyntaxWarning could be raised for invalid octals too. It is a more obvious error, and less common, so it may have lesser deprecation period.

Ok, I updated my PR: Use SyntaxWarning for invalid octal sequence.

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning. Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError. codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning. * The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions. * Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.

vstinner · 2022-11-03T12:39:14Z

Hum, I messed up my PR so I squashed two commits and I fixed my PR:

invalid octal sequence and invalid escape sequence now both emit SyntaxWarning (and will emit SyntaxError in the future)
codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged

I also mentioned the future convertion to SyntaxError in the doc (What's New / NEWS entries).

vstinner · 2022-11-03T15:04:02Z

@mdickinson @serhiy-storchaka @hugovk: Would you mind to review my PR?

serhiy-storchaka · 2022-11-03T15:29:34Z

Did you remove all pyc files?

find -name '*.py[co]' -exec rm -rf '{}' +

vstinner · 2022-11-03T16:12:20Z

Let me try these commands:

export PYTHONWARNINGS=error::SyntaxWarning 
git clean -fdx
./configure --with-pydebug CFLAGS=-O0
make
make regen-all PYTHON_FOR_REGEN=./python
make

./python -c 's = "\477"'

The last command fails as expected with:

  File "<string>", line 1
    s = "\477"
        ^^^^^^
SyntaxError: invalid octal escape sequence '\477'

vstinner · 2022-11-03T16:16:50Z

Oops, test_string_literals didn't work when run with ./python -W error::SyntaxWarning: it's now fixed.

I ran the test suite with:

export PYTHONWARNINGS=error::SyntaxWarning 
./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui

Note: I would prefer to run the whole test suite with -Werror, but it fails even without this PR :-( Python currently emits many deprecation warnings for example :-(

Fixes this warning experienced with Python 3.12 (python/cpython#98401, python/cpython#99011): faa_cs_aan.py:690: SyntaxWarning: invalid escape sequence '\.' email_match = re.search('For Inquiries: ([0-9a-z._-]+@[0-9a-z.-]+)\.?$',

Use r-strings for all regular expressions. Fixes these warnings experienced with Python 3.12 (python/cpython#98401, python/cpython#99011, https://docs.python.org/3/whatsnew/3.12.html#other-language-changes point 2): run_tests.py:200: SyntaxWarning: invalid escape sequence '\d' FINAL_LINE_RE = re.compile('status=(\d+)$') run_tests.py:441: SyntaxWarning: invalid escape sequence '\*' re.match('^\* daemon .+ \*$', line) or line == ''): Change-Id: I71ddfb1a2ca62654378ae67a99e9aeb4ce7b7394 Reviewed-on: https://chromium-review.googlesource.com/c/crashpad/crashpad/+/6254063 Commit-Queue: Mark Mentovai <[email protected]> Reviewed-by: Nico Weber <[email protected]>

All other regular expressions in the file already used r-strings, but this newer one added in beba95a did not. Fixes this warning experienced with Python ≥3.12 (python/cpython#98401, python/cpython#99011, python/cpython@a60ddd31be7f): bisect-builds.py:1260: SyntaxWarning: invalid escape sequence '\.' if not re.search("\.apks?$", apk_name): Change-Id: Ia8f1981efde3b14d5c36c3dfc2bca7737fd4cb89 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/6647645 Commit-Queue: Mark Mentovai <[email protected]> Reviewed-by: Kuan Huang <[email protected]> Cr-Commit-Position: refs/heads/main@{#1474605}

Since 61fad56 (https://chromium-review.googlesource.com/c/5848450, 2024-09-11), autoninja under Python 3.12 presents these warnings: ``` …/autoninja.py:73: SyntaxWarning: invalid escape sequence '\s' m = re.match('instance\s*=\s*projects/([^/]*)/instances/.*', line) …/autoninja.py:92: SyntaxWarning: invalid escape sequence '\s' m = re.match('SISO_PROJECT=\s*(\S*)\s*', line) ``` This warning appears in Python 3.12 ([1], [2], [3]). '\s' and '\S' are not valid escape sequences in strings. r'\s' and r'\S' are valid in regular expressions, but outside of raw strings, they would need to be written as '\\s' and '\\S'. There is no reason to not use raw strings in this case, so the new regular expression pattern strings introduced in 61fad56 are changed to raw strings. [1] https://docs.python.org/3/whatsnew/3.12.html#:~:text=A%20backslash%2Dcharacter%20pair%20that%20is%20not%20a%20valid%20escape%20sequence%20now%20generates%20a%20SyntaxWarning%2C%20instead%20of%20DeprecationWarning. [2] python/cpython#98401 [3] python/cpython#99011 Bug: b/364318216 Change-Id: I0f237976fe9c39208541ae78205f5bdbf126fa82 Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/5859159 Commit-Queue: Mark Mentovai <[email protected]> Reviewed-by: Philipp Wollermann <[email protected]> Auto-Submit: Mark Mentovai <[email protected]>

vstinner requested review from pablogsal and lysnikolaou as code owners November 2, 2022 16:21

bedevere-bot mentioned this pull request Nov 2, 2022

Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Closed

bedevere-bot added the awaiting core review label Nov 2, 2022

This was referenced Nov 2, 2022

gh-98401: Reject invalid escape sequences in strings #98404

Closed

Raise non-silent warning for invalid escape sequences #77093

Open

lysnikolaou reviewed Nov 2, 2022

View reviewed changes

Parser/string_parser.c Outdated Show resolved Hide resolved

serhiy-storchaka reviewed Nov 3, 2022

View reviewed changes

serhiy-storchaka approved these changes Nov 3, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Nov 3, 2022

Fix test_string_literals

d24f799

vstinner merged commit a60ddd3 into python:main Nov 3, 2022

bedevere-bot removed the awaiting merge label Nov 3, 2022

vstinner deleted the syntax_warning branch November 3, 2022 16:53

chenhengqi mentioned this pull request Oct 31, 2023

tools/*.py: fix invalid escape sequences in regexps iovisor/bcc#4789

Merged

benoit9126 mentioned this pull request Nov 9, 2023

Syntax warning because of "\S" in Python 3.12 keygen-sh/py-machineid#9

Closed

terryjreedy mentioned this pull request May 23, 2024

Invalid escape sequences raise SyntaxWarning #119458

Closed

bebound mentioned this pull request Sep 14, 2024

gh-123858: Improve Doc: SyntaxWarning is emitted during bytecode generation #122844

Open

Uh oh!

gh-98401: Invalid escape sequences emits SyntaxWarning #99011

gh-98401: Invalid escape sequences emits SyntaxWarning #99011

Uh oh!

Conversation

vstinner commented Nov 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vstinner commented Nov 2, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

Uh oh!

serhiy-storchaka Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

vstinner Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Nov 3, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

serhiy-storchaka commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

serhiy-storchaka commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

vstinner commented Nov 3, 2022

Uh oh!

Uh oh!

vstinner commented Nov 2, 2022 •

edited

Loading