bpo-37764: Fix infinite loop when parsing unstructured email headers. #15239

epicfaace · 2019-08-12T23:04:04Z

Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either:

a case without trailing whitespace
an invalid encoded word

https://bugs.python.org/issue37764

This fix should also be backported to 3.7 and 3.8

https://bugs.python.org/issue37764

Automerge-Triggered-By: @maxking

maxking

Thanks for this PR.
I've made inline comments. Please also see my comments on BPO.

Lib/email/_header_value_parser.py

Lib/test/test_email/test__header_value_parser.py

bedevere-bot · 2019-08-16T06:10:08Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

epicfaace · 2019-08-19T18:12:23Z

I have made the requested changes; please review again

bedevere-bot · 2019-08-19T18:12:26Z

Thanks for making the requested changes!

@maxking: please review the changes made to this pull request.

Lib/email/_header_value_parser.py

maxking · 2019-08-21T22:49:33Z

Lib/email/_header_value_parser.py

@@ -1039,7 +1039,7 @@ def get_encoded_word(value):
        raise errors.HeaderParseError(
            "expected encoded word but found {}".format(value))
    remstr = ''.join(remainder)
-    if len(remstr) > 1 and remstr[0] in hexdigits and remstr[1] in hexdigits:
+    if len(remstr) > 1 and remstr[0] in hexdigits and remstr[1] in hexdigits and tok.count('?') < 2:


Probably needs to be split on multiple line for pep8.

Done.

Is there a way to check for this automatically? What's the standard I should usually apply? (for example, this file doesn't have two blank lines between each class, which also violates pep8)

Lib/email/_header_value_parser.py

bedevere-bot · 2019-08-21T22:52:30Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

DeltaF1 · 2019-08-22T21:14:19Z

I applied this patch to a local install of python 3.7, and the problem seemed to persist with my test case. The attached file has the email I parsed, with some content redacted for privacy. It appears to be very mangled, but even so the library should probably not fall into an infinite loop if it receives mangled data.

Hopefully you can reproduce the error, otherwise it might just be an artefact of my patching method.

mangled_message.txt

epicfaace · 2019-08-23T02:13:32Z

I applied this patch to a local install of python 3.7, and the problem seemed to persist with my test case. The attached file has the email I parsed, with some content redacted for privacy. It appears to be very mangled, but even so the library should probably not fall into an infinite loop if it receives mangled data.

Hopefully you can reproduce the error, otherwise it might just be an artefact of my patching method.

mangled_message.txt

I did add a test in this PR with that mangled message, but it doesn't seem to hang. Either it's an artefact with your patching method, or a different issue with 3.7.

epicfaace · 2019-08-23T02:13:43Z

I have made the requested changes; please review again.

maxking

LGTM, thanks @epicfaace !

miss-islington · 2019-08-31T15:25:38Z

Thanks @epicfaace for the PR 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8.
🐍🍒⛏🤖

miss-islington · 2019-08-31T15:25:49Z

Sorry, @epicfaace, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker c5b242f87f31286ad38991bc3868cf4cfbf2b681 3.8

miss-islington · 2019-08-31T15:25:51Z

Sorry @epicfaace, I had trouble checking out the 3.7 backport branch.
Please backport using cherry_picker on command line.
cherry_picker c5b242f87f31286ad38991bc3868cf4cfbf2b681 3.7

…aders. (pythonGH-15239) Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either: - a case without trailing whitespace - an invalid encoded word https://bugs.python.org/issue37764 This fix should also be backported to 3.7 and 3.8 https://bugs.python.org/issue37764 (cherry picked from commit c5b242f) Co-authored-by: Ashwin Ramaswami <[email protected]>

bedevere-bot · 2019-09-03T04:35:35Z

GH-15654 is a backport of this pull request to the 3.7 branch.

…aders. (GH-15239) (GH-15654) …aders. (GH-15239) Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either: - a case without trailing whitespace - an invalid encoded word https://bugs.python.org/issue37764 This fix should also be backported to 3.7 and 3.8 https://bugs.python.org/issue37764 (cherry picked from commit c5b242f) Co-authored-by: Ashwin Ramaswami <[email protected]> https://bugs.python.org/issue37764

…aders. (pythonGH-15239) Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either: - a case without trailing whitespace - an invalid encoded word https://bugs.python.org/issue37764 This fix should also be backported to 3.7 and 3.8 https://bugs.python.org/issue37764 (cherry picked from commit c5b242f) Co-authored-by: Ashwin Ramaswami <[email protected]>

bedevere-bot · 2019-09-05T00:41:24Z

GH-15686 is a backport of this pull request to the 3.8 branch.

…aders. (GH-15239) (GH-15686) Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either: - a case without trailing whitespace - an invalid encoded word https://bugs.python.org/issue37764 This fix should also be backported to 3.7 and 3.8 https://bugs.python.org/issue37764 (cherry picked from commit c5b242f) Co-authored-by: Ashwin Ramaswami <[email protected]>

…pythonGH-15239) Fixes a case in which email._header_value_parser.get_unstructured hangs the system for some invalid headers. This covers the cases in which the header contains either: - a case without trailing whitespace - an invalid encoded word https://bugs.python.org/issue37764 This fix should also be backported to 3.7 and 3.8 https://bugs.python.org/issue37764

bpo-37764: fix hang case in get_unstructured

fd5a82a

epicfaace requested a review from a team as a code owner August 12, 2019 23:04

the-knights-who-say-ni added the CLA signed label Aug 12, 2019

bedevere-bot added the awaiting review label Aug 12, 2019

fix: rename variable

0baad95

maxking suggested changes Aug 16, 2019

View reviewed changes

Lib/email/_header_value_parser.py Outdated Show resolved Hide resolved

Lib/email/_header_value_parser.py Outdated Show resolved Hide resolved

Lib/test/test_email/test__header_value_parser.py Outdated Show resolved Hide resolved

bedevere-bot added awaiting changes and removed awaiting review labels Aug 16, 2019

epicfaace added 5 commits August 19, 2019 16:35

fix: parse unstructured word correctly, fix 1st bug

b7874bc

fix: handle invalid encoded word

364c549

fix format

c55361e

fix typo

2d5966f

update test

5a84182

bedevere-bot added awaiting change review and removed awaiting changes labels Aug 19, 2019

maxking suggested changes Aug 21, 2019

View reviewed changes

bedevere-bot removed the awaiting change review label Aug 21, 2019

bedevere-bot added the awaiting changes label Aug 21, 2019

epicfaace added 5 commits August 23, 2019 01:59

format for pep8

7155ac5

rename to valid_ew

eca9a64

add infinite loop comments

18cef4c

rename to InvalidEwError

941da1c

add tests for message_from_string

6f9da90

maxking approved these changes Aug 31, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Aug 31, 2019

maxking added needs backport to 3.7 labels Aug 31, 2019

maxking changed the title ~~bpo-37764: fix hang case in get_unstructured~~ bpo-37764: Fix infinite loop when parsing unstrucuted email headers. Aug 31, 2019

maxking changed the title ~~bpo-37764: Fix infinite loop when parsing unstrucuted email headers.~~ bpo-37764: Fix infinite loop when parsing unstructured email headers. Aug 31, 2019

maxking added the 🤖 automerge label Aug 31, 2019

miss-islington merged commit c5b242f into python:master Aug 31, 2019

bedevere-bot removed the awaiting merge label Aug 31, 2019

miss-islington self-assigned this Aug 31, 2019

maxking added the type-security A security issue label Aug 31, 2019

epicfaace mentioned this pull request Sep 3, 2019

[3.7] bpo-37764: Fix infinite loop when parsing unstructured email headers. (GH-15239) #15654

Merged

bedevere-bot removed the needs backport to 3.7 label Sep 3, 2019

epicfaace mentioned this pull request Sep 3, 2019

[3.8] bpo-21315: Fix parsing of encoded words with missing leading ws (GH-13425) #15655

Merged

epicfaace deleted the issue37764 branch September 3, 2019 04:42

bedevere-bot removed the needs backport to 3.8 label Sep 5, 2019

Uh oh!

bpo-37764: Fix infinite loop when parsing unstructured email headers. #15239

bpo-37764: Fix infinite loop when parsing unstructured email headers. #15239

Uh oh!

Conversation

epicfaace commented Aug 12, 2019 • edited by miss-islington Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxking left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Aug 16, 2019

Uh oh!

epicfaace commented Aug 19, 2019

Uh oh!

bedevere-bot commented Aug 19, 2019

Uh oh!

Uh oh!

Uh oh!

maxking Aug 21, 2019

Choose a reason for hiding this comment

Uh oh!

epicfaace Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Aug 21, 2019

Uh oh!

DeltaF1 commented Aug 22, 2019

Uh oh!

epicfaace commented Aug 23, 2019

Uh oh!

epicfaace commented Aug 23, 2019

Uh oh!

maxking left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Aug 31, 2019

Uh oh!

miss-islington commented Aug 31, 2019

Uh oh!

miss-islington commented Aug 31, 2019

Uh oh!

bedevere-bot commented Sep 3, 2019

Uh oh!

bedevere-bot commented Sep 5, 2019

Uh oh!

Uh oh!

epicfaace commented Aug 12, 2019 •

edited by miss-islington

Loading

epicfaace Aug 23, 2019 •

edited

Loading