Skip to content

gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin #94386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 5, 2022
Merged

gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin #94386

merged 2 commits into from
Jul 5, 2022

Conversation

pablogsal
Copy link
Member

@pablogsal pablogsal commented Jun 28, 2022 •
edited
Loading

@pablogsal pablogsal requested a review from lysnikolaou as a code owner June 28, 2022 13:21
@pablogsal pablogsal added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Jun 28, 2022
@pablogsal
Copy link
Member Author

pablogsal commented Jun 28, 2022 •
edited
Loading

I'm not adding a test (for now) because this doesn't reproduce nicely on the test suite.

@pablogsal pablogsal marked this pull request as draft June 28, 2022 13:46
@pablogsal
pythongh-94360: Fix a tokenizer crash when reading encoded files with…
a7cf92c 
… syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>
@pablogsal pablogsal marked this pull request as ready for review June 28, 2022 15:23
@zware
Copy link
Member

zware commented Jun 28, 2022

Confirmed that this resolves the crash as produced by my reduced reproducer in main and 3.11, and without the change to Parser/pegen_errors.c (which doesn't exist in 3.10) it also resolves both reproducers for 3.10.

@pablogsal
Copy link
Member Author

Thanks for checking @zware ! ♥️

ambv
ambv reviewed Jul 5, 2022
View reviewed changes
Parser/pegen_errors.c
@@ -259,15 +259,15 @@ get_error_line_from_tokenizer_buffers(Parser *p, Py_ssize_t lineno)
const char* buf_end = p->tok->fp_interactive ? p->tok->interactive_src_end : p->tok->inp;

for (int i = 0; i < relative_lineno - 1; i++) {
char *new_line = strchr(cur_line, '\n') + 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NULL + 1 is beautifully strongly typed, huh? 😎

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6lujn3.jpg

ambv
ambv reviewed Jul 5, 2022
View reviewed changes
@pablogsal pablogsal merged commit 36fcde6 into python:main Jul 5, 2022
@pablogsal pablogsal deleted the gh-94360 branch July 5, 2022 16:39
@miss-islington
Copy link
Contributor

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

@bedevere-bot
Copy link

GH-94573 is a backport of this pull request to the 3.11 branch.

@miss-islington
Copy link
Contributor

Sorry, @pablogsal, I could not cleanly backport this to 3.10 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 36fcde61ba48c4e918830691ecf4092e4e3b9b99 3.10

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jul 5, 2022
@pablogsal @miss-islington
pythongh-94360: Fix a tokenizer crash when reading encoded files with…
2012ce5 
… syntax errors from stdin (pythonGH-94386)

* pythongh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>

* nitty nit

Co-authored-by: Łukasz Langa <[email protected]>
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
miss-islington added a commit that referenced this pull request Jul 5, 2022
@miss-islington @pablogsal
gh-94360: Fix a tokenizer crash when reading encoded files with synta…
d49c99f 
…x errors from stdin (GH-94386)

* gh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>

* nitty nit

Co-authored-by: Łukasz Langa <[email protected]>
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
@bedevere-bot
Copy link

GH-94574 is a backport of this pull request to the 3.10 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label Jul 5, 2022
pablogsal added a commit to pablogsal/cpython that referenced this pull request Jul 5, 2022
@pablogsal
[3.10] pythongh-94360: Fix a tokenizer crash when reading encoded fil…
005fc4b 
…es with syntax errors from stdin (pythonGH-94386)

* pythongh-94360: Fix a tokenizer crash when reading encoded files with syntax errors from stdin

Signed-off-by: Pablo Galindo <[email protected]>

* nitty nit

Co-authored-by: Łukasz Langa <[email protected]>.
(cherry picked from commit 36fcde6)

Co-authored-by: Pablo Galindo Salgado <[email protected]>
ambv pushed a commit that referenced this pull request Jul 5, 2022
@pablogsal
[3.10] gh-94360: Fix a tokenizer crash when reading encoded files wit…
697e78c 
…h syntax errors from stdin (GH-94386) (GH-94574)

Signed-off-by: Pablo Galindo <[email protected]>
Co-authored-by: Pablo Galindo Salgado <[email protected]>
Co-authored-by: Łukasz Langa <[email protected]>

(cherry picked from commit 36fcde6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ambv ambv ambv left review comments

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Assignees

@pablogsal pablogsal

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Tokenizer crash when redirecting input to stdin
5 participants
@pablogsal @zware @miss-islington @bedevere-bot @ambv