Make errors due to full parser stack identifiable

[Trundle]

BPO	11343
Nosy	@loewis, @birkenfeld, @terryjreedy, @ncoghlan, @orsenthil, @marienz, @pitrou, @benjaminp, @aroberge, @bitdancer, @Trundle, @ericsnowcurrently, @pablogsal, @iritkatriel
Files	issue11343.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-02-26.22:39:50.959>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Make errors due to full parser stack identifiable'
updated_at = <Date 2021-07-08.16:46:57.711>
user = 'https://github.com/Trundle'

bugs.python.org fields:

activity = <Date 2021-07-08.16:46:57.711>
actor = 'pablogsal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2011-02-26.22:39:50.959>
creator = 'Trundle'
dependencies = []
files = ['21387']
hgrepos = []
issue_num = 11343
keywords = ['patch']
message_count = 29.0
messages = ['129604', '130076', '132063', '134778', '140456', '140469', '140470', '140479', '140480', '140496', '140498', '140499', '140500', '140501', '140502', '140504', '140506', '140507', '140517', '140631', '140635', '140646', '396883', '397125', '397133', '397151', '397152', '397153', '397154']
nosy_count = 15.0
nosy_names = ['loewis', 'georg.brandl', 'terry.reedy', 'ncoghlan', 'orsenthil', 'marienz', 'pitrou', 'benjamin.peterson', 'aroberge', 'Arfrever', 'r.david.murray', 'Trundle', 'eric.snow', 'pablogsal', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11343'
versions = ['Python 3.11']

[Trundle]

Currently, if the parser's internal stack is full (as it happens when
the parser tries to parse a deeply nested structure), the parser
writes an error message to stderr and a bare MemoryError is
raised. That way, it is really hard for REPLs do handle input that
causes the parser to exhaust its stack correctly (see e.g. python -c "print('(' * 100 + ')' * 100)" | python -c "import code; code.interact()").

In my opinion, a SyntaxError would be great because that way one can
easily see where in the source code the error is caused. One reason
against that (stated by MvL in msg1687) is that it's really
syntactically correct Python and only the parser can't parse it. But
then, the same is also true when there are too many indentation levels
in which case an IndentationError is raised (which is a subclass of
SyntaxError).

I have attached two patches: The first still raises a MemoryError
(for the sake of backward compatibility) but adds a message ("too many
opening parens") to it. The second raises a SyntaxError instead.

[terryjreedy]

I agree that compile-time stack exhaustion is different from runtime object-heap exhaustion and could/should have a different error.

I agree with Martin (from 2000) that SyntaxError is not right either. Perhaps a new ParseError subclass thereof. I believe IndentationError was added since 2000. Its doc is "Base class for syntax errors related to incorrect indentation." and it has TabError as a subclass, so its use for too many indents (not really 'incorrect') is not obvious. But having the position marked (if it would be) would be a plus.

I presume REPL == read-eval-print-loop (from Google). Would a new error help such programs (like code.interact, or IDLE)?

[Trundle]

On Fri, Mar 4, 2011 at 9:30 PM, Terry J. Reedy <report@bugs.python.org> wrote:

I agree with Martin (from 2000) that SyntaxError is not right either. Perhaps a new ParseError subclass thereof.

I added a new ParserError that inherits from SyntaxError.

I presume REPL == read-eval-print-loop (from Google). Would a new error help such programs (like code.interact, or IDLE)?

Yes, I meant a read-eval-print-loop. A new error would help and if
it's a subclass of SyntaxError, they most likely need no changes at
all to handle that new error, as they have to deal with SyntaxErrors
anyway.

[Trundle]

FWIW, this also affects ast.literal_eval().

[pitrou]

This looks like a rather good idea, although I'm not sure it deserves a new global exception type.

[benjaminp]

Just make it a SyntaxError.

[benjaminp]

Oh, I see Martin disagrees. SyntaxError is raised for anything which Python won't compile. For example, too many arguments gets a SyntaxError.

[ncoghlan]

+1 for a new exception type to indicate "this may technically be legal Python, but this Python implementation cannot handle it correctly"

Whatever exception type we add, it would be nice to also be able to use it if someone eventually fixes the compiler recursion crasher, so I'd like to paint this particular bikeshed as the more general "SyntaxLimitError".

Possible docs phrasing:

exception:: SyntaxLimitError

Raised when the compilation and code generation process encounters an error due to internal limitations of the current implementation (e.g. the CPython parser has an upper limit on the number of nested sets of parentheses it can handle in a single expression, but no such limitation exists in the Python language reference).
This is a subclass of :exc:`SyntaxError`.

[birkenfeld]

I like SyntaxLimitError much better than ParserError.

[benjaminp]

2011/7/16 Nick Coghlan <report@bugs.python.org>:

Nick Coghlan <ncoghlan@gmail.com> added the comment:

+1 for a new exception type to indicate "this may technically be legal Python, but this Python implementation cannot handle it correctly"

Whatever exception type we add, it would be nice to also be able to use it if someone eventually fixes the compiler recursion crasher, so I'd like to paint this particular bikeshed as the more general "SyntaxLimitError".

What is the point of a new exception? When would you ever want to
catch it as opposed to a regular SyntaxError? You're going to have to
change the code either way.

Moreover, all Python implementations are going to have to place some
limits on stack depth and recursion, so it's not really an
implementation detail.

The Python language doesn't make an mention of limited memory, but no
one is going to suggest a MemoryLimitError, which is an
"implementation detail".

[ncoghlan]

It's important to remember that other implementations treat CPython as
the "gold standard" for compatibility purposes. If we declare
something to be an ordinary SyntaxError, then that carries strong
implications for what other implementations should do.

Some kinds of errors are inherently implementation specific
(MemoryError and SystemError spring to mind). No sane implementor is
going to try to match CPython like-for-like when it comes to those.
SyntaxError, however, is typically defined by the language definition,
not the CPython implementation of it. By introducing a new exception
type, we're explicitly telling other implementations "Look, this is
our problem that we don't plan to fix as it doesn't typically arise in
real code, but please don't deliberately cripple your own
implementation just to match this behaviour". I'm strongly with MvL on
this one - SyntaxError itself should never be raised for legal Python
code just because the CPython bytecode generation toolchain isn't able
to handle it properly.

[benjaminp]

2011/7/16 Nick Coghlan <report@bugs.python.org>:

Nick Coghlan <ncoghlan@gmail.com> added the comment:

It's important to remember that other implementations treat CPython as
the "gold standard" for compatibility purposes. If we declare
something to be an ordinary SyntaxError, then that carries strong
implications for what other implementations should do.

Some kinds of errors are inherently implementation specific
(MemoryError and SystemError spring to mind). No sane implementor is
going to try to match CPython like-for-like when it comes to those.
SyntaxError, however, is typically defined by the language definition,
not the CPython implementation of it. By introducing a new exception
type, we're explicitly telling other implementations "Look, this is
our problem that we don't plan to fix as it doesn't typically arise in
real code, but please don't deliberately cripple your own
implementation just to match this behaviour". I'm strongly with MvL on
this one - SyntaxError itself should never be raised for legal Python
code just because the CPython bytecode generation toolchain isn't able
to handle it properly.

Implementations are quite good at figuring this stuff out themselves.
We don't need a whole new exception (new docs, one more thing in the
builtin namespace) to "send a message" to other implementations. If
some implementation is not part of the language, then its test just
just be marked cpython_only. It shouldn't be brought into public API
and the language.

[ncoghlan]

It also makes it clear to users whether they've just run up against a
limitation of the implementation they're using or whether what they've
written is genuinely illegal code. They are NOT the same thing.
Attempting to conflate them into one exception for the mere sake of
not adding a new exception type is a false economy. Take it to
python-dev if you want, but I will strongly oppose this check going in
with it raising an ordinary SyntaxError instead of a new subclass.