Custom str subclass as keyword argument leads to System Error

[jeff5]

When arguments are given by keyword using a dictionary, CPython requires that the key be of type str, or a sub-class. If a custom str subclass defines a badly-behaved __eq__, and is used that way, a SystemError (null argument) can result, which is not expected in the interpreter.

This report stems from a discussion on discuss.python.org about the types of keyword that ought to be allowed -- it's not code I actually want to run.

Reproducer

# kwargsbomb.py Use of a str subclass as a keyword

class S(str):
    def __eq__(self, other):
        print(other)
        return True
    def __hash__(self):
        return 42

u = "my.domain.name"

a = u.split(**{'sep':'.'})     # ok
print(a)

b = u.split(**{S('sep'):'.'})  # ok
print(b)

c = u.split(**{S('xxx'):'.'})  # fails
print(c)

Output

['my', 'domain', 'name']
['my', 'domain', 'name']
sep
Traceback (most recent call last):
  File " ... \kwargsbomb.py", line 18, in <module>
    c = u.split(**{S('xxx'):'.'})  # fails
        ^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: null argument to internal routine

Environment

• CPython versions: 3.10.2 3.11.0b4 (in Idle and at the prompt)
• Operating system and architecture: Windows 10 64-bit

[JelleZijlstra]

I'm not sure this should be supported. The discuss.python.org discussion was focused on strings that aren't valid identifiers, not on subclasses of str.

Restricting kwargs to only str may allow some optimizations and saves us from possible subtle bugs (what if a str subclass overrides __hash__). In addition, there's no known use case for using str subclasses.

I suggest we:

• Document that kwargs must be exactly of type str
• Fix the code so that attempting to use a str subclass raises TypeError instead of SystemError

[serhiy-storchaka]

The user should not be able to provoke a SystemError from pure Python code. This exception is reserved for misuse of the C API. There is a bug somewhere in _PyArg_UnpackKeywords(). In other examples it can lead to crash, memory corruption, or just silent produce incorrect result.

[jeff5]

The discuss.python.org discussion was focused on strings that aren't valid identifiers, not on subclasses of str.

It starts with the question why integer keys are not allowed. When I looked where this is enforced, I was interested to find that it accepted sub-classes of str. So I tried it, idly, and idle restarted (in 3.8).

Restricting kwargs to only str may allow some optimizations and saves us from possible subtle bugs (what if a str subclass overrides __hash__).

I agree, and favour that, but I cannot say for certain no-one is relying on it. Also @serhiy-storchaka is right that something is wrong elsewhere, and may go wrong in other circumstances, which this daft trick just serves to discover.

[serhiy-storchaka]

Thank you for your report @jeff5. It was an interesting bug.

[merwok]

See also #96397