Discrepancy between `__args__` of `typing.Callable` and `collections.abc.Callable`

[AlexWaygood]

On both Python 3.11 and main @ b57105a:

>>> import typing, collections.abc
>>> Ts = typing.TypeVarTuple("Ts")
>>> typing.Callable[[*Ts], None].__args__
(*Ts, <class 'NoneType'>)
>>> collections.abc.Callable[[*Ts], None].__args__
(*Ts, None)

According to this assertion in the test suite for typing.py, the behaviour of typing.Callable is correct here, whereas the behaviour of collections.abc.Callable is incorrect:

cpython/Lib/test/test_typing.py

Lines 1290 to 1292 in b57105a

	a = Callable[[*Ts], None]
	b = Callable[[Unpack[Ts]], None]
	self.assertEqual(a.__args__, (*Ts, type(None)))

However, the test is currently only run with typing.Callable, whereas it should be run with collections.abc.Callable as well.

Cc. @mrahtz, who wrote this test. This may also be of interest to @sobolevn :)

[sobolevn]

Oh no 🔥 🚒 😆

[sobolevn]

This is not about Callable, the same happens with other types.GenericAlias objects:

>>> import typing

>>> list[None].__args__
(None,)
>>> typing.List[None].__args__
(<class 'NoneType'>,)

>>> set[None]
set[None]
>>> typing.Set[None]
typing.Set[NoneType]

[Eclips4]

Problem is here:

cpython/Lib/typing.py

Lines 160 to 166 in 9666387

	def _type_convert(arg, module=None, *, allow_special_forms=False):
	"""For converting None to type(None), and strings to ForwardRef."""
	if arg is None:
	return type(None)
	if isinstance(arg, str):
	return ForwardRef(arg, module=module, is_class=allow_special_forms)
	return arg

[Eclips4]

So, we can roughly delete this two lines from _type_convert and some other places in typing.py. Of course, test_typing.py need some fixes (replace in many places type(None) to None).
But, let's see what others say.
With this fixes, example from Alex works fine:

>>> import typing, collections.abc
>>> Ts = typing.TypeVarTuple("Ts")
>>> typing.Callable[[*Ts], None].__args__
(*Ts, None)
>>> collections.abc.Callable[[*Ts], None].__args__
(*Ts, None)

[sobolevn]

No, it is not a problem :)

The problem is in types.GenericAlias, but we are not sure (yet) that it is a bug.
typing has this behaviour for a long time.

[AlexWaygood]

@sobolevn is correct. The objects in typing.py are much older than types.GenericAlias, so if we need to change one of the two, it should be types.GenericAlias rather than the objects in typing.py.

But the fact that all types.GenericAlias objects behave the same way here makes me think that this was probably a deliberate decision.

[sunmy2019]

The behavior is specified here:

https://peps.python.org/pep-0484/#using-none

When used in a type hint, the expression None is considered equivalent to type(None).

https://peps.python.org/pep-0483/#pragmatics

Where a type is expected, None can be substituted for type(None); e.g. Union[t1, None] == Union[t1, type(None)].

https://peps.python.org/pep-0483/#fundamental-building-blocks

Optional[t1]. Alias for Union[t1, None], i.e. Union[t1, type(None)].

But I do not find any wording about None in the PEP of Type Hinting Generics.

This sounds like a bug to me. I think the correct behavior here is to make

collections.abc.Callable[[*Ts], None] == collections.abc.Callable[[*Ts], type(None)]

evaluates to True, no matter how the underlying works.

Perhaps a PEP is needed to resolve this?

[mrahtz]

Fwiw as the one who wrote the original test, I don't have a strong opinion about this - I don't think there's anything about the way that TypeVarTuples interacts with Callable which makes it matter whether it's None vs NoneType.

My personal vote is that it should be None, because it seems simpler and more intuitive: if you put Foo in the args to anything, including Callable, you should expect to see Foo in the __args__. In everyday life I just use None everywhere and it works fine, so I'd expect it to just be fine here too with no complications.

I do agree with @sunmy2019 that it seems correct for this to be the case:

collections.abc.Callable[[*Ts], None] == collections.abc.Callable[[*Ts], type(None)]

But I'd expect that to be true because None == type(None); I'm a bit surprised to find that it isn't:

>>> None is type(None)
False
>>> None == type(None)
False

This seems super weird to me considering that those PEPs @sunmy2019 is quoting explicitly says they should be equivalent. How difficult would this be to change?

[sobolevn]

It is not hard at all, but this is breaking change. We have types.GenericAlias with this behavior since 3.9:

Python 3.9.15 (main, Nov 22 2022, 17:18:20) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> list[None].__args__[0]
>>> import typing
>>> typing.List[None].__args__[0]
<class 'NoneType'>

I am pretty sure many libraries rely on this. I've tried pydantic:

>>> from pydantic import BaseModel
>>> class My(BaseModel):
...    x: list[None]
... 
>>> import types
>>> class Other(BaseModel):
...    y: list[types.NoneType]
... 
>>> My(x=[None])
My(x=[None])
>>> My(x=[type(None)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for My
x -> 0
  value is not None (type=type_error.not_none)
>>> Other(y=[None])
Other(y=[None])
>>> Other(y=[type(None)])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Other
y -> 0
  value is not None (type=type_error.not_none)

Looks like this exact lib handles list[type(None)] correctly.