Cannot specify return type with Union[..., NotImplemented] (and others, e.g. Ellipsis)

[kamahen]

[UPDATE: The reported behavior happens only with --warn-return-any or --strict.]

Mypy gives warning: Returning Any from function declared to return "bool" for

class C:
    def m(self) -> bool:
        return NotImplemented

So, try this:

from typing import Union
class C:
    def m(self) -> Union[bool, type(NotImplemented)]:
        return NotImplemented

which gives error

q.py:3: error: invalid type comment or annotation
q.py:3: note: Suggestion: use type[...] instead of type(...)

Following mypy's suggestion results in:

q.py:3: error: Return type becomes "Union[bool, Type[Any]]" due to an unfollowed import
q.py:4: warning: Returning Any from function declared to return "Union[bool, Type[Any]]"

which also gives a runtime error: TypeError: Parameters to generic types must be types. Got NotImplemented. or (when attempting type[...]) TypeError: 'type' object is not subscriptable.

(See also pull #4770, which attempts to fix some magic methods)

There seem to be two issues:

1. Only magic methods can return NotImplemented without a complaint from mypy
2. If the programmer tries to get rid of the mypy complaint by specifying -> Union[bool, type(NotImplemented)], mypy doesn't like that and suggests changing it to -> Union[bool, type[NotImplemented]], which gets a runtime error (Type[NotImplemented] doesn't work either).

[Strilanc]

A similar issue occurs with Ellipsis. For example, if you try to specify the type of a method which operates on slices possibly involving ellipses, there's no way to say "I return a tuple containing items that are an integer, a slice instance, or Ellipsis".

I suggest that these, and other, builtin singleton values be handle the same way that None is handled. If you say Union[int, None], that's the same as saying Optional[int] even though technically you should say type(None) or NoneType instead of None because None is a value not a type.

[Strilanc]

Using the string 'NotImplemented' appears to work when invoking mypy and at runtime.

Union[int, 'NotImplemented']

[ilevkivskyi]

Yes, I think we can allow NotImplemented as a singleton (you can open an issue on typing tracker for the runtime part). But Ellipsis will be harder, since it already has special meaning, for example Tuple[int, ...] is a variable length tuple. How would we distinguish it from a tuple of two elements? (A similar problem appears for Callable).

[whatisaphone]

I suggest mypy continues to treat ... as it does today, but treats the long spelled-out form Ellipsis as the type of the singleton. In other words:

foo: Tuple[int, ...] = (2, 3, 4)
foo: Tuple[int, Ellipsis] = (2, ...)

Switching them around would cause errors in both cases:

foo: Tuple[int, Ellipsis] = (2, 3, 4)  # error: 3 elements found, but only 2 expected
foo: Tuple[int, ...] = (2, ...)  # error: type(Ellipsis) is not assignable to int

[Jackenmen]

Could also just support usage of type(Ellipsis)/type(...) as a type alias (maybe explicit type alias, i.e. PEP 613), so that one could do:

# the commented version could be used instead,
# if this were to use explicit type aliases feature:
# EllipsisType: TypeAlias = type(Ellipsis)
EllipsisType = type(Ellipsis)

foo: Tuple[int, EllipsisType] = (2, ...)

How does that look?

I suggest mypy continues to treat ... as it does today, but treats the long spelled-out form Ellipsis as the type of the singleton.

As far as I know, this would not be really possible as Tuple[int, ...] and Tuple[int, Ellipsis] mean the same at runtime.

[hauntsaninja]

Some more discussion at https://mail.python.org/archives/list/[email protected]/thread/277RFXNQYLT6D3R4EYZZOPPGX56SNIK6/

[julian-klode]

So, NotImplemented worked fine as a return type in e.g. Union[int, NotImplemented] in mypy 0.782, but now in mypy 0.790 it says

apt/package.py:473: error: Variable "builtins.NotImplemented" is not valid as a type
apt/package.py:473: note: See https://mypy.readthedocs.io/en/latest/common_issues.html#variables-vs-type-aliases

Making it a Literal[NotImplemented] does not help:

apt/package.py:473: error: Parameter 1 of Literal[...] is invalid

I'm a bit unhappy.

[hauntsaninja]

mypy_primer bisects this to python/typeshed#4222 to fix python/typeshed#3315
Previously NotImplemented was just Any, which is why you got the complaint with --warn-return-any.

Seems like we should sprinkle some more special casing on this. Not the greatest workaround, but you could use NotImplementedT = Any. You could also dangerously rely on typeshed implementation details here and do some if TYPE_CHECKING stuff.
In most cases, I think the best thing to do is to not put NotImplemented in the return type and just type ignore if using with --warn-return-any. (You should only want to be able to put NotImplemented in your return type if you want all callers to be aware of that and have to check that the returned value is NotImplemented)

[julian-klode]

Yeah, um, in my case I'm not sure what the code is doing, as it wraps a cmp-style function, and the callers then do > 0, for __gt__ for example, which produces a TypeError with TypeError: '>' not supported between instances of 'NotImplementedType' and 'int'. I think that code is buggy, as the __gt__ functions and friends should forward NotImplemented return, but then mypy never gave me a bug about that :/

[EPronovost]

@julian-klode Not sure about your situation but I just encountered issues related to NotImplemented after upgrading to mypy 0.790. Here is the commit that fixed things for me. numba/numba@55b6eda

[ItsDrike]

@julian-klode Not sure about your situation but I just encountered issues related to NotImplemented after upgrading to mypy 0.790. Here is the commit that fixed things for me. numba/numba@55b6eda

This is interesting, yet really weird. It means that mypy will handle something like this:

def foo() -> bool:
    return NotImplemented

as valid type-wise, even though this function never actually returned what was expected of it (being a bool). This probably shouldn't be the case. Even though it may be helpful since it allows for marking the return type of __gt__ or other comparison methods as simple bool even though it may also return NotImplemented, but I'm a lot more inclined to just use a union with NotImplemented and a bool which is more explicit and doesn't allow weird behavior like this.

Also, since I didn't see any quick fix for this, if someone needs it, this is how I was able to get a mypy compliant annotation that works without issues:

NotImplementedType = type(NotImplemented)

def foo() -> NotImplementedType:
    ...

I'm relying on type(NotImplemented) to get the internal builtin NotImplementedType because for some reason, it isn't included in the types stdlib module which is the place I'd expect it to be defined.

Edit: Apparently NotImplementedType is in types since python 3.10, however I don't expect most people to already be on that version so the solution above is still the one you'd want to use unless you've already migrated to purely 3.10, in which case you can switch to from types import NotImplementedType and use it directly.

[flisboac]

@ItsDrike In mypy 0.930, def foo() -> bool works (but should it? I think not).

But NotImplementedType = type(NotImplemented) fails with the message:

Variable "main.NotImplementedType" is not valid as a type
See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

[hauntsaninja]

mypy is basically working as expected here. NotImplemented is allowed to duck type to better integrate with Python's data model. Any follow-ups are perhaps better discussed over at typeshed.

[alexshpilkin]

@hauntsaninja

NotImplemented is allowed to duck type to better integrate with Python's data model. Any follow-ups are perhaps better discussed over at typeshed.

This does not make any sense whatsoever to me, sorry. Can you elaborate?

Is every type in Mypy implicitly a union with NotImplementedType (and if so, where is that documented, if anywhere)? Are the double-dispatch arithmetic and comparison dunders special-cased (and if so, how do I get that treatment for my own methods or functions, given Mypy doesn’t seem to recognize NotImplementedType)? Is there a sense of “duck typing” in which NotImplemented does legitimately act like a bool, an int, a fractions.Fraction, and every other arithmetic-like (not a technical term) type now known or heretofore invented?

I don’t mean to be confrontational, I really don’t, but these are the best of my guesses at what you could mean and they all sound kind of stupid to me now that I’ve written them out. What am I missing?

[JelleZijlstra]

Basically NotImplemented is treated like Any: https://github.com/python/typeshed/blob/2cd8f26e6671724351dd8d2b92f22738d8e85df8/stdlib/builtins.pyi#L1257

[alexshpilkin]

@JelleZijlstra Ah, I see, so indeed every type is implicitly a union with { NotImplemented }. This is ... a bit surprising (given None does not get the same treatment), but fundamentally okay. Probably should be documented, though.

@hauntsaninja What about Ellipsis, though? Should it be reported as a separate issue? Because it seems to me that you still cannot write a reasonable type for lambda: Ellipsis, let alone lambda x: x + 1 if x else Ellipsis.

[erictraut]

types.EllipsisType was added in Python 3.10.

[alexshpilkin]

@erictraut So typeshed just needs to be changed to stop faking it (in https://github.com/python/typeshed/blob/2cd8f26e6671724351dd8d2b92f22738d8e85df8/stdlib/builtins.pyi#L1908)? OK, I see.