Generic class and Union TypeVar

[geckyan]

Description

I've been trying to make "type-safe" ("type checkable") interfaces for custom parsing facilities, wrote generic Parser[ParsingT, ParsedT] and then MultiParser[ParsingT, Union[...]] to compose parsers. Latter gives me weird errors. I don't know whether it's a bug or my way of using generic classes has flaws. The code below will tell more... :)

Minimal code to reproduce:

from typing import Generic, TypeVar, Union

BarT = TypeVar('BarT')

class BasicFoo(Generic[BarT]):
    pass

class Foo(BasicFoo[BarT]):
    def add(self, sub_foo: BasicFoo[BarT]):
        pass

foo = Foo[Union[int, str]]()
foo.add(Foo[int]()) # Error: Argument of type 'Foo[int]' cannot be assigned to parameter 'sub_foo' of type 'BasicFoo[Union[int, str]]'. Type 'int' cannot be assigned to type 'Union[int, str]'.
foo.add(Foo[str]()) # Error: Argument of type 'Foo[str]' cannot be assigned to parameter 'sub_foo' of type 'BasicFoo[Union[int, str]]'. Type 'str' cannot be assigned to type 'Union[int, str]'.

Expected behaviour

No errors... I suppose so.

Context:

• Python 3.8.2
• VS Code 1.43.1 (user setup)
• pyright 1.1.28

[erictraut]

I tried your example above with mypy, another popular type checker, and I verified that Pyright is behaving consistently. So I'm confident Pyright is doing the right thing in this case, even though it might be somewhat unintuitive. Here's what's going on...

Unless otherwise specified, a declared TypeVar is considered "invariant". That means that once the type of the TypeVar is established within a context, all other places where that TypeVar is used must match it exactly. You can't match it with a narrower type (e.g. a subclass) or wider type (e.g. an ancestor class or a union).

In your example above, when you allocated foo, the type checker associates BarT with the type Union[int, str]. All subsequent uses of BarT relating to foo must also be of that type. When you call the add method on foo, you're providing an argument of type BasicFoo[int], but int is "narrower" than the required type (Union[int, str]).

If you want to allow the use of narrower types in your classes, you need to mark the TypeVar BarT as "covariant". Here's how that would look:
BarT = TypeVar('BarT', covariant=True)

For more details on invariance, covariance, and contravariance, see PEP 483 and 484.

[geckyan]

Thank you very much for the clear explaining! 😃👍

I think "Type A cannot be assigned to Union[A, ...]" message was a little bit misleading for me because it is correct to put A() value to a variable of type Union[A, ...]:

val: Union[int, str] = 10

[erictraut]

I can see why that error message might be confusing. I've added code to provide additional context. I think this will make things clearer. Let me know what you think.

[geckyan]

That is much clearer. It gives a nice hint about context of type error. I can imagine as someone try type variables, see "Type variable X is *variant" and then read TypeVar documentation... 🤓

[Congee]

That is much clearer. It gives a nice hint about context of type error. I can imagine as someone try type variables, see "Type variable X is *variant" and then read TypeVar documentation... 🤓

You are ..... right. I am reading the TypeVar documentation. How can I make sure there's no type error on the assignment of a to b?

a: Dict[str, str] = {'': ''}
b: Dict[Union[str, int], Union[float, str]] = a  # type error here at `a`
_ = b

c: str = ''
d: Union[str, int] = c
_ = d

Expression of type "Dict[str, str]" cannot be assigned to declared type "Dict[str | int, float | str]"
  TypeVar "_KT" is invariant
    Type "str" cannot be assigned to type "str | int"
[Pyright (reportGeneralTypeIssues):]

@erictraut Thank you for your fantastic work. Can you please kindly help here?

[erictraut]

First, let me make sure you understand why this is an error. It may not be obvious from the TypeVar documentation. The value a has a type of Dict[str, str]. That means only str keys and values can be inserted into this dictionary. Any attempt to insert non-str keys or values will violate the type.

In the next line, you attempt to assign a to b, which has a declared type of Dict[Union[str, int], Union[float, str]]. Note that you have not made a copy of a. Both a and b point to the same object. But now b is a type that allows you to insert non-str keys and values, which "corrupts" the object that a points to, violating the type guarantees.

There are two workarounds here:

1. Declare a's type as the same as b.

a: Dict[Union[str, int], Union[float, str]] = {"": ""}
b: Dict[Union[str, int], Union[float, str]] = a

(or use a type alias to make it more redable)

MyDict = Dict[Union[str, int], Union[float, str]]
a: MyDict = {"": ""}
b: MyDict = a

2. If you don't want to declare the values of a as a union, you can use a Mapping type.

a: Dict[Union[str, int], str] = {"": ""}
b: Mapping[Union[str, int], Union[float, str]] = a

The Mapping type is similar to a Dict except that it's read-only. That guarantees that you won't violate the type guarantees for a.

[Congee]

That copy enlightened me. I forgot b can mutate a. And the Covariance & Contravariance section in PEP484 does mention what's happening here.

So, as a library consumer, I went by creating a copy of a b: Dict[Union[str, int], Union[float, str]] = {**a}, because I have no control over a and performance is not mission critical.

From the perspective of a library author, the Mapping approach is great! Though I was confused why the key of Mapping cannot be a covariant, your comment explained why python/typing#273

Thank you so much for detailed answers and patience. I appreciate it!