Generic class with constrained type vars

[wkschwartz]

On Python 3.7.0 and mypy 0.620, I'm getting some error messages I don't understand. The Mypy documentation and PEP 484 didn't clear it up for me. Maybe it's a bug in Mypy. Maybe I don't understand type erasure. In case it's the former, here's a minimal example. In case it's the latter, please help.

In the following examples, the fact that m does not return is not critical. In the real code that gave me these errors, m is an abstract method, which also doesn't change anything important.

Put the following examples in mypy-test.py to get the corresponding output from running mypyt mypy-test.py. The first and the last examples are the ones that are baffling me.

classmethod with constrained type variable

from typing import TypeVar, Generic
T = TypeVar('T', int, str)
class K(Generic[T]):
	@classmethod
	def m(cls) -> T:
		raise NotImplementedError

Mypy output:

mypy-test.py:5: error: The erased type of self 'def [T in (builtins.int, builtins.str)] () -> mypy-test.K[builtins.int*]' is not a supertype of its class 'Type[mypy-test.K[T`1]]'
mypy-test.py:5: error: The erased type of self 'def [T in (builtins.int, builtins.str)] () -> mypy-test.K[builtins.str*]' is not a supertype of its class 'Type[mypy-test.K[T`1]]'

classmethod with generic self

from typing import TypeVar, Generic
S = TypeVar('S')
T = TypeVar('T', int, str)
class K(Generic[T]):
	@classmethod
	def m(cls: S) -> T:
		raise NotImplementedError

produces no error.

classmethod with unconstrained type variable

from typing import TypeVar, Generic
T = TypeVar('T')
class K(Generic[T]):
	@classmethod
	def m(cls) -> T:
		raise NotImplementedError

produces no error.

instance method with constrained type

from typing import TypeVar, Generic
T = TypeVar('T', int, str)
class K(Generic[T]):
	def __init__(self, t: T) -> None:
		self.t = t

produces

mypy-test.py:5: error: Need type annotation for 't'

instance method with unconstrained type

from typing import TypeVar, Generic
T = TypeVar('T')
class K(Generic[T]):
	def __init__(self, t: T) -> None:
		self.t = t

produces no error.

instance method with constrained type and inheritance

from typing import TypeVar, Generic
T = TypeVar('T', int, str)
class K(Generic[T]):
	def __init__(self, t: T) -> None:
		self.t: T = t

class L(K[T]):
	def __init__(self, t: T) -> None:
		self.u = 1
		super().__init__(t=t)

produces

mypy-test.py:10: error: Argument "t" to "__init__" of "K" has incompatible type "int"; expected "T"
mypy-test.py:10: error: Argument "t" to "__init__" of "K" has incompatible type "str"; expected "T"

[ilevkivskyi]

The first and the last examples are the ones that are baffling me.

Apart from the fact that using a generic class type variable in class methods is typically a bad idea (although not formally wrong), I think the first and last examples are bugs. Both are results of how mypy checks functions with constrained type variables.

Also I think @elazarg might help with the first one if he has time.

[wkschwartz]

Thanks for responding so quickly! I'll # type: ignore those for now.

Apart from the fact that using a generic class type variable in class methods is typically a bad idea (although not formally wrong)

In my actual code, the class method m is part of an abstract base class for wrapping other libraries for a particular type of functionality I need to interface with. Type T parameterizes a common type among Python libraries for my use case; it's usually either an int or str. Other methods in the base class expect inputs of type T, so I figured I should indicate that the input type T on those methods is the same as the output type T from class method m. For this type of use case, why would using a generic class type variable in the class method be a bad idea? Is there another way to handle this use case?

[ilevkivskyi]

For this type of use case, why would using a generic class type variable in the class method be a bad idea?

This is probably fine. However I would anyway try refactoring the code (if possible) to make m a normal instance method. This is because type variables are something tightly associated with instances, not classes (this is however a matter of style, since a subclass can bind the type variable as well).

[wkschwartz]

I would anyway try refactoring the code (if possible) to make m a normal instance method.

My intention with m is that it return a list of available/installed external programs known to the class available to be requested by command-line interfaces or other client code. I need the type variable because this all happens through third-party libraries, each of which refers to these external programs differently (some with strings, others with C enum integers). I could provide m as a static method, but that seemed unnecessarily to constrain future subclasses. I'm not sure how I could do this at all as an instance method: the CLI would have to instantiate an instance with fake data to obtain the information needed to tell users who ask for --help what data they can process. Thoughts?

[ilevkivskyi]

It looks like you are too worried :-) I said typically and if possible. In your situation it is perfectly OK.

[wkschwartz]

I just tested the examples in the original post using mypy 0.630. The first example no longer generates errors. The fourth and last examples continue to generate errors. Maybe related to fixing #5309?

[hvassard]

Hi there ! Still facing the last issue (7 years later lmao) with mypy 1.10.1 and python 3.11.9. Here's my code and mypy output :

Content of src/domain/models/searchable_string_item_list.py (parent class with constrained type)

from collections.abc import Iterable
from typing import TypeVar
from src.domain.models.key_word import KeyWord
from src.domain.models.regex import Regex

# KeyWord and Regex are 2 classes implementing interfaces
T = TypeVar("T", KeyWord, Regex)

class SearchableStringItemList(list[T]):
    def __init__(self, iterable: Iterable[T]) -> None:
        super().__init__(iterable)

# other instance methods not pasted here
# ...

Content of src/domain/models/key_word_list.py (child class no.1 inheriting from SearchableStringItemList)

from collections.abc import Iterable
from src.domain.models.key_word import KeyWord
from src.domain.models.searchable_string_item_list import SearchableStringItemList

class KeyWordList(SearchableStringItemList[KeyWord]):
    def __init__(self, iterable: Iterable[KeyWord]) -> None:
        if not all(isinstance(item, KeyWord) for item in iterable):
            raise TypeError(
                "Cannot create a KeyWordList from an iterable if it contains other types "
                f"than KeyWord. Received types : { {type(item) for item in iterable} }"
            )
        super().__init__(iterable)

# other instance methods not pasted here
# ...

Content of src/domain/models/regex_list.py (child class no.2 inheriting from SearchableStringItemList)

from collections.abc import Iterable
from src.domain.models.key_word import KeyWord
from src.domain.models.searchable_string_item_list import SearchableStringItemList

class KeyWordList(SearchableStringItemList[KeyWord]):
    def __init__(self, iterable: Iterable[KeyWord]) -> None:
        if not all(isinstance(item, KeyWord) for item in iterable):
            raise TypeError(
                "Cannot create a KeyWordList from an iterable if it contains other types "
                f"than KeyWord. Received types : { {type(item) for item in iterable} }"
            )
        super().__init__(iterable)

mypy output :

src\domain\models\searchable_string_item_list.py: note: In member "__init__" of class "SearchableStringItemList":
src\domain\models\searchable_string_item_list.py:21:26: error: Argument 1 to "__init__" of "list" has incompatible type "Iterable[KeyWord]"; expected "Iterable[T]"  [arg-type]
            super().__init__(iterable)
                             ^~~~~~~~
src\domain\models\searchable_string_item_list.py:21:26: error: Argument 1 to "__init__" of "list" has incompatible type "Iterable[Regex]"; expected "Iterable[T]"  [arg-type]  
            super().__init__(iterable)
                             ^~~~~~~~
Found 2 errors in 1 file

As mentioned above, adding a type ignore comment removes this mypy error and my code works perfectly fine :

Modified content of src/domain/models/searchable_string_item_list.py (parent class with type ignore comment)

class SearchableStringItemList(list[T]):
    def __init__(self, iterable: Iterable[T]) -> None:
        super().__init__(iterable) # type:ignore [arg-type]

# other instance methods not pasted here
# ...

If you have any idea of a better solution than adding a type ignore, I'd love to hear it !

Thanks 🙏