Mypy now treats classes with __getitem__ as iterable
#13485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexWaygood
reviewed
Aug 23, 2022
test-data/unit/check-protocols.test
Outdated
| reveal_type(list(a for a in ReturnsTuple())) # N: Revealed type is "builtins.list[Tuple[builtins.int, builtins.str]]" | ||
|
|
||
| reveal_type(list(a for a in WrongGetItemSig())) # N: Revealed type is "builtins.list[Any]" \ | ||
| # E: "WrongGetItemSig" has no attribute "__iter__" (not iterable) |
There was a problem hiding this comment.
Hmm, this error message isn't ideal anymore, with this change. Could the error message be changed to just "WrongGetItemSig" is not iterable?
There was a problem hiding this comment.
Error messages are just insanely complex. I did my best in a new just-pushed iteration.
We can later further improve them.
test-data/unit/check-protocols.test
Outdated
Comment on lines
2076
to
2204
| [case testOtherFeaturesAreNotAffectedByOldStyleIterableProtocol] | ||
| from typing import Iterator | ||
| class A: | ||
| def __getitem__(self, arg: int) -> int: pass | ||
| def f() -> Iterator[int]: | ||
| yield from A() # E: "yield from" cannot be applied to "A" | ||
| [builtins fixtures/list.pyi] |
There was a problem hiding this comment.
Isn't this testing for incorrect behaviour? It seems like yield from works fine with old-style iteration at runtime.
>>> class Foo:
... def __getitem__(self, item):
... return self
...
>>> def func():
... yield from Foo()
...
>>> f = func()
>>>
>>> for _ in range(5):
... print(next(f))
...
<__main__.Foo object at 0x000002380E473CD0>
<__main__.Foo object at 0x000002380E473CD0>
<__main__.Foo object at 0x000002380E473CD0>
<__main__.Foo object at 0x000002380E473CD0>
<__main__.Foo object at 0x000002380E473CD0>There was a problem hiding this comment.
Oh wow, I didn't think that it works with yield from! TIL
There was a problem hiding this comment.
Even this works:
>>> class A:
... def __getitem__(self, arg):
... if arg > 2:
... raise IndexError
... return arg
...
>>> a, b, c = A()Crazy!
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
I don't like the message change:
I will revert it, we can work on a better message later on. |
Makes sense! |
This comment has been minimized.
This comment has been minimized.
sobolevn
commented
Aug 23, 2022
| @@ -27,6 +27,18 @@ print(list(reversed(A()))) | |||
| \['c', 'b', 'a'] | |||
| \['f', 'o', 'o'] | |||
|
|
|||
| [case testOldIterableAndIterableProtocolAreIncompatible] | |||
There was a problem hiding this comment.
This is something I want to discuss.
I can make Iterable and __getitem__ behave like subtypes. But this will be a lie!
Because Iterable suppose to have __iter__ method, which does not really exist on classes with only __getitem__.
So, any direct invokations of iterable.__iter__ will fail.
What should we do here?
- Make life easier and accept
__getitem__everywhere whereIterableis expected - Keep it as-is
- Find some middle-ground. For example, accept
__getitem__everywhere whereIterableis expected, but be very strict about explicit.__iter__method access.
There was a problem hiding this comment.
However, 2. does not make much sence. It won't solve problems with enumerate, list, and custom functions, where iteration is used.
This comment has been minimized.
This comment has been minimized.
|
For context, there was a recent (somewhat heated) discussion on this feature: https://discuss.python.org/t/deprecate-old-style-iteration-protocol/17863/ |
|
@thomkeh thanks! The general consensus there is to "keep things as they are" (and they have +1 from me, I don't see any reasons to remove this behaviour) Some thoughts about the implementation:
|
This comment has been minimized.
This comment has been minimized.
|
Ok, I will have to analyze new stubtest errors, there might be problems... |
|
Diff from mypy_primer, showing the effect of this PR on open source code: sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/domains/std.py: note: In member "run" of class "Glossary":
+ sphinx/domains/std.py:364:20: error: "ViewList" has no attribute "startswith" [attr-defined]
+ sphinx/domains/std.py:376:38: error: List item 0 has incompatible type "Tuple[ViewList, str, int]"; expected "Tuple[str, str, int]" [list-item]
+ sphinx/domains/std.py:385:47: error: Argument 1 to "append" of "list" has incompatible type "Tuple[ViewList, str, int]"; expected "Tuple[str, str, int]" [arg-type]
+ sphinx/domains/std.py:396:50: error: "ViewList" has no attribute "lstrip" [attr-defined]
+ sphinx/domains/std.py:410:13: error: Incompatible types in assignment (expression has type "str", variable has type "ViewList") [assignment]
+ sphinx/domains/std.py:411:48: error: Argument 1 to "split_term_classifiers" has incompatible type "ViewList"; expected "str" [arg-type]
+ sphinx/domains/std.py:419:34: error: Incompatible types in assignment (expression has type "ViewList", variable has type "str") [assignment]
discord.py (https://github.com/Rapptz/discord.py)
- discord/mentions.py:134: error: Incompatible types in assignment (expression has type "bool", target has type "List[Union[Any, int]]")
+ discord/mentions.py:134: error: Incompatible types in assignment (expression has type "bool", target has type "List[Any]")
- discord/mentions.py:136: error: Incompatible types in assignment (expression has type "List[str]", target has type "List[Union[Any, int]]")
prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/collections.py:184: error: Incompatible return value type (got "defaultdict[Union[Any, Type[T]], List[Any]]", expected "Union[List[T], Dict[Type[T], T]]")
+ src/prefect/utilities/collections.py:184: error: Incompatible return value type (got "defaultdict[Any, List[Any]]", expected "Union[List[T], Dict[Type[T], T]]")
xarray (https://github.com/pydata/xarray)
+ xarray/core/accessor_str.py:579: error: Incompatible return value type (got "DataArray", expected "T_DataArray") [return-value]
spark (https://github.com/apache/spark)
+ python/pyspark/ml/linalg/__init__.py:192: error: Unused "type: ignore" comment
+ python/pyspark/ml/linalg/__init__.py:401: error: Argument 1 to "dot" of "SparseVector" has incompatible type "DenseVector"; expected "Iterable[float]" [arg-type]
+ python/pyspark/ml/linalg/__init__.py:436: error: Argument 1 to "squared_distance" of "SparseVector" has incompatible type "DenseVector"; expected "Iterable[float]" [arg-type]
+ python/pyspark/ml/linalg/__init__.py:757: error: <nothing> has no attribute "array" [attr-defined]
+ python/pyspark/ml/linalg/__init__.py:758: error: Value of type variable "_SCT" of "zeros" cannot be "bool" [type-var]
+ python/pyspark/ml/linalg/__init__.py:758: error: "Iterable[float]" has no attribute "size" [attr-defined]
+ python/pyspark/ml/linalg/__init__.py:760: error: Value of type "Iterable[float]" is not indexable [index]
+ python/pyspark/ml/linalg/__init__.py:763: error: Value of type "Iterable[float]" is not indexable [index]
+ python/pyspark/mllib/linalg/__init__.py:201: error: Unused "type: ignore" comment
+ python/pyspark/mllib/linalg/__init__.py:454: error: Argument 1 to "dot" of "SparseVector" has incompatible type "DenseVector"; expected "Iterable[float]" [arg-type]
+ python/pyspark/mllib/linalg/__init__.py:489: error: Argument 1 to "squared_distance" of "SparseVector" has incompatible type "DenseVector"; expected "Iterable[float]" [arg-type]
+ python/pyspark/mllib/linalg/__init__.py:864: error: <nothing> has no attribute "array" [attr-defined]
+ python/pyspark/mllib/linalg/__init__.py:865: error: Value of type variable "_SCT" of "zeros" cannot be "bool" [type-var]
+ python/pyspark/mllib/linalg/__init__.py:865: error: "Iterable[float]" has no attribute "size" [attr-defined]
+ python/pyspark/mllib/linalg/__init__.py:867: error: Value of type "Iterable[float]" is not indexable [index]
+ python/pyspark/mllib/linalg/__init__.py:870: error: Value of type "Iterable[float]" is not indexable [index]
pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/internals/managers.py:897: error: Argument 1 to "BlockPlacement" has incompatible type "BlockPlacement"; expected "Union[int, slice, ndarray[Any, Any]]" [arg-type]
+ pandas/core/internals/managers.py:1150: error: No overload variant of "__setitem__" of "list" matches argument types "BlockPlacement", "Any" [call-overload]
+ pandas/core/internals/managers.py:1150: note: Possible overload variants:
+ pandas/core/internals/managers.py:1150: note: def __setitem__(self, SupportsIndex, Optional[ndarray[Any, Any]], /) -> None
+ pandas/core/internals/managers.py:1150: note: def __setitem__(self, slice, Iterable[Optional[ndarray[Any, Any]]], /) -> None
+ pandas/core/internals/blocks.py:409: error: Argument 1 to "BlockPlacement" has incompatible type "BlockPlacement"; expected "Union[int, slice, ndarray[Any, Any]]" [arg-type]
ibis (https://github.com/ibis-project/ibis)
- ibis/expr/types/relations.py:778: error: Argument 1 to "difference" of "frozenset" has incompatible type "Schema"; expected "Iterable[object]"
- ibis/expr/types/relations.py:778: note: Following member(s) of "Schema" have conflicts:
- ibis/expr/types/relations.py:778: note: Expected:
- ibis/expr/types/relations.py:778: note: def __iter__(self) -> Iterator[object]
- ibis/expr/types/relations.py:778: note: Got:
- ibis/expr/types/relations.py:778: note: def __iter__(self) -> Iterable[str]
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This feature keeps coming up:
enumeratedoesn't understand classes with __len__ and __getitem__ #13451__getitem__as not iterables #11695My work is based on #10386
Closes #10386
Closes #2220
I've tried to cover major cases in tests, but any suggestions are welcome :)