Mypy doesn't recognize a class as iterable when it implements only __getitem__

[akaptur]

The following code runs fine, but mypy produce an error:

from typing import List

class Foo(object):
    def __init__(self):
        self.version = [1, 2, 3]

    def __getitem__(self, k):
        return self.version[k]


def list_foo(f):
    # type: (Foo) -> List[int]
    return list(f)

print list_foo(Foo())

example.py: note: In function "list_foo":
example.py:13: error: No overload variant of "list" matches argument types [example.Foo]
Fail

[gvanrossum]

We should support this. It is part of the Python spec for iter(): https://docs.python.org/3/library/functions.html#iter

Without a second argument, object must be a collection object which supports the iteration protocol (the iter() method), or it must support the sequence protocol (the __getitem__() method with integer arguments starting at 0). [emphasis mine]

[JukkaL]

We currently require an explicit Iterable base class, as mypy doesn't have structural subtyping. A potential quick fix would be to not complain about unimplemented __iter__ in a subclass of Iterable if there is a suitable __getitem__.

A better fix would require structural subtyping (or protocols), but that's going to be harder. And we'd need another special case for __getitem__.

[chadrik]

I ran into this problem with sre_parse.SubPattern using mypy 0.580. Is there an adjustment that still needs to be made to structural subtyping to recognize this, or should the stubs be "fixed" for objects like these?

[ilevkivskyi]

This is not really a mypy problem. If you ask Python at runtime

class A:
    def __iter__(self):
        ...
class B:
    def __getitem__(self):
        ...
isinstance(A(), collections.abc.Iterable)  # True
isinstance(B(), collections.abc.Iterable)  # False

There were long discussions about this but there is still no decision about this. If you control the expected type you can define your own protocol:

class OtherIterable(Protocol[T]):
    def __getitem__(self, index: int) -> T:
        ...
WideIterable = Union[Iterable[T], OtherIterable[T]]

def fun(arg: WideIterable[str]) -> None:
    ...

class One:
    def __iter__(self) -> Iterator[str]:
        ...
class Other:
    def __getitem__(self, index: int) -> str:
        ...

fun(One())  # OK
fun(Other())  # OK

or whatever else you want.

[chadrik]

So what's the solution for sre_parse.parse? To be clear, the problem is that I get false errors in mypy when running the following code:

for pattern in sre_parse.parse('(\w+)(\d+)'):
    print(pattern)

Mypy prints:

error: Iterable expected
error: "SubPattern" has no attribute "__iter__

[JelleZijlstra]

We should probably just lie in the stubs for sre_parse and claim that SubPattern is Iterable.

[pmhahn]

Same problem for Exceptions in Python2:

d = {0: 0}
try:
  d[1]
except KeyError as ex:
  (key,) = ex
# error: 'builtins.KeyError' object is not iterable

I can work-around that by explicitly accessing ex.args, but I still get that error for existing code.
But as exceptions in Python3 no longer work like this, I will have to update that code anyway in the future.

[Recursing]

Has there been any progress on this issue?

[glesica]

This problem still appears to exist in version 0.790. Specifically, the sorted function appears not to like getting an ItemsView instance even though it is technically iterable and the whole thing works at runtime.

Edit: turns out I had an unrelated type error that was causing this problem, never mind me!

[hasansalimkanmaz]

Any solution or workaround to this? I get the same error. I don't want to ignore these errors by type: ignore as it seems to be a bad practice.

[ahelwer]

For sre_parse.SubPattern I silenced this error by changing:

for x in subpattern:
  ...

to

for x in subpattern.data:
  ...

not really ideal since it accesses a field that is probably intended to be private/internal, but it works.

[hauntsaninja]

There was some more recent discussion in python/typeshed#7813

The outcome is python/typeshed#7817 , which gives a not terrible workaround — just add an iter:

class Foo:
    def __init__(self):
        self.version = [1, 2, 3]

    def __getitem__(self, k):
        return self.version[k]

def your_code_that_uses_foo(f: Foo) -> list[int]:
    # return list(f)
    return list(iter(f))

print(list_foo(Foo()))

If you maintain an iterable class, just add an __iter__ method. You can do this by inheriting from collections.abc.Sequence or by adding your own:

def __iter__(self) -> Iterator[T]:
    try:
        for i in itertools.count():
            yield self[i]
    except IndexError:
        pass

[udifuchs]

I just ran into this issue. In my case, I have control over the class, so I added an __iter__ implementation.
The naive implementation is:

    def __iter__(self):
        return iter(self[:])

In my case __getitem__ returns a different value for slices and for integer indices.
Indeed, the following is closer to how python generates the iterator:

    def __iter__(self):
        return iter(self[i] for i, _item in enumerate(self[:]))

I don't how python actually implements this, so there might be other cases which are not covered.