Dealing with modules that don't expose type objects

[JukkaL]

Some modules (for example, re) don't expose some of the underlying type objects, which makes it tricky to annotate code that uses these types. Mypy defines custom type aliases in typing for some of the more common types (including pattern and match object types for re), but this does not easily extend to 3rd party libraries.

Example:

>>> import re
>>> r = re.compile('x')
>>> type(r)
<class '_sre.SRE_Pattern'>
>>> import _sre
>>> _sre.SRE_Pattern
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'SRE_Pattern'

A related issue is types that are implicit, such as protocol types / structural types (in case we support them). Where should a structural type needed for annotating a library module be defined?

Also, generic library types pose a similar problem, since vanilla Python classes don't support indexing, and we'd have to use the string escape syntax to use a library class as a generic type.

Here are alternatives I've came up with:

• Fall back to the type Any if the type object is not available. This is clearly suboptimal.
• Define the type in typing or a related module as an alias. This does not scale to an arbitrary number of modules, but it would probably work for the most common cases. We could fall back to Any elsewhere.
• Define a parallel module hierarchy for additional typing-related definitions, such as typing.re for re. Not sure how this will work if these modules come from multiple sources. Should this hierarchy be managed centrally?
• Expose the type objects of standard library modules in Python 3.5 (as was suggested by Guido). This does not help with earlier Python releases. In addition to just exposing the types they should also have meaningful names, unlike _sre.SRE_Pattern in the above example.

[ambv]

My vote is for option 4. If the type object is still unavailable, falling back to Any is sadly the only option. Could we somehow use a decorator in a stub to inform the type checker that a class/function is a stub for a different name?

This looks like it's blocked on a decision what to do with protocols. @gvanrossum?

[JukkaL]

More discussion here: https://github.com/JukkaL/typing/issues/18

[gvanrossum]

So we're not doing protocols this time around. Let's add a typing.re submodule that defines Match and Pattern, and also let's add a typing.io submodule for IO and its subclasses.

Separately, we should add re.Match and re.Pattern in 3.5 to the regular stdlib re module. But if you want to use them as generic types (e.g. Pattern[str] or Pattern[bytes]) you must import the types from typing.re.

[JukkaL]

At least the following modules could also benefit from having typing.<module> submodules, since they contain classes that are generic in mypy stubs:

• asyncio
• collections
• urllib.parse
• weakref
• difflib (the stub is in a pull request)

There are most likely others that don't have any stubs yet. Maybe we should go through the standard library (or at least the most commonly used modules) and try to find additional examples of generic classes. Once the set of typing submodules is finalized in 3.5, I guess it's going to be difficult to add new ones for existing modules.

[gvanrossum]

At least for the duration of 3.5, this will be in provisional mode which means we can still make changes. (We do this for asyncio in 3.4 and it's great -- and nobody has complained so far.)

Also, adding new submodules (that have to be explicitly imported to be used) sounds like a fine thing to do at any point in the future. At some point it may be better to make the implementations themselves generic, so people can start using e.g. re.Pattern[str] rather than typing.re.Pattern[str]. But that may be a ways off and I propose not to do this for 3.5 (though authors of new 3.5 modules are welcome to use typing, as long as they use it correctly and run the checker).

One subtlety is, do we require import typing.re or should import typing implicitly import these submodules? (That could be expensive if there are many of them.) As a compromise, maybe have typing.re and typing.io pre-imported but require explicit import of all others? (There's a reason those have two-letter names. :-)

[ambv]

Fixed in e2e6fc4.