Thoughts about type hints in the standard library

[srittau]

In my opinion, one long term goal should be to integrate the standard library type hints into the standard library itself. The advantage of having type hints directly in Python are the same advantages as for third-party stubs: tighter coupling and prevention of divergence, better capturing an author's intent, better availability, not needing to maintain multiple versions of stubs.

At this point, this issue is not so much intended as a concrete plan, but for a collection of ideas, thoughts, and challenges. Any concrete plan should take the form of a PEP and be discussed on python-dev.

Here are some of my thoughts.

Stubs vs. inline type hints

Extension modules will need to use type stubs. An open question here is the relation between the argument clinic and stubs, since they both fulfill the same function to a certain degree, but with different goals. Maybe it's possible to generate stubs automatically?

There is a similar question about using stubs vs inline type hints for Python modules. One advantage of using stubs is consistency with extension modules, and that it gives a bit more flexibility. But what about Python modules using type hints? Maybe it's also possible to use automatic generation using something like my very incomplete stub-extractor?

Incremental Adoption

I believe that it would make sense to introduce type hints slowly to the standard library: Start with modules that already have high quality stubs in typeshed and that are actively maintained in Python, by core developers that are willing to support type hints in the future. This has the advantage of slowly introducing type hints to core developers who didn't have contact with them before, and doesn't force skeptical core developers to use them. It also prevents us from having low-quality type hints in Python, or from deteriorating type hint quality.

The downside of an incremental adoption is confusion of where to find which stub and a higher burden to contribute fixes for non-core developers (but a lower burden for core developers). The latter is also a good reason only to add stubs that are already high quality, since they are less likely to require changes.

Scope

What should be part of the stubs? Only public API? Or should it be like our typeshed policy where we include non-documented members on request? Should we include docstrings?

My vision

My vision (at the moment): Use stub files for every module, only include public and semi-public API, and include docstring. This way, we have a clearly defined API, easy access to types and documentation for tools, and an easily accessible and comprehensive reference during development. docs.python.org's documentation could focus more on teaching each module and delegate more of its reference responsibilities to the stub files.

[Akuli]

Please do not integrate the stubs to CPython. I once contributed a tiny bugfix to CPython, and I will never do it again:

• You need to sign a CLA (including writing your street address onto it, which is just ridiculous). I didn't want to do this, which meant that the CPython maintainers were very reluctant to merge my PR, although they did end up doing it anyway (after rewriting most of it in the code review).
• You need to file a bug on bugs.python.org. If you don't, your PR will not be reviewed.
• All CPython pull requests need to be written following templates exactly. The title must start with bpo-nnnnn, where nnnnn is the bugs.python.org number.
• Every new contributor needs to read long walls of text describing this process. Those are not just common sense; they really contain weird gotchas that I wasn't used to.

Compare that to having the stubs in typeshed:

• Submit a pull request and it's likely reviewed within a couple hours.

[srittau]

@Akuli If we would integrate it in the stdlib, it's still a long way off. But I definitely share your concerns about the process. One possibility I see is to keep typeshed as a "staging ground" for the stdlib types.

[srittau]

Coincidentally there was a good talk today at PyCon by @meredydd called "Writing Good Documentation for Developers" that goes into the difference between "Reference" (the Python docs) and "API Docs" (potentially the stubs).

[Avasam]

As a discussion point about docstrings: They're not currently included in typeshed, and I believe that makes sense: IDEs (like the Python Language Server on VSCode), will pickup the docstrings from the source code anyway. Adding them to typeshed would only make files bigger and harder to maintain (ie: having to ensure the stubs docstrings stay in sync with source docstrings).

There are however, two exceptions:

1. C extension code
2. Dynamically generated code

OpenCV and pywin32 (third party libraries) being two great examples of it. They are, however, both massive stubs, and adding dosctrings for everything doesn't seem viable (too much of a chore, and makes the files even bigger than they already are.

As a middle ground, docstrings could automatically be added upon upload for modules/classes/methods/functions/etc. that don't have a direct pure python source to reference.

Maybe that's something the Python Language Server could pull dynamically and that's not something for typeshed to care about. Idk about PyCharm.

[jakebailey]

IDEs (like the Python Language Server on VSCode), will pickup the docstrings from the source code anyway

I'm not working on Pylance/pyright anymore, but I wouldn't use this as a reason to omit docstrings from stubs; the reason that IDEs have to do things like executing code is because end users (rightfully!) want docstrings in their editors.

For the longest time, pyright and Pylance didn't have docstrings for anything in typeshed at all, but users of course in masse requested them, so we invented ways to map stubs back to the underlying python (covering most of the standard library), and I added something that could provide them for the compiled libraries at runtime by running the interpreter (which I believe is now used for any and all things in the standard lib, as it's more accurate than the mapping). Doing that was a lot easier than to get typeshed / the PEPs to change, especially in the timeframe the users expected the feature.

So, it would be sort of a shame to take that as precedent for why stubs shouldn't have docstrings (in and of itself); the only reason IDEs have to run the code is because stubs don't contain docs. There are of course other rationales both ways, but this one in particular is circular.

I now work on TypeScript, which does put docs in its "stubs" (declarations), both for what is like the "standard library" and untyped npm packages in DefinitelyTyped, and it's worked so far quite well (people do send PRs to fix things). But, the languages are different; the buy-in to types is much higher comparatively (which increases the likelihood someone can contribute a fix to DT), and the "standard library" doesn't change all that much (though Node's types are on DT, docs and all, and change frequently).

[hauntsaninja]

Thanks for chiming in Jake! Before docstrings discussion takes off too much, I propose we move to #4881

[gpshead]

A related discussion from the CPython side: https://discuss.python.org/t/type-annotations-in-the-stdlib/21487