Missing stubs in Python 3.8 standard library

[hauntsaninja]

If a module is undocumented, feel free to strike it off (#1019 (comment)).

Filtered list:

• ctypes.macholib (undocumented, vendored macholib)
• ctypes.macholib.dyld (undocumented, vendored macholib)
• ctypes.macholib.dylib (undocumented, vendored macholib)
• ctypes.macholib.framework (undocumented, vendored macholib)
• curses.has_key (stubs use _curses.has_key)
• email.base64mime (undocumented)
• email.quoprimime (undocumented)
• json.scanner (undocumented, implementation detail)
• json.tool
• multiprocessing.forkserver (concept documented, but neither module nor class)
• multiprocessing.heap (undocumented, multiprocessing utility)
• multiprocessing.popen_fork (undocumented, multiprocessing utility)
• multiprocessing.popen_forkserver (undocumented, multiprocessing utility)
• multiprocessing.popen_spawn_posix (undocumented, multiprocessing utility)
• multiprocessing.popen_spawn_win32 (undocumented, multiprocessing utility)
• multiprocessing.reduction (undocumented, multiprocessing utility)
• multiprocessing.resource_sharer (undocumented, multiprocessing utility)
• multiprocessing.resource_tracker (undocumented, multiprocessing utility)
• multiprocessing.util (undocumented, multiprocessing utility)
• sqlite3.dump (undocumented, sqlite3 utility)
• tkinter.colorchooser
• tkinter.dnd
• tkinter.scrolledtext
• tkinter.simpledialog
• tkinter.tix
• xml.dom.domreg (module undocumented, but documented methods reexported from xml.dom)
• xml.dom.expatbuilder (undocumented, see also Missing standard library xml #3787)
• xml.dom.minicompat (undocumented, implementation detail)
• xml.dom.xmlbuilder (undocumented)
• xml.sax.expatreader (undocumented)

Full list:

{'_abc',
 '_asyncio',
 '_blake2',
 '_bz2',
 '_codecs_cn',
 '_codecs_hk',
 '_codecs_iso2022',
 '_codecs_jp',
 '_codecs_kr',
 '_codecs_tw',
 '_collections',
 '_collections_abc',
 '_contextvars',
 '_crypt',
 '_ctypes',
 '_curses_panel',
 '_datetime',
 '_dbm',
 '_elementtree',
 '_functools',
 '_gdbm',
 '_hashlib',
 '_io',
 '_locale',
 '_lsprof',
 '_lzma',
 '_md5',
 '_multibytecodec',
 '_multiprocessing',
 '_opcode',
 '_pickle',
 '_posixshmem',
 '_pyio',
 '_queue',
 '_scproxy',
 '_sha1',
 '_sha256',
 '_sha3',
 '_sha512',
 '_signal',
 '_socket',
 '_sqlite3',
 '_sre',
 '_ssl',
 '_statistics',
 '_string',
 '_strptime',
 '_struct',
 '_symtable',
 '_sysconfigdata__darwin_darwin',
 '_uuid',
 '_xxsubinterpreters',
 'asyncio.__main__',
 'ctypes._aix',
 'ctypes._endian',
 'ctypes.macholib',
 'ctypes.macholib.dyld',
 'ctypes.macholib.dylib',
 'ctypes.macholib.framework',
 'curses.has_key',
 'distutils._msvccompiler',
 'distutils.command.install_egg_info',
 'distutils.command.upload',
 'distutils.config',
 'distutils.msvc9compiler',
 'distutils.versionpredicate',
 'email._encoded_words',
 'email._header_value_parser',
 'email._parseaddr',
 'email._policybase',
 'email.base64mime',
 'email.quoprimime',
 'encodings.aliases',
 'encodings.ascii',
 'encodings.base64_codec',
 'encodings.big5',
 'encodings.big5hkscs',
 'encodings.bz2_codec',
 'encodings.charmap',
 'encodings.cp037',
 'encodings.cp1006',
 'encodings.cp1026',
 'encodings.cp1125',
 'encodings.cp1140',
 'encodings.cp1250',
 'encodings.cp1251',
 'encodings.cp1252',
 'encodings.cp1253',
 'encodings.cp1254',
 'encodings.cp1255',
 'encodings.cp1256',
 'encodings.cp1257',
 'encodings.cp1258',
 'encodings.cp273',
 'encodings.cp424',
 'encodings.cp437',
 'encodings.cp500',
 'encodings.cp720',
 'encodings.cp737',
 'encodings.cp775',
 'encodings.cp850',
 'encodings.cp852',
 'encodings.cp855',
 'encodings.cp856',
 'encodings.cp857',
 'encodings.cp858',
 'encodings.cp860',
 'encodings.cp861',
 'encodings.cp862',
 'encodings.cp863',
 'encodings.cp864',
 'encodings.cp865',
 'encodings.cp866',
 'encodings.cp869',
 'encodings.cp874',
 'encodings.cp875',
 'encodings.cp932',
 'encodings.cp949',
 'encodings.cp950',
 'encodings.euc_jis_2004',
 'encodings.euc_jisx0213',
 'encodings.euc_jp',
 'encodings.euc_kr',
 'encodings.gb18030',
 'encodings.gb2312',
 'encodings.gbk',
 'encodings.hex_codec',
 'encodings.hp_roman8',
 'encodings.hz',
 'encodings.idna',
 'encodings.iso2022_jp',
 'encodings.iso2022_jp_1',
 'encodings.iso2022_jp_2',
 'encodings.iso2022_jp_2004',
 'encodings.iso2022_jp_3',
 'encodings.iso2022_jp_ext',
 'encodings.iso2022_kr',
 'encodings.iso8859_1',
 'encodings.iso8859_10',
 'encodings.iso8859_11',
 'encodings.iso8859_13',
 'encodings.iso8859_14',
 'encodings.iso8859_15',
 'encodings.iso8859_16',
 'encodings.iso8859_2',
 'encodings.iso8859_3',
 'encodings.iso8859_4',
 'encodings.iso8859_5',
 'encodings.iso8859_6',
 'encodings.iso8859_7',
 'encodings.iso8859_8',
 'encodings.iso8859_9',
 'encodings.johab',
 'encodings.koi8_r',
 'encodings.koi8_t',
 'encodings.koi8_u',
 'encodings.kz1048',
 'encodings.latin_1',
 'encodings.mac_arabic',
 'encodings.mac_centeuro',
 'encodings.mac_croatian',
 'encodings.mac_cyrillic',
 'encodings.mac_farsi',
 'encodings.mac_greek',
 'encodings.mac_iceland',
 'encodings.mac_latin2',
 'encodings.mac_roman',
 'encodings.mac_romanian',
 'encodings.mac_turkish',
 'encodings.mbcs',
 'encodings.oem',
 'encodings.palmos',
 'encodings.ptcp154',
 'encodings.punycode',
 'encodings.quopri_codec',
 'encodings.raw_unicode_escape',
 'encodings.rot_13',
 'encodings.shift_jis',
 'encodings.shift_jis_2004',
 'encodings.shift_jisx0213',
 'encodings.tis_620',
 'encodings.undefined',
 'encodings.unicode_escape',
 'encodings.utf_16',
 'encodings.utf_16_be',
 'encodings.utf_16_le',
 'encodings.utf_32',
 'encodings.utf_32_be',
 'encodings.utf_32_le',
 'encodings.utf_7',
 'encodings.utf_8_sig',
 'encodings.uu_codec',
 'encodings.zlib_codec',
 'ensurepip.__main__',
 'ensurepip._uninstall',
 'idlelib',
 'idlelib.__main__',
 'idlelib.autocomplete',
 'idlelib.autocomplete_w',
 'idlelib.autoexpand',
 'idlelib.browser',
 'idlelib.calltip',
 'idlelib.calltip_w',
 'idlelib.codecontext',
 'idlelib.colorizer',
 'idlelib.config',
 'idlelib.config_key',
 'idlelib.configdialog',
 'idlelib.debugger',
 'idlelib.debugger_r',
 'idlelib.debugobj',
 'idlelib.debugobj_r',
 'idlelib.delegator',
 'idlelib.dynoption',
 'idlelib.editor',
 'idlelib.filelist',
 'idlelib.format',
 'idlelib.grep',
 'idlelib.help',
 'idlelib.help_about',
 'idlelib.history',
 'idlelib.hyperparser',
 'idlelib.idle',
 'idlelib.iomenu',
 'idlelib.macosx',
 'idlelib.mainmenu',
 'idlelib.multicall',
 'idlelib.outwin',
 'idlelib.parenmatch',
 'idlelib.pathbrowser',
 'idlelib.percolator',
 'idlelib.pyparse',
 'idlelib.pyshell',
 'idlelib.query',
 'idlelib.redirector',
 'idlelib.replace',
 'idlelib.rpc',
 'idlelib.run',
 'idlelib.runscript',
 'idlelib.scrolledlist',
 'idlelib.search',
 'idlelib.searchbase',
 'idlelib.searchengine',
 'idlelib.sidebar',
 'idlelib.squeezer',
 'idlelib.stackviewer',
 'idlelib.statusbar',
 'idlelib.textview',
 'idlelib.tooltip',
 'idlelib.tree',
 'idlelib.undo',
 'idlelib.window',
 'idlelib.zoomheight',
 'idlelib.zzdummy',
 'importlib._bootstrap',
 'importlib._bootstrap_external',
 'json.scanner',
 'json.tool',
 'lib2to3.__main__',
 'lib2to3.btm_matcher',
 'lib2to3.btm_utils',
 'lib2to3.fixer_base',
 'lib2to3.fixer_util',
 'lib2to3.fixes',
 'lib2to3.fixes.fix_apply',
 'lib2to3.fixes.fix_asserts',
 'lib2to3.fixes.fix_basestring',
 'lib2to3.fixes.fix_buffer',
 'lib2to3.fixes.fix_dict',
 'lib2to3.fixes.fix_except',
 'lib2to3.fixes.fix_exec',
 'lib2to3.fixes.fix_execfile',
 'lib2to3.fixes.fix_exitfunc',
 'lib2to3.fixes.fix_filter',
 'lib2to3.fixes.fix_funcattrs',
 'lib2to3.fixes.fix_future',
 'lib2to3.fixes.fix_getcwdu',
 'lib2to3.fixes.fix_has_key',
 'lib2to3.fixes.fix_idioms',
 'lib2to3.fixes.fix_import',
 'lib2to3.fixes.fix_imports',
 'lib2to3.fixes.fix_imports2',
 'lib2to3.fixes.fix_input',
 'lib2to3.fixes.fix_intern',
 'lib2to3.fixes.fix_isinstance',
 'lib2to3.fixes.fix_itertools',
 'lib2to3.fixes.fix_itertools_imports',
 'lib2to3.fixes.fix_long',
 'lib2to3.fixes.fix_map',
 'lib2to3.fixes.fix_metaclass',
 'lib2to3.fixes.fix_methodattrs',
 'lib2to3.fixes.fix_ne',
 'lib2to3.fixes.fix_next',
 'lib2to3.fixes.fix_nonzero',
 'lib2to3.fixes.fix_numliterals',
 'lib2to3.fixes.fix_operator',
 'lib2to3.fixes.fix_paren',
 'lib2to3.fixes.fix_print',
 'lib2to3.fixes.fix_raise',
 'lib2to3.fixes.fix_raw_input',
 'lib2to3.fixes.fix_reduce',
 'lib2to3.fixes.fix_reload',
 'lib2to3.fixes.fix_renames',
 'lib2to3.fixes.fix_repr',
 'lib2to3.fixes.fix_set_literal',
 'lib2to3.fixes.fix_standarderror',
 'lib2to3.fixes.fix_sys_exc',
 'lib2to3.fixes.fix_throw',
 'lib2to3.fixes.fix_tuple_params',
 'lib2to3.fixes.fix_types',
 'lib2to3.fixes.fix_unicode',
 'lib2to3.fixes.fix_urllib',
 'lib2to3.fixes.fix_ws_comma',
 'lib2to3.fixes.fix_xrange',
 'lib2to3.fixes.fix_xreadlines',
 'lib2to3.fixes.fix_zip',
 'lib2to3.main',
 'lib2to3.patcomp',
 'lib2to3.pgen2.conv',
 'lib2to3.refactor',
 'multiprocessing.forkserver',
 'multiprocessing.heap',
 'multiprocessing.popen_fork',
 'multiprocessing.popen_forkserver',
 'multiprocessing.popen_spawn_posix',
 'multiprocessing.popen_spawn_win32',
 'multiprocessing.reduction',
 'multiprocessing.resource_sharer',
 'multiprocessing.resource_tracker',
 'multiprocessing.util',
 'sqlite3.dump',
 'tkinter.__main__',
 'tkinter.colorchooser',
 'tkinter.dnd',
 'tkinter.scrolledtext',
 'tkinter.simpledialog',
 'tkinter.tix',
 'turtledemo',
 'turtledemo.__main__',
 'turtledemo.bytedesign',
 'turtledemo.chaos',
 'turtledemo.clock',
 'turtledemo.colormixer',
 'turtledemo.forest',
 'turtledemo.fractalcurves',
 'turtledemo.lindenmayer',
 'turtledemo.minimal_hanoi',
 'turtledemo.nim',
 'turtledemo.paint',
 'turtledemo.peace',
 'turtledemo.penrose',
 'turtledemo.planet_and_moon',
 'turtledemo.rosette',
 'turtledemo.round_dance',
 'turtledemo.sorting_animate',
 'turtledemo.tree',
 'turtledemo.two_canvases',
 'turtledemo.yinyang',
 'venv.__main__',
 'xml.dom.domreg',
 'xml.dom.expatbuilder',
 'xml.dom.minicompat',
 'xml.dom.xmlbuilder',
 'xml.sax._exceptions',
 'xml.sax.expatreader',
 'xxlimited',
 'xxsubtype'}

In particular, the xml modules have actually been requested (#3787) so that's probably where one should start.

(edit: dumping the code for finding stdlib modules here, since it's not straightforward and I find I've needed it a couple times. for finding typeshed stdlib modules, look at stubtest)

def get_stdlib(submodules: bool = False):
    all_stdlib = set()
    importers = filter(
        lambda x: isinstance(x, importlib.machinery.FileFinder),
        pkgutil.iter_importers()
    )
    for importer in importers:
        modules = {m for m, _ in pkgutil.iter_importer_modules(importer)}
        # Hacky, but whatever
        if "json" in modules or "_json" in modules:
            if submodules:
                all_stdlib.update(m.name for m in pkgutil.walk_packages([importer.path]))
            else:
                all_stdlib.update(modules)
    all_stdlib.update(sys.builtin_module_names)
    return all_stdlib

[srittau]

Great, thanks!

[srittau]

I've struck all undocumented modules in the filtered list, which basically only left the tkinter modules. I'm working on a PR with a few other additions.

[CraftSpider]

I'm working on a PR with the tkinter modules

[hauntsaninja]

(edit: I realised I'd filtered all the lib2to3 modules. Even though lib2to3 is deprecated, we should probably try and get the non specific fix modules in #2510, so I've added them as checkboxes to the filtered list)

[Akuli]

I checked all remaining tkinter modules.

[AlexWaygood]

I'm removing the deprecated modules from the filtered list at the top of this issue, as I think it would be a waste of time for a contributor to prepare a PR adding stubs for a module that's shortly to be removed from CPython altogether.

(If a contributor has a valid use case for those modules, then it's obviously fine for them to be added, but I don't think there's value in them being added just for completeness's sake.)

[JelleZijlstra]

All the remaining modules are done or struck through as undocumented. Can we close this issue?

[AlexWaygood]

All the remaining modules are done or struck through as undocumented. Can we close this issue?

I won't oppose that!

[kkirsche]

I'm planning to probably address a number of the remaining undocumented modules in the name of completeness. I feel it's important to work towards a complete view of types in hopes of one day making type information more easily retrievable from online docs (e.g. https://docs.python.org/3/library/os.html)

If there is a different focus you'd prefer me to work on though, just let me know

[Akuli]

If you are bored, it might be good to fix the remaining CI errors on #4776 and finally get it merged. It's been open for very long :) Ask on #4776 if you're interested and you don't know how to get started.

[AlexWaygood]

It also looks like our emoji stubs are out of date for the latest release (see the CI errors on #8359). A PR updating our emoji stubs for the latest version would be welcome :)

[kkirsche]

It also looks like our emoji stubs are out of date for the latest release (see the CI errors on #8359). A PR updating our emoji stubs for the latest version would be welcome :)

This should be fixed by #8426 once the tests are done running.

[hauntsaninja]

A lot of our third party stubs are incomplete. You could try getting packages you're familiar with to pass with ignore_missing_stub = false e.g.

typeshed/stubs/ujson/METADATA.toml

Line 4 in f074ef6

ignore_missing_stub = false

This will make sure CI complains if we're missing definitions.

There are also per-package allowlists with existing errors that could be fixed.