Missing stubs in Python 3.6 standard library

[JelleZijlstra]

The following modules in the Python 3.6 standard library do not have stubs:

• _bootlocale
• _collections_abc
• _compat_pickle
• _osx_support
• _pydecimal
• _pyio
• _sitebuiltins
• _strptime
• _sysconfigdata_m_darwin_darwin
• _threading_local
• _weakrefset
• aifc
• bdb
• binhex
• cgitb
• chunk
• cmd
• codeop
• compileall
• copyreg
• crypt
• dummy_threading
• formatter
• genericpath
• imaplib
• imghdr
• lzma
• macpath
• macurl2path
• mailbox
• mailcap
• modulefinder
• netrc
• nntplib
• ntpath
• nturl2path
• optparse
• pickletools
• poplib
• pty
• pydoc
• quopri
• reprlib
• sched
• smtpd
• sndhdr
• sre_compile
• sre_constants
• sre_parse
• statistics
• stringprep
• sunau
• symbol
• symtable
• tabnanny
• telnetlib
• this
• timeit
• trace
• tracemalloc
• tty
• uu
• wave
• xdrlib
• zipapp

Found this with https://github.com/JelleZijlstra/stubcheck. Some of these are private or aren't intended to be imported.

[JelleZijlstra]

Added statistics in #1021.

[JukkaL]

These modules seem to be internal to CPython and thus not a priority:

• Modules starting with _.
• genericpath (used through os.path)
• sre_compile
• sre_constants
• sre_parse

[JelleZijlstra]

Just realized this is missing modules implemented in C.

[Dushistov]

Also https://docs.python.org/3/library/internet.html is missing,
in particular xmlrpc

[matthiaskramm]

This list is a good idea. I'm going to do something similar for Python 2

[srittau]

I will also use this bug report to plug typeshed-completion, which tries to document the completion status of the standard library.

[CraftSpider]

Most of the public check-marks on this issue seem to be complete. On my local machine, the current list of missing packages (Windows 10, Python 3.6.9, unfiltered):

• __phello__.foo - Only used for testing the __hello__ frozen module
• _collections_abc - pending Add _collections_abc module #4000
• _pyio
• _sitebuiltins
• _strptime
• idlelib
• pydoc_data
• turtledemo

msilib is a public windows-specific module, in the original post I believe the _sysconfigdata_* module is both private and os-specific. typeshed-completion also seems to list a number of packages as 'missing' that are now in typeshed.

[CraftSpider]

Adding on the list of modules implemented in C (Same source as previous):

• _asyncio
• _bz2
• _ctypes
• _ctypes_test
• _distutils_findvs
• _elementtree
• _hashlib
• _lzma
• _msi
• _multiprocessing
• _overlapped
• _socket
• _sqlite3
• _ssl
• _testbuffer
• _testcapi
• _testconsole
• _testimportmultiple
• _testmultiphase
• _tkinter

[dkg]

This list seems to be valid for 3.7 as well as 3.6. At least, i'm trying to typecheck some code that uses mailbox and running into the missing stub on 3.7.

[martinburchell]

Should xml.dom be in this list?

[petsuter]

Also winreg?

[CraftSpider]

@JelleZijlstra Mind if I update your checklist to match all the stub files now in typeshed? All the non-private modules have been completed

[JelleZijlstra]

Yes, feel free to edit the message (I think you should have permission to do that now?).

[Akuli]

I recently contributed support for tkinter.font, and seems like it never was on any of the checkbox lists here. There is tkinter/font.py in the standard library, but it seems like the standard library never imports tkinter.font except in Lib/tkinter/test/test_tkinter.

Maybe someone should check if there are other modules missing from the list.

[petsuter]

xml:

• xml.etree.ElementTree: the ElementTree API, a simple and lightweight XML processor
• xml.dom: the DOM API definition
• xml.dom.minidom: a minimal DOM implementation
• xml.dom.pulldom: support for building partial DOM trees
• xml.sax: SAX2 base classes and convenience functions
• xml.parsers.expat: the Expat parser binding

(see #3787)

[Akuli]

pydoc_data support was added recently in #4351

Should there be stubs for idlelib? It's the implementation of IDLE, and it's not supposed to be used for other projects (even though there are some stackoverflow answers that use it).

[hauntsaninja]

We're also missing various submodules (eg, just got bitten by asyncio.sslproto)

[srittau]

@JelleZijlstra Should we close this? The only open stubs are private and this would probably need to be updated for 3.9 anyways.

[JelleZijlstra]

Sounds good. We should do something about the missing submodules though (like asyncio.sslproto and various xml things mentioned above).

[hauntsaninja]

Pulled a list of missing submodules in Python 3.8 on darwin (after filtering out anything with the word test in it):

(Also I was wrong about asyncio.sslproto. We have it, I just got bitten by implicit reexport).

{'_abc',
 '_asyncio',
 '_blake2',
 '_bz2',
 '_codecs_cn',
 '_codecs_hk',
 '_codecs_iso2022',
 '_codecs_jp',
 '_codecs_kr',
 '_codecs_tw',
 '_collections',
 '_collections_abc',
 '_contextvars',
 '_crypt',
 '_ctypes',
 '_curses_panel',
 '_datetime',
 '_dbm',
 '_elementtree',
 '_functools',
 '_gdbm',
 '_hashlib',
 '_io',
 '_locale',
 '_lsprof',
 '_lzma',
 '_md5',
 '_multibytecodec',
 '_multiprocessing',
 '_opcode',
 '_pickle',
 '_posixshmem',
 '_pyio',
 '_queue',
 '_scproxy',
 '_sha1',
 '_sha256',
 '_sha3',
 '_sha512',
 '_signal',
 '_socket',
 '_sqlite3',
 '_sre',
 '_ssl',
 '_statistics',
 '_string',
 '_strptime',
 '_struct',
 '_symtable',
 '_sysconfigdata__darwin_darwin',
 '_uuid',
 '_xxsubinterpreters',
 'asyncio.__main__',
 'ctypes._aix',
 'ctypes._endian',
 'ctypes.macholib',
 'ctypes.macholib.dyld',
 'ctypes.macholib.dylib',
 'ctypes.macholib.framework',
 'curses.has_key',
 'distutils._msvccompiler',
 'distutils.command.install_egg_info',
 'distutils.command.upload',
 'distutils.config',
 'distutils.msvc9compiler',
 'distutils.versionpredicate',
 'email._encoded_words',
 'email._header_value_parser',
 'email._parseaddr',
 'email._policybase',
 'email.base64mime',
 'email.quoprimime',
 'encodings.aliases',
 'encodings.ascii',
 'encodings.base64_codec',
 'encodings.big5',
 'encodings.big5hkscs',
 'encodings.bz2_codec',
 'encodings.charmap',
 'encodings.cp037',
 'encodings.cp1006',
 'encodings.cp1026',
 'encodings.cp1125',
 'encodings.cp1140',
 'encodings.cp1250',
 'encodings.cp1251',
 'encodings.cp1252',
 'encodings.cp1253',
 'encodings.cp1254',
 'encodings.cp1255',
 'encodings.cp1256',
 'encodings.cp1257',
 'encodings.cp1258',
 'encodings.cp273',
 'encodings.cp424',
 'encodings.cp437',
 'encodings.cp500',
 'encodings.cp720',
 'encodings.cp737',
 'encodings.cp775',
 'encodings.cp850',
 'encodings.cp852',
 'encodings.cp855',
 'encodings.cp856',
 'encodings.cp857',
 'encodings.cp858',
 'encodings.cp860',
 'encodings.cp861',
 'encodings.cp862',
 'encodings.cp863',
 'encodings.cp864',
 'encodings.cp865',
 'encodings.cp866',
 'encodings.cp869',
 'encodings.cp874',
 'encodings.cp875',
 'encodings.cp932',
 'encodings.cp949',
 'encodings.cp950',
 'encodings.euc_jis_2004',
 'encodings.euc_jisx0213',
 'encodings.euc_jp',
 'encodings.euc_kr',
 'encodings.gb18030',
 'encodings.gb2312',
 'encodings.gbk',
 'encodings.hex_codec',
 'encodings.hp_roman8',
 'encodings.hz',
 'encodings.idna',
 'encodings.iso2022_jp',
 'encodings.iso2022_jp_1',
 'encodings.iso2022_jp_2',
 'encodings.iso2022_jp_2004',
 'encodings.iso2022_jp_3',
 'encodings.iso2022_jp_ext',
 'encodings.iso2022_kr',
 'encodings.iso8859_1',
 'encodings.iso8859_10',
 'encodings.iso8859_11',
 'encodings.iso8859_13',
 'encodings.iso8859_14',
 'encodings.iso8859_15',
 'encodings.iso8859_16',
 'encodings.iso8859_2',
 'encodings.iso8859_3',
 'encodings.iso8859_4',
 'encodings.iso8859_5',
 'encodings.iso8859_6',
 'encodings.iso8859_7',
 'encodings.iso8859_8',
 'encodings.iso8859_9',
 'encodings.johab',
 'encodings.koi8_r',
 'encodings.koi8_t',
 'encodings.koi8_u',
 'encodings.kz1048',
 'encodings.latin_1',
 'encodings.mac_arabic',
 'encodings.mac_centeuro',
 'encodings.mac_croatian',
 'encodings.mac_cyrillic',
 'encodings.mac_farsi',
 'encodings.mac_greek',
 'encodings.mac_iceland',
 'encodings.mac_latin2',
 'encodings.mac_roman',
 'encodings.mac_romanian',
 'encodings.mac_turkish',
 'encodings.mbcs',
 'encodings.oem',
 'encodings.palmos',
 'encodings.ptcp154',
 'encodings.punycode',
 'encodings.quopri_codec',
 'encodings.raw_unicode_escape',
 'encodings.rot_13',
 'encodings.shift_jis',
 'encodings.shift_jis_2004',
 'encodings.shift_jisx0213',
 'encodings.tis_620',
 'encodings.undefined',
 'encodings.unicode_escape',
 'encodings.utf_16',
 'encodings.utf_16_be',
 'encodings.utf_16_le',
 'encodings.utf_32',
 'encodings.utf_32_be',
 'encodings.utf_32_le',
 'encodings.utf_7',
 'encodings.utf_8_sig',
 'encodings.uu_codec',
 'encodings.zlib_codec',
 'ensurepip.__main__',
 'ensurepip._uninstall',
 'idlelib',
 'idlelib.__main__',
 'idlelib.autocomplete',
 'idlelib.autocomplete_w',
 'idlelib.autoexpand',
 'idlelib.browser',
 'idlelib.calltip',
 'idlelib.calltip_w',
 'idlelib.codecontext',
 'idlelib.colorizer',
 'idlelib.config',
 'idlelib.config_key',
 'idlelib.configdialog',
 'idlelib.debugger',
 'idlelib.debugger_r',
 'idlelib.debugobj',
 'idlelib.debugobj_r',
 'idlelib.delegator',
 'idlelib.dynoption',
 'idlelib.editor',
 'idlelib.filelist',
 'idlelib.format',
 'idlelib.grep',
 'idlelib.help',
 'idlelib.help_about',
 'idlelib.history',
 'idlelib.hyperparser',
 'idlelib.idle',
 'idlelib.iomenu',
 'idlelib.macosx',
 'idlelib.mainmenu',
 'idlelib.multicall',
 'idlelib.outwin',
 'idlelib.parenmatch',
 'idlelib.pathbrowser',
 'idlelib.percolator',
 'idlelib.pyparse',
 'idlelib.pyshell',
 'idlelib.query',
 'idlelib.redirector',
 'idlelib.replace',
 'idlelib.rpc',
 'idlelib.run',
 'idlelib.runscript',
 'idlelib.scrolledlist',
 'idlelib.search',
 'idlelib.searchbase',
 'idlelib.searchengine',
 'idlelib.sidebar',
 'idlelib.squeezer',
 'idlelib.stackviewer',
 'idlelib.statusbar',
 'idlelib.textview',
 'idlelib.tooltip',
 'idlelib.tree',
 'idlelib.undo',
 'idlelib.window',
 'idlelib.zoomheight',
 'idlelib.zzdummy',
 'importlib._bootstrap',
 'importlib._bootstrap_external',
 'json.scanner',
 'json.tool',
 'lib2to3.__main__',
 'lib2to3.btm_matcher',
 'lib2to3.btm_utils',
 'lib2to3.fixer_base',
 'lib2to3.fixer_util',
 'lib2to3.fixes',
 'lib2to3.fixes.fix_apply',
 'lib2to3.fixes.fix_asserts',
 'lib2to3.fixes.fix_basestring',
 'lib2to3.fixes.fix_buffer',
 'lib2to3.fixes.fix_dict',
 'lib2to3.fixes.fix_except',
 'lib2to3.fixes.fix_exec',
 'lib2to3.fixes.fix_execfile',
 'lib2to3.fixes.fix_exitfunc',
 'lib2to3.fixes.fix_filter',
 'lib2to3.fixes.fix_funcattrs',
 'lib2to3.fixes.fix_future',
 'lib2to3.fixes.fix_getcwdu',
 'lib2to3.fixes.fix_has_key',
 'lib2to3.fixes.fix_idioms',
 'lib2to3.fixes.fix_import',
 'lib2to3.fixes.fix_imports',
 'lib2to3.fixes.fix_imports2',
 'lib2to3.fixes.fix_input',
 'lib2to3.fixes.fix_intern',
 'lib2to3.fixes.fix_isinstance',
 'lib2to3.fixes.fix_itertools',
 'lib2to3.fixes.fix_itertools_imports',
 'lib2to3.fixes.fix_long',
 'lib2to3.fixes.fix_map',
 'lib2to3.fixes.fix_metaclass',
 'lib2to3.fixes.fix_methodattrs',
 'lib2to3.fixes.fix_ne',
 'lib2to3.fixes.fix_next',
 'lib2to3.fixes.fix_nonzero',
 'lib2to3.fixes.fix_numliterals',
 'lib2to3.fixes.fix_operator',
 'lib2to3.fixes.fix_paren',
 'lib2to3.fixes.fix_print',
 'lib2to3.fixes.fix_raise',
 'lib2to3.fixes.fix_raw_input',
 'lib2to3.fixes.fix_reduce',
 'lib2to3.fixes.fix_reload',
 'lib2to3.fixes.fix_renames',
 'lib2to3.fixes.fix_repr',
 'lib2to3.fixes.fix_set_literal',
 'lib2to3.fixes.fix_standarderror',
 'lib2to3.fixes.fix_sys_exc',
 'lib2to3.fixes.fix_throw',
 'lib2to3.fixes.fix_tuple_params',
 'lib2to3.fixes.fix_types',
 'lib2to3.fixes.fix_unicode',
 'lib2to3.fixes.fix_urllib',
 'lib2to3.fixes.fix_ws_comma',
 'lib2to3.fixes.fix_xrange',
 'lib2to3.fixes.fix_xreadlines',
 'lib2to3.fixes.fix_zip',
 'lib2to3.main',
 'lib2to3.patcomp',
 'lib2to3.pgen2.conv',
 'lib2to3.refactor',
 'multiprocessing.forkserver',
 'multiprocessing.heap',
 'multiprocessing.popen_fork',
 'multiprocessing.popen_forkserver',
 'multiprocessing.popen_spawn_posix',
 'multiprocessing.popen_spawn_win32',
 'multiprocessing.reduction',
 'multiprocessing.resource_sharer',
 'multiprocessing.resource_tracker',
 'multiprocessing.util',
 'sqlite3.dump',
 'tkinter.__main__',
 'tkinter.colorchooser',
 'tkinter.dnd',
 'tkinter.scrolledtext',
 'tkinter.simpledialog',
 'tkinter.tix',
 'turtledemo',
 'turtledemo.__main__',
 'turtledemo.bytedesign',
 'turtledemo.chaos',
 'turtledemo.clock',
 'turtledemo.colormixer',
 'turtledemo.forest',
 'turtledemo.fractalcurves',
 'turtledemo.lindenmayer',
 'turtledemo.minimal_hanoi',
 'turtledemo.nim',
 'turtledemo.paint',
 'turtledemo.peace',
 'turtledemo.penrose',
 'turtledemo.planet_and_moon',
 'turtledemo.rosette',
 'turtledemo.round_dance',
 'turtledemo.sorting_animate',
 'turtledemo.tree',
 'turtledemo.two_canvases',
 'turtledemo.yinyang',
 'venv.__main__',
 'xml.dom.domreg',
 'xml.dom.expatbuilder',
 'xml.dom.minicompat',
 'xml.dom.xmlbuilder',
 'xml.sax._exceptions',
 'xml.sax.expatreader',
 'xxlimited',
 'xxsubtype'}

List filtered to things that could maybe help people:

'ctypes.macholib',
'ctypes.macholib.dyld',
'ctypes.macholib.dylib',
'ctypes.macholib.framework',
'curses.has_key',
'distutils._msvccompiler',
'distutils.command.install_egg_info',
'distutils.command.upload',
'distutils.config',
'distutils.msvc9compiler',
'distutils.versionpredicate',
'email.base64mime',
'email.quoprimime',
'json.scanner',
'json.tool',
'multiprocessing.forkserver',
'multiprocessing.heap',
'multiprocessing.popen_fork',
'multiprocessing.popen_forkserver',
'multiprocessing.popen_spawn_posix',
'multiprocessing.popen_spawn_win32',
'multiprocessing.reduction',
'multiprocessing.resource_sharer',
'multiprocessing.resource_tracker',
'multiprocessing.util',
'sqlite3.dump',
'tkinter.colorchooser',
'tkinter.dnd',
'tkinter.scrolledtext',
'tkinter.simpledialog',
'tkinter.tix',
'xml.dom.domreg',
'xml.dom.expatbuilder',
'xml.dom.minicompat',
'xml.dom.xmlbuilder',
'xml.sax.expatreader',

[srittau]

@hauntsaninja Do you want to open a new ticket for 3.8, so we could get a fresh start?

[srittau]

Also, I think we should leave out undocumented modules from this list. (But we can do that in the new issue.) While it's fine to add such a module, if someone cares to annotate it, our general policy is to add such undocumented features only on request, so I don't think we need to keep an issue around for those.

[hauntsaninja]

Sure!