Improve CLI, refactor and document stubgen (#6256)

This PR does several "infrastructure" changes to `stubgen` tool:
* Update CLI and source discovery/collection to match the `mypy` CLI
* Refactor the logic in `stubgen.main()` into independent functions
* Use semantically analyzed Python ASTs
* Separate inference of signatures from docs to a separate module `stubdoc.py`
* Move some functions from `stubgen.py` to `stubutil.py` and reorganize the latter
* Clean-up the test runner to make more use of `DataSuite`
* Add documentation for the `stubgen` script

This also does few smaller things:
* Minimize number of repeated hard-coded constants
* Removed duplicate TODOs
* Added dosctrings to `stubgenc.py` and (new) `stubdoc.py`
* Avoid mutable defaults

This is not a pure refactoring, turning the semantic analysis on required some (although relatively small) changes in logic (because the sources should be semantically analyzed as a whole). It also required couple minor changes in `semanal.py` and `build.py`.

Author: ilevkivskyi

docs/source/index.rst

@@ -53,6 +53,7 @@ Mypy is a static type checker for Python 3 and Python 2.7.
    mypy_daemon
    installed_packages
    extending_mypy
+   stubgen
 
 .. toctree::
    :maxdepth: 2

docs/source/stubgen.rst

@@ -0,0 +1,143 @@
+.. _stugen:
+
+Automatic stub generation
+=========================
+
+Stub files (see `PEP 484 <https://www.python.org/dev/peps/pep-0484/#stub-files>`_)
+are files containing only type hints not the actual runtime implementation.
+They can be useful for C extension modules, third-party modules whose authors
+have not yet added type hints, etc.
+
+Mypy comes with a ``stubgen`` tool for automatic generation of
+stub files (``.pyi`` files) from Python source files. For example,
+this source file:
+
+.. code-block:: python
+
+   from other_module import dynamic
+
+   BORDER_WIDTH = 15
+
+   class Window:
+       parent = dynamic()
+       def __init__(self, width, hight):
+           self.width = width
+           self.hight = hight
+
+   def create_empty() -> Window:
+       return Window(0, 0)
+
+will be transformed into the following stub file:
+
+.. code-block:: python
+
+   from typing import Any
+
+   BORDER_WIDTH: int = ...
+
+   class Window:
+       parent: Any = ...
+       width: Any = ...
+       height: Any: ...
+       def __init__(self, width, height) -> None: ...
+
+   def create_empty() -> Window: ...
+
+In most cases, the auto-generated stub files require manual check for
+completeness. This section documents stubgen's command line interface.
+You can view a quick summary of the available flags by running
+``stubgen --help``.
+
+.. note::
+
+   Stubgen tool is still experimental and will evolve. Command line flags
+   are liable to change between releases.
+
+Specifying what to stub
+***********************
+
+By default, you can specify for what code you want to generate
+stub files by passing in the paths to the sources::
+
+    $ stubgen foo.py bar.py some_directory
+
+Note that directories are checked recursively.
+
+Stubgen also lets you specify modules for stub generation in two
+other ways. The relevant flags are:
+
+``-m MODULE``, ``--module MODULE``
+    Asks stubgen to generate stub file for the provided module. This flag
+    may be repeated multiple times.
+
+    Stubgen *will not* recursively generate stubs for any submodules of
+    the provided module.
+
+``-p PACKAGE``, ``--package PACKAGE``
+    Asks stubgen to generate stubs for the provided package. This flag may
+    be repeated multiple times.
+
+    Stubgen *will* recursively generate stubs for all submodules of
+    the provided package. This flag is identical to ``--module`` apart from
+    this behavior.
+
+.. note::
+
+   You can use either module/package mode or source code mode, these two
+   can't be mixed together in the same stubgen invocation.
+
+Specifying how to generate stubs
+********************************
+
+By default stubgen will try to import the modules and packages given.
+This has an advantage of possibility to discover and stub also C modules.
+By default stubgen will use mypy to semantically analyze the Python
+sources found. To alter this behavior, you can use following flags:
+
+``--no-import``
+    Don't try to import modules, instead use mypy's normal mechanisms to find
+    sources. This will not find any C extension modules. Stubgen also uses
+    runtime introspection to find actual value of ``__all__``, so with this flag
+    the set of re-exported names may be incomplete. This flag will be useful if
+    importing the module causes an error.
+
+``--parse-only``
+    Don't perform mypy semantic analysis of source files. This may generate
+    worse stubs: in particular some module, class, and function aliases may
+    be typed as variables with ``Any`` type. This can be useful if semantic
+    analysis causes a critical mypy error.
+
+``--doc-dir PATH``
+    Try to infer function and class signatures by parsing .rst documentation
+    in ``PATH``. This may result in better stubs, but currently only works for
+    C modules.
+
+Additional flags
+****************
+
+``--py2``
+    Run stubgen in Python 2 mode (the default is Python 3 mode).
+
+``--ignore-errors``
+    Ignore any errors when trying to generate stubs for modules and packages.
+    This may be useful for C modules where runtime introspection is used
+    intensively.
+
+``--include-private``
+    Generate stubs for objects and members considered private (with single
+    leading underscore and no trailing underscores).
+
+``--search-path PATH``
+    Specify module search directories, separated by colons (currently only
+    used if ``--no-import`` is given).
+
+``--python-executable PATH``
+    Use Python interpreter at ``PATH`` for module finding and runtime
+    introspection (has no effect with ``--no-import``). Currently only works
+    for Python 2. In Python 3 mode only the default interpreter will be used.
+
+``-o PATH``, ``--output PATH``
+    Change the output directory. By default the stubs are written in
+    ``./out`` directory. The output directory will be created if it didn't
+    exist. Existing stubs in the output directory will be overwritten without
+    warning.

mypy/build.py

@@ -1463,7 +1463,8 @@ def __init__(self,
                  # as error reporting should be avoided.
                  temporary: bool = False,
                  ) -> None:
-        assert id or path or source is not None, "Neither id, path nor source given"
+        if not temporary:
+            assert id or path or source is not None, "Neither id, path nor source given"
         self.manager = manager
         State.order_counter += 1
         self.order = State.order_counter

mypy/newsemanal/semanal.py

@@ -245,7 +245,9 @@ def __init__(self,
         self.incomplete_namespaces = incomplete_namespaces
         self.postpone_nested_functions_stack = [FUNCTION_BOTH_PHASES]
         self.postponed_functions_stack = []
-        self.all_exports = set()  # type: Set[str]
+        self.all_exports = []  # type: List[str]
+        # Map from module id to list of explicitly exported names (i.e. names in __all__).
+        self.export_map = {}  # type: Dict[str, List[str]]
         self.plugin = plugin
         # If True, process function definitions. If False, don't. This is used
         # for processing module top levels in fine-grained incremental mode.
@@ -317,6 +319,8 @@ def visit_file(self, file_node: MypyFile, fnam: str, options: Options,
                     if name not in self.all_exports:
                         g.module_public = False
 
+            self.export_map[self.cur_mod_id] = self.all_exports
+            self.all_exports = []
             del self.options
             del self.patches
             del self.cur_mod_node
@@ -3899,7 +3903,7 @@ def add_exports(self, exp_or_exps: Union[Iterable[Expression], Expression]) -> N
         exps = [exp_or_exps] if isinstance(exp_or_exps, Expression) else exp_or_exps
         for exp in exps:
             if isinstance(exp, StrExpr):
-                self.all_exports.add(exp.value)
+                self.all_exports.append(exp.value)
 
     def check_no_global(self, n: str, ctx: Context,
                         is_overloaded_func: bool = False) -> None:

mypy/nodes.py

@@ -652,6 +652,8 @@ class Decorator(SymbolNode, Statement):
 
     func = None  # type: FuncDef                # Decorated function
     decorators = None  # type: List[Expression] # Decorators (may be empty)
+    # Some decorators are removed by semanal, keep the original here.
+    original_decorators = None  # type: List[Expression]
     # TODO: This is mostly used for the type; consider replacing with a 'type' attribute
     var = None  # type: Var                     # Represents the decorated function obj
     is_overload = False
@@ -661,6 +663,7 @@ def __init__(self, func: FuncDef, decorators: List[Expression],
         super().__init__()
         self.func = func
         self.decorators = decorators
+        self.original_decorators = decorators.copy()
         self.var = var
         self.is_overload = False
 

mypy/semanal.py

@@ -242,7 +242,9 @@ def __init__(self,
         self.missing_modules = missing_modules
         self.postpone_nested_functions_stack = [FUNCTION_BOTH_PHASES]
         self.postponed_functions_stack = []
-        self.all_exports = set()  # type: Set[str]
+        self.all_exports = []  # type: List[str]
+        # Map from module id to list of explicitly exported names (i.e. names in __all__).
+        self.export_map = {}  # type: Dict[str, List[str]]
         self.plugin = plugin
         # If True, process function definitions. If False, don't. This is used
         # for processing module top levels in fine-grained incremental mode.
@@ -314,6 +316,8 @@ def visit_file(self, file_node: MypyFile, fnam: str, options: Options,
                     if name not in self.all_exports:
                         g.module_public = False
 
+            self.export_map[self.cur_mod_id] = self.all_exports
+            self.all_exports = []
             del self.options
             del self.patches
             del self.cur_mod_node
@@ -3707,7 +3711,7 @@ def add_exports(self, exp_or_exps: Union[Iterable[Expression], Expression]) -> N
         exps = [exp_or_exps] if isinstance(exp_or_exps, Expression) else exp_or_exps
         for exp in exps:
             if isinstance(exp, StrExpr):
-                self.all_exports.add(exp.value)
+                self.all_exports.append(exp.value)
 
     def check_no_global(self, n: str, ctx: Context,
                         is_overloaded_func: bool = False) -> None: