diff --git a/README.md b/README.md index 6eb71e9..ce261c2 100644 --- a/README.md +++ b/README.md @@ -1,54 +1,40 @@ # docstub -> [!NOTE] -> In early development! +> [!NOTE] In early development! +> Expect bugs, missing features, and incomplete documentation. +> Docstub is still evaluating which features it needs to support as the community gives feedback. +> Several features are experimental and included to make adoption of docstub easier. +> Long-term, some of these might be discouraged or removed as docstub matures. -A command line tool to generate Python stub files (PYI) from type descriptions -in NumPyDoc style docstrings. +docstub is a command-line tool to generate [Python stub files](https://typing.python.org/en/latest/guides/writing_stubs.html) (i.e., PYI files) from type descriptions found in [numpydoc](https://numpydoc.readthedocs.io)-style docstrings. +Many packages in the scientific Python ecosystem already describe expected parameter and return types in their docstrings. +Docstub aims to take advantage of these and help with the adoption of type annotations. +It does so by supporting widely used readable conventions such as `array of dtype` or `iterable of int(s)` which it translates into valid type annotations. -## Installation -To try out docstub, for now, we recommend installing docstub directly from this -repo: +## Installation & getting started -```shell -pip install 'docstub [optional] @ git+https://github.com/scientific-python/docstub' -``` +Please refer to the [user guide](doc/user_guide.md) to get started with docstub. -## Usage & configuration - -```shell -cd examples/ -docstub example_pkg/ -``` -will create stub files for `example_pkg/` in `examples/example_pkg-stubs/`. -For now, refer to `docstub --help` for more. - - -### Declare imports and synonyms - -Types in docstrings can and are used without having to import them. However, -when docstub creates stub files from these docstrings it actually needs to -know how to import those unknown types. - -> [!TIP] -> docstub already knows about types in Python's `typing` or `collections.abc` -> modules. That means you can just use types like `Literal` or `Sequence`. - -While docstub is smart enough to find some types via static analysis of -definitions in the given source directory, it must be told about other types -for now. To do so, refer to the syntax and comments in the -`default_config.toml`. +## Contributing +The best way you can help and contribute right now is by trying docstub out! +Feedback to what features might still be missing or where it breaks for you would be greatly appreciated. +Pointers to where the documentation is confusing and unclear. -## Contributing +Since docstub is still in early development there isn't an official contribution guide yet. +Features and API are still being heavily extended and the internal structure is still somewhat in flux. +That said, if that only entices you, feel free to open a PR. +But please do check in with an issue before you do so. -TBD +Our project follows the [Scientific Python's Code of Conduct](https://scientific-python.org/code_of_conduct/). ## Acknowledgements Thanks to [docs2stubs](https://github.com/gramster/docs2stubs) by which this project was heavily inspired and influenced. + +And thanks to CZI for supporting this work with an [EOSS grant](https://chanzuckerberg.com/eoss/proposals/from-library-to-protocol-scikit-image-as-an-api-reference/). diff --git a/doc/command_line.md b/doc/command_line.md new file mode 100644 index 0000000..c6e641f --- /dev/null +++ b/doc/command_line.md @@ -0,0 +1,36 @@ +# Command line reference + +Running +``` +docstub --help +``` +will print + + + + +```plain +Usage: docstub [OPTIONS] PACKAGE_PATH + + Generate Python stub files with type annotations from docstrings. + + Given a path `PACKAGE_PATH` to a Python package, generate stub files for it. + Type descriptions in docstrings will be used to fill in missing inline type + annotations or to override them. + +Options: + --version Show the version and exit. + -o, --out-dir PATH Set output directory explicitly. Otherwise, stubs are + generated inplace. + --config PATH Set configuration file explicitly. + --group-errors Group identical errors together and list where they + occured. Will delay showing errors until all files have + been processed. Otherwise, simply report errors as the + occur. + --allow-errors INT Allow this many or fewer errors. If docstub reports + more, exit with error code '1'. [default: 0; x>=0] + -v, --verbose Print more details (repeatable). + -h, --help Show this message and exit. +``` + + diff --git a/doc/typing_syntax.md b/doc/typing_syntax.md new file mode 100644 index 0000000..682aef6 --- /dev/null +++ b/doc/typing_syntax.md @@ -0,0 +1,124 @@ +# Typing syntax in docstrings + +> [!NOTE] In early development! +> Expect bugs, missing features, and incomplete documentation. +> Docstub is still evaluating which features it needs to support as the community gives feedback. +> Several features are experimental and included to make adoption of docstub easier. +> Long-term, some of these might be discouraged or removed as docstub matures. + +Docstub defines its own [grammar](../src/docstub/doctype.lark) to parse and transform type information in docstrings into valid type annotations. +This grammar fully supports [Python's conventional typing syntax](https://typing.python.org/en/latest/index.html). +So any type annotation that is valid in Python, can be used in a docstrings as is. +In addition, docstub extends this syntax with several "natural language" expressions that are commonly used in the scientific Python ecosystem. + +Docstrings are expected to follow the NumPyDoc style: +``` +Section name +------------ +name : annotation, optional, extra_info + Description. +``` + +- `name` might be the name of a parameter or attribute. + Other sections like "Returns" or "Yields" are supported. +- `annotation` the actual type information that will be transformed into the type annotation. +- `optional` and `extra_info` can be appended to provide additional information. + Their presence and content doesn't currently affect the resulting type annotation. + + +## Unions + +In addition to Python's conventional shorthand `|` syntax for [union types](https://typing.python.org/en/latest/spec/concepts.html#union-types), you can use `or` to join types. + +| Docstring type | Python type annotation | +|----------------|------------------------| +| `X or Y` | `X \| Y` | +| `int or float` | `int \| float` | + + +## Containers + +The content of containers can be typed using a `CONTAINER of X` like form. +This extends the basic subscription syntax for [generics](https://typing.python.org/en/latest/spec/generics.html#generics). + +| Docstring type | Python type annotation | +|-------------------------|------------------------| +| `CONTAINER of X` | `CONTAINER[X]` | +| `CONTAINER of (X or Y)` | `CONTAINER[X \| Y]` | + +For the simple case `CONTAINER of X`, where `X` is a name, you can append `(s)` to indicate the plural form. +E.g., `list of float(s)`. + +Variants of for [**tuples**](https://typing.python.org/en/latest/spec/tuples.html) + +| Docstring type | Python type annotation | +|---------------------|------------------------| +| `tuple of (X, Y)` | `tuple[X, Y]` | +| `tuple of (X, ...)` | `tuple[X, ...]` | + +and **mappings** exist. + +| Docstring type | Python type annotation | +|----------------------|------------------------| +| `MAPPING of {X: Y}` | `MAPPING[X, Y]` | +| `dict of {str: int}` | `dict[str, int]` | + + +> [!TIP] +> While it is possible to nest these variants repeatedly, it is discouraged to do so to keep type descriptions readable. +> For complex annotations with nested containers, consider using Python's conventional syntax. +> In the future, docstub may warn against or disallow nesting these natural language variants. + + +## Shape and dtype syntax for arrays + +This expression allows adding shape and datatype information for data structures like [NumPy arrays](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html). + +`array` and `ndarray`, and `array-like` and `array_like` can be used interchange-ably. + +| Docstring type | Python type annotation | +|-----------------------------|------------------------| +| `array of DTYPE` | `ndarray[DTYPE]` | +| `ndarray of dtype DTYPE` | `ndarray[DTYPE]` | +| `array-like of DTYPE` | `ArrayLike[DTYPE]` | +| `array_like of dtype DTYPE` | `ArrayLike[DTYPE]` | + +> [!NOTE] +> Noting the **shape** of an array in the docstring is supported. +> However, Python's typing system is not yet able to express this information. +> It is therefore not included in the resulting type annotation. + +| Docstring type | Python type annotation | +|--------------------------|------------------------| +| `(3,) array of DTYPE` | `ndarray[DTYPE]` | +| `(X, Y) array of DTYPE` | `ndarray[DTYPE]` | +| `([P,] M, N) array-like` | `ArrayLike` | +| `(M, ...) ndarray` | `ArrayLike` | + + +## Literals + +[Literals](https://typing.python.org/en/latest/spec/literal.html#literals) indicate a concrete value instead of type. +Instead of using [`typing.Literal`](https://docs.python.org/3/library/typing.html#typing.Literal), you can enclose literal values in `{...}` in docstrings. + +| Docstring type | Python type annotation | +|----------------|------------------------| +| `{1, 2, 3}` | `Literal[1, 2, 3]` | +| `{1, 2, 3}` | `Literal[1, 2, 3]` | + +> [!TIP] +> Enclosing a single value `{X}` is currently allowed but discouraged. +> Instead consider the more explicit `Literal[X]`. + + +## reStructuredText role + +Since docstrings are also used to generate documentation with Sphinx, you may want to use [restructuredText roles](https://docutils.sourceforge.io/docs/ref/rst/roles.html) in your type annotations. +Docstub allows for this anywhere where a qualified name can be used. + +| Docstring type | Python type annotation | +|----------------------|------------------------| +| `` `X` `` | `X` | +| ``:ref:`X` `` | `X` | +| ``:class:`Y.X` `` | `Y.X` | +| ``:py:class:`Y.X` `` | `Y.X` | diff --git a/doc/user_guide.md b/doc/user_guide.md new file mode 100644 index 0000000..93ddac8 --- /dev/null +++ b/doc/user_guide.md @@ -0,0 +1,114 @@ +# User guide + +> [!NOTE] In early development! +> Expect bugs, missing features, and incomplete documentation. +> Docstub is still evaluating which features it needs to support as the community gives feedback. +> Several features are experimental and included to make adoption of docstub easier. +> Long-term, some of these might be discouraged or removed as docstub matures. + + +## Installation + +While a docstub package is already available on PyPI, we recommend trying out docstub by installing directly from GitHub with + +```shell +pip install 'docstub [optional] @ git+https://github.com/scientific-python/docstub' +``` + +If you want to pin to a certain commit you can append `@COMMIT_SHA` to the repo URL above. + + +## Getting started + +Consider a simple example with the following documented function + + + + +```python +# example.py + +def example_metric(image, *, mask=None, sigma=1.0, method='standard'): + """Pretend to calculate a local metric between two images. + + Parameters + ---------- + image : array-like + First image. + mask : array of dtype uint8, optional + Second image. + sigma : float or Iterable of float, optional + Sigma value for each dimension in `image`. A single value is broadcast + to all dimensions. + method : {'standard', 'modified'}, optional, default = 'standard' + The method to use for calculating the metric. + + Returns + ------- + met : ndarray of dtype float + """ + pass +``` + + + +Feeding this input to docstub with + +```shell +docstub simple_script.py +``` + +will create `example.pyi` in the same directory + + + + +```python +# File generated with docstub + +from collections.abc import Iterable +from typing import Literal + +import numpy as np +from numpy.typing import ArrayLike, NDArray + +def example_metric( + image: ArrayLike, + *, + mask: NDArray[np.uint8] | None = ..., + sigma: float | Iterable[float] = ..., + method: Literal["standard", "modified"] = ... +) -> NDArray[float]: ... +``` + + + +There are several interesting things to note here: + +- Many existing conventions that the scientific Python ecosystem uses, will work out of the box. + In this case, docstub knew how to translate `array-like`, `array of dtype uint8` into a valid type annotation in the stub file. + In a similar manner, `or` can be used as a "natural language" alternative to `|`. + You can find more details in [Typing syntax in docstrings](typing_syntax.md). + +- Optional arguments that default to `None` are recognized and a `| None` is appended automatically if the type doesn't include it already. + +- Common container types from Pythons standard library such as `Iterable` can be used and a necessary import will be added automatically. + + +## Importing types + +TBD + + +## Adding your own aliases for docstring descriptions + +TBD + + +## Adopting docstub gradually + +TBD + +`--group-errors` + +`--allow-errors` diff --git a/tests/test_doc.py b/tests/test_doc.py new file mode 100644 index 0000000..684bc75 --- /dev/null +++ b/tests/test_doc.py @@ -0,0 +1,71 @@ +"""Test documentation in doc/.""" + +import re +from pathlib import Path + +import click + +from docstub._cli import main as docstub_main + +PROJECT_ROOT = Path(__file__).parent.parent + + +def test_getting_started_example(tmp_path): + # Load user guide + md_file = PROJECT_ROOT / "doc/user_guide.md" + with md_file.open("r") as io: + md_content = io.read() + + # Extract code block for example.py + regex_py = ( + r"" + r"\n+```python(.*)```\n+" + r"" + ) + matches_py = re.findall(regex_py, md_content, flags=re.DOTALL) + assert len(matches_py) == 1 + py_source = matches_py[0] + + # Create example.py and run docstub on it + py_file = tmp_path / "example.py" + with py_file.open("x") as io: + io.write(py_source) + docstub_main([str(py_file)], standalone_mode=False) + + # Load created PYI file, this is what we expect to find in the user guide's + # code block for example.pyi + pyi_file = py_file.with_suffix(".pyi") + assert pyi_file.is_file() + with pyi_file.open("r") as io: + expected_pyi = io.read().strip() + + # Extract code block for example.pyi from guide + regex_pyi = ( + r"" + r"\n+```python(.*)```\n+" + r"" + ) + matches_pyi = re.findall(regex_pyi, md_content, flags=re.DOTALL) + assert len(matches_pyi) == 1 + actual_pyi = matches_pyi[0].strip() + + assert expected_pyi == actual_pyi + + +def test_command_line_help(): + ctx = click.Context(docstub_main, info_name="docstub") + expected_help = f""" +```plain +{docstub_main.get_help(ctx)} +``` +""".strip() + md_file = PROJECT_ROOT / "doc/command_line.md" + with md_file.open("r") as io: + md_content = io.read() + + regex = r"(.*)" + matches = re.findall(regex, md_content, flags=re.DOTALL) + assert len(matches) == 1 + + actual_help = matches[0].strip() + assert actual_help == expected_help