Typing information for pyosmium

[wiktorn]

This provides *,pyi files with type annotations. As pyosmium is still supporting Python 2.7 I decided to implement type annotations as separate files.

I found this quite useful and informative when using with PyCharm.

For now, I decided to ship *.pyi files together with *.py files. Python community is working on target solution

[lonvia]

I'm not entirely happy with all the code duplication. Keeping the py and pyi files in sync will be a pain. So I'm not sure how to go forward with this. I'm not against adding typing information but it needs to be maintainable.

[wiktorn]

I'm not happy too. Though:

• it's easiest way to maintain compatibility with wide range of python versions
• it makes easy to annotate pure C(++) functions that doesn't have python representation

We can also try with this approach:
https://mypy.readthedocs.io/en/latest/python2.html#type-checking-python-2-code

But then we need to add typing module backport as runtime requirement.

Then we would need to have stub methods for pure C++ functions.

[wiktorn]

I've managed to inline all annotations to *py files, where they do exists:
https://github.com/osmcode/pyosmium/compare/master...wiktorn:typing_annotations_inline?expand=1

For all pure C++ modules I kept the *pyi files.

Does it improve situation to manageable level?

[wiktorn]

To follow up on this. I was able to use mypy to verify the completeness of the annotations but only for some definitions of completeness, i.e. only if all arguments and fields in *.pyi files are annotated. There was no signature check and missing or superfluous annotations.

I also tried pytype but it fails loading C modules. I've also tried mypy's stubgen but this generates only empty stubs. I'll try to debug either of it to get it working and automate generation of *.pyi information first.

[wiktorn]

I've just found python/mypy#5814. Looks promising and was merged just 2 days ago. I gave it a try on current (boost_python) version and results are better than before, but still far from perfect (and even acceptable), as all C++ function signatures are guessed as (*args, **kwargs). I hope that it will be better with pybind as this was primary goal of mentioned issue.

[lonvia]

Thanks for looking into this. If you want to play with the pybind version, the current state is on branch https://github.com/osmcode/pyosmium/tree/pybind-rewrite. It only works with python3 at the moment and some parts are still missing but it should be enough to play with mypy. You need to put a copy of pybind11 in contrib/pybind11 yourself. I still need to figure out how to best handle this external dependency (submodule, copy, external install, ...).

[wiktorn]

Just to let you know. My current plan to move this forward is:

• extend mypy stubgen so it will generate flawless pyi files for C/C++ classes generated by pybind11 (Improve stub files generated by stubgenc for pybind11 modules python/mypy#5850 and one more planned with support for overloaded functions)
• add inline declarations in Python classes in comments, so this will be Python27 compatibile
• make invocation of stubgen task of packaging pyosmium

So this will achieve the goal of not having to think about pyi files when implementing changes.

Another approach to consider is to extend pybind11 with the ability to generate pyi files for classes it generates, but this way above my current C++ skills

[lonvia]

Sounds all good to me. And I see that your mypy change got merged already. I couldn't help with pybind11 either, as my C++ skills do not include that level of template magic.

[wiktorn]

Using master branch of mypy/stubgenc it is possible now to get almost perfect stubs.

The only outstanding broken annotation are in lib/osmium.cc:

def apply(reader: osmium.io.Reader, handler: osmium::handler::NodeLocationsForWays<osmium::index::map::Map<unsignedlong, osmium: :Location>, osmium: :index::map::Dummy<unsignedlong, osmium: :Location>>) -> None: ...
def apply(reader: osmium.io.Reader, node_handler: osmium::handler::NodeLocationsForWays<osmium::index::map::Map<unsignedlong, osmium: :Location>, osmium: :index::map::Dummy<unsignedlong, osmium: :Location>>, handler: BaseHandler) -> None: ...

But I didn't find a way to provide proper types here (like it was possible elsewhere by adding py::import statements).

The other outstanding problem is that pybind generates docstring for stl iterators that doesn't give any type information, but that something I hope can by addressed on pybind side.

I've added step in setup.py / build_ext that generates stubs for C++ modules. For Python modules - typing information is included as comments to maintain compatibility with Python2.

I've added build-requirements.txt and test-requirements.txt so it is easy to setup build/test environment with pip install -r build-requirements.txt etc.

PS. We will have to wait for mypy > v0.660 to get functionality from: python/mypy#5975

[lonvia]

Closing in favour of #211.