Docstrings checking

[xmnlab]

Accordingly pydocstyle, currently there are 3445 docstings issues inside the project.

I have started to fix that #1996

I will open one PR for each backend dosctring fixing

[xmnlab]

Currently there are a lot of docstrings issue inside ibis. Accordingly pydocstyle, currently there are 3064 docstrings error inside the project.

Ibis CI now is checking just files at ibis/ and ibis/omniscidb but for the other backends, a "precise" validation should be done locally using pydocstyle. and it takes times.

My point of view is that a docstring validations is similar to code validation. In ibis we have already code format validation, and black checking, also pydocstyle validation for two folders.

Ibis also has pre-commit hooks for black, flake8 and isort.

That helps a lot to keep review simple and easier.

I think that would be good to work on that into 2 phases.

1. fix in a simple way the docstring issues and
2. improve and standardize (as much as possible) the docstrings.

the 1st step is important because after this fixing every new contributions will automatically be checked the docstrings on CI.

for the phase 2, we need to discuss best practices and create a document specifying the ibis documentation guideline.

[xmnlab]

Following the discussion on #2018 :

There are a lot of functions and method inside Ibis with no docstrings or with docstrings that doesn't fit in the numpy specifications.

it is an evidence that it is hard for reviewers to check that manually also maybe it also not clear for reviewers that ibis use numpy docstring standard for docstrings.

[jreback]

pls show a sample of the warnings

the linters oftentimes show IMHO a lot of useless things that simply should be ignored

we already do quite a lot of linking so need to be quite mindful of adding baggage

[xmnlab]

in the PR #2018 also I am proposing to keep as private functions that don't need to be visible to end-users.

But if that functions need to be visible to these functions should be very well documented and the documentation should be updated.

https://docs.ibis-project.org/api.html seems to be out of date.

[xmnlab]

here is the current log from pydocstyle: https://pastebin.com/uUsRHkhP

[xmnlab]

the linters oftentimes show IMHO a lot of useless things that simply should be ignored

the problem is do it in a subjective approach .. also my opinion could be different than yours.

using something objective makes the review workflow easier and it guarantees at least a minimum standard level for the code.

[jreback]

you are missing the point

the linters check for too much stuff here

you are going to cause massive changes for IMHO not much gain and a lot of pain from contributors

[xmnlab]

the checks just contemplate the numpy docstring specification.

I think all these changes will have a good impact, for the documentation, for new contributors, for reviewers, for users ...

as I am proposing that .. I am working on fixing these issues .. I am trying to work on one backend each week.

[jreback]

what exactly is the problem with the current lint checks?

[xmnlab]

currently on CI pydocstyle just run for ibis/ and ibis/omniscidb/ .. the docstrings there were already fixed.

related to #2018 these are the current docstring errors: https://pastebin.com/pB7Qgc4y

as you can see numpy docstring specification specify that all public functions methods or classes need a docstring.

[xmnlab]

@toryhaavik @scottcode @cpcloud any thought about this?

[scottcode]

@xmnlab I appreciate your enthusiasm for docstrings as well as automated style checkers. I especially like our use of black and isort, because they impose some consistency without any added burden to the developer. If something doesn't fit the style for those two, they automatically fix them. The fixes are pretty uncontroversial: for example, reorder the import statements or change whitespace/linebreaks in the code.

The problem with pydocstyle is that it complains about a whole bunch of stuff that I don't see much value in. I don't care if the first line ends in a period or not. It doesn't bother me much if some docstrings fit in one line including the quotes while others don't. If the style checker doesn't automatically make uncontroversial fixes to these kind of issues, then I'd prefer not having my commits hung up on such tiny style details.

Some things it finds can be important, like missing docstrings, but even in that case, sometimes method names are self-explanatory enough. Especially in a batch of many very similar methods, ensuring docstrings on every single one might actually more distracting than helpful.

The few ways in which standardization would be helpful don't seem to be checked by pydocstyle anyway. Like consistently using either :param arg: style argument documentation versus more informal-looking as below:

Parameters
-----------
arg:   str Name of table

(Which are we supposed to use, by the way?)

In short, I'd be on board with taking a little more care with our docstrings in ways that add value, but let's not overburden ourselves with minutiae.

[scottcode]

Also, I think using pydocstyle in the way you have with fixing docstrings for the Pyspark backend can lead to unintended consequences, like making a bunch of methods private just so the style checker will ignore them. I don't think all those methods that you changed to private in #2018 really ought to be private.