Type hints

[kmike]

I've tried to add PEP-484 type hints in this PR (checked using mypy). They showed how tricky are our string types :) For each string there are several options:

• str
• bytes
• typing.AnyStr
• typing.Text (or six.text_type)
• w3lib._types.String (a custom type I had to add)

There are several interesting points revealed by these type checks, e.g. that html_body_declared_encoding only worked in Python 3 because stdlib encodings.normalize_encoding accept binary values (I'm not sure this stdlib module is even documented).

---

Update: this PR is rather useless for the end users because of python/mypy#1190; it also adds typing runtime dependency which could be good to avoid. Switch to stub files?

[kmike]

Get off annoying bot, these imports are used. You're not aware of PEP-484.

[codecov-io]

Current coverage is 94.07%

Merging #61 into master will increase coverage by 0.41%

Powered by Codecov. Last updated by ec0230a...825c1cb

[lopuhin]

A nitpick - this is not reflected by the type

[kmike]

Yeah; currently mypy assumes that None is a valid value for any type, so Tuple[str, bytes] allows None, None. I recall using Union[Tuple[str, bytes], Tuple[None, None]] in some other place, but I'm not sure it works.

[lopuhin]

Ah, didn't know that, thanks! mypy behavior makes sense for tuple, although not that I would expect it in general :)

[kmike]

They have docs for this: http://mypy.readthedocs.io/en/latest/planned_features.html. Let me try Union again :)

[kmike]

it returns a weird error for this line if annotation is changed to Union:

bom_enc, bom = read_bom(html_body_str)

w3lib/encoding.py: note: In function "html_to_unicode":
w3lib/encoding.py:274: error: 'Union[Tuple[builtins.str, builtins.bytes], Tuple[void, void]]' object is not iterable

It looks related to Union handling, not to Tuple[None, None] handling because it also fails if Tuple[str, str] is used as a second option for Union.

[lopuhin]

So mypy does not see that all union "kinds" here are iterable, that's a pity.

[kmike]

relevant mypy issue: python/mypy#1575

[eliasdorneles]

❤️

[eliasdorneles]

it also adds typing runtime dependency which could be good to avoid

if we want to avoid the runtime dependency, we could use a hack like:

if os.getenv('ENABLE_TYPE_CHECKS'):
    from typing import ...

it'd make the imports kinda ugly, tho.

[lucywang000]

FTR: I'll continue this PR soon.

[kmike]

FTR: There was a lot of new development in past 2 years on typing. Now type annotations from third-party libraries can be even more useful, as mypy implemented support for https://www.python.org/dev/peps/pep-0561/ - it is possible to check user code against annotations of third-party libraries.

[yozachar]

Bumping to close outdated PR.