Statement is unreachable when checking if Match[str].group(...) type is None

[Jackenmen]

Hi, I am not entirely sure if this is the right place to report it so please tell me if I should submit the issue somewhere else.

Issue type: Bug
Problematic code:

from typing import Match, Optional
import re


def some_function(some_match: Match[str]) -> bool:
    maybe_match_group = some_match.group(1)
    if maybe_match_group is None:
        group_matched = True
    else:
        group_matched = False
    return group_matched


pattern = re.compile("(str)?")
match = pattern.fullmatch("")
if match is not None:
    print(some_function(match))

Error message:

main.py:8: error: Statement is unreachable

Expected behaviour:
No error as the statement is reachable (and it's used when you run this code). I tried using Match[Optional[str]] to fix this but it's not valid argument for Match.
Another thing to note is that this code will not error if I'll just use Match without specifying type of string.

Python version:: 3.7.4
Mypy version: 0.730, occurs on master as well
Mypy flags: --warn-unreachable

[Michael0x2a]

This is a legitimate bug/limitation in mypy. In short, what's happening is that the Match.group(...) function is currently defined in typeshed to only ever return a str or a byte, depending on whether you have a Match[str] or a Match[byte].

This is obviously incorrect, as you noted. And due to this incorrect annotation, the --warn-unreachable flag ends up assuming that the "if the group is None" branch will never be run.

We actually did experiment with having the return be an Optional, but it ended up causing regressions in various codebases we were testing on. E.g. some snippet of code was using a regex where there would always be N groups and so rightfully skipped the 'is this group None?' checks.

The reason why not specifying the type of the string works is because such a type is equivalent to doing Match[Any]. So, group() ends up also returning Any.

cc @ilevkivskyi, since we were just talking about this the other day. Maybe writing a plugin is the right approach after all? Doing re.compile(...).groups will at least get you the upper bound on the number of groups there'll be, and doing sre_parse.parse(...) seems to get us an AST of the regex we could perhaps try analyzing.

[ilevkivskyi]

OK, it looks like we need some compromise here, possible plan is:

• Infer non-optional type for .group() and .group(0)
• Infer precise type for .group(n) when a plugin can
• Fallback to Any in other cases
• For multi-argument calls infer precise type when plugin can
• Fallback to optional type in multi-argument calls

This should:

• fix all bugs with --warn-unreachable
• not generate too many false positives for optional types
• still have some decent type checking for str/bytes

@JelleZijlstra @srittau

[AlexWaygood]

Typeshed improvements mean that this can no longer be reproduced in 0.920+