Recursive types not working correctly

[Waszker]

Bug Report

Related to #731
The new recursive types support (enabled by flag --enable-recursive-aliases) does not behave correctly.

To Reproduce

from typing import Dict, List, Union

JSON = Union[Dict[str, 'JSON'], List['JSON'], str, int, float, bool, None]

some_data = {'eggs': 'spam'}
json_data: JSON = {'foo': 'bar', 'python': some_data}

Expected Behavior

The MyPy should not detect any errors since some_data is a valid JSON representation and can be safely nested in another JSON.

Actual Behavior

$ mypy --enable-recursive-aliases main.py                                                                                                                                                                        
main.py:6: error: Dict entry 1 has incompatible type "str": "Dict[str, str]"; expected "str": "Union[Dict[str, JSON], List[JSON], str, float, None]"
Found 1 error in 1 file (checked 1 source file)

Please note that explicitly marking some_data as JSON does not return the error, e.g.

from typing import Dict, List, Union

JSON = Union[Dict[str, 'JSON'], List['JSON'], str, int, float, bool, None]

some_data: JSON = {'eggs': 'spam'}
json_data: JSON = {'foo': 'bar', 'python': some_data}  # no error detected

Your Environment

• Mypy version used: 0.981
• Mypy command-line flags: --enable-recursive-aliases
• Mypy configuration options from mypy.ini (and other config files): -
• Python version used: 3.10

[erictraut]

I think mypy is doing the correct thing here. In your first sample, some_data has no type annotation, so its type is inferred to be dict[str, str]. The type parameters for dict are invariant, so they must match exactly. Since some_data is dict[str, str], it is not compatible with dict[str, JSON]. If you add the type annotation for some_data as you do in your second example, then it will work fine. Also, if you inline the expression for some_data rather than creating a separate variable, mypy can use the type context to infer that its type should be dict[str, JSON] in that context.

[Waszker]

I think mypy is doing the correct thing here. In your first sample, some_data has no type annotation, so its type is inferred to be dict[str, str]. The type parameters for dict are invariant, so they must match exactly. Since some_data is dict[str, str], it is not compatible with dict[str, JSON]. If you add the type annotation for some_data as you do in your second example, then it will work fine. Also, if you inline the expression for some_data rather than creating a separate variable, mypy can use the type context to infer that its type should be dict[str, JSON] in that context.

Thank you for the answer. How should I proceed with the JSON type definition then? There are some functions in my code that have annotated return value as dict[str, str] and I'd like them to be accepted as part of JSON structure. Should I just cast them explicitly?

[erictraut]

Casting is one option. Another option is possible if you don't need the resulting type to be mutable. You could define another type for a read-only JSON value.

JSON_RO = Union[
    Mapping[str, "JSON_RO"], Sequence["JSON_RO"], str, int, float, bool, None
]

some_data = {"eggs": "spam"}
json_data: JSON_RO = {"foo": "bar", "python": some_data}

This works because the type parameters for Mapping and Sequence are covariant.

[smheidrich]

@erictraut The problem with that is that you often genuinely don't want generic mappings and sequences to be considered valid JSON-like types, because certain JSON libraries won't accept them.

Python's own builtin json module, for example, insists on lists and dicts (or subclasses thereof) and will fail if you e.g. give it another kind of mapping:

import json
from typing import Mapping, Sequence, Union

JSON_RO = Union[
    Mapping[str, "JSON_RO"], Sequence["JSON_RO"], str, int, float, bool, None
]

class MyMapping(Mapping[str, int]):
    def __init__(self):
        self.data = {"a": 1, "b": 2}

    def __iter__(self):
        return iter(self.data)

    def __getitem__(self, key):
        return self.data[key]

    def __len__(self):
        return len(self.data)

def dumps(j: JSON_RO):
    json.dumps(j)

dumps(MyMapping())

Mypy doesn't complain about this, of course, but it fails at runtime with

Traceback (most recent call last):
  File "/home/smheidrich/json-type.py", line 29, in <module>
    dumps(MyMapping())
  File "/home/smheidrich/json-type.py", line 26, in dumps
    json.dumps(j)
  File "/usr/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type MyMapping is not JSON serializable

So in that case, using list and dict instead of Sequence and Mapping would be quite deliberate and allow Mypy to catch the invalid usage.

Is there a way to fix the type alias while keeping it limited to lists and dicts (or subclasses) specifically? 🤔