Add basic end-to-end support for Literal types (#5947)

This pull request primarily adds support for string, int, bool, and
None Literal types in the parsing and semantic analysis layer. It also
adds a small smidge of logic to the subtyping and overlapping types
parts of the typechecking layer -- just enough so we can start using
Literal types end-to-end and write some tests. It also adds a 
"skip path normalization" option to our test cases.

Basically, my previous diff made a bunch of gross changes in a
horizontal way to lay out a foundation; this diff makes a bunch of
gross changes in a vertical way to create a proof-of-concept of the
proof-of-concept.

I'll probably need to submit a few follow-up PRs cleaning up some
stuff in the semantic analysis layer, but my hope is to switch focus
on fleshing out the type checking layer shortly after this PR lands.

Specific changes made:

1. I added a new 'RawLiteralType' synthetic type meant to represent any
   literal expression that appears in the earliest stages of semantic
   analysis. If these 'RawLiteralTypes' appear in a "Literal[...]" context,
   they transformed into regular 'LiteralTypes' during phase 2 of semantic
   analysis (turning UnboundTypes into actual types). If they appear
   outside of the correct context, we report an error instead.

   I also added a string field to UnboundType to keep track of similar
   information. Basically, if we have `Foo["bar"]`, we don't actually
   know immediately whether or not that "bar" is meant to be string
   literal vs a forward reference. (Foo could be a regular class or
   an alias for 'Literal').

   Note: I wanted to avoid having to introduce yet another type, so
   looked into perhaps attaching even more metadata to UnboundType
   or perhaps to LiteralType. Both of those approaches ended up looking
   pretty messy though, so I went with this.

2. As a consequence, some of the syntax checking logic had to move
   from the parsing layer to the semantic analysis layer, and some of
   the existing error messages changed slightly.

3. The PEP, at some point, provisionally rejected having Literals
   contain other Literal types. For example, something like this:

       RegularIds = Literal[1, 2, 3, 4]
       AdminIds = Literal[100, 101, 102]
       AllIds = Literal[RegularIds, AdminIds, 30, 32]

   This was mainly because I thought this would be challenging to
   implement, but it turned out to be easier then expected -- it
   happened almost by accident while I was implementing support
   for 'Literal[1, 2, 3]'.

   I can excise this part out if we think supporting this kind
   of thing is a fundamentally bad idea.

4. I also penciled in some minimal code to the subtyping and 
    overlapping types logic.

This diff also tweaks some of our test-case logic: we can now specify
we want to skip path normalization on certain test cases.

Specifically, the problem was that mypy attempts to normalize
paths by replacing all instances of '\' with '/' so that the
output when running the tests on Windows matches the specified
errors. This is what we want to most of the time: except when we want
to print out Literal types containing strings with slashes.

I thought about maybe changing the output of mypy in general so
it always uses '/' for paths in error outputs, even on Windows:
this would mean we would no longer have to do path normalization.
However, I wasn't convinced this was the right thing to do: using
'\' on Windows technically *is* the right thing to do, and I didn't
want to complicate the codebase by forcing us to keep track of when
to use os.sep vs '/'.

I don't think I'll add too many of these test cases, so I decided
to just go with a localized solution instead of changing mypy's
error output as a whole.

Author: Michael0x2a

mypy/exprtotype.py

@@ -1,13 +1,14 @@
 """Translate an Expression to a Type value."""
 
 from mypy.nodes import (
-    Expression, NameExpr, MemberExpr, IndexExpr, TupleExpr,
+    Expression, NameExpr, MemberExpr, IndexExpr, TupleExpr, IntExpr, FloatExpr, UnaryExpr,
     ListExpr, StrExpr, BytesExpr, UnicodeExpr, EllipsisExpr, CallExpr,
     get_member_expr_fullname
 )
 from mypy.fastparse import parse_type_comment
 from mypy.types import (
-    Type, UnboundType, TypeList, EllipsisType, AnyType, Optional, CallableArgument, TypeOfAny
+    Type, UnboundType, TypeList, EllipsisType, AnyType, Optional, CallableArgument, TypeOfAny,
+    LiteralType, RawLiteralType,
 )
 
 
@@ -37,7 +38,12 @@ def expr_to_unanalyzed_type(expr: Expression, _parent: Optional[Expression] = No
     name = None  # type: Optional[str]
     if isinstance(expr, NameExpr):
         name = expr.name
-        return UnboundType(name, line=expr.line, column=expr.column)
+        if name == 'True':
+            return RawLiteralType(True, 'builtins.bool', line=expr.line, column=expr.column)
+        elif name == 'False':
+            return RawLiteralType(False, 'builtins.bool', line=expr.line, column=expr.column)
+        else:
+            return UnboundType(name, line=expr.line, column=expr.column)
     elif isinstance(expr, MemberExpr):
         fullname = get_member_expr_fullname(expr)
         if fullname:
@@ -108,11 +114,26 @@ def expr_to_unanalyzed_type(expr: Expression, _parent: Optional[Expression] = No
     elif isinstance(expr, (StrExpr, BytesExpr, UnicodeExpr)):
         # Parse string literal type.
         try:
-            result = parse_type_comment(expr.value, expr.line, None)
-            assert result is not None
+            node = parse_type_comment(expr.value, expr.line, None)
+            assert node is not None
+            if isinstance(node, UnboundType) and node.original_str_expr is None:
+                node.original_str_expr = expr.value
+            return node
         except SyntaxError:
+            return RawLiteralType(expr.value, 'builtins.str', line=expr.line, column=expr.column)
+    elif isinstance(expr, UnaryExpr):
+        typ = expr_to_unanalyzed_type(expr.expr)
+        if isinstance(typ, RawLiteralType) and isinstance(typ.value, int) and expr.op == '-':
+            typ.value *= -1
+            return typ
+        else:
             raise TypeTranslationError()
-        return result
+    elif isinstance(expr, IntExpr):
+        return RawLiteralType(expr.value, 'builtins.int', line=expr.line, column=expr.column)
+    elif isinstance(expr, FloatExpr):
+        # Floats are not valid parameters for RawLiteralType, so we just
+        # pass in 'None' for now. We'll report the appropriate error at a later stage.
+        return RawLiteralType(None, 'builtins.float', line=expr.line, column=expr.column)
     elif isinstance(expr, EllipsisExpr):
         return EllipsisType(expr.line)
     else:

mypy/fastparse.py

@@ -32,7 +32,7 @@
 )
 from mypy.types import (
     Type, CallableType, AnyType, UnboundType, TupleType, TypeList, EllipsisType, CallableArgument,
-    TypeOfAny, Instance,
+    TypeOfAny, Instance, RawLiteralType,
 )
 from mypy import defaults
 from mypy import messages
@@ -53,6 +53,9 @@
         Expression as ast3_Expression,
         Str,
         Index,
+        Num,
+        UnaryOp,
+        USub,
     )
 except ImportError:
     if sys.version_info.minor > 2:
@@ -1138,12 +1141,46 @@ def visit_Name(self, n: Name) -> Type:
         return UnboundType(n.id, line=self.line)
 
     def visit_NameConstant(self, n: NameConstant) -> Type:
-        return UnboundType(str(n.value))
+        if isinstance(n.value, bool):
+            return RawLiteralType(n.value, 'builtins.bool', line=self.line)
+        else:
+            return UnboundType(str(n.value), line=self.line)
+
+    # UnaryOp(op, operand)
+    def visit_UnaryOp(self, n: UnaryOp) -> Type:
+        # We support specifically Literal[-4] and nothing else.
+        # For example, Literal[+4] or Literal[~6] is not supported.
+        typ = self.visit(n.operand)
+        if isinstance(typ, RawLiteralType) and isinstance(n.op, USub):
+            if isinstance(typ.value, int):
+                typ.value *= -1
+                return typ
+        self.fail(TYPE_COMMENT_AST_ERROR, self.line, getattr(n, 'col_offset', -1))
+        return AnyType(TypeOfAny.from_error)
+
+    # Num(number n)
+    def visit_Num(self, n: Num) -> Type:
+        # Could be either float or int
+        numeric_value = n.n
+        if isinstance(numeric_value, int):
+            return RawLiteralType(numeric_value, 'builtins.int', line=self.line)
+        elif isinstance(numeric_value, float):
+            # Floats and other numbers are not valid parameters for RawLiteralType, so we just
+            # pass in 'None' for now. We'll report the appropriate error at a later stage.
+            return RawLiteralType(None, 'builtins.float', line=self.line)
+        else:
+            self.fail(TYPE_COMMENT_AST_ERROR, self.line, getattr(n, 'col_offset', -1))
+            return AnyType(TypeOfAny.from_error)
 
     # Str(string s)
     def visit_Str(self, n: Str) -> Type:
-        return (parse_type_comment(n.s.strip(), self.line, self.errors) or
-                AnyType(TypeOfAny.from_error))
+        try:
+            node = parse_type_comment(n.s.strip(), self.line, errors=None)
+            if isinstance(node, UnboundType) and node.original_str_expr is None:
+                node.original_str_expr = n.s
+            return node or AnyType(TypeOfAny.from_error)
+        except SyntaxError:
+            return RawLiteralType(n.s, 'builtins.str', line=self.line)
 
     # Subscript(expr value, slice slice, expr_context ctx)
     def visit_Subscript(self, n: ast3.Subscript) -> Type:

mypy/indirection.py

@@ -90,6 +90,9 @@ def visit_tuple_type(self, t: types.TupleType) -> Set[str]:
     def visit_typeddict_type(self, t: types.TypedDictType) -> Set[str]:
         return self._visit(t.items.values()) | self._visit(t.fallback)
 
+    def visit_raw_literal_type(self, t: types.RawLiteralType) -> Set[str]:
+        assert False, "Unexpected RawLiteralType after semantic analysis phase"
+
     def visit_literal_type(self, t: types.LiteralType) -> Set[str]:
         return self._visit(t.fallback)
 

mypy/meet.py

@@ -235,6 +235,13 @@ def is_none_typevar_overlap(t1: Type, t2: Type) -> bool:
     elif isinstance(right, CallableType):
         right = right.fallback
 
+    if isinstance(left, LiteralType) and isinstance(right, LiteralType):
+        return left == right
+    elif isinstance(left, LiteralType):
+        left = left.fallback
+    elif isinstance(right, LiteralType):
+        right = right.fallback
+
     # Finally, we handle the case where left and right are instances.
 
     if isinstance(left, Instance) and isinstance(right, Instance):

mypy/messages.py

@@ -19,7 +19,7 @@
 from mypy.erasetype import erase_type
 from mypy.errors import Errors
 from mypy.types import (
-    Type, CallableType, Instance, TypeVarType, TupleType, TypedDictType,
+    Type, CallableType, Instance, TypeVarType, TupleType, TypedDictType, LiteralType,
     UnionType, NoneTyp, AnyType, Overloaded, FunctionLike, DeletedType, TypeType,
     UninhabitedType, TypeOfAny, ForwardRef, UnboundType
 )
@@ -297,6 +297,8 @@ def format_bare(self, typ: Type, verbosity: int = 0) -> str:
                                                  self.format_bare(item_type)))
             s = 'TypedDict({{{}}})'.format(', '.join(items))
             return s
+        elif isinstance(typ, LiteralType):
+            return str(typ)
         elif isinstance(typ, UnionType):
             # Only print Unions as Optionals if the Optional wouldn't have to contain another Union
             print_as_optional = (len(typ.items) -

mypy/semanal_newtype.py

@@ -5,7 +5,7 @@
 
 from typing import Tuple, Optional
 
-from mypy.types import Type, Instance, CallableType, NoneTyp, TupleType
+from mypy.types import Type, Instance, CallableType, NoneTyp, TupleType, AnyType, TypeOfAny
 from mypy.nodes import (
     AssignmentStmt, NewTypeExpr, CallExpr, NameExpr, RefExpr, Context, StrExpr, BytesExpr,
     UnicodeExpr, Block, FuncDef, Argument, TypeInfo, Var, SymbolTableNode, GDEF, MDEF, ARG_POS
@@ -107,13 +107,21 @@ def check_newtype_args(self, name: str, call: CallExpr, context: Context) -> Opt
             has_failed = True
 
         # Check second argument
+        msg = "Argument 2 to NewType(...) must be a valid type"
         try:
             unanalyzed_type = expr_to_unanalyzed_type(args[1])
         except TypeTranslationError:
-            self.fail("Argument 2 to NewType(...) must be a valid type", context)
+            self.fail(msg, context)
             return None
+
         old_type = self.api.anal_type(unanalyzed_type)
 
+        # The caller of this function assumes that if we return a Type, it's always
+        # a valid one. So, we translate AnyTypes created from errors into None.
+        if isinstance(old_type, AnyType) and old_type.type_of_any == TypeOfAny.from_error:
+            self.fail(msg, context)
+            return None
+
         return None if has_failed else old_type
 
     def build_newtype_typeinfo(self, name: str, old_type: Type, base_type: Instance) -> TypeInfo:

mypy/semanal_pass3.py

@@ -22,7 +22,7 @@
 )
 from mypy.types import (
     Type, Instance, AnyType, TypeOfAny, CallableType, TupleType, TypeVarType, TypedDictType,
-    UnionType, TypeType, Overloaded, ForwardRef, TypeTranslator, function_type
+    UnionType, TypeType, Overloaded, ForwardRef, TypeTranslator, function_type, LiteralType,
 )
 from mypy.errors import Errors, report_internal_error
 from mypy.options import Options
@@ -704,6 +704,13 @@ def visit_typeddict_type(self, t: TypedDictType) -> Type:
         assert isinstance(fallback, Instance)
         return TypedDictType(items, t.required_keys, fallback, t.line, t.column)
 
+    def visit_literal_type(self, t: LiteralType) -> Type:
+        if self.check_recursion(t):
+            return AnyType(TypeOfAny.from_error)
+        fallback = self.visit_instance(t.fallback, from_fallback=True)
+        assert isinstance(fallback, Instance)
+        return LiteralType(t.value, fallback, t.line, t.column)
+
     def visit_union_type(self, t: UnionType) -> Type:
         if self.check_recursion(t):
             return AnyType(TypeOfAny.from_error)

mypy/server/astmerge.py

@@ -59,6 +59,7 @@
     Type, SyntheticTypeVisitor, Instance, AnyType, NoneTyp, CallableType, DeletedType, PartialType,
     TupleType, TypeType, TypeVarType, TypedDictType, UnboundType, UninhabitedType, UnionType,
     Overloaded, TypeVarDef, TypeList, CallableArgument, EllipsisType, StarType, LiteralType,
+    RawLiteralType,
 )
 from mypy.util import get_prefix, replace_object_state
 from mypy.typestate import TypeState
@@ -391,6 +392,9 @@ def visit_typeddict_type(self, typ: TypedDictType) -> None:
             value_type.accept(self)
         typ.fallback.accept(self)
 
+    def visit_raw_literal_type(self, t: RawLiteralType) -> None:
+        assert False, "Unexpected RawLiteralType after semantic analysis phase"
+
     def visit_literal_type(self, typ: LiteralType) -> None:
         typ.fallback.accept(self)
 

mypy/subtypes.py

@@ -327,8 +327,11 @@ def visit_typeddict_type(self, left: TypedDictType) -> bool:
         else:
             return False
 
-    def visit_literal_type(self, t: LiteralType) -> bool:
-        raise NotImplementedError()
+    def visit_literal_type(self, left: LiteralType) -> bool:
+        if isinstance(self.right, LiteralType):
+            return left == self.right
+        else:
+            return self._is_subtype(left.fallback, self.right)
 
     def visit_overloaded(self, left: Overloaded) -> bool:
         right = self.right
@@ -1172,7 +1175,10 @@ def visit_typeddict_type(self, left: TypedDictType) -> bool:
         return self._is_proper_subtype(left.fallback, right)
 
     def visit_literal_type(self, left: LiteralType) -> bool:
-        raise NotImplementedError()
+        if isinstance(self.right, LiteralType):
+            return left == self.right
+        else:
+            return self._is_proper_subtype(left.fallback, self.right)
 
     def visit_overloaded(self, left: Overloaded) -> bool:
         # TODO: What's the right thing to do here?

mypy/test/data.py

@@ -42,6 +42,7 @@ def parse_test_case(case: 'DataDrivenTestCase') -> None:
         join = posixpath.join  # type: ignore
 
     out_section_missing = case.suite.required_out_section
+    normalize_output = True
 
     files = []  # type: List[Tuple[str, str]] # path and contents
     output_files = []  # type: List[Tuple[str, str]] # path and contents for output files
@@ -98,8 +99,11 @@ def parse_test_case(case: 'DataDrivenTestCase') -> None:
             full = join(base_path, m.group(1))
             deleted_paths.setdefault(num, set()).add(full)
         elif re.match(r'out[0-9]*$', item.id):
+            if item.arg == 'skip-path-normalization':
+                normalize_output = False
+
             tmp_output = [expand_variables(line) for line in item.data]
-            if os.path.sep == '\\':
+            if os.path.sep == '\\' and normalize_output:
                 tmp_output = [fix_win_path(line) for line in tmp_output]
             if item.id == 'out' or item.id == 'out1':
                 output = tmp_output
@@ -147,6 +151,7 @@ def parse_test_case(case: 'DataDrivenTestCase') -> None:
     case.expected_rechecked_modules = rechecked_modules
     case.deleted_paths = deleted_paths
     case.triggered = triggered or []
+    case.normalize_output = normalize_output
 
 
 class DataDrivenTestCase(pytest.Item):  # type: ignore  # inheriting from Any
@@ -168,6 +173,10 @@ class DataDrivenTestCase(pytest.Item):  # type: ignore  # inheriting from Any
     # Files/directories to clean up after test case; (is directory, path) tuples
     clean_up = None  # type: List[Tuple[bool, str]]
 
+    # Whether or not we should normalize the output to standardize things like
+    # forward vs backward slashes in file paths for Windows vs Linux.
+    normalize_output = True
+
     def __init__(self,
                  parent: 'DataSuiteCollector',
                  suite: 'DataSuite',

mypy/test/testcheck.py

@@ -80,6 +80,7 @@
     'check-ctypes.test',
     'check-dataclasses.test',
     'check-final.test',
+    'check-literal.test',
 ]
 
 
@@ -177,7 +178,8 @@ def run_case_once(self, testcase: DataDrivenTestCase,
             assert sys.path[0] == plugin_dir
             del sys.path[0]
 
-        a = normalize_error_messages(a)
+        if testcase.normalize_output:
+            a = normalize_error_messages(a)
 
         # Make sure error messages match
         if incremental_step == 0:

mypy/test/testcmdline.py

@@ -85,14 +85,19 @@ def test_python_cmdline(testcase: DataDrivenTestCase) -> None:
                 actual_output_content = output_file.read().splitlines()
             normalized_output = normalize_file_output(actual_output_content,
                                                       os.path.abspath(test_temp_dir))
-            if testcase.suite.native_sep and os.path.sep == '\\':
-                normalized_output = [fix_cobertura_filename(line) for line in normalized_output]
-            normalized_output = normalize_error_messages(normalized_output)
+            # We always normalize things like timestamp, but only handle operating-system
+            # specific things if requested.
+            if testcase.normalize_output:
+                if testcase.suite.native_sep and os.path.sep == '\\':
+                    normalized_output = [fix_cobertura_filename(line)
+                                         for line in normalized_output]
+                normalized_output = normalize_error_messages(normalized_output)
             assert_string_arrays_equal(expected_content.splitlines(), normalized_output,
                                        'Output file {} did not match its expected output'.format(
                                            path))
     else:
-        out = normalize_error_messages(err + out)
+        if testcase.normalize_output:
+            out = normalize_error_messages(err + out)
         obvious_result = 1 if out else 0
         if obvious_result != result:
             out.append('== Return code: {}'.format(result))

mypy/test/testmerge.py

@@ -93,7 +93,8 @@ def run_case(self, testcase: DataDrivenTestCase) -> None:
             # Verify that old AST nodes are removed from the expression type map.
             assert expr not in new_types
 
-        a = normalize_error_messages(a)
+        if testcase.normalize_output:
+            a = normalize_error_messages(a)
 
         assert_string_arrays_equal(
             testcase.output, a,