Revamp implementation of generic functions. (#581)

This change revises parsing and type substitution for generic functions.  Fix a number of bugs in the existing implementation.

The implementation of generic bounds-safe interfaces required that all calls involving checked arguments be supplied type arguments.  This increased the number of code changes required.  In unchecked or _Bounds_only contexts, allow the type arguments to be omitted and supply void as the omitted type arguments. This allows existing code to work "as is" (i.e. keep implicit casts to _Array_ptr<void>).  We may revisit this at some point

Details:

1. Parsing of generic type applications depended on detecting an  occurrence of a generic function to parse a type argument list.  Reimplement parsing of generic type  applications so that it does not depend on information from semantic checking.

   Treat generic type applications as postfix expressions, with the grammar production:

    postfix-expression:
      postfix-expression < type argument list>

   where type argument list consists of a comma separated list of zero or more type arguments.

   In C, this isn't ambiguous because the `<` operator cannot be followed by a type name or '>'.  We just need to check the first token after '<' to see if it is a type name or '>'.  Of course, there is an ambiguity in
the C grammar that an identifier could either by a type name or variable name.  We haven't made this worse,
      
   I don't believe this is ambiguous for C++ assuming that recognizing a template application has higher precedence. A handful of clang tests that check compiler diagnostics for C++ templates broke, so I've enabled this production only for Checked C for now.
   
   The parsing change partly addresses Github issue #559 (address of operator on function with generic type parameters).  We now allow the address-of operator to be applied to generic functions with type arguments.

2. Refactor the code for applying a generic function type to type arguments. The implementation of type substitution was buggy and didn't substitute type arguments into bounds expressions.

   Move the substitution code from Parse into Sema so that we have access to TreeTransform. Also move the semantic checking for type applications to Sema.

   Use TreeTransform-based transformation to do type substitution for type applications so that it is correct for bounds expression (so that bounds expressions are rewritten).

   When checking call expressions, we expand count/byte_count bounds expressions to standard forms after doing substitution.  In the case where the interface type for a parameter is used to construct the
parameter bounds expression, we need to cast the argument to the interface type before expansion.  Otherwise we end up with arithmetic on void pointers when generic bounds-safe interfaces are used.

3.  Improve diagnostics:
   Add diagnostics for the different ways redeclarations of functions  can be incompatible due generic quantifiers.
      - The quantifiers can be mismatched.
      - The number of type variables can be mismatched.
      - One type can be declared as generic and the other type as
     non-generic.
    
    Add a new diagnostic message for the case where a non-generic function is applied to type arguments.

4. Fix bugs in the existing implementation:
   1. The calculation of nesting depths of type variables (the number   of generic quantifiers enclosing a type variable) was wrong.   All type variables were being assigned a depth of 0, which would cause substitution in types involving higher-order polymorphic functions to go wrong.
   2. Fix type compatibility for generic, itype generic, and non-generic  function types.
   - Generic types are compatible with generic types.
   - Non-generic types are compatible with non-generic types.
   - Itype generic types are compatible with other itype generic types or non-generic types.
   - All other combinations of generic, itype generics, and non-generic functions are incompatible.
   Also, merging of function types wasn't working properly for itype generic functions.
   3.  Dumping of itype generic functions differed from generic functions.
   4. Scope handling of scopes created for generic and itype generic functions was often wrong.  In particular, we weren't always exiting the scope in some cases.  I fixed all the spots I could find, but the   handling of this remains brittle and bug prone.  The issue is that we treat the quantifier as being part of the decl spec, so we introduce  the new scope during parsing of decl specifiers.  That means we have to be sure to exit the scope when we are done with the declaration.
   5. When adding function declarations to scopes, we didn't properly go to the enclosing scope when the function was an `itype forall`.
   6. The Profile function for type variables wasn't including whether the type variable is for a bounds-safe interface. This caused  `_Itype_for_any` type variables and `_For_any type` variables to become confused with each other, which led to test failures.

Testing:
- Add more tests of AST construction for generic functions, including new tests for higher-order generic functions (generic functions that take other generic functions as arguments).
- Add tests of AST construction for type applications of generic functions, where type arguments are substited for type parameters.
- In the Checked C repo, add tests of new diagnostics for types that are incompatible due properties of generic types.
- Update existing tests of AST dumping of type variables.
- Passed automated testing for update Checked C, Checked C clang, and and clang tests on Linux.  Passed testing on Windows x64 locally.

Author: dtarditi

include/clang/AST/Type.h

@@ -1762,6 +1762,8 @@ class Type : public ExtQualsTypeCommonBase {
   bool isFunctionType() const;
   bool isFunctionNoProtoType() const { return getAs<FunctionNoProtoType>(); }
   bool isFunctionProtoType() const { return getAs<FunctionProtoType>(); }
+  bool isGenericFunctionType() const;
+  bool isItypeGenericFunctionType() const;
   bool isPointerType() const;
   bool isCheckedPointerType() const;
   bool isUncheckedPointerType() const;
@@ -3562,6 +3564,9 @@ class FunctionProtoType : public FunctionType, public llvm::FoldingSetNode {
   unsigned getNumTypeVars() const { return NumTypeVars; }
   bool isGenericFunction() const { return GenericFunction; }
   bool isItypeGenericFunction() const { return ItypeGenericFunction; }
+  bool isNonGenericFunction() const {
+    return !(GenericFunction || ItypeGenericFunction);
+  }
 
   ArrayRef<QualType> getParamTypes() const {
     return llvm::makeArrayRef(param_type_begin(), param_type_end());
@@ -3847,7 +3852,7 @@ class TypeVariableType : public Type, public llvm::FoldingSetNode {
   // prototype scope depth, this keeps track of the depth of forany scope.
   unsigned int depth;
   unsigned int index;
-  bool isBoundsInterfaceType;
+  bool isBoundsInterfaceType; // TODO: pack this into a bitfield.
 protected:
   TypeVariableType(unsigned int inDepth, unsigned int inIndex, bool inBoundsInterface)
     : Type(TypeVariable, QualType(), false, false, false, false),
@@ -3868,11 +3873,13 @@ class TypeVariableType : public Type, public llvm::FoldingSetNode {
   }
 
   void Profile(llvm::FoldingSetNodeID &ID) {
-    Profile(ID, depth, index);
+    Profile(ID, depth, index, isBoundsInterfaceType);
   }
-  static void Profile(llvm::FoldingSetNodeID &ID, unsigned int inDepth, unsigned int inIndex) {
+  static void Profile(llvm::FoldingSetNodeID &ID, unsigned int inDepth, unsigned int inIndex,
+                      bool isBoundsInterfaceType) {
     ID.AddInteger(inDepth);
     ID.AddInteger(inIndex);
+    ID.AddBoolean(isBoundsInterfaceType);
   }
 
   static bool classof(const Type *T) { return T->getTypeClass() == TypeVariable; }
@@ -6011,6 +6018,16 @@ inline bool Type::isCompoundType() const {
 inline bool Type::isFunctionType() const {
   return isa<FunctionType>(CanonicalType);
 }
+inline bool Type::isGenericFunctionType() const {
+   if (const FunctionProtoType *FPT = getAs<FunctionProtoType>())
+     return FPT->isGenericFunction();
+   return false;
+}
+inline bool Type::isItypeGenericFunctionType() const {
+   if (const FunctionProtoType *FPT = getAs<FunctionProtoType>())
+     return FPT->isItypeGenericFunction();
+   return false;
+}
 inline bool Type::isPointerType() const {
   return isa<PointerType>(CanonicalType);
 }

include/clang/Basic/DiagnosticParseKinds.td

@@ -1210,12 +1210,6 @@ def err_expected_list_of_types_expr_for_generic_function : Error<
 def err_type_function_comma_or_greater_expected : Error<
   "expected , or >">;
 
-def err_type_list_and_type_variable_num_mismatch : Error<
-  "mismatch between the number of types listed and number of type variables">;
-
-def note_type_variables_declared_at : Note<
-  "type variable(s) declared here">;
-
  // Checked C pragmas
 def err_pragma_checked_scope_invalid_argument : Error<
   "unexpected argument '%0' to '#pragma CHECKED_SCOPE'; "

include/clang/Basic/DiagnosticSemaKinds.td

@@ -9806,6 +9806,30 @@ def err_bounds_type_annotation_lost_checking : Error<
   def no_prototype_generic_function : Error<
     "expected prototype for a generic function">;
 
+  def err_decl_conflicting_generic_non_generic_functions : Error<
+    "conflicting non-generic and generic declarations of %0" >;
+
+  def err_decl_conflicting_type_variable_count : Error<
+    "conflicting numbers of type variables for declarations of %0" >;
+
+  def err_expected_type_argument_list_for_generic_call : Error<
+    "expected a type argument list for a generic function call">;
+
+  def err_expected_type_argument_list_for_itype_generic_call : Error<
+    "expected a type argument list for a bounds-safe interface call in a checked scope">;
+
+  def err_type_list_and_type_variable_num_mismatch : Error<
+    "mismatch between the number of types listed and number of type variables">;
+
+  def note_type_variables_declared_at : Note<
+    "type variable(s) declared here">;
+
+  def err_type_args_for_non_generic_expression : Error<
+    "type arguments supplied for non-generic function or expression">;
+
+  def err_type_args_limited : Error<
+    "type arguments only supported for variables and function names right now">;
+
 } // end of Checked C Category
 
 } // end of sema component.

include/clang/Parse/Parser.h

@@ -996,6 +996,10 @@ class Parser : public CodeCompletionHandler {
   /// point for skipping past a simple-declaration.
   void SkipMalformedDecl();
 
+  // If the current scope is a Checked C _Forany or _Itypeforany scope, exit it.
+  // TODO: this probably doesn't belong in the parser.
+  void ExitQuantifiedTypeScope(DeclSpec &DS);
+
 private:
   //===--------------------------------------------------------------------===//
   // Lexing and parsing of C++ inline methods.
@@ -1749,7 +1753,7 @@ class Parser : public CodeCompletionHandler {
   ExprResult ParseBoundsCastExpression();
 
   ExprResult ParseBoundsExpression();
-  bool ParseItypeAndGenericFunctionExpression(ExprResult &Res);
+  ExprResult ParseGenericTypeArgumentList(ExprResult TypeFunc, SourceLocation Loc);
 
   QualType SubstituteTypeVariable(QualType QT,
     SmallVector<DeclRefExpr::GenericInstInfo::TypeArgument, 4> &typeNames);
@@ -1776,7 +1780,6 @@ class Parser : public CodeCompletionHandler {
 
   ExprResult ParseReturnValueExpression();
 
-
   //===--------------------------------------------------------------------===//
   // clang Expressions
 

include/clang/Sema/Sema.h

@@ -4809,6 +4809,12 @@ class Sema {
 
   BoundsExpr *CheckNonModifyingBounds(BoundsExpr *Bounds, Expr *E);
 
+  ExprResult ActOnTypeApplication(ExprResult TypeFunc, SourceLocation Loc,
+                     ArrayRef<DeclRefExpr::GenericInstInfo::TypeArgument> Args);
+
+  QualType SubstituteTypeArgs(QualType QT,
+                ArrayRef<DeclRefExpr::GenericInstInfo::TypeArgument> TypeArgs);
+
   bool AbstractForFunctionType(BoundsAnnotations &BA,
                                ArrayRef<DeclaratorChunk::ParamInfo> Params);
   /// \brief Take a bounds expression with positional parameters from a function

lib/AST/ASTContext.cpp

@@ -2519,7 +2519,7 @@ QualType ASTContext::getPointerType(QualType T, CheckedPointerKind kind) const {
 QualType ASTContext::getTypeVariableType(unsigned int inDepth, unsigned int inIndex, 
                                           const bool isInBoundsSafeInterface) const {
   llvm::FoldingSetNodeID ID;
-  TypeVariableType::Profile(ID, inDepth, inIndex);
+  TypeVariableType::Profile(ID, inDepth, inIndex, isInBoundsSafeInterface);
 
   void *InsertPos = nullptr;
   if (TypeVariableType *PT = TypeVariableTypes.FindNodeOrInsertPos(ID, InsertPos))
@@ -8075,7 +8075,7 @@ QualType ASTContext::mergeFunctionParameterTypes(QualType lhs, QualType rhs,
                     /*BlockReturnType=*/false, IgnoreBounds);
 }
 
-QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs, 
+QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs,
                                         bool OfBlockPointer,
                                         bool Unqualified,
                                         bool IgnoreBounds) {
@@ -8100,7 +8100,7 @@ QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs,
     retType = mergeTypes(lbase->getReturnType(), rbase->getReturnType(), false,
                          Unqualified,/*BlockReturnType=*/false, IgnoreBounds);
   if (retType.isNull()) return QualType();
-  
+
   if (Unqualified)
     retType = retType.getUnqualifiedType();
 
@@ -8110,7 +8110,7 @@ QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs,
     LRetType = LRetType.getUnqualifiedType();
     RRetType = RRetType.getUnqualifiedType();
   }
-  
+
   if (getCanonicalType(retType) != LRetType)
     allLTypes = false;
   if (getCanonicalType(retType) != RRetType)
@@ -8148,25 +8148,44 @@ QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs,
 
   FunctionType::ExtInfo einfo = lbaseInfo.withNoReturn(NoReturn);
   unsigned NumTypeVars = 0;
+  bool IsITypeGenericFunction = false;
+
   if (lproto && rproto) { // two C99 style function prototypes
     assert(!lproto->hasExceptionSpec() && !rproto->hasExceptionSpec() &&
            "C++ shouldn't be here");
+
     // Compatible functions must have the same number of parameters
     if (lproto->getNumParams() != rproto->getNumParams())
       return QualType();
 
-    // Compatible functions must have the same number of type variables.    
-    if (lproto->getNumTypeVars() != rproto->getNumTypeVars()) {
-      if (lproto->isItypeGenericFunction() && !rproto->isItypeGenericFunction()) {
-        allRTypes = false;
-        NumTypeVars = lproto->getNumTypeVars();
-      } else if (!lproto->isItypeGenericFunction() && rproto->isItypeGenericFunction()) {
-        allLTypes = false;
-        NumTypeVars = rproto->getNumTypeVars();
-      } else {
+    // Compatible functions must be:
+    // 1. Both nongeneric, or
+    // 2. Both generic, or
+    // 3. Both itype generic, or
+    // 4. One is non-generic and the other has an itype.
+    // For cases 1-3, the number of type variables must match.
+
+    if ((lproto->isNonGenericFunction() && rproto->isNonGenericFunction()) ||
+        (lproto->isGenericFunction() && rproto->isGenericFunction()) ||
+        (lproto->isItypeGenericFunction() && rproto->isItypeGenericFunction())) {
+      if (lproto->getNumTypeVars() != rproto->getNumTypeVars())
         return QualType();
+      else  {
+        NumTypeVars = lproto->getNumTypeVars();
+        IsITypeGenericFunction = lproto->isItypeGenericFunction();
       }
-    }
+    } else if (lproto->isItypeGenericFunction() &&
+               rproto->isNonGenericFunction()) {
+      allRTypes = false;
+      NumTypeVars = lproto->getNumTypeVars();
+      IsITypeGenericFunction = true;
+    } else if (lproto->isNonGenericFunction() &&
+            rproto->isItypeGenericFunction()) {
+      allLTypes = false;
+      NumTypeVars = rproto->getNumTypeVars();
+      IsITypeGenericFunction = true;
+    } else
+      return QualType();
 
     // Variadic and non-variadic functions aren't compatible
     if (lproto->isVariadic() != rproto->isVariadic())
@@ -8263,7 +8282,7 @@ QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs,
         }
       }
     }
-      
+
     if (allLTypes) return lhs;
     if (allRTypes) return rhs;
 
@@ -8275,7 +8294,7 @@ QualType ASTContext::mergeFunctionTypes(QualType lhs, QualType rhs,
       EPI.ParamAnnots = bounds.data();
     EPI.ReturnAnnots = ReturnAnnots;
     EPI.numTypeVars = NumTypeVars;
-
+    EPI.ItypeGenericFunction = IsITypeGenericFunction;
     return getFunctionType(retType, types, EPI);
   }
 

lib/AST/ASTDumper.cpp

@@ -1221,9 +1221,11 @@ void ASTDumper::VisitFunctionDecl(const FunctionDecl *D) {
     }
   }
 
-  // If the function is generic function, dump information about type variable.
-  // Type variable is stored as a TypedefDecl.
-  if (D->isGenericFunction() && D->getNumTypeVars() > 0) {
+  // If the function is generic function or an itype generic function, dump
+  // information about type variables. The type variable name is stored as a
+  // TypedefDecl.
+  if ((D->isGenericFunction() || D->isItypeGenericFunction()) &&
+      D->getNumTypeVars() > 0) {
     for (const TypedefDecl* Typevar : D->typeVariables()) {
       dumpChild([=] {
         OS << "TypeVariable";

lib/AST/TypePrinter.cpp

@@ -943,11 +943,10 @@ void TypePrinter::printUnresolvedUsingAfter(const UnresolvedUsingType *T,
 
 void TypePrinter::printTypeVariableBefore(const TypeVariableType *T,
                                              raw_ostream &OS) {
-  OS << "(" << T->GetDepth() << ", " << T->GetIndex();
+  OS << "(" << T->GetDepth() << ", " << T->GetIndex() << ")";
   if (T->IsBoundsInterfaceType()) {
-    OS << " __BoundsInterfaceScope(true)";
+    OS << " __BoundsInterface";
   }
-  OS << ")";
 }
 
 void TypePrinter::printTypeVariableAfter(const TypeVariableType *T, raw_ostream &OS) { }