Merge master into database branch (#1312)

* Remove old qualified name

* Node storage

* Class and scope to use AST map

* Library analysis

* Fix SO

* Keep small AST with imports

* AST reduction

* Final field

* Initial

* Reload

* Ignore post-final requests

* Drop AST

* Remove local variables

* Test fixes

* Fix overload match

* Tests

* Add locks

* Remove local variables

* Drop file content to save memory

* Cache PEP hints

* Recreate AST

* Fix specialization

* Fix locations

* usings

* Test fixes

* Add options to keep data in memory

* Fix test

* Fix lambda parameters

* Fix argument set
Fix global scope node

* Fix overload doc

* Fix stub merge errors

* Fix async issues

* Undo some changes

* Fix test

* Fix race condition

* Partial

* Models and views

* Restore log null checks

* Fix merge conflict

* Fix merge issue

* Null check

* Partial

* Partial

* Partial

* Fix test

* Partial

* Partial

* First test

* Baseline comparison

* Builtins

* Partial

* Type fixes

* Fix type names, part I

* Qualified name

* Properly write variables

* Partial

* Construct module from model

* Test

* Variable creations

* Factories

* Factories

* Split construction

* Restore

* Save builtins

* Test passes

* Qualified name

* Better export detection

* Test fixes

* More consistent qualified names

* Sys test

* Demo

* Complete sys write/read

* Partial

* Partial

* Test staility

* Perf bug

* Baseline, remove debug code, deactivate db

* Test fixes

* Test fix

* Simplify a bit

* Baselines and use : separator

* Baselines

* PR feedback

* Merge master

* Remove registry reference

* PR feedback

* Avoid empty node in empty global scope (#1226)

* Add null checks

* Avoid null node in empty global scope

* Remove unrelated change

* Lock content since it is locked in other cases (#1227)

* Fix addBrackets setting (#1241)

* Fix addBrackets

* Remove unrelated change

* Usings

* Handle library analysis better (#1246)

* Trying to augment argument set with Evaluator and Expression context (#1259)

* Moving anyStr to its own method to prevent FromTypeVar from being called with an ArgumentSet that has no context

* Trying to augment argument set with context wherever possible

* Setting expression in argument set, removing null

* Renaming to WithoutContext and updating comment

* Check for null args in PythonFunctionOverload

* When returning in init method, report a diagnostic error (#1261)

* When returning in init method, report a diagnostic error

* Adding check and tests for returning None before adding error for a return statement in the init function

* Cleaning up tests

* Give Diagnostic messages on improper usage of Generic (#1248)

* Adding diagnostic message for improper use Generic.

* Adding tests to make sure no diagnostics on valid uses of Generic

* Adding diagnostic error on binary operations with incompatible types (#1254)

* Adding diagnostic error for binary operations with incompatible types - e.g 5 + 'str'

* When NewType is called and the first argument is not a string, make a diagnostic message (#1260)

* Adding diagnostic error if the user calls NewType with the first arg not of string type

* Fix typo in TROUBLESHOOTING.md (#1285)

* Multiple analysis fixes (#1297)

Multiple analysis fixes.
- Fixes #1151: Reliable way to determine final analysis of a module
- Fixes #1201: NRE in SymbolCollector.AddProperty
- Fixes #1228: NRE in TryFindMissingDependencies
- Fixes #1294: Handle valid case for reloading AST
- Fixes #1295: FindReferences doesn't work on package init members
- Fixes #1296: DependencyResolver graph misses intermediate imports
- Part of the fix for #1174: All references are not listed when using ms language server (fixes in analysis itself are also required for this one)

* Rework search path resolution (#1289)

* first working path classfier, breaks existing path code for now

* inherit Comparer instead of implementing both

* don't do path depth comparison, preserve user ordering as before

* switch to using new path classification code in main module resolution

* fix typo

* clean up comment about paths in Server.cs

* use String.Split instead of regex when parsing script output

* test ordering of user provided paths

* add new normalize and trim helper, check preconditions as debug asserts rather than commenting on them

* use Split extension, don't MaybeEnumerate paths

* No diagnostic message when Generic is called with no args (#1305)

* No diagnostic message when Generic is called with no args

* When a class inherits from something that is not a class, give a diagnostic error (#1277)

* When analyzing class definitions, give diagnostic if bases are not all class types

* PR feedback

Author: Mikhail Arkhipov

TROUBLESHOOTING.md

@@ -21,7 +21,7 @@ Python is a requirement for the language server to run. In VS Code, an interpret
 _must_ be selected in order for the language server to properly initialize. If your
 language server fails to start, be sure that you have selected an interpreter.
 
-The language server can only run on platforms where the .NET Core can run. This rougly means:
+The language server can only run on platforms where the .NET Core can run. This roughly means:
 
 - Windows, 32/64 bit
 - macOS, 64 bit

src/Analysis/Ast/Impl/Analyzer/ActivityTracker.cs

@@ -19,48 +19,21 @@
 
 namespace Microsoft.Python.Analysis.Analyzer {
     internal static class ActivityTracker {
-        private static readonly Dictionary<string, AnalysisState> _modules = new Dictionary<string, AnalysisState>();
+        private static readonly HashSet<string> _modules = new HashSet<string>();
         private static readonly object _lock = new object();
         private static bool _tracking;
         private static Stopwatch _sw;
 
-        private struct AnalysisState {
-            public int Count;
-            public bool IsComplete;
-        }
-
         public static void OnEnqueueModule(string path) {
             if (string.IsNullOrEmpty(path)) {
                 return;
             }
 
             lock (_lock) {
-                if (!_modules.TryGetValue(path, out var st)) {
-                    _modules[path] = default;
-                } else {
-                    st.IsComplete = false;
-                }
-            }
-        }
-
-        public static void OnModuleAnalysisComplete(string path) {
-            lock (_lock) {
-                if (_modules.TryGetValue(path, out var st)) {
-                    st.Count++;
-                    st.IsComplete = true;
-                }
+                _modules.Add(path);
             }
         }
 
-        public static bool IsAnalysisComplete {
-            get {
-                lock (_lock) {
-                    return _modules.All(m => m.Value.IsComplete);
-                }
-            }
-        }
-
-
         public static void StartTracking() {
             lock (_lock) {
                 if (!_tracking) {
@@ -71,22 +44,15 @@ public static void StartTracking() {
             }
         }
 
-        public static void EndTracking() {
+        public static (int modulesCount, double totalMilliseconds) EndTracking() {
             lock (_lock) {
                 if (_tracking) {
                     _sw?.Stop();
                     _tracking = false;
                 }
-            }
-        }
 
-        public static int ModuleCount {
-            get {
-                lock (_lock) {
-                    return _modules.Count;
-                }
+                return (_modules.Count, _sw?.Elapsed.TotalMilliseconds ?? 0);
             }
         }
-        public static double MillisecondsElapsed => _sw?.Elapsed.TotalMilliseconds ?? 0;
     }
 }

src/Analysis/Ast/Impl/Analyzer/AnalysisModuleKey.cs

@@ -15,39 +15,56 @@
 
 using System;
 using System.Diagnostics;
+using Microsoft.Python.Analysis.Documents;
 using Microsoft.Python.Analysis.Modules;
 using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Core;
 
 namespace Microsoft.Python.Analysis.Analyzer {
     [DebuggerDisplay("{Name} : {FilePath}")]
-    internal struct AnalysisModuleKey : IEquatable<AnalysisModuleKey> {
+    internal readonly struct AnalysisModuleKey : IEquatable<AnalysisModuleKey> {
+        private enum KeyType { Default, Typeshed, LibraryAsDocument }
+
+        private readonly KeyType _type;
         public string Name { get; }
         public string FilePath { get; }
-        public bool IsTypeshed { get; }
+        public bool IsTypeshed => _type == KeyType.Typeshed;
+        public bool IsLibraryAsDocument => _type == KeyType.LibraryAsDocument;
 
         public AnalysisModuleKey(IPythonModule module) {
             Name = module.Name;
             FilePath = module.ModuleType == ModuleType.CompiledBuiltin ? null : module.FilePath;
-            IsTypeshed = module is StubPythonModule stub && stub.IsTypeshed;
+            _type = module is StubPythonModule stub && stub.IsTypeshed
+                ? KeyType.Typeshed
+                : module.ModuleType == ModuleType.Library && module is IDocument document && document.IsOpen
+                    ? KeyType.LibraryAsDocument
+                    : KeyType.Default;
         }
 
         public AnalysisModuleKey(string name, string filePath, bool isTypeshed) {
             Name = name;
             FilePath = filePath;
-            IsTypeshed = isTypeshed;
+            _type = isTypeshed ? KeyType.Typeshed : KeyType.Default;
+        }
+
+        private AnalysisModuleKey(string name, string filePath, KeyType type) {
+            Name = name;
+            FilePath = filePath;
+            _type = type;
         }
 
+        public AnalysisModuleKey GetLibraryAsDocumentKey() => new AnalysisModuleKey(Name, FilePath, KeyType.LibraryAsDocument);
+
         public bool Equals(AnalysisModuleKey other)
-            => Name.EqualsOrdinal(other.Name) && FilePath.PathEquals(other.FilePath) && IsTypeshed == other.IsTypeshed;
+            => Name.EqualsOrdinal(other.Name) && FilePath.PathEquals(other.FilePath) && _type == other._type;
 
         public override bool Equals(object obj) => obj is AnalysisModuleKey other && Equals(other);
 
         public override int GetHashCode() {
             unchecked {
                 var hashCode = (Name != null ? Name.GetHashCode() : 0);
                 hashCode = (hashCode * 397) ^ (FilePath != null ? FilePath.GetPathHashCode() : 0);
-                hashCode = (hashCode * 397) ^ IsTypeshed.GetHashCode();
+                hashCode = (hashCode * 397) ^ _type.GetHashCode();
                 return hashCode;
             }
         }

src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs

@@ -26,6 +26,7 @@ internal interface IAnalyzable {
         /// <summary>
         /// Notifies document that its analysis is now complete.
         /// </summary>
-        void NotifyAnalysisComplete(int version, ModuleWalker walker, bool isFinalPass);
+        /// <param name="analysis">Document analysis</param>
+        void NotifyAnalysisComplete(IDocumentAnalysis analysis);
     }
 }

src/Analysis/Ast/Impl/Analyzer/Definitions/IPythonAnalyzer.cs

@@ -20,14 +20,15 @@
 using Microsoft.Python.Analysis.Diagnostics;
 using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Core.Collections;
+using Microsoft.Python.Parsing.Ast;
 
 namespace Microsoft.Python.Analysis.Analyzer {
     public interface IPythonAnalyzer {
         Task WaitForCompleteAnalysisAsync(CancellationToken cancellationToken = default);
         /// <summary>
         /// Schedules module for analysis. Module will be scheduled if version of AST is greater than the one used to get previous analysis
         /// </summary>
-        void EnqueueDocumentForAnalysis(IPythonModule module, int version);
+        void EnqueueDocumentForAnalysis(IPythonModule module, PythonAst ast, int version);
 
         /// <summary>
         /// Schedules module for analysis for its existing AST, but with new dependencies.

src/Analysis/Ast/Impl/Analyzer/EmptyAnalysis.cs

@@ -30,7 +30,7 @@ public EmptyAnalysis(IServiceContainer services, IDocument document) {
             Document = document ?? throw new ArgumentNullException(nameof(document));
             GlobalScope = new EmptyGlobalScope(document);
             Ast = AstUtilities.MakeEmptyAst(document.Uri);
-            ExpressionEvaluator = new ExpressionEval(services, document);
+            ExpressionEvaluator = new ExpressionEval(services, document, Ast);
         }
 
         public IDocument Document { get; }

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs

@@ -89,11 +89,11 @@ public IMember GetValueFromLambda(LambdaExpression expr) {
         public IMember GetValueFromClassCtor(IPythonClassType cls, CallExpression expr) {
             SymbolTable.Evaluate(cls.ClassDefinition);
             // Determine argument types
-            var args = ArgumentSet.Empty;
+            var args = ArgumentSet.Empty(expr, this);
             var init = cls.GetMember<IPythonFunctionType>(@"__init__");
             if (init != null) {
                 using (OpenScope(cls.DeclaringModule, cls.ClassDefinition, out _)) {
-                    var a = new ArgumentSet(init, 0, new PythonInstance(cls), expr, Module, this);
+                    var a = new ArgumentSet(init, 0, new PythonInstance(cls), expr, this);
                     if (a.Errors.Count > 0) {
                         // AddDiagnostics(Module.Uri, a.Errors);
                     }
@@ -108,7 +108,7 @@ private IMember GetValueFromBound(IPythonBoundType t, CallExpression expr) {
                 case IPythonFunctionType fn:
                     return GetValueFromFunctionType(fn, t.Self, expr);
                 case IPythonPropertyType p:
-                    return GetValueFromProperty(p, t.Self);
+                    return GetValueFromProperty(p, t.Self, expr);
                 case IPythonIteratorType _ when t.Self is IPythonCollection seq:
                     return seq.GetIterator();
             }
@@ -201,10 +201,10 @@ public IMember GetValueFromFunctionType(IPythonFunctionType fn, IPythonInstance
             return UnknownType;
         }
 
-        private IMember GetValueFromProperty(IPythonPropertyType p, IPythonInstance instance) {
+        private IMember GetValueFromProperty(IPythonPropertyType p, IPythonInstance instance, CallExpression expr) {
             // Function may not have been walked yet. Do it now.
             SymbolTable.Evaluate(p.FunctionDefinition);
-            return instance.Call(p.Name, ArgumentSet.Empty);
+            return instance.Call(p.Name, ArgumentSet.Empty(expr, this));
         }
 
 

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs

@@ -16,12 +16,15 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using Microsoft.Python.Analysis.Diagnostics;
 using Microsoft.Python.Analysis.Specializations.Typing;
 using Microsoft.Python.Analysis.Specializations.Typing.Types;
 using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Analysis.Values;
 using Microsoft.Python.Core;
+using Microsoft.Python.Parsing;
 using Microsoft.Python.Parsing.Ast;
+using ErrorCodes = Microsoft.Python.Analysis.Diagnostics.ErrorCodes;
 
 namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
     internal sealed partial class ExpressionEval {
@@ -59,40 +62,58 @@ private IMember GetValueFromGeneric(IMember target, Expression expr) {
         }
 
         /// <summary>
-        /// Given generic type and list of indices in the expression like
-        /// Generic[T1, T2, ...] or List[str] creates generic class base
-        /// (if the former) on specific type (if the latter).
+        /// Returns whether the arguments to Generic are valid
         /// </summary>
-        private IMember CreateSpecificTypeFromIndex(IGenericType gt, IReadOnlyList<IMember> indices, Expression expr) {
-            // See which ones are generic parameters as defined by TypeVar() 
-            // and which are specific types. Normally there should not be a mix.
-            var genericTypeArgs = indices.OfType<IGenericTypeDefinition>().ToArray();
-            var specificTypes = indices.Where(i => !(i is IGenericTypeDefinition)).OfType<IPythonType>().ToArray();
-
-            if (genericTypeArgs.Length > 0 && genericTypeArgs.Length != indices.Count) {
-                // TODO: report that some type arguments are not declared with TypeVar.
+        private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeDefinition> genericTypeArgs, IReadOnlyList<IMember> args, Expression expr) {
+            // All arguments to Generic must be type parameters
+            // e.g. Generic[T, str] throws a runtime error
+            if (genericTypeArgs.Count != args.Count) {
+                ReportDiagnostics(Module.Uri, new DiagnosticsEntry(
+                    Resources.GenericNotAllTypeParameters,
+                    GetLocation(expr).Span,
+                    ErrorCodes.TypingGenericArguments,
+                    Severity.Error,
+                    DiagnosticSource.Analysis));
+                return false;
             }
-            if (specificTypes.Length > 0 && specificTypes.Length != indices.Count) {
-                // TODO: report that arguments are not specific types or are not declared.
+
+            // All arguments to Generic must be distinct
+            if (genericTypeArgs.Distinct().Count() != genericTypeArgs.Count) {
+                ReportDiagnostics(Module.Uri, new DiagnosticsEntry(
+                   Resources.GenericNotAllUnique,
+                   GetLocation(expr).Span,
+                   ErrorCodes.TypingGenericArguments,
+                   Severity.Error,
+                   DiagnosticSource.Analysis));
+                return false;
             }
 
+            return true;
+        }
+
+        /// <summary>
+        /// Given generic type and list of arguments in the expression like
+        /// Generic[T1, T2, ...] or List[str] creates generic class base
+        /// (if the former) on specific type (if the latter).
+        /// </summary>
+        private IMember CreateSpecificTypeFromIndex(IGenericType gt, IReadOnlyList<IMember> args, Expression expr) {
+            var genericTypeArgs = args.OfType<IGenericTypeDefinition>().ToArray();
+
             if (gt.Name.EqualsOrdinal("Generic")) {
-                // Generic[T1, T2, ...] expression. Create generic base for the class.
-                if (genericTypeArgs.Length > 0) {
-                    return new GenericClassParameter(genericTypeArgs, Module);
-                } else {
-                    // TODO: report too few type arguments for Generic[].
+                if (!GenericClassParameterValid(genericTypeArgs, args, expr)) {
                     return UnknownType;
                 }
+
+                // Generic[T1, T2, ...] expression. Create generic base for the class.
+                return new GenericClassParameter(genericTypeArgs, Module);
             }
 
             // For other types just use supplied arguments
-            if (indices.Count > 0) {
-                return gt.CreateSpecificType(new ArgumentSet(indices));
+            if (args.Count > 0) {
+                return gt.CreateSpecificType(new ArgumentSet(args, expr, this));
             }
-            // TODO: report too few type arguments for the generic expression.
-            return UnknownType;
 
+            return UnknownType;
         }
 
         private IReadOnlyList<IMember> EvaluateIndex(IndexExpression expr) {
@@ -104,7 +125,10 @@ private IReadOnlyList<IMember> EvaluateIndex(IndexExpression expr) {
                 }
             } else {
                 var index = GetValueFromExpression(expr.Index);
-                indices.Add(index ?? UnknownType);
+                // Don't count null indexes as arguments
+                if (index != null) {
+                    indices.Add(index);
+                }
             }
             return indices;
         }
@@ -132,7 +156,7 @@ private IMember CreateClassInstance(PythonClassType cls, IReadOnlyList<IMember>
 
             var argSet = initOverload != null
                     ? new ArgumentSet(initFunc, 0, null, callExpr, this)
-                    : new ArgumentSet(constructorArguments);
+                    : new ArgumentSet(constructorArguments, callExpr, this);
 
             argSet.Evaluate();
             var specificType = cls.CreateSpecificType(argSet);
@@ -194,7 +218,7 @@ private static IReadOnlyList<IPythonType> GetSpecificTypeFromArgumentValue(objec
                     var itemType = iter.GetIterator().Next.GetPythonType();
                     if (!itemType.IsUnknown()) {
                         specificTypes.Add(itemType);
-                    } else if(argumentValue is IPythonInstance inst) {
+                    } else if (argumentValue is IPythonInstance inst) {
                         specificTypes.Add(inst.GetPythonType());
                     }
                     break;