Report unresolved imports (#593)

* Part 7

* Buildable

* PR feedback

* Merge conflict

* Fix #446

* Fix #446

* Part 8

* Part 9

* Buildable

* Part 10

* Part 11

* Part 12

* Buildable

* Part 14

* First passing test

* Simplify configuration

* Style

* Fix test and move code to folders

* Builtins import

* Fix #470

* Fluents

* Add search path

* Import analysis, part I

* Simplify builtins handling

* Remove IMember

* Handle import specific

* More tests

* Add typeshed

* Renames

* Make sure lazy modules are loaded

* Renames

* Move/rename

* Rework importing

* Derivation rework

* Part 2

* Part 3

* Buildable

* Module members

* Async walk

* Imports test pass

* Remove lazy types

* Fix from import

* Stubs

* Double overloads

* Fix datetime test

* Couple more tests + fluents

* Few more tests

* Additionl test + union type

* Built-in scrape tests

* Full stdlib scrape test

* Complete async AST walker

* Conditional defines test + variable loc cleanup

* More stub tests
Fix stub loading for packages (port from DDG)
Split walker into multiple files

* Add some (broken mostly) tests from DDG

* Move document tests

* Function arg eval, part I

* Instance/factory

* Builds

* Test fixes

* Fix static and instance call eval

* More tests

* More ported tests

* Specialize builtin functions

* Make walkers common and handle nested functions

* Moar tests

* Parser fixes + more tests

* Handle negative numbers

* Fix null ref

* Basic list support

* Few more list tests

* Basic iterators

* Support __iter__

* Iterators

* Fix couple of tests

* Add decorator test

* Generics, part I

* Generics, part 2

* Generics, part 3

* Basic TypeVar test

* Typings, part 4

* Fix test

* Generics, part 6

* Generics, part 7

* More tests (failing)

* Forward ref fixes

* Reorg

* Improve symbol resolution + test fixes

* Test fixes

* Dictionary, part I

* Part 11

* Fix test

* Tests

* Tests

* More dict work

* List ctor

* Skip some tests for now

* Fix iterators

* Tuple slicing

* Polish type comparo in return types

* Add Mapping and tests

* Add Iterable

* Fix typo

* Add Iterator[T] + test

* Simplify typing types

* Class reduction

* Fix tests

* Test fix

* Handle 'with' statement

* Handle try-except

* Class method inheritance + NewType

* Container types

* Containers test

* Tests

* Handle generic type alias

* Named tuple

* Global/non-local

* Handle tuples in for
Handle custom iterators

* Basic generator

* Any/AnyStr

* Test fixes

* Type/Optional/etc handling

* Proper doc population

* Tests + range

* Argument match

* Basic argset and diagnostics

* Argset tests

* Exclude WIP

* Exclude WIP

* Arg eval

* Arg match, part 2

* Tests and generic arg comparisons

* Function eval with arguments

* Baselines

* Fix test

* Undo AST formatting change and update baseline

* LS cleanup 1

* Fix list ctor argument unpacking

* Cleanup 2

* Builds

* Partial completions

* Partial

* Partial

* Simple test

* Tests

* Basic startup

* Clean up a bit

* Remove debug code

* Port formatter tests

* Fix tokenizer crash

* Async fixes

* Hover

* Basic hover

* Adjust expression options

* Basic signature help

* Fix class/instance

* Update test

* Fix builtin creation exception

* Fix tests

* Actually provide declared module

* Completion test (partial)

* Undo

* Fix null await
Fix override completions + test

* Exports filtering
Prevent augmenting imported types

* Filter variables & exports

* Ported tests

* Test fixes

* More ported tests

* Fix exception completions

* Import completions

* Scope completions

* With completions

* Test fixes

* WIP

* Test fix

* Better arg match

* Temp disable WIP

* First cut

* Fix type leak

* WIP

* Remove ConfigureAwait and handle canceled and failed in the analysis notifications

* WIP

* Generic class base

* Generic forward reference resolution

* Suppress completion in strings + test

* Prevent recursion on generic resolution
Better match arguments

* Handle call expression in generics

* Relax condition as it happens in tensorflow

* Fix typeshed version search
Make writing cached modules async
Fix module doc fetching

* Hover tests

* Fix prom import hover

* Hover tests

* Synchronize test cache writing

* First cut

* Test

* Fixes

* Add tests for os.path
Null ref fix

* Fix cache check

* Improve resolution of builtins and typing in stubs

* Merge tests

* Add ntst for requests

* Handle typeshed better

* Fix custom stub handling

* Better sync

* Move files

* Fix parameter locations

* Hover improvement

* PEP hints

* One more test for PEP hints

* Better handle hover over import as

* Text based generic constraints

* Handle with better with generic stubs

* Undo debug

* Handle non-binary open()
Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet.

* Output syntax errors

* Properly clear

* - Fix async issue with analysis completion
- Clean up diagnostics service interface
- Use real DS in tests

* Use proper scope when analyzing module

* Severity mapping and reporting

* Add publishing test
Add NSubstitute
Move all services to the same namespace.

* Unused var

* Test forced publish on close

* Fix typo

* Update test framework

* Import location

* Remove incorrect reference

* Diagnostic severity mapping test
Fix post-mortem earlier PR comment

* Minor fixes

* Move interface to the main class part

* Flicker reduction

* - Correct reported unresolved import name
- Add tests

* PR feedback

Author: Mikhail Arkhipov

src/Analysis/Ast/Impl/Analyzer/Definitions/IExpressionEvaluator.cs

@@ -67,6 +67,8 @@ public interface IExpressionEvaluator {
         IPythonModule Module { get; }
         IPythonInterpreter Interpreter { get; }
         IServiceContainer Services { get; }
+
+        void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry);
         IEnumerable<DiagnosticsEntry> Diagnostics { get; }
     }
 }

src/Analysis/Ast/Impl/Analyzer/DocumentAnalysis.cs

@@ -18,14 +18,11 @@
 using System.IO;
 using System.Linq;
 using Microsoft.Python.Analysis.Analyzer.Evaluation;
-using Microsoft.Python.Analysis.Analyzer.Expressions;
 using Microsoft.Python.Analysis.Diagnostics;
-using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Analysis.Documents;
 using Microsoft.Python.Analysis.Values;
 using Microsoft.Python.Core;
 using Microsoft.Python.Core.Diagnostics;
-using Microsoft.Python.Core.Text;
 using Microsoft.Python.Parsing;
 using Microsoft.Python.Parsing.Ast;
 
@@ -67,6 +64,11 @@ public DocumentAnalysis(IDocument document, int version, IGlobalScope globalScop
         /// Expression evaluator used in the analysis.
         /// </summary>
         public IExpressionEvaluator ExpressionEvaluator { get; }
+
+        /// <summary>
+        /// Analysis diagnostics.
+        /// </summary>
+        public IEnumerable<DiagnosticsEntry> Diagnostics => ExpressionEvaluator.Diagnostics;
         #endregion
     }
 
@@ -88,5 +90,4 @@ public EmptyAnalysis(IServiceContainer services, IDocument document) {
         public IExpressionEvaluator ExpressionEvaluator { get; }
         public IEnumerable<DiagnosticsEntry> Diagnostics => Enumerable.Empty<DiagnosticsEntry>();
     }
-
 }

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs

@@ -33,7 +33,7 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
     /// Helper class that provides methods for looking up variables
     /// and types in a chain of scopes during analysis.
     /// </summary>
-    internal sealed partial class ExpressionEval : IExpressionEvaluator {
+    internal sealed partial class ExpressionEval: IExpressionEvaluator {
         private readonly Stack<Scope> _openScopes = new Stack<Scope>();
         private readonly List<DiagnosticsEntry> _diagnostics = new List<DiagnosticsEntry>();
         private readonly object _lock = new object();
@@ -229,11 +229,11 @@ private async Task<IMember> GetValueFromConditionalAsync(ConditionalExpression e
             return trueValue ?? falseValue;
         }
 
-        private void ReportDiagnostics(Uri documentUri, IEnumerable<DiagnosticsEntry> entries) {
+        public void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry) {
             // Do not add if module is library, etc. Only handle user code.
             if (Module.ModuleType == ModuleType.User) {
                 lock (_lock) {
-                    _diagnostics.AddRange(entries);
+                    _diagnostics.Add(entry);
                 }
             }
         }

src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs

@@ -33,37 +33,36 @@ public async Task<bool> HandleFromImportAsync(FromImportStatement node, Cancella
             }
 
             var rootNames = node.Root.Names;
-            IImportSearchResult imports = null;
             if (rootNames.Count == 1) {
                 var rootName = rootNames[0].Name;
                 if (rootName.EqualsOrdinal("__future__")) {
                     return false;
                 }
             }
 
-            imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node);
-            // If we are processing stub, ignore imports of the original module.
-            // For example, typeshed stub for sys imports sys.
-            if (Module.ModuleType == ModuleType.Stub && imports is ModuleImport mi && mi.Name == Module.Name) {
-                return false;
-            }
-
+            var imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node);
             switch (imports) {
+                case ModuleImport moduleImport when moduleImport.FullName == Module.Name && Module.ModuleType == ModuleType.Stub:
+                    // If we are processing stub, ignore imports of the original module.
+                    // For example, typeshed stub for 'sys' imports sys.
+                    break;
                 case ModuleImport moduleImport when moduleImport.FullName == Module.Name:
                     ImportMembersFromSelf(node);
-                    return false;
+                    break;
                 case ModuleImport moduleImport:
                     await ImportMembersFromModuleAsync(node, moduleImport.FullName, cancellationToken);
-                    return false;
+                    break;
                 case PossibleModuleImport possibleModuleImport:
-                    await HandlePossibleImportAsync(node, possibleModuleImport, cancellationToken);
-                    return false;
+                    await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, Eval.GetLoc(node.Root), cancellationToken);
+                    break;
                 case PackageImport packageImports:
                     await ImportMembersFromPackageAsync(node, packageImports, cancellationToken);
-                    return false;
-                default:
-                    return false;
+                    break;
+                case ImportNotFound notFound:
+                    MakeUnresolvedImport(null, notFound.FullName, Eval.GetLoc(node.Root));
+                    break;
             }
+            return false;
         }
 
         private void ImportMembersFromSelf(FromImportStatement node) {

src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs

@@ -18,13 +18,17 @@
 using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Python.Analysis.Core.DependencyResolution;
+using Microsoft.Python.Analysis.Diagnostics;
 using Microsoft.Python.Analysis.Modules;
 using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Analysis.Values;
+using Microsoft.Python.Core;
+using Microsoft.Python.Parsing;
 using Microsoft.Python.Parsing.Ast;
+using ErrorCodes = Microsoft.Python.Analysis.Diagnostics.ErrorCodes;
 
 namespace Microsoft.Python.Analysis.Analyzer.Handlers {
-    internal sealed partial class ImportHandler: StatementHandler {
+    internal sealed partial class ImportHandler : StatementHandler {
         public ImportHandler(AnalysisWalker walker) : base(walker) { }
 
         public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToken cancellationToken = default) {
@@ -54,14 +58,14 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
                         Eval.DeclareVariable(memberName, Module, VariableSource.Declaration, location);
                         break;
                     case ModuleImport moduleImport:
-                        module = await HandleImportAsync(node, moduleImport, cancellationToken);
+                        module = await HandleImportAsync(moduleImport, location, cancellationToken);
                         break;
                     case PossibleModuleImport possibleModuleImport:
-                        module = await HandlePossibleImportAsync(node, possibleModuleImport, cancellationToken);
+                        module = await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, location, cancellationToken);
                         break;
                     default:
                         // TODO: Package import?
-                        MakeUnresolvedImport(memberName, moduleImportExpression);
+                        MakeUnresolvedImport(memberName, moduleImportExpression.MakeString(), Eval.GetLoc(moduleImportExpression));
                         break;
                 }
 
@@ -72,20 +76,21 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
             return false;
         }
 
-        private async Task<IPythonModule> HandleImportAsync(ImportStatement node, ModuleImport moduleImport, CancellationToken cancellationToken) {
+        private async Task<IPythonModule> HandleImportAsync(ModuleImport moduleImport, LocationInfo location, CancellationToken cancellationToken) {
             var module = await ModuleResolution.ImportModuleAsync(moduleImport.FullName, cancellationToken);
             if (module == null) {
-                MakeUnresolvedImport(moduleImport.FullName, node);
+                MakeUnresolvedImport(moduleImport.FullName, moduleImport.FullName, location);
                 return null;
             }
             return module;
         }
 
-        private async Task<IPythonModule> HandlePossibleImportAsync(Node node, PossibleModuleImport possibleModuleImport, CancellationToken cancellationToken) {
+        private async Task<IPythonModule> HandlePossibleImportAsync(
+            PossibleModuleImport possibleModuleImport, string moduleName, LocationInfo location, CancellationToken cancellationToken) {
             var fullName = possibleModuleImport.PrecedingModuleFullName;
             var module = await ModuleResolution.ImportModuleAsync(possibleModuleImport.PrecedingModuleFullName, cancellationToken);
             if (module == null) {
-                MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, node);
+                MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, moduleName, location);
                 return null;
             }
 
@@ -95,7 +100,7 @@ private async Task<IPythonModule> HandlePossibleImportAsync(Node node, PossibleM
                 var childModule = module.GetMember<IPythonModule>(namePart);
                 if (childModule == null) {
                     var unresolvedModuleName = string.Join(".", nameParts.Take(i + 1).Prepend(fullName));
-                    MakeUnresolvedImport(unresolvedModuleName, node);
+                    MakeUnresolvedImport(unresolvedModuleName, moduleName, location);
                     return null;
                 }
                 module = childModule;
@@ -143,7 +148,12 @@ private void AssignImportedVariables(IPythonModule module, DottedName moduleImpo
             }
         }
 
-        private void MakeUnresolvedImport(string name, Node node)
-            => Eval.DeclareVariable(name, new SentinelModule(name, Eval.Services), VariableSource.Import, Eval.GetLoc(node));
+        private void MakeUnresolvedImport(string variableName, string moduleName, LocationInfo location) {
+            if (!string.IsNullOrEmpty(variableName)) {
+                Eval.DeclareVariable(variableName, new SentinelModule(moduleName, Eval.Services), VariableSource.Import, location);
+            }
+            Eval.ReportDiagnostics(Eval.Module.Uri, new DiagnosticsEntry(
+                Resources.ErrorUnresolvedImport.FormatInvariant(moduleName), location.Span, ErrorCodes.UnresolvedImport, Severity.Warning));
+        }
     }
 }

src/Analysis/Ast/Impl/Definitions/IDocumentAnalysis.cs

@@ -13,7 +13,9 @@
 // See the Apache Version 2.0 License for specific language governing
 // permissions and limitations under the License.
 
+using System.Collections.Generic;
 using Microsoft.Python.Analysis.Analyzer;
+using Microsoft.Python.Analysis.Diagnostics;
 using Microsoft.Python.Analysis.Documents;
 using Microsoft.Python.Analysis.Values;
 using Microsoft.Python.Parsing.Ast;
@@ -49,5 +51,10 @@ public interface IDocumentAnalysis {
         /// Expression evaluator used in the analysis.
         /// </summary>
         IExpressionEvaluator ExpressionEvaluator { get; }
+
+        /// <summary>
+        /// Analysis diagnostics.
+        /// </summary>
+        IEnumerable<DiagnosticsEntry> Diagnostics { get; }
     }
 }

src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs

@@ -0,0 +1,45 @@
+// Copyright(c) Microsoft Corporation
+// All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the License); you may not use
+// this file except in compliance with the License. You may obtain a copy of the
+// License at http://www.apache.org/licenses/LICENSE-2.0
+//
+// THIS CODE IS PROVIDED ON AN  *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS
+// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY
+// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+// MERCHANTABILITY OR NON-INFRINGEMENT.
+//
+// See the Apache Version 2.0 License for specific language governing
+// permissions and limitations under the License.
+
+using System.Collections.Generic;
+using Microsoft.Python.Core;
+using Microsoft.Python.Parsing;
+
+namespace Microsoft.Python.Analysis.Diagnostics {
+    public sealed class DiagnosticsSeverityMap {
+        private readonly Dictionary<string, Severity> _map = new Dictionary<string, Severity>();
+
+        public DiagnosticsSeverityMap() { }
+
+        public DiagnosticsSeverityMap(string[] errors, string[] warnings, string[] information, string[] disabled) {
+            _map.Clear();
+            // disabled > error > warning > information
+            foreach (var x in information.MaybeEnumerate()) {
+                _map[x] = Severity.Information;
+            }
+            foreach (var x in warnings.MaybeEnumerate()) {
+                _map[x] = Severity.Warning;
+            }
+            foreach (var x in errors.MaybeEnumerate()) {
+                _map[x] = Severity.Error;
+            }
+            foreach (var x in disabled.MaybeEnumerate()) {
+                _map[x] = Severity.Suppressed;
+            }
+        }
+        public Severity GetEffectiveSeverity(string code, Severity defaultSeverity)
+            => _map.TryGetValue(code, out var severity) ? severity : defaultSeverity;
+    }
+}

src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs

@@ -21,5 +21,6 @@ public static class ErrorCodes {
         public const string UnknownParameterName = "unknown-parameter-name";
         public const string ParameterAlreadySpecified = "parameter-already-specified";
         public const string ParameterMissing = "parameter-missing";
+        public const string UnresolvedImport = "unresolved-import";
     }
 }

src/Analysis/Ast/Impl/Diagnostics/IDiagnosticsService.cs

@@ -38,5 +38,12 @@ public interface IDiagnosticsService {
         /// the diagnostic publishing to the client.
         /// </summary>
         int PublishingDelay { get; set; }
+
+        /// <summary>
+        /// Provides map of error codes to severity when user wants
+        /// to override default severity settings or suppress particular
+        /// diagnostics completely.
+        /// </summary>
+        DiagnosticsSeverityMap DiagnosticsSeverityMap { get; set; }
     }
 }

src/Analysis/Ast/Impl/Modules/PythonModule.cs

@@ -385,7 +385,7 @@ private void Parse(CancellationToken cancellationToken) {
 
                 // Do not report issues with libraries or stubs
                 if (sink != null) {
-                    _diagnosticsService?.Replace(Uri, _parseErrors);
+                    _diagnosticsService?.Replace(Uri, _parseErrors.Concat(Analysis.Diagnostics));
                 }
 
                 _parsingTask = null;
@@ -460,6 +460,11 @@ public virtual bool NotifyAnalysisComplete(IDocumentAnalysis analysis) {
                     OnAnalysisComplete();
                     ContentState = State.Analyzed;
 
+                    // Do not report issues with libraries or stubs
+                    if (ModuleType == ModuleType.User) {
+                        _diagnosticsService?.Replace(Uri, _parseErrors.Concat(Analysis.Diagnostics));
+                    }
+
                     var tcs = _analysisTcs;
                     _analysisTcs = null;
                     tcs.TrySetResult(analysis);