Skip to content
This repository was archived by the owner on Apr 14, 2022. It is now read-only.

Commit 037d248

Browse files
author
Mikhail Arkhipov
authored
Report unresolved imports (#593)
* Part 7 * Buildable * PR feedback * Merge conflict * Fix #446 * Fix #446 * Part 8 * Part 9 * Buildable * Part 10 * Part 11 * Part 12 * Buildable * Part 14 * First passing test * Simplify configuration * Style * Fix test and move code to folders * Builtins import * Fix #470 * Fluents * Add search path * Import analysis, part I * Simplify builtins handling * Remove IMember * Handle import specific * More tests * Add typeshed * Renames * Make sure lazy modules are loaded * Renames * Move/rename * Rework importing * Derivation rework * Part 2 * Part 3 * Buildable * Module members * Async walk * Imports test pass * Remove lazy types * Fix from import * Stubs * Double overloads * Fix datetime test * Couple more tests + fluents * Few more tests * Additionl test + union type * Built-in scrape tests * Full stdlib scrape test * Complete async AST walker * Conditional defines test + variable loc cleanup * More stub tests Fix stub loading for packages (port from DDG) Split walker into multiple files * Add some (broken mostly) tests from DDG * Move document tests * Function arg eval, part I * Instance/factory * Builds * Test fixes * Fix static and instance call eval * More tests * More ported tests * Specialize builtin functions * Make walkers common and handle nested functions * Moar tests * Parser fixes + more tests * Handle negative numbers * Fix null ref * Basic list support * Few more list tests * Basic iterators * Support __iter__ * Iterators * Fix couple of tests * Add decorator test * Generics, part I * Generics, part 2 * Generics, part 3 * Basic TypeVar test * Typings, part 4 * Fix test * Generics, part 6 * Generics, part 7 * More tests (failing) * Forward ref fixes * Reorg * Improve symbol resolution + test fixes * Test fixes * Dictionary, part I * Part 11 * Fix test * Tests * Tests * More dict work * List ctor * Skip some tests for now * Fix iterators * Tuple slicing * Polish type comparo in return types * Add Mapping and tests * Add Iterable * Fix typo * Add Iterator[T] + test * Simplify typing types * Class reduction * Fix tests * Test fix * Handle 'with' statement * Handle try-except * Class method inheritance + NewType * Container types * Containers test * Tests * Handle generic type alias * Named tuple * Global/non-local * Handle tuples in for Handle custom iterators * Basic generator * Any/AnyStr * Test fixes * Type/Optional/etc handling * Proper doc population * Tests + range * Argument match * Basic argset and diagnostics * Argset tests * Exclude WIP * Exclude WIP * Arg eval * Arg match, part 2 * Tests and generic arg comparisons * Function eval with arguments * Baselines * Fix test * Undo AST formatting change and update baseline * LS cleanup 1 * Fix list ctor argument unpacking * Cleanup 2 * Builds * Partial completions * Partial * Partial * Simple test * Tests * Basic startup * Clean up a bit * Remove debug code * Port formatter tests * Fix tokenizer crash * Async fixes * Hover * Basic hover * Adjust expression options * Basic signature help * Fix class/instance * Update test * Fix builtin creation exception * Fix tests * Actually provide declared module * Completion test (partial) * Undo * Fix null await Fix override completions + test * Exports filtering Prevent augmenting imported types * Filter variables & exports * Ported tests * Test fixes * More ported tests * Fix exception completions * Import completions * Scope completions * With completions * Test fixes * WIP * Test fix * Better arg match * Temp disable WIP * First cut * Fix type leak * WIP * Remove ConfigureAwait and handle canceled and failed in the analysis notifications * WIP * Generic class base * Generic forward reference resolution * Suppress completion in strings + test * Prevent recursion on generic resolution Better match arguments * Handle call expression in generics * Relax condition as it happens in tensorflow * Fix typeshed version search Make writing cached modules async Fix module doc fetching * Hover tests * Fix prom import hover * Hover tests * Synchronize test cache writing * First cut * Test * Fixes * Add tests for os.path Null ref fix * Fix cache check * Improve resolution of builtins and typing in stubs * Merge tests * Add ntst for requests * Handle typeshed better * Fix custom stub handling * Better sync * Move files * Fix parameter locations * Hover improvement * PEP hints * One more test for PEP hints * Better handle hover over import as * Text based generic constraints * Handle with better with generic stubs * Undo debug * Handle non-binary open() Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet. * Output syntax errors * Properly clear * - Fix async issue with analysis completion - Clean up diagnostics service interface - Use real DS in tests * Use proper scope when analyzing module * Severity mapping and reporting * Add publishing test Add NSubstitute Move all services to the same namespace. * Unused var * Test forced publish on close * Fix typo * Update test framework * Import location * Remove incorrect reference * Diagnostic severity mapping test Fix post-mortem earlier PR comment * Minor fixes * Move interface to the main class part * Flicker reduction * - Correct reported unresolved import name - Add tests * PR feedback
1 parent 87c46f9 commit 037d248

22 files changed

+311
-91
lines changed

‎src/Analysis/Ast/Impl/Analyzer/Definitions/IExpressionEvaluator.cs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,8 @@ public interface IExpressionEvaluator {
6767
IPythonModule Module { get; }
6868
IPythonInterpreter Interpreter { get; }
6969
IServiceContainer Services { get; }
70+
71+
void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry);
7072
IEnumerable<DiagnosticsEntry> Diagnostics { get; }
7173
}
7274
}

‎src/Analysis/Ast/Impl/Analyzer/DocumentAnalysis.cs

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,14 +18,11 @@
1818
using System.IO;
1919
using System.Linq;
2020
using Microsoft.Python.Analysis.Analyzer.Evaluation;
21-
using Microsoft.Python.Analysis.Analyzer.Expressions;
2221
using Microsoft.Python.Analysis.Diagnostics;
23-
using Microsoft.Python.Analysis.Types;
2422
using Microsoft.Python.Analysis.Documents;
2523
using Microsoft.Python.Analysis.Values;
2624
using Microsoft.Python.Core;
2725
using Microsoft.Python.Core.Diagnostics;
28-
using Microsoft.Python.Core.Text;
2926
using Microsoft.Python.Parsing;
3027
using Microsoft.Python.Parsing.Ast;
3128

@@ -67,6 +64,11 @@ public DocumentAnalysis(IDocument document, int version, IGlobalScope globalScop
6764
/// Expression evaluator used in the analysis.
6865
/// </summary>
6966
public IExpressionEvaluator ExpressionEvaluator { get; }
67+
68+
/// <summary>
69+
/// Analysis diagnostics.
70+
/// </summary>
71+
public IEnumerable<DiagnosticsEntry> Diagnostics => ExpressionEvaluator.Diagnostics;
7072
#endregion
7173
}
7274

@@ -88,5 +90,4 @@ public EmptyAnalysis(IServiceContainer services, IDocument document) {
8890
public IExpressionEvaluator ExpressionEvaluator { get; }
8991
public IEnumerable<DiagnosticsEntry> Diagnostics => Enumerable.Empty<DiagnosticsEntry>();
9092
}
91-
9293
}

‎src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
3333
/// Helper class that provides methods for looking up variables
3434
/// and types in a chain of scopes during analysis.
3535
/// </summary>
36-
internal sealed partial class ExpressionEval : IExpressionEvaluator {
36+
internal sealed partial class ExpressionEval: IExpressionEvaluator {
3737
private readonly Stack<Scope> _openScopes = new Stack<Scope>();
3838
private readonly List<DiagnosticsEntry> _diagnostics = new List<DiagnosticsEntry>();
3939
private readonly object _lock = new object();
@@ -229,11 +229,11 @@ private async Task<IMember> GetValueFromConditionalAsync(ConditionalExpression e
229229
return trueValue ?? falseValue;
230230
}
231231

232-
private void ReportDiagnostics(Uri documentUri, IEnumerable<DiagnosticsEntry> entries) {
232+
public void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry) {
233233
// Do not add if module is library, etc. Only handle user code.
234234
if (Module.ModuleType == ModuleType.User) {
235235
lock (_lock) {
236-
_diagnostics.AddRange(entries);
236+
_diagnostics.Add(entry);
237237
}
238238
}
239239
}

‎src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -33,37 +33,36 @@ public async Task<bool> HandleFromImportAsync(FromImportStatement node, Cancella
3333
}
3434

3535
var rootNames = node.Root.Names;
36-
IImportSearchResult imports = null;
3736
if (rootNames.Count == 1) {
3837
var rootName = rootNames[0].Name;
3938
if (rootName.EqualsOrdinal("__future__")) {
4039
return false;
4140
}
4241
}
4342

44-
imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node);
45-
// If we are processing stub, ignore imports of the original module.
46-
// For example, typeshed stub for sys imports sys.
47-
if (Module.ModuleType == ModuleType.Stub && imports is ModuleImport mi && mi.Name == Module.Name) {
48-
return false;
49-
}
50-
43+
var imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node);
5144
switch (imports) {
45+
case ModuleImport moduleImport when moduleImport.FullName == Module.Name && Module.ModuleType == ModuleType.Stub:
46+
// If we are processing stub, ignore imports of the original module.
47+
// For example, typeshed stub for 'sys' imports sys.
48+
break;
5249
case ModuleImport moduleImport when moduleImport.FullName == Module.Name:
5350
ImportMembersFromSelf(node);
54-
return false;
51+
break;
5552
case ModuleImport moduleImport:
5653
await ImportMembersFromModuleAsync(node, moduleImport.FullName, cancellationToken);
57-
return false;
54+
break;
5855
case PossibleModuleImport possibleModuleImport:
59-
await HandlePossibleImportAsync(node, possibleModuleImport, cancellationToken);
60-
return false;
56+
await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, Eval.GetLoc(node.Root), cancellationToken);
57+
break;
6158
case PackageImport packageImports:
6259
await ImportMembersFromPackageAsync(node, packageImports, cancellationToken);
63-
return false;
64-
default:
65-
return false;
60+
break;
61+
case ImportNotFound notFound:
62+
MakeUnresolvedImport(null, notFound.FullName, Eval.GetLoc(node.Root));
63+
break;
6664
}
65+
return false;
6766
}
6867

6968
private void ImportMembersFromSelf(FromImportStatement node) {

‎src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs

Lines changed: 21 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,17 @@
1818
using System.Threading;
1919
using System.Threading.Tasks;
2020
using Microsoft.Python.Analysis.Core.DependencyResolution;
21+
using Microsoft.Python.Analysis.Diagnostics;
2122
using Microsoft.Python.Analysis.Modules;
2223
using Microsoft.Python.Analysis.Types;
2324
using Microsoft.Python.Analysis.Values;
25+
using Microsoft.Python.Core;
26+
using Microsoft.Python.Parsing;
2427
using Microsoft.Python.Parsing.Ast;
28+
using ErrorCodes = Microsoft.Python.Analysis.Diagnostics.ErrorCodes;
2529

2630
namespace Microsoft.Python.Analysis.Analyzer.Handlers {
27-
internal sealed partial class ImportHandler: StatementHandler {
31+
internal sealed partial class ImportHandler : StatementHandler {
2832
public ImportHandler(AnalysisWalker walker) : base(walker) { }
2933

3034
public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToken cancellationToken = default) {
@@ -54,14 +58,14 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
5458
Eval.DeclareVariable(memberName, Module, VariableSource.Declaration, location);
5559
break;
5660
case ModuleImport moduleImport:
57-
module = await HandleImportAsync(node, moduleImport, cancellationToken);
61+
module = await HandleImportAsync(moduleImport, location, cancellationToken);
5862
break;
5963
case PossibleModuleImport possibleModuleImport:
60-
module = await HandlePossibleImportAsync(node, possibleModuleImport, cancellationToken);
64+
module = await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, location, cancellationToken);
6165
break;
6266
default:
6367
// TODO: Package import?
64-
MakeUnresolvedImport(memberName, moduleImportExpression);
68+
MakeUnresolvedImport(memberName, moduleImportExpression.MakeString(), Eval.GetLoc(moduleImportExpression));
6569
break;
6670
}
6771

@@ -72,20 +76,21 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
7276
return false;
7377
}
7478

75-
private async Task<IPythonModule> HandleImportAsync(ImportStatement node, ModuleImport moduleImport, CancellationToken cancellationToken) {
79+
private async Task<IPythonModule> HandleImportAsync(ModuleImport moduleImport, LocationInfo location, CancellationToken cancellationToken) {
7680
var module = await ModuleResolution.ImportModuleAsync(moduleImport.FullName, cancellationToken);
7781
if (module == null) {
78-
MakeUnresolvedImport(moduleImport.FullName, node);
82+
MakeUnresolvedImport(moduleImport.FullName, moduleImport.FullName, location);
7983
return null;
8084
}
8185
return module;
8286
}
8387

84-
private async Task<IPythonModule> HandlePossibleImportAsync(Node node, PossibleModuleImport possibleModuleImport, CancellationToken cancellationToken) {
88+
private async Task<IPythonModule> HandlePossibleImportAsync(
89+
PossibleModuleImport possibleModuleImport, string moduleName, LocationInfo location, CancellationToken cancellationToken) {
8590
var fullName = possibleModuleImport.PrecedingModuleFullName;
8691
var module = await ModuleResolution.ImportModuleAsync(possibleModuleImport.PrecedingModuleFullName, cancellationToken);
8792
if (module == null) {
88-
MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, node);
93+
MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, moduleName, location);
8994
return null;
9095
}
9196

@@ -95,7 +100,7 @@ private async Task<IPythonModule> HandlePossibleImportAsync(Node node, PossibleM
95100
var childModule = module.GetMember<IPythonModule>(namePart);
96101
if (childModule == null) {
97102
var unresolvedModuleName = string.Join(".", nameParts.Take(i + 1).Prepend(fullName));
98-
MakeUnresolvedImport(unresolvedModuleName, node);
103+
MakeUnresolvedImport(unresolvedModuleName, moduleName, location);
99104
return null;
100105
}
101106
module = childModule;
@@ -143,7 +148,12 @@ private void AssignImportedVariables(IPythonModule module, DottedName moduleImpo
143148
}
144149
}
145150

146-
private void MakeUnresolvedImport(string name, Node node)
147-
=> Eval.DeclareVariable(name, new SentinelModule(name, Eval.Services), VariableSource.Import, Eval.GetLoc(node));
151+
private void MakeUnresolvedImport(string variableName, string moduleName, LocationInfo location) {
152+
if (!string.IsNullOrEmpty(variableName)) {
153+
Eval.DeclareVariable(variableName, new SentinelModule(moduleName, Eval.Services), VariableSource.Import, location);
154+
}
155+
Eval.ReportDiagnostics(Eval.Module.Uri, new DiagnosticsEntry(
156+
Resources.ErrorUnresolvedImport.FormatInvariant(moduleName), location.Span, ErrorCodes.UnresolvedImport, Severity.Warning));
157+
}
148158
}
149159
}

‎src/Analysis/Ast/Impl/Definitions/IDocumentAnalysis.cs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@
1313
// See the Apache Version 2.0 License for specific language governing
1414
// permissions and limitations under the License.
1515

16+
using System.Collections.Generic;
1617
using Microsoft.Python.Analysis.Analyzer;
18+
using Microsoft.Python.Analysis.Diagnostics;
1719
using Microsoft.Python.Analysis.Documents;
1820
using Microsoft.Python.Analysis.Values;
1921
using Microsoft.Python.Parsing.Ast;
@@ -49,5 +51,10 @@ public interface IDocumentAnalysis {
4951
/// Expression evaluator used in the analysis.
5052
/// </summary>
5153
IExpressionEvaluator ExpressionEvaluator { get; }
54+
55+
/// <summary>
56+
/// Analysis diagnostics.
57+
/// </summary>
58+
IEnumerable<DiagnosticsEntry> Diagnostics { get; }
5259
}
5360
}

‎src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
// Copyright(c) Microsoft Corporation
2+
// All rights reserved.
3+
//
4+
// Licensed under the Apache License, Version 2.0 (the License); you may not use
5+
// this file except in compliance with the License. You may obtain a copy of the
6+
// License at http://www.apache.org/licenses/LICENSE-2.0
7+
//
8+
// THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS
9+
// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY
10+
// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
11+
// MERCHANTABILITY OR NON-INFRINGEMENT.
12+
//
13+
// See the Apache Version 2.0 License for specific language governing
14+
// permissions and limitations under the License.
15+
16+
using System.Collections.Generic;
17+
using Microsoft.Python.Core;
18+
using Microsoft.Python.Parsing;
19+
20+
namespace Microsoft.Python.Analysis.Diagnostics {
21+
public sealed class DiagnosticsSeverityMap {
22+
private readonly Dictionary<string, Severity> _map = new Dictionary<string, Severity>();
23+
24+
public DiagnosticsSeverityMap() { }
25+
26+
public DiagnosticsSeverityMap(string[] errors, string[] warnings, string[] information, string[] disabled) {
27+
_map.Clear();
28+
// disabled > error > warning > information
29+
foreach (var x in information.MaybeEnumerate()) {
30+
_map[x] = Severity.Information;
31+
}
32+
foreach (var x in warnings.MaybeEnumerate()) {
33+
_map[x] = Severity.Warning;
34+
}
35+
foreach (var x in errors.MaybeEnumerate()) {
36+
_map[x] = Severity.Error;
37+
}
38+
foreach (var x in disabled.MaybeEnumerate()) {
39+
_map[x] = Severity.Suppressed;
40+
}
41+
}
42+
public Severity GetEffectiveSeverity(string code, Severity defaultSeverity)
43+
=> _map.TryGetValue(code, out var severity) ? severity : defaultSeverity;
44+
}
45+
}

‎src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,5 +21,6 @@ public static class ErrorCodes {
2121
public const string UnknownParameterName = "unknown-parameter-name";
2222
public const string ParameterAlreadySpecified = "parameter-already-specified";
2323
public const string ParameterMissing = "parameter-missing";
24+
public const string UnresolvedImport = "unresolved-import";
2425
}
2526
}

‎src/Analysis/Ast/Impl/Diagnostics/IDiagnosticsService.cs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,5 +38,12 @@ public interface IDiagnosticsService {
3838
/// the diagnostic publishing to the client.
3939
/// </summary>
4040
int PublishingDelay { get; set; }
41+
42+
/// <summary>
43+
/// Provides map of error codes to severity when user wants
44+
/// to override default severity settings or suppress particular
45+
/// diagnostics completely.
46+
/// </summary>
47+
DiagnosticsSeverityMap DiagnosticsSeverityMap { get; set; }
4148
}
4249
}

‎src/Analysis/Ast/Impl/Modules/PythonModule.cs

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -385,7 +385,7 @@ private void Parse(CancellationToken cancellationToken) {
385385

386386
// Do not report issues with libraries or stubs
387387
if (sink != null) {
388-
_diagnosticsService?.Replace(Uri, _parseErrors);
388+
_diagnosticsService?.Replace(Uri, _parseErrors.Concat(Analysis.Diagnostics));
389389
}
390390

391391
_parsingTask = null;
@@ -460,6 +460,11 @@ public virtual bool NotifyAnalysisComplete(IDocumentAnalysis analysis) {
460460
OnAnalysisComplete();
461461
ContentState = State.Analyzed;
462462

463+
// Do not report issues with libraries or stubs
464+
if (ModuleType == ModuleType.User) {
465+
_diagnosticsService?.Replace(Uri, _parseErrors.Concat(Analysis.Diagnostics));
466+
}
467+
463468
var tcs = _analysisTcs;
464469
_analysisTcs = null;
465470
tcs.TrySetResult(analysis);

‎src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ internal async Task InitializeAsync(CancellationToken cancellationToken = defaul
4242

4343
// Initialize built-in
4444
var moduleName = BuiltinTypeId.Unknown.GetModuleName(_interpreter.LanguageVersion);
45-
var modulePath = ModuleCache.GetCacheFilePath(_interpreter.Configuration.InterpreterPath ?? "python.exe");
45+
var modulePath = ModuleCache.GetCacheFilePath(_interpreter.Configuration.InterpreterPath);
4646

4747
var b = new BuiltinsPythonModule(moduleName, modulePath, _services);
4848
_modules[BuiltinModuleName] = BuiltinsModule = b;

‎src/Analysis/Ast/Test/FluentAssertions/AssertionsFactory.cs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,5 +32,8 @@ internal static class AssertionsFactory {
3232
public static VariableAssertions Should(this IVariable v) => new VariableAssertions(v);
3333

3434
public static RangeAssertions Should(this Range? range) => new RangeAssertions(range);
35+
36+
public static SourceSpanAssertions Should(this SourceSpan span) => new SourceSpanAssertions(span);
37+
public static SourceSpanAssertions Should(this SourceSpan? span) => new SourceSpanAssertions(span.Value);
3538
}
3639
}

‎src/LanguageServer/Test/FluentAssertions/SourceSpanAssertions.cs renamed to ‎src/Analysis/Ast/Test/FluentAssertions/SourceSpanAssertions.cs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@
1818
using Microsoft.Python.Core.Text;
1919
using static Microsoft.Python.Analysis.Tests.FluentAssertions.AssertionsUtilities;
2020

21-
namespace Microsoft.Python.LanguageServer.Tests.FluentAssertions {
22-
internal sealed class SourceSpanAssertions {
21+
namespace Microsoft.Python.Analysis.Tests.FluentAssertions {
22+
public sealed class SourceSpanAssertions {
2323
public SourceSpan? Subject { get; }
2424

2525
public SourceSpanAssertions(SourceSpan? span) {

0 commit comments

Comments
 (0)