Skip to content
This repository was archived by the owner on Apr 14, 2022. It is now read-only.

Commit 7707351

Browse files
AlexanderSherAlexanderSher
authored
Add dependency resolution to the analyzer (#621)
* Add dependency resolution to the analyzer * Don't wait on missing Typeshed files * Merge "De-duplicate completions (#602)" * Fix empty signature tooltip after function call (#616) Fixes #614. * - Address CR Comments - Merge "Port import tests from old LS (#606)" * Resolve issue when specific types shared data after creation off generic template (#600) * Buildable * Part 14 * First passing test * Simplify configuration * Style * Fix test and move code to folders * Builtins import * Fix #470 * Fluents * Add search path * Import analysis, part I * Simplify builtins handling * Remove IMember * Handle import specific * More tests * Add typeshed * Renames * Make sure lazy modules are loaded * Renames * Move/rename * Rework importing * Derivation rework * Part 2 * Part 3 * Buildable * Module members * Async walk * Imports test pass * Remove lazy types * Fix from import * Stubs * Double overloads * Fix datetime test * Couple more tests + fluents * Few more tests * Additionl test + union type * Built-in scrape tests * Full stdlib scrape test * Complete async AST walker * Conditional defines test + variable loc cleanup * More stub tests Fix stub loading for packages (port from DDG) Split walker into multiple files * Add some (broken mostly) tests from DDG * Move document tests * Function arg eval, part I * Instance/factory * Builds * Test fixes * Fix static and instance call eval * More tests * More ported tests * Specialize builtin functions * Make walkers common and handle nested functions * Moar tests * Parser fixes + more tests * Handle negative numbers * Fix null ref * Basic list support * Few more list tests * Basic iterators * Support __iter__ * Iterators * Fix couple of tests * Add decorator test * Generics, part I * Generics, part 2 * Generics, part 3 * Basic TypeVar test * Typings, part 4 * Fix test * Generics, part 6 * Generics, part 7 * More tests (failing) * Forward ref fixes * Reorg * Improve symbol resolution + test fixes * Test fixes * Dictionary, part I * Part 11 * Fix test * Tests * Tests * More dict work * List ctor * Skip some tests for now * Fix iterators * Tuple slicing * Polish type comparo in return types * Add Mapping and tests * Add Iterable * Fix typo * Add Iterator[T] + test * Simplify typing types * Class reduction * Fix tests * Test fix * Handle 'with' statement * Handle try-except * Class method inheritance + NewType * Container types * Containers test * Tests * Handle generic type alias * Named tuple * Global/non-local * Handle tuples in for Handle custom iterators * Basic generator * Any/AnyStr * Test fixes * Type/Optional/etc handling * Proper doc population * Tests + range * Argument match * Basic argset and diagnostics * Argset tests * Exclude WIP * Exclude WIP * Arg eval * Arg match, part 2 * Tests and generic arg comparisons * Function eval with arguments * Baselines * Fix test * Undo AST formatting change and update baseline * LS cleanup 1 * Fix list ctor argument unpacking * Cleanup 2 * Builds * Partial completions * Partial * Partial * Simple test * Tests * Basic startup * Clean up a bit * Remove debug code * Port formatter tests * Fix tokenizer crash * Async fixes * Hover * Basic hover * Adjust expression options * Basic signature help * Fix class/instance * Update test * Fix builtin creation exception * Fix tests * Actually provide declared module * Completion test (partial) * Undo * Fix null await Fix override completions + test * Exports filtering Prevent augmenting imported types * Filter variables & exports * Ported tests * Test fixes * More ported tests * Fix exception completions * Import completions * Scope completions * With completions * Test fixes * WIP * Test fix * Better arg match * Temp disable WIP * First cut * Fix type leak * WIP * Remove ConfigureAwait and handle canceled and failed in the analysis notifications * WIP * Generic class base * Generic forward reference resolution * Suppress completion in strings + test * Prevent recursion on generic resolution Better match arguments * Handle call expression in generics * Relax condition as it happens in tensorflow * Fix typeshed version search Make writing cached modules async Fix module doc fetching * Hover tests * Fix prom import hover * Hover tests * Synchronize test cache writing * First cut * Test * Fixes * Add tests for os.path Null ref fix * Fix cache check * Improve resolution of builtins and typing in stubs * Merge tests * Add ntst for requests * Handle typeshed better * Fix custom stub handling * Better sync * Move files * Fix parameter locations * Hover improvement * PEP hints * One more test for PEP hints * Better handle hover over import as * Text based generic constraints * Handle with better with generic stubs * Undo debug * Handle non-binary open() Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet. * Output syntax errors * Properly clear * - Fix async issue with analysis completion - Clean up diagnostics service interface - Use real DS in tests * Use proper scope when analyzing module * Severity mapping and reporting * Add publishing test Add NSubstitute Move all services to the same namespace. * Unused var * Test forced publish on close * Fix typo * Update test framework * Import location * Remove incorrect reference * Diagnostic severity mapping test Fix post-mortem earlier PR comment * Minor fixes * Better handle return types in classes created from templates. * Move interface to the main class part * Dynamic return type * Add hover and signature tests * Baseline update * Flicker reduction * - Correct reported unresolved import name - Add tests * PR feedback * Resolve merge issues * Fix completion doc + test * Restore formatting * Bunch of null checks * Fix generic base classes Add tests and changes to couple more cases * PR feedback * Remove double assigning * Don't allow non-rooted modules to be added as part of reload
1 parent 4afc1ef commit 7707351

File tree

130 files changed

+2998
-1299
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

130 files changed

+2998
-1299
lines changed

‎src/Analysis/Ast/Impl/Analyzer/AnalysisWalker.cs

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,17 +74,17 @@ public override async Task<bool> WalkAsync(ForStatement node, CancellationToken
7474
return await base.WalkAsync(node, cancellationToken);
7575
}
7676

77-
public override Task<bool> WalkAsync(FromImportStatement node, CancellationToken cancellationToken = default)
78-
=> ImportHandler.HandleFromImportAsync(node, cancellationToken);
77+
public override async Task<bool> WalkAsync(FromImportStatement node, CancellationToken cancellationToken = default)
78+
=> ImportHandler.HandleFromImport(node, cancellationToken);
7979

8080
public override Task<bool> WalkAsync(GlobalStatement node, CancellationToken cancellationToken = default)
8181
=> NonLocalHandler.HandleGlobalAsync(node, cancellationToken);
8282

8383
public override Task<bool> WalkAsync(IfStatement node, CancellationToken cancellationToken = default)
8484
=> ConditionalHandler.HandleIfAsync(node, cancellationToken);
8585

86-
public override Task<bool> WalkAsync(ImportStatement node, CancellationToken cancellationToken = default)
87-
=> ImportHandler.HandleImportAsync(node, cancellationToken);
86+
public override async Task<bool> WalkAsync(ImportStatement node, CancellationToken cancellationToken = default)
87+
=> ImportHandler.HandleImport(node, cancellationToken);
8888

8989
public override Task<bool> WalkAsync(NonlocalStatement node, CancellationToken cancellationToken = default)
9090
=> NonLocalHandler.HandleNonLocalAsync(node, cancellationToken);

‎src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs

Lines changed: 1 addition & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -20,39 +20,10 @@ namespace Microsoft.Python.Analysis.Analyzer {
2020
/// Represents document that can be analyzed asynchronously.
2121
/// </summary>
2222
internal interface IAnalyzable {
23-
/// <summary>
24-
/// Expected version of the analysis when asynchronous operations complete.
25-
/// Typically every change to the document or documents that depend on it
26-
/// increment the expected version. At the end of the analysis if the expected
27-
/// version is still the same, the analysis is applied to the document and
28-
/// becomes available to consumers.
29-
/// </summary>
30-
int ExpectedAnalysisVersion { get; }
31-
32-
/// <summary>
33-
/// Notifies document that analysis is now pending. Typically document increments
34-
/// the expected analysis version. The method can be called repeatedly without
35-
/// calling `CompleteAnalysis` first. The method is invoked for every dependency
36-
/// in the chain to ensure that objects know that their dependencies have been
37-
/// modified and the current analysis is no longer up to date.
38-
/// </summary>
39-
void NotifyAnalysisPending();
40-
4123
/// <summary>
4224
/// Notifies document that its analysis is now complete.
4325
/// </summary>
4426
/// <param name="analysis">Document analysis</param>
45-
/// <returns>True if analysis was accepted, false if is is out of date.</returns>
46-
bool NotifyAnalysisComplete(IDocumentAnalysis analysis);
47-
48-
/// <summary>
49-
/// Notifies module that analysis has been canceled.
50-
/// </summary>
51-
void NotifyAnalysisCanceled();
52-
53-
/// <summary>
54-
/// Notifies module that analysis has thrown an exception.
55-
/// </summary>
56-
void NotifyAnalysisFailed(Exception ex);
27+
void NotifyAnalysisComplete(IDocumentAnalysis analysis);
5728
}
5829
}

‎src/Analysis/Ast/Impl/Analyzer/Definitions/IPythonAnalyzer.cs

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,17 +16,21 @@
1616
using System.Threading;
1717
using System.Threading.Tasks;
1818
using Microsoft.Python.Analysis.Documents;
19+
using Microsoft.Python.Analysis.Types;
20+
using Microsoft.Python.Parsing.Ast;
1921

2022
namespace Microsoft.Python.Analysis.Analyzer {
2123
public interface IPythonAnalyzer {
24+
Task WaitForCompleteAnalysisAsync(CancellationToken cancellationToken = default);
25+
2226
/// <summary>
23-
/// Analyze single document.
27+
/// Schedules module for re-analysis
2428
/// </summary>
25-
Task AnalyzeDocumentAsync(IDocument document, CancellationToken cancellationToken);
29+
void EnqueueDocumentForAnalysis(IPythonModule module, PythonAst ast, int version, CancellationToken cancellationToken = default);
2630

2731
/// <summary>
28-
/// Analyze document with dependents.
32+
///
2933
/// </summary>
30-
Task AnalyzeDocumentDependencyChainAsync(IDocument document, CancellationToken cancellationToken);
34+
Task<IDocumentAnalysis> GetAnalysisAsync(IPythonModule module, int waitTime = 200, CancellationToken cancellationToken = default);
3135
}
3236
}

‎src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,9 +160,9 @@ public async Task<IMember> GetValueFromFunctionTypeAsync(IPythonFunctionType fn,
160160
fn.IsStub || !string.IsNullOrEmpty(fn.Overloads[args.OverloadIndex].GetReturnDocumentation(null))) {
161161

162162
if (fn.IsSpecialized && fn is PythonFunctionType ft) {
163-
foreach (var module in ft.Dependencies) {
163+
foreach (var moduleName in ft.Dependencies) {
164164
cancellationToken.ThrowIfCancellationRequested();
165-
await Interpreter.ModuleResolution.ImportModuleAsync(module, cancellationToken);
165+
Interpreter.ModuleResolution.GetOrLoadModule(moduleName);
166166
}
167167
}
168168

‎src/Analysis/Ast/Impl/Analyzer/Expressions/ExpressionFinder.cs

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -270,12 +270,12 @@ public override bool Walk(ImportStatement node) {
270270
SaveStmt(node, true);
271271

272272
if (_options.ImportNames) {
273-
foreach (var n in node.Names.MaybeEnumerate()) {
273+
foreach (var n in node.Names) {
274274
n?.Walk(this);
275275
}
276276
}
277277
if (_options.ImportAsNames) {
278-
foreach (var n in node.AsNames.MaybeEnumerate()) {
278+
foreach (var n in node.AsNames) {
279279
n?.Walk(this);
280280
}
281281
}
@@ -291,13 +291,14 @@ public override bool Walk(FromImportStatement node) {
291291
SaveStmt(node, true);
292292

293293
if (_options.ImportNames) {
294-
node.Root?.Walk(this);
294+
node.Root.Walk(this);
295295
}
296296

297-
foreach (var n in node.Names.MaybeEnumerate()) {
297+
foreach (var n in node.Names) {
298298
n?.Walk(this);
299299
}
300-
foreach (var n in node.AsNames.MaybeEnumerate()) {
300+
301+
foreach (var n in node.AsNames) {
301302
n?.Walk(this);
302303
}
303304

‎src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs

Lines changed: 16 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
using System.Diagnostics;
1717
using System.Linq;
1818
using System.Threading;
19-
using System.Threading.Tasks;
2019
using Microsoft.Python.Analysis.Core.DependencyResolution;
2120
using Microsoft.Python.Analysis.Modules;
2221
using Microsoft.Python.Analysis.Types;
@@ -26,9 +25,9 @@
2625

2726
namespace Microsoft.Python.Analysis.Analyzer.Handlers {
2827
internal sealed partial class ImportHandler {
29-
public async Task<bool> HandleFromImportAsync(FromImportStatement node, CancellationToken cancellationToken = default) {
28+
public bool HandleFromImport(FromImportStatement node, CancellationToken cancellationToken = default) {
3029
cancellationToken.ThrowIfCancellationRequested();
31-
if (node.Root == null || node.Names == null || Module.ModuleType == ModuleType.Specialized) {
30+
if (Module.ModuleType == ModuleType.Specialized) {
3231
return false;
3332
}
3433

@@ -46,13 +45,13 @@ public async Task<bool> HandleFromImportAsync(FromImportStatement node, Cancella
4645
await ImportMembersFromSelfAsync(node, cancellationToken);
4746
break;
4847
case ModuleImport moduleImport:
49-
await ImportMembersFromModuleAsync(node, moduleImport.FullName, cancellationToken);
48+
ImportMembersFromModule(node, moduleImport.FullName, cancellationToken);
5049
break;
5150
case PossibleModuleImport possibleModuleImport:
52-
await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, Eval.GetLoc(node.Root), cancellationToken);
51+
HandlePossibleImport(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, Eval.GetLoc(node.Root));
5352
break;
5453
case PackageImport packageImports:
55-
await ImportMembersFromPackageAsync(node, packageImports, cancellationToken);
54+
ImportMembersFromPackage(node, packageImports);
5655
break;
5756
case ImportNotFound notFound:
5857
MakeUnresolvedImport(null, notFound.FullName, Eval.GetLoc(node.Root));
@@ -85,17 +84,17 @@ private async Task ImportMembersFromSelfAsync(FromImportStatement node, Cancella
8584
// Consider 'from . import path as path' in os.pyi in typeshed.
8685
var import = ModuleResolution.CurrentPathResolver.GetModuleImportFromModuleName($"{Module.Name}.{importName}");
8786
if (!string.IsNullOrEmpty(import?.FullName)) {
88-
member = await ModuleResolution.ImportModuleAsync(import.FullName, cancellationToken);
87+
member = ModuleResolution.GetOrLoadModule(import.FullName);
8988
}
9089
}
9190
Eval.DeclareVariable(memberName, member ?? Eval.UnknownType, VariableSource.Declaration, Eval.GetLoc(names[i]));
9291
}
9392
}
9493

95-
private async Task ImportMembersFromModuleAsync(FromImportStatement node, string moduleName, CancellationToken cancellationToken = default) {
94+
private void ImportMembersFromModule(FromImportStatement node, string moduleName, CancellationToken cancellationToken = default) {
9695
var names = node.Names;
9796
var asNames = node.AsNames;
98-
var module = await ModuleResolution.ImportModuleAsync(moduleName, cancellationToken);
97+
var module = ModuleResolution.GetOrLoadModule(moduleName);
9998
if (module == null) {
10099
return;
101100
}
@@ -104,7 +103,7 @@ private async Task ImportMembersFromModuleAsync(FromImportStatement node, string
104103
// TODO: warn this is not a good style per
105104
// TODO: https://docs.python.org/3/faq/programming.html#what-are-the-best-practices-for-using-import-in-a-module
106105
// TODO: warn this is invalid if not in the global scope.
107-
await HandleModuleImportStarAsync(module, cancellationToken);
106+
HandleModuleImportStar(module, cancellationToken);
108107
return;
109108
}
110109

@@ -122,7 +121,7 @@ private async Task ImportMembersFromModuleAsync(FromImportStatement node, string
122121
}
123122
}
124123

125-
private async Task HandleModuleImportStarAsync(IPythonModule module, CancellationToken cancellationToken = default) {
124+
private void HandleModuleImportStar(IPythonModule module, CancellationToken cancellationToken = default) {
126125
foreach (var memberName in module.GetMemberNames()) {
127126
cancellationToken.ThrowIfCancellationRequested();
128127

@@ -135,14 +134,14 @@ private async Task HandleModuleImportStarAsync(IPythonModule module, Cancellatio
135134

136135
member = member ?? Eval.UnknownType;
137136
if (member is IPythonModule m) {
138-
await ModuleResolution.ImportModuleAsync(m.Name, cancellationToken);
137+
ModuleResolution.GetOrLoadModule(m.Name);
139138
}
140139

141140
Eval.DeclareVariable(memberName, member, VariableSource.Import, module.Location);
142141
}
143142
}
144143

145-
private async Task ImportMembersFromPackageAsync(FromImportStatement node, PackageImport packageImport, CancellationToken cancellationToken = default) {
144+
private void ImportMembersFromPackage(FromImportStatement node, PackageImport packageImport) {
146145
var names = node.Names;
147146
var asNames = node.AsNames;
148147

@@ -158,13 +157,10 @@ private async Task ImportMembersFromPackageAsync(FromImportStatement node, Packa
158157
var memberName = memberReference.Name;
159158
var location = Eval.GetLoc(memberReference);
160159

161-
ModuleImport moduleImport;
162-
IPythonType member;
163-
if ((moduleImport = packageImport.Modules.FirstOrDefault(mi => mi.Name.EqualsOrdinal(importName))) != null) {
164-
member = await ModuleResolution.ImportModuleAsync(moduleImport.FullName, cancellationToken);
165-
} else {
166-
member = Eval.UnknownType;
167-
}
160+
var moduleImport = packageImport.Modules.FirstOrDefault(mi => mi.Name.EqualsOrdinal(importName));
161+
var member = moduleImport != null
162+
? ModuleResolution.GetOrLoadModule(moduleImport.FullName)
163+
: Eval.UnknownType;
168164

169165
Eval.DeclareVariable(memberName, member, VariableSource.Import, location);
170166
}

‎src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
using System;
1717
using System.Linq;
1818
using System.Threading;
19-
using System.Threading.Tasks;
2019
using Microsoft.Python.Analysis.Core.DependencyResolution;
2120
using Microsoft.Python.Analysis.Diagnostics;
2221
using Microsoft.Python.Analysis.Modules;
@@ -31,9 +30,9 @@ namespace Microsoft.Python.Analysis.Analyzer.Handlers {
3130
internal sealed partial class ImportHandler : StatementHandler {
3231
public ImportHandler(AnalysisWalker walker) : base(walker) { }
3332

34-
public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToken cancellationToken = default) {
33+
public bool HandleImport(ImportStatement node, CancellationToken cancellationToken = default) {
3534
cancellationToken.ThrowIfCancellationRequested();
36-
if (node.Names == null || Module.ModuleType == ModuleType.Specialized) {
35+
if (Module.ModuleType == ModuleType.Specialized) {
3736
return false;
3837
}
3938

@@ -58,10 +57,10 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
5857
Eval.DeclareVariable(memberName, Module, VariableSource.Declaration, location);
5958
break;
6059
case ModuleImport moduleImport:
61-
module = await HandleImportAsync(moduleImport, location, cancellationToken);
60+
module = HandleImport(moduleImport, location);
6261
break;
6362
case PossibleModuleImport possibleModuleImport:
64-
module = await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, location, cancellationToken);
63+
module = HandlePossibleImport(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, location);
6564
break;
6665
default:
6766
// TODO: Package import?
@@ -73,22 +72,23 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
7372
AssignImportedVariables(module, moduleImportExpression, asNameExpression);
7473
}
7574
}
75+
7676
return false;
7777
}
7878

79-
private async Task<IPythonModule> HandleImportAsync(ModuleImport moduleImport, LocationInfo location, CancellationToken cancellationToken) {
80-
var module = await ModuleResolution.ImportModuleAsync(moduleImport.FullName, cancellationToken);
81-
if (module == null) {
82-
MakeUnresolvedImport(moduleImport.FullName, moduleImport.FullName, location);
83-
return null;
79+
private IPythonModule HandleImport(ModuleImport moduleImport, LocationInfo location) {
80+
var module = ModuleResolution.GetOrLoadModule(moduleImport.FullName);
81+
if (module != null) {
82+
return module;
8483
}
85-
return module;
84+
85+
MakeUnresolvedImport(moduleImport.FullName, moduleImport.FullName, location);
86+
return null;
8687
}
8788

88-
private async Task<IPythonModule> HandlePossibleImportAsync(
89-
PossibleModuleImport possibleModuleImport, string moduleName, LocationInfo location, CancellationToken cancellationToken) {
89+
private IPythonModule HandlePossibleImport(PossibleModuleImport possibleModuleImport, string moduleName, LocationInfo location) {
9090
var fullName = possibleModuleImport.PrecedingModuleFullName;
91-
var module = await ModuleResolution.ImportModuleAsync(possibleModuleImport.PrecedingModuleFullName, cancellationToken);
91+
var module = ModuleResolution.GetOrLoadModule(possibleModuleImport.PrecedingModuleFullName);
9292
if (module == null) {
9393
MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, moduleName, location);
9494
return null;

0 commit comments

Comments
 (0)