Add dependency resolution to the analyzer #621

AlexanderSher · 2019-02-15T14:51:33Z

No description provided.

MikhailArkhipov · 2019-02-15T17:52:50Z

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs

+            public ModuleKey(IPythonModule module) {
+                Name = module.Name;
+                FilePath = module.ModuleType == ModuleType.CompiledBuiltin ? null : module.FilePath;
+                IsTypeshed = module.ModuleType == ModuleType.Stub;
            }


What about user stubs ie stubs next to the user code (pyi) and not in the typeshed paths. They would have module.ModuleType == ModuleType.Stub

Good point. Do you think we should add another flag for Typeshed?

Gets complicated... Is it used to tell which module resolution to use? Maybe just attach the module resolution reference.

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs

MikhailArkhipov · 2019-02-15T18:30:53Z

If you wish, merge #617 here, it is related.

MikhailArkhipov · 2019-02-15T18:47:11Z

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs

+                            var analysis = await entry.GetAnalysisAsync(cts.Token);
+                            lock (_syncObj) {
+                                if (entry.Version == analysis.Version) {
+                                    return analysis;


So it goes in and out of lock. Do we care if entry changes in the dictionary between we fetched it and when we reached entry.Version == analysis.Version ?

Hm, I can't find where entry is changed outside of the lock.

I mean code fetches entry inside lock, then goes out. At this time theoretically entry can get replaced in the dictionary with something else. Or not. Just checking.

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs

MikhailArkhipov · 2019-02-15T18:56:30Z

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs

+            public Task<ImmutableArray<ModuleKey>> FindDependenciesAsync(AnalysisEntry value, CancellationToken cancellationToken) {
+                var dependencies = new HashSet<ModuleKey>();
+                var module = value.Module;
+                var isTypeshed = module.ModuleType == ModuleType.Stub; // TODO: This is not the correct way to determine Typeshed.


I think ModuleKey has IsTypeshed. Or maybe ModuleKey could just hold the desired resolution service instance.

Sure it can, but it doesn't solve the question how to determine which resolution service instance is the right one.

I guess when module is first created as a stub we can attach the stub resolution service to it so when we need to find dependencies we'll just use the service attached to the module.

Good idea, will do.

- Merge "Port import tests from old LS (microsoft#606)"

…ric template (microsoft#600) * Buildable * Part 14 * First passing test * Simplify configuration * Style * Fix test and move code to folders * Builtins import * Fix microsoft#470 * Fluents * Add search path * Import analysis, part I * Simplify builtins handling * Remove IMember * Handle import specific * More tests * Add typeshed * Renames * Make sure lazy modules are loaded * Renames * Move/rename * Rework importing * Derivation rework * Part 2 * Part 3 * Buildable * Module members * Async walk * Imports test pass * Remove lazy types * Fix from import * Stubs * Double overloads * Fix datetime test * Couple more tests + fluents * Few more tests * Additionl test + union type * Built-in scrape tests * Full stdlib scrape test * Complete async AST walker * Conditional defines test + variable loc cleanup * More stub tests Fix stub loading for packages (port from DDG) Split walker into multiple files * Add some (broken mostly) tests from DDG * Move document tests * Function arg eval, part I * Instance/factory * Builds * Test fixes * Fix static and instance call eval * More tests * More ported tests * Specialize builtin functions * Make walkers common and handle nested functions * Moar tests * Parser fixes + more tests * Handle negative numbers * Fix null ref * Basic list support * Few more list tests * Basic iterators * Support __iter__ * Iterators * Fix couple of tests * Add decorator test * Generics, part I * Generics, part 2 * Generics, part 3 * Basic TypeVar test * Typings, part 4 * Fix test * Generics, part 6 * Generics, part 7 * More tests (failing) * Forward ref fixes * Reorg * Improve symbol resolution + test fixes * Test fixes * Dictionary, part I * Part 11 * Fix test * Tests * Tests * More dict work * List ctor * Skip some tests for now * Fix iterators * Tuple slicing * Polish type comparo in return types * Add Mapping and tests * Add Iterable * Fix typo * Add Iterator[T] + test * Simplify typing types * Class reduction * Fix tests * Test fix * Handle 'with' statement * Handle try-except * Class method inheritance + NewType * Container types * Containers test * Tests * Handle generic type alias * Named tuple * Global/non-local * Handle tuples in for Handle custom iterators * Basic generator * Any/AnyStr * Test fixes * Type/Optional/etc handling * Proper doc population * Tests + range * Argument match * Basic argset and diagnostics * Argset tests * Exclude WIP * Exclude WIP * Arg eval * Arg match, part 2 * Tests and generic arg comparisons * Function eval with arguments * Baselines * Fix test * Undo AST formatting change and update baseline * LS cleanup 1 * Fix list ctor argument unpacking * Cleanup 2 * Builds * Partial completions * Partial * Partial * Simple test * Tests * Basic startup * Clean up a bit * Remove debug code * Port formatter tests * Fix tokenizer crash * Async fixes * Hover * Basic hover * Adjust expression options * Basic signature help * Fix class/instance * Update test * Fix builtin creation exception * Fix tests * Actually provide declared module * Completion test (partial) * Undo * Fix null await Fix override completions + test * Exports filtering Prevent augmenting imported types * Filter variables & exports * Ported tests * Test fixes * More ported tests * Fix exception completions * Import completions * Scope completions * With completions * Test fixes * WIP * Test fix * Better arg match * Temp disable WIP * First cut * Fix type leak * WIP * Remove ConfigureAwait and handle canceled and failed in the analysis notifications * WIP * Generic class base * Generic forward reference resolution * Suppress completion in strings + test * Prevent recursion on generic resolution Better match arguments * Handle call expression in generics * Relax condition as it happens in tensorflow * Fix typeshed version search Make writing cached modules async Fix module doc fetching * Hover tests * Fix prom import hover * Hover tests * Synchronize test cache writing * First cut * Test * Fixes * Add tests for os.path Null ref fix * Fix cache check * Improve resolution of builtins and typing in stubs * Merge tests * Add ntst for requests * Handle typeshed better * Fix custom stub handling * Better sync * Move files * Fix parameter locations * Hover improvement * PEP hints * One more test for PEP hints * Better handle hover over import as * Text based generic constraints * Handle with better with generic stubs * Undo debug * Handle non-binary open() Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet. * Output syntax errors * Properly clear * - Fix async issue with analysis completion - Clean up diagnostics service interface - Use real DS in tests * Use proper scope when analyzing module * Severity mapping and reporting * Add publishing test Add NSubstitute Move all services to the same namespace. * Unused var * Test forced publish on close * Fix typo * Update test framework * Import location * Remove incorrect reference * Diagnostic severity mapping test Fix post-mortem earlier PR comment * Minor fixes * Better handle return types in classes created from templates. * Move interface to the main class part * Dynamic return type * Add hover and signature tests * Baseline update * Flicker reduction * - Correct reported unresolved import name - Add tests * PR feedback * Resolve merge issues * Fix completion doc + test * Restore formatting * Bunch of null checks * Fix generic base classes Add tests and changes to couple more cases * PR feedback

src/Parsing/Impl/Ast/FunctionDefinition.cs

MikhailArkhipov · 2019-02-18T07:12:41Z

src/Parsing/Impl/Ast/FunctionDefinition.cs

@@ -204,7 +214,7 @@ public FunctionDefinition(NameExpression name, Parameter[] parameters)
                    await NameExpression.WalkAsync(walker, cancellationToken);
                }

-                foreach (var p in _parameters.MaybeEnumerate()) {
+                foreach (var p in Parameters) {
                    await p.WalkAsync(walker, cancellationToken);


src/Parsing/Impl/Ast/GlobalStatement.cs

src/Parsing/Impl/Ast/ImportStatement.cs

MikhailArkhipov · 2019-02-18T07:33:46Z

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerEntry.cs

+                Version = version;
+            }
+
+            Version = version;


Setting version twice?

Copy/paste issue :)

MikhailArkhipov · 2019-02-18T07:34:13Z

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzerEntry.cs

+                Version = version;
+            }
+
+            Version = version;


As above, twice?

Yep, one time is enough.

MikhailArkhipov · 2019-02-18T16:05:25Z

OK, approving pending the merge conflicts.

* Add dependency resolution to the analyzer * Don't wait on missing Typeshed files * Merge "De-duplicate completions (microsoft#602)" * Fix empty signature tooltip after function call (microsoft#616) Fixes microsoft#614. * - Address CR Comments - Merge "Port import tests from old LS (microsoft#606)" * Resolve issue when specific types shared data after creation off generic template (microsoft#600) * Buildable * Part 14 * First passing test * Simplify configuration * Style * Fix test and move code to folders * Builtins import * Fix microsoft#470 * Fluents * Add search path * Import analysis, part I * Simplify builtins handling * Remove IMember * Handle import specific * More tests * Add typeshed * Renames * Make sure lazy modules are loaded * Renames * Move/rename * Rework importing * Derivation rework * Part 2 * Part 3 * Buildable * Module members * Async walk * Imports test pass * Remove lazy types * Fix from import * Stubs * Double overloads * Fix datetime test * Couple more tests + fluents * Few more tests * Additionl test + union type * Built-in scrape tests * Full stdlib scrape test * Complete async AST walker * Conditional defines test + variable loc cleanup * More stub tests Fix stub loading for packages (port from DDG) Split walker into multiple files * Add some (broken mostly) tests from DDG * Move document tests * Function arg eval, part I * Instance/factory * Builds * Test fixes * Fix static and instance call eval * More tests * More ported tests * Specialize builtin functions * Make walkers common and handle nested functions * Moar tests * Parser fixes + more tests * Handle negative numbers * Fix null ref * Basic list support * Few more list tests * Basic iterators * Support __iter__ * Iterators * Fix couple of tests * Add decorator test * Generics, part I * Generics, part 2 * Generics, part 3 * Basic TypeVar test * Typings, part 4 * Fix test * Generics, part 6 * Generics, part 7 * More tests (failing) * Forward ref fixes * Reorg * Improve symbol resolution + test fixes * Test fixes * Dictionary, part I * Part 11 * Fix test * Tests * Tests * More dict work * List ctor * Skip some tests for now * Fix iterators * Tuple slicing * Polish type comparo in return types * Add Mapping and tests * Add Iterable * Fix typo * Add Iterator[T] + test * Simplify typing types * Class reduction * Fix tests * Test fix * Handle 'with' statement * Handle try-except * Class method inheritance + NewType * Container types * Containers test * Tests * Handle generic type alias * Named tuple * Global/non-local * Handle tuples in for Handle custom iterators * Basic generator * Any/AnyStr * Test fixes * Type/Optional/etc handling * Proper doc population * Tests + range * Argument match * Basic argset and diagnostics * Argset tests * Exclude WIP * Exclude WIP * Arg eval * Arg match, part 2 * Tests and generic arg comparisons * Function eval with arguments * Baselines * Fix test * Undo AST formatting change and update baseline * LS cleanup 1 * Fix list ctor argument unpacking * Cleanup 2 * Builds * Partial completions * Partial * Partial * Simple test * Tests * Basic startup * Clean up a bit * Remove debug code * Port formatter tests * Fix tokenizer crash * Async fixes * Hover * Basic hover * Adjust expression options * Basic signature help * Fix class/instance * Update test * Fix builtin creation exception * Fix tests * Actually provide declared module * Completion test (partial) * Undo * Fix null await Fix override completions + test * Exports filtering Prevent augmenting imported types * Filter variables & exports * Ported tests * Test fixes * More ported tests * Fix exception completions * Import completions * Scope completions * With completions * Test fixes * WIP * Test fix * Better arg match * Temp disable WIP * First cut * Fix type leak * WIP * Remove ConfigureAwait and handle canceled and failed in the analysis notifications * WIP * Generic class base * Generic forward reference resolution * Suppress completion in strings + test * Prevent recursion on generic resolution Better match arguments * Handle call expression in generics * Relax condition as it happens in tensorflow * Fix typeshed version search Make writing cached modules async Fix module doc fetching * Hover tests * Fix prom import hover * Hover tests * Synchronize test cache writing * First cut * Test * Fixes * Add tests for os.path Null ref fix * Fix cache check * Improve resolution of builtins and typing in stubs * Merge tests * Add ntst for requests * Handle typeshed better * Fix custom stub handling * Better sync * Move files * Fix parameter locations * Hover improvement * PEP hints * One more test for PEP hints * Better handle hover over import as * Text based generic constraints * Handle with better with generic stubs * Undo debug * Handle non-binary open() Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet. * Output syntax errors * Properly clear * - Fix async issue with analysis completion - Clean up diagnostics service interface - Use real DS in tests * Use proper scope when analyzing module * Severity mapping and reporting * Add publishing test Add NSubstitute Move all services to the same namespace. * Unused var * Test forced publish on close * Fix typo * Update test framework * Import location * Remove incorrect reference * Diagnostic severity mapping test Fix post-mortem earlier PR comment * Minor fixes * Better handle return types in classes created from templates. * Move interface to the main class part * Dynamic return type * Add hover and signature tests * Baseline update * Flicker reduction * - Correct reported unresolved import name - Add tests * PR feedback * Resolve merge issues * Fix completion doc + test * Restore formatting * Bunch of null checks * Fix generic base classes Add tests and changes to couple more cases * PR feedback * Remove double assigning * Don't allow non-rooted modules to be added as part of reload

AlexanderSher added 4 commits February 15, 2019 08:48

Add dependency resolution to the analyzer

a7e7c0f

Don't wait on missing Typeshed files

256640f

Merge "De-duplicate completions (microsoft#602)"

d5604d2

Fix empty signature tooltip after function call (microsoft#616)

e2c12dc

Fixes microsoft#614.

MikhailArkhipov reviewed Feb 15, 2019

View reviewed changes

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs Show resolved Hide resolved

MikhailArkhipov reviewed Feb 15, 2019

View reviewed changes

src/Analysis/Ast/Impl/Analyzer/PythonAnalyzer.cs Outdated Show resolved Hide resolved

MikhailArkhipov reviewed Feb 15, 2019

View reviewed changes

MikhailArkhipov mentioned this pull request Feb 16, 2019

Fix #601 #622

Merged

AlexanderSher and others added 2 commits February 18, 2019 00:29

- Address CR Comments

b22ec54

- Merge "Port import tests from old LS (microsoft#606)"

MikhailArkhipov reviewed Feb 18, 2019

View reviewed changes

src/Parsing/Impl/Ast/FunctionDefinition.cs Show resolved Hide resolved

MikhailArkhipov reviewed Feb 18, 2019

View reviewed changes

src/Parsing/Impl/Ast/GlobalStatement.cs Show resolved Hide resolved

MikhailArkhipov reviewed Feb 18, 2019

View reviewed changes

src/Parsing/Impl/Ast/ImportStatement.cs Show resolved Hide resolved

MikhailArkhipov reviewed Feb 18, 2019

View reviewed changes

MikhailArkhipov approved these changes Feb 18, 2019

View reviewed changes

AlexanderSher and others added 3 commits February 18, 2019 10:21

Remove double assigning

48c9c40

Don't allow non-rooted modules to be added as part of reload

518ef94

Merge branch 'master' into DependencyResolution

e31a457

AlexanderSher merged commit 7707351 into microsoft:master Feb 19, 2019

AlexanderSher deleted the DependencyResolution branch February 19, 2019 20:45

MikhailArkhipov mentioned this pull request Feb 19, 2019

Subsystem to analyze import dependencies and provide order of analysis #432

Closed

Add dependency resolution to the analyzer #621

Add dependency resolution to the analyzer #621

Uh oh!

Conversation

AlexanderSher commented Feb 15, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MikhailArkhipov commented Feb 15, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikhailArkhipov commented Feb 18, 2019

Uh oh!

Uh oh!