Skip to content
This repository was archived by the owner on Nov 4, 2024. It is now read-only.

Commit 5280d34

Browse files
author
Mikhail Arkhipov
authored
Merge persistence work into master (microsoft#1482)
* Persistent analysis, part I (microsoft#1224) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * Merge master into database branch (microsoft#1312) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * Avoid empty node in empty global scope (microsoft#1226) * Add null checks * Avoid null node in empty global scope * Remove unrelated change * Lock content since it is locked in other cases (microsoft#1227) * Fix addBrackets setting (microsoft#1241) * Fix addBrackets * Remove unrelated change * Usings * Handle library analysis better (microsoft#1246) * Trying to augment argument set with Evaluator and Expression context (microsoft#1259) * Moving anyStr to its own method to prevent FromTypeVar from being called with an ArgumentSet that has no context * Trying to augment argument set with context wherever possible * Setting expression in argument set, removing null * Renaming to WithoutContext and updating comment * Check for null args in PythonFunctionOverload * When returning in init method, report a diagnostic error (microsoft#1261) * When returning in init method, report a diagnostic error * Adding check and tests for returning None before adding error for a return statement in the init function * Cleaning up tests * Give Diagnostic messages on improper usage of Generic (microsoft#1248) * Adding diagnostic message for improper use Generic. * Adding tests to make sure no diagnostics on valid uses of Generic * Adding diagnostic error on binary operations with incompatible types (microsoft#1254) * Adding diagnostic error for binary operations with incompatible types - e.g 5 + 'str' * When NewType is called and the first argument is not a string, make a diagnostic message (microsoft#1260) * Adding diagnostic error if the user calls NewType with the first arg not of string type * Fix typo in TROUBLESHOOTING.md (microsoft#1285) * Multiple analysis fixes (microsoft#1297) Multiple analysis fixes. - Fixes microsoft#1151: Reliable way to determine final analysis of a module - Fixes microsoft#1201: NRE in SymbolCollector.AddProperty - Fixes microsoft#1228: NRE in TryFindMissingDependencies - Fixes microsoft#1294: Handle valid case for reloading AST - Fixes microsoft#1295: FindReferences doesn't work on package init members - Fixes microsoft#1296: DependencyResolver graph misses intermediate imports - Part of the fix for microsoft#1174: All references are not listed when using ms language server (fixes in analysis itself are also required for this one) * Rework search path resolution (microsoft#1289) * first working path classfier, breaks existing path code for now * inherit Comparer instead of implementing both * don't do path depth comparison, preserve user ordering as before * switch to using new path classification code in main module resolution * fix typo * clean up comment about paths in Server.cs * use String.Split instead of regex when parsing script output * test ordering of user provided paths * add new normalize and trim helper, check preconditions as debug asserts rather than commenting on them * use Split extension, don't MaybeEnumerate paths * No diagnostic message when Generic is called with no args (microsoft#1305) * No diagnostic message when Generic is called with no args * When a class inherits from something that is not a class, give a diagnostic error (microsoft#1277) * When analyzing class definitions, give diagnostic if bases are not all class types * PR feedback * Treat persistent module as regular specialized (microsoft#1334) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Fix merge issue * Remove debug code * Store location of module members in the database (microsoft#1339) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Add location converter abstraction * Store member location * Basic locations test * Navigation * Add test * Update baselines * Port changes from dbtype * Multiple fixes and additional tests for persistence (microsoft#1372) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Add location converter abstraction * Store member location * Fix merge issue * Basic locations test * Navigation * Add test * Update baselines * Type restore - initial * Remove debug code * Partial * Fix stub merge * Various model fixes * Improve module handling * Qualified name improvements * Fix unbound case * Improve stub merge * Fix qualified names of typing * Handle stub-only modules * Add tests for io, re and sys * Better handle named tuple * Handle module circular references * Handle else in platform and version clauses + handle it in symbol collector * Formatting * Fix walk of multi-level if statement * Unify package search in imports * Fix tests * Undo change * Port changes from dbtype * Partial * Named tuple support * Baseline updates * Debug code * Support types * Properly compare class member declaring type * Nested classes and functions persistence * Undo debug * Fix numpy restore * Baselines * Fix tests * Update AnyStr test reflecting changes to AnyStr behavior * Exclude baselines from git diff * Fix gitattr * Move git setting to root * Try no path * Add RDT count to the analysis_complete event (microsoft#1396) * Add RDT count to the analysis_complete event * Use property instead of method * Test fixes * Undo change * Additional stub merge fixes * Baseline updates * Fix goto def * Protect specific type creation * Track documentaton source * More reliable tests + simplification * Prevent crashes when the AST happens to be null (microsoft#1405) * Remove all references to dropped modules and force GC on reload (microsoft#1402) * First attempt at ensuring moduled dropped during reload are GC'd * Make sure AST is set after content is cleared * More tests passing * ModuleWalker updates * Also don't merge against specialized modules, do builtin _astMap reset before reanalyzing instead of after any analysis * Don't call dispose directly, it's done elsewhere and has no effect * Force GC via normal session instead of waiting for a hardcoded time * Un-refactor function since it's not being called externally anymore * Formatting/usings * PR feedback * Undo IsBuiltin, some things continued to be referenced with that changed * Move reload logic down into PythonAnalyzer, remove added public inferface items * Mode AddReference to PythonType as override * Remove dead code, make all ResetAnalyzer calls full * Typo * Cleanup * Basic classification * Fix merge error * Module unique id fixes * Stricted check to save analysis * Revert "Fix tests" This reverts commit 247a8c3. * Revert "Unify package search in imports" This reverts commit 67fed10. * Don't clear scope variables with inner classes * Fix typo * Add persistence tests for 60+ system modules (microsoft#1429) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Add location converter abstraction * Store member location * Fix merge issue * Basic locations test * Navigation * Add test * Update baselines * Type restore - initial * Remove debug code * Partial * Fix stub merge * Various model fixes * Improve module handling * Qualified name improvements * Fix unbound case * Improve stub merge * Fix qualified names of typing * Handle stub-only modules * Add tests for io, re and sys * Better handle named tuple * Handle module circular references * Handle else in platform and version clauses + handle it in symbol collector * Formatting * Fix walk of multi-level if statement * Unify package search in imports * Fix tests * Undo change * Port changes from dbtype * Partial * Named tuple support * Baseline updates * Debug code * Support types * Properly compare class member declaring type * Nested classes and functions persistence * Undo debug * Fix numpy restore * Baselines * Fix tests * Update AnyStr test reflecting changes to AnyStr behavior * Exclude baselines from git diff * Fix gitattr * Move git setting to root * Try no path * Test fixes * Undo change * Additional stub merge fixes * Baseline updates * Fix goto def * Protect specific type creation * Track documentaton source * More reliable tests + simplification * Typo * Cleanup * Basic classification * Fix merge error * Module unique id fixes * Stricted check to save analysis * Revert "Fix tests" This reverts commit 247a8c3. * Revert "Unify package search in imports" This reverts commit 67fed10. * Don't clear scope variables with inner classes * Fix typo * Many new tests * Fix collections test * Fix CTypes * Initial * Update test * Merge issues * Fix CTypes again * Fix null bases * Tell between class members with/without self better * TypeVar support * Add bound/covariant * Fix Enum reassignments * Fix Random * Fix import * over local declarations (Socket) * Move interface * Fix reference search * Enable asyncio test * More tests * Enable dataclasses * Add inspect and gzip * More tests * Add setting * Add handling of import position relative to the variable * Caching level changes * Update baselines * Test update (we no longer reassign classes) * Fix typing tests * Update baseline * Fix missing keys issue (microsoft#1451) * Typevar fix * Persistence of generics and named tuples (microsoft#1459) * Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Add location converter abstraction * Store member location * Fix merge issue * Basic locations test * Navigation * Add test * Update baselines * Type restore - initial * Remove debug code * Partial * Fix stub merge * Various model fixes * Improve module handling * Qualified name improvements * Fix unbound case * Improve stub merge * Fix qualified names of typing * Handle stub-only modules * Add tests for io, re and sys * Better handle named tuple * Handle module circular references * Handle else in platform and version clauses + handle it in symbol collector * Formatting * Fix walk of multi-level if statement * Unify package search in imports * Fix tests * Undo change * Port changes from dbtype * Partial * Named tuple support * Baseline updates * Debug code * Support types * Properly compare class member declaring type * Nested classes and functions persistence * Undo debug * Fix numpy restore * Baselines * Fix tests * Update AnyStr test reflecting changes to AnyStr behavior * Exclude baselines from git diff * Fix gitattr * Move git setting to root * Try no path * Test fixes * Undo change * Additional stub merge fixes * Baseline updates * Fix goto def * Protect specific type creation * Track documentaton source * More reliable tests + simplification * Typo * Cleanup * Basic classification * Fix merge error * Module unique id fixes * Stricted check to save analysis * Revert "Fix tests" This reverts commit 247a8c3. * Revert "Unify package search in imports" This reverts commit 67fed10. * Don't clear scope variables with inner classes * Fix typo * Many new tests * Fix collections test * Fix CTypes * Initial * Update test * Merge issues * Fix CTypes again * Fix null bases * Tell between class members with/without self better * TypeVar support * Add bound/covariant * Fix Enum reassignments * Fix Random * Fix import * over local declarations (Socket) * Move interface * Fix reference search * Enable asyncio test * More tests * Enable dataclasses * Add inspect and gzip * More tests * Add setting * Add handling of import position relative to the variable * Caching level changes * Partial * Rework * Fix null * Update baselines * Functools pass * Reverse stub merge * Partial * Partial ctypes * Update variables post merge * Fix ctypes * Merge issues * Fix typevar * Add caching on construction * Assorted fixes * Named tuples * Work around os stub oddness * Fix generic qualified names * Ignore lambdas * Support named tuples as bases * Baselines * Test updates * Named tuple tests * PR feedback * Using * Move diagnostics to GenericBase * PR feedback * - Fix microsoft#1455: AF: Library module of type Stub has been analyzed already! (microsoft#1470) - Restore task handling in unit tests - Throw exception from Debug.Fail instead of FailFast * Update baselines since __spec__ member was added * Undo some of changes to match master. * Fix null ref * Resolve conflicts * Fix null ref on bases due to new AnyStr behavior * Handling dependencies in module persistence (microsoft#1471) * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Add location converter abstraction * Store member location * Fix merge issue * Basic locations test * Navigation * Add test * Update baselines * Type restore - initial * Remove debug code * Partial * Fix stub merge * Various model fixes * Improve module handling * Qualified name improvements * Fix unbound case * Improve stub merge * Fix qualified names of typing * Handle stub-only modules * Add tests for io, re and sys * Better handle named tuple * Handle module circular references * Handle else in platform and version clauses + handle it in symbol collector * Formatting * Fix walk of multi-level if statement * Unify package search in imports * Fix tests * Undo change * Port changes from dbtype * Partial * Named tuple support * Baseline updates * Debug code * Support types * Properly compare class member declaring type * Nested classes and functions persistence * Undo debug * Fix numpy restore * Baselines * Fix tests * Update AnyStr test reflecting changes to AnyStr behavior * Exclude baselines from git diff * Fix gitattr * Move git setting to root * Try no path * Test fixes * Undo change * Additional stub merge fixes * Baseline updates * Fix goto def * Protect specific type creation * Track documentaton source * More reliable tests + simplification * Typo * Cleanup * Basic classification * Fix merge error * Module unique id fixes * Stricted check to save analysis * Revert "Fix tests" This reverts commit 247a8c3. * Revert "Unify package search in imports" This reverts commit 67fed10. * Don't clear scope variables with inner classes * Fix typo * Many new tests * Fix collections test * Fix CTypes * Initial * Update test * Merge issues * Fix CTypes again * Fix null bases * Tell between class members with/without self better * TypeVar support * Add bound/covariant * Fix Enum reassignments * Fix Random * Fix import * over local declarations (Socket) * Move interface * Fix reference search * Enable asyncio test * More tests * Enable dataclasses * Add inspect and gzip * More tests * Add setting * Add handling of import position relative to the variable * Caching level changes * Partial * Rework * Fix null * Update baselines * Functools pass * Reverse stub merge * Partial * Partial ctypes * Update variables post merge * Fix ctypes * Merge issues * Fix typevar * Add caching on construction * Assorted fixes * Named tuples * Work around os stub oddness * Fix generic qualified names * Ignore lambdas * Support named tuples as bases * Baselines * Test updates * Named tuple tests * PR feedback * Using * Separate creation and population for better handling of forward references. * Add dependency provider * Merge issues * Better class creation * Partial * Dependency providers * Add IsPersistent flag * Make special global scope * Move dependency fetch earlier * Replace AST walk by interface * Make analysis sequence closer to master * Undo some * Undo debug * Show dependencies in log * null check * Update tests * Make sure factory looks in correct scope Enable assertions in tests * Separate stub dependencies * Don't try and merge from empty stub * Match generic parameters by name * Use qualified names in generics Minor tests cleanup * Back to names * Update tests * usings * Fix dependencies for stub * Null check * PR feedback * PR feedback + fix merge issue in generics * Test fixes + PR feedback * Baseline updates * PR feedback * Restore original order * Order * Merge issues * Make cancellation closer to original * Pass cancellation to module walker * Simplify * Upgrade to 3.0 * Undo change + .NET 3 * Get back to 2.0 * Update components * Display caching level Round time to reasonble fractions. * Fix null ref due to conflicting changes * Null ref fix * Simplify * Pass AST whish is required during analysis when module does not have its own yet. * tensorflow test * Fix from import model Add tensorflow test * Undo debug * Improve goto def in import and fix class base processing (microsoft#1529) * Update signing project to 3.0 * Upgrade to .NET 3.0 (microsoft#1521) * Upgrade to 3.0 * Get back to 2.0 * Add global.json for .NET SDK * Ensure FileInfo is not relative in DirectoryInfoProxy (microsoft#1519) * Ensure FileInfo is not relative in DirectoryInfoProxy * Add tests * Hadle import * Update dependencies (microsoft#1526) * Handle from import * Add test * Fix bad find and replace in AssertionsUtilities (microsoft#1527) * Merge issues * Add test (fails) * using * Fix merge of classes * Fix regression in bases Split merging into separate class * Fix null refs * Make common CanUpdateAnalysis check * Usings * Test fix * Remove CompletionItemKind.None * Add info to asserts * PR feedback * Undo test change, enable class reassignment Fix stub merge for TypeVar * Prevent merge of unrelated types * more tests * Fix variable update * Null check * Baselines * Move analysis complete event * More stable enumeration * Improve restoring imports in tests * Make extension for tests * Add URI null check * Fix completion typo * Add assert on unresolved modules * Add more details to assert * Using * To 3.7 * Implement check for big/little endian * Sync * Synchronization Fix import model restore * Add hard assert * Fix exception * Fix typo * Add lock
1 parent 448c9cc commit 5280d34

File tree

223 files changed

+8043
-1988
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

223 files changed

+8043
-1988
lines changed

.gitattributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.json -diff

build/Common.Build.Core.settings

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,8 @@
99
</PropertyGroup>
1010

1111
<!-- This file is shared across desktop .NET and .NET Core builds.
12-
Please do not put Windows-specific or dekstop-specific settings
13-
here (these incluse C#, C++ or VSSDK settings and targets -->
12+
Please do not put Windows-specific or desktop-specific settings
13+
here (these include C#, C++ or VS SDK settings and targets -->
1414
<PropertyGroup>
1515
<Configuration Condition="'$(Configuration)' == ''">Debug</Configuration>
1616
<VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>

src/Analysis/Ast/Impl/Analyzer/Definitions/IAnalyzable.cs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,18 @@
1313
// See the Apache Version 2.0 License for specific language governing
1414
// permissions and limitations under the License.
1515

16+
using Microsoft.Python.Analysis.Dependencies;
17+
1618
namespace Microsoft.Python.Analysis.Analyzer {
1719
/// <summary>
1820
/// Represents document that can be analyzed asynchronously.
1921
/// </summary>
2022
internal interface IAnalyzable {
23+
/// <summary>
24+
/// Returns object that can calculate dependencies of this entry.
25+
/// </summary>
26+
IDependencyProvider DependencyProvider { get; }
27+
2128
/// <summary>
2229
/// Notifies document that analysis is about to begin.
2330
/// </summary>

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,6 @@
1717
using System.Diagnostics;
1818
using System.Linq;
1919
using Microsoft.Python.Analysis.Documents;
20-
using Microsoft.Python.Analysis.Extensions;
2120
using Microsoft.Python.Analysis.Modules;
2221
using Microsoft.Python.Analysis.Types;
2322
using Microsoft.Python.Analysis.Values;

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Collections.cs

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ public IMember GetValueFromList(ListExpression expression) {
6262
var value = GetValueFromExpression(item) ?? UnknownType;
6363
contents.Add(value);
6464
}
65-
return PythonCollectionType.CreateList(Module.Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
65+
return PythonCollectionType.CreateList(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
6666
}
6767

6868
public IMember GetValueFromDictionary(DictionaryExpression expression) {
@@ -72,7 +72,7 @@ public IMember GetValueFromDictionary(DictionaryExpression expression) {
7272
var value = GetValueFromExpression(item.SliceStop) ?? UnknownType;
7373
contents[key] = value;
7474
}
75-
return new PythonDictionary(Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
75+
return new PythonDictionary(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
7676
}
7777

7878
private IMember GetValueFromTuple(TupleExpression expression) {
@@ -81,7 +81,7 @@ private IMember GetValueFromTuple(TupleExpression expression) {
8181
var value = GetValueFromExpression(item) ?? UnknownType;
8282
contents.Add(value);
8383
}
84-
return PythonCollectionType.CreateTuple(Module.Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
84+
return PythonCollectionType.CreateTuple(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
8585
}
8686

8787
public IMember GetValueFromSet(SetExpression expression) {
@@ -90,7 +90,7 @@ public IMember GetValueFromSet(SetExpression expression) {
9090
var value = GetValueFromExpression(item) ?? UnknownType;
9191
contents.Add(value);
9292
}
93-
return PythonCollectionType.CreateSet(Interpreter, contents, exact: expression.Items.Count <= MaxCollectionSize);
93+
return PythonCollectionType.CreateSet(Module, contents, exact: expression.Items.Count <= MaxCollectionSize);
9494
}
9595

9696
public IMember GetValueFromGenerator(GeneratorExpression expression) {
@@ -110,14 +110,14 @@ public IMember GetValueFromComprehension(Comprehension node) {
110110
switch (node) {
111111
case ListComprehension lc:
112112
var v1 = GetValueFromExpression(lc.Item) ?? UnknownType;
113-
return PythonCollectionType.CreateList(Interpreter, new[] { v1 });
113+
return PythonCollectionType.CreateList(Module, new[] { v1 });
114114
case SetComprehension sc:
115115
var v2 = GetValueFromExpression(sc.Item) ?? UnknownType;
116-
return PythonCollectionType.CreateSet(Interpreter, new[] { v2 });
116+
return PythonCollectionType.CreateSet(Module, new[] { v2 });
117117
case DictionaryComprehension dc:
118118
var k = GetValueFromExpression(dc.Key) ?? UnknownType;
119119
var v = GetValueFromExpression(dc.Value) ?? UnknownType;
120-
return new PythonDictionary(new PythonDictionaryType(Interpreter), new Dictionary<IMember, IMember> { { k, v } });
120+
return new PythonDictionary(new PythonDictionaryType(Interpreter.ModuleResolution.BuiltinsModule), new Dictionary<IMember, IMember> { { k, v } });
121121
}
122122

123123
return UnknownType;

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Constants.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323

2424
namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
2525
internal sealed partial class ExpressionEval {
26-
public IPythonInstance GetConstantFromLiteral(Expression expr, LookupOptions options) {
26+
public IPythonInstance GetConstantFromLiteral(Expression expr) {
2727
if (expr is ConstantExpression ce) {
2828
switch (ce.Value) {
2929
case string s:

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs

Lines changed: 15 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -62,11 +62,12 @@ private IMember GetValueFromGeneric(IMember target, Expression expr) {
6262
}
6363

6464
/// <summary>
65-
/// Returns whether the arguments to Generic are valid
65+
/// Determines if arguments to Generic are valid
6666
/// </summary>
67+
// TODO: move check to GenericClassBase. This requires extensive changes to SpecificTypeConstructor.
6768
private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeParameter> genericTypeArgs, IReadOnlyList<IMember> args, Expression expr) {
68-
// All arguments to Generic must be type parameters
69-
// e.g. Generic[T, str] throws a runtime error
69+
// All arguments to Generic must be type parameters
70+
// e.g. Generic[T, str] throws a runtime error
7071
if (genericTypeArgs.Count != args.Count) {
7172
ReportDiagnostics(Module.Uri, new DiagnosticsEntry(
7273
Resources.GenericNotAllTypeParameters,
@@ -77,14 +78,14 @@ private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeParameter> gen
7778
return false;
7879
}
7980

80-
// All arguments to Generic must be distinct
81+
// All arguments to Generic must be distinct
8182
if (genericTypeArgs.Distinct().Count() != genericTypeArgs.Count) {
8283
ReportDiagnostics(Module.Uri, new DiagnosticsEntry(
83-
Resources.GenericNotAllUnique,
84-
GetLocation(expr).Span,
85-
ErrorCodes.TypingGenericArguments,
86-
Severity.Warning,
87-
DiagnosticSource.Analysis));
84+
Resources.GenericNotAllUnique,
85+
GetLocation(expr).Span,
86+
ErrorCodes.TypingGenericArguments,
87+
Severity.Warning,
88+
DiagnosticSource.Analysis));
8889
return false;
8990
}
9091

@@ -97,24 +98,19 @@ private bool GenericClassParameterValid(IReadOnlyList<IGenericTypeParameter> gen
9798
/// (if the former) on specific type (if the latter).
9899
/// </summary>
99100
private IMember CreateSpecificTypeFromIndex(IGenericType gt, IReadOnlyList<IMember> args, Expression expr) {
101+
// TODO: move check to GenericClassBase. This requires extensive changes to SpecificTypeConstructor.
100102
if (gt.Name.EqualsOrdinal("Generic")) {
101103
var genericTypeArgs = args.OfType<IGenericTypeParameter>().ToArray();
102104
if (!GenericClassParameterValid(genericTypeArgs, args, expr)) {
103105
return UnknownType;
104106
}
105-
106-
// Generic[T1, T2, ...] expression. Create generic base for the class.
107-
return new GenericClassParameter(genericTypeArgs, Module);
108-
}
109-
110-
// For other types just use supplied arguments
111-
if (args.Count > 0) {
112-
return gt.CreateSpecificType(new ArgumentSet(args, expr, this));
107+
// Generic[T1, T2, ...] expression. Create generic base for the class.
108+
return new GenericClassBase(genericTypeArgs, Module.Interpreter);
113109
}
114110

115-
return UnknownType;
111+
// For other types just use supplied arguments
112+
return args.Count > 0 ? gt.CreateSpecificType(new ArgumentSet(args, expr, this)) : UnknownType;
116113
}
117-
118114
private IReadOnlyList<IMember> EvaluateIndex(IndexExpression expr) {
119115
var indices = new List<IMember>();
120116
if (expr.Index is TupleExpression tex) {

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Operators.cs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,9 +105,9 @@ private IMember GetValueFromBinaryOp(Expression expr) {
105105

106106
switch (leftTypeId) {
107107
case BuiltinTypeId.List:
108-
return PythonCollectionType.CreateConcatenatedList(Module.Interpreter, lc, rc);
108+
return PythonCollectionType.CreateConcatenatedList(Module, lc, rc);
109109
case BuiltinTypeId.Tuple:
110-
return PythonCollectionType.CreateConcatenatedTuple(Module.Interpreter, lc, rc);
110+
return PythonCollectionType.CreateConcatenatedTuple(Module, lc, rc);
111111
}
112112
}
113113

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs

Lines changed: 18 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ public ExpressionEval(IServiceContainer services, IPythonModule module, PythonAs
5555
public ModuleSymbolTable SymbolTable { get; } = new ModuleSymbolTable();
5656
public IPythonType UnknownType => Interpreter.UnknownType;
5757
public Location DefaultLocation { get; }
58+
public IPythonModule BuiltinsModule => Interpreter.ModuleResolution.BuiltinsModule;
5859

5960
public LocationInfo GetLocationInfo(Node node) => node?.GetLocation(this) ?? LocationInfo.Empty;
6061

@@ -188,7 +189,7 @@ public IMember GetValueFromExpression(Expression expr, LookupOptions options = L
188189
m = null;
189190
break;
190191
default:
191-
m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr, options);
192+
m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr);
192193
break;
193194
}
194195
if (m == null) {
@@ -236,29 +237,32 @@ private IMember GetValueFromMember(MemberExpression expr) {
236237
return null;
237238
}
238239

239-
IPythonInstance instance = null;
240240
var m = GetValueFromExpression(expr.Target);
241-
if (m is IPythonType typeInfo) {
242-
var member = typeInfo.GetMember(expr.Name);
243-
// If container is class/type info rather than the instance, then the method is an unbound function.
244-
// Example: C.f where f is a method of C. Compare to C().f where f is bound to the instance of C.
245-
if (member is PythonFunctionType f && !f.IsStatic && !f.IsClassMethod) {
246-
f.AddReference(GetLocationOfName(expr));
247-
return f.ToUnbound();
248-
}
249-
instance = new PythonInstance(typeInfo);
241+
if (m == null) {
242+
return UnknownType;
250243
}
251244

252-
instance = instance ?? m as IPythonInstance;
253-
var type = m?.GetPythonType(); // Try inner type
254-
245+
var type = m.GetPythonType();
255246
var value = type?.GetMember(expr.Name);
256247
type?.AddMemberReference(expr.Name, this, GetLocationOfName(expr));
257248

258249
if (type is IPythonModule) {
259250
return value;
260251
}
261252

253+
IPythonInstance instance = null;
254+
if (m == type) {
255+
// If container is class/type info rather than the instance, then the method is an unbound function.
256+
// Example: C.f where f is a method of C. Compare to C().f where f is bound to the instance of C.
257+
if (value is PythonFunctionType f && f.DeclaringType != null && !f.IsStatic && !f.IsClassMethod) {
258+
f.AddReference(GetLocationOfName(expr));
259+
return f.ToUnbound();
260+
}
261+
instance = new PythonInstance(type);
262+
}
263+
264+
instance = instance ?? m as IPythonInstance;
265+
262266
// Class type GetMember returns a type. However, class members are
263267
// mostly instances (consider self.x = 1, x is an instance of int).
264268
// However, it is indeed possible to have them as types, like in

src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -58,14 +58,14 @@ public void HandleAssignment(AssignmentStatement node) {
5858
}
5959

6060
foreach (var ne in node.Left.OfType<NameExpression>()) {
61+
IScope scope;
6162
if (Eval.CurrentScope.NonLocals[ne.Name] != null) {
62-
Eval.LookupNameInScopes(ne.Name, out var scope, LookupOptions.Nonlocal);
63+
Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Nonlocal);
6364
scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne));
6465
continue;
6566
}
66-
6767
if (Eval.CurrentScope.Globals[ne.Name] != null) {
68-
Eval.LookupNameInScopes(ne.Name, out var scope, LookupOptions.Global);
68+
Eval.LookupNameInScopes(ne.Name, out scope, LookupOptions.Global);
6969
scope?.Variables[ne.Name].Assign(value, Eval.GetLocationOfName(ne));
7070
continue;
7171
}

src/Analysis/Ast/Impl/Analyzer/Handlers/ConditionalHandler.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,6 @@ public ConditionalHandler(AnalysisWalker walker) : base(walker) {
2525
}
2626

2727
public bool HandleIf(IfStatement node)
28-
=> node.WalkIfWithSystemConditions(Walker, Ast.LanguageVersion, _platformService.IsWindows);
28+
=> node.WalkIfWithSystemConditions(Walker, Ast.LanguageVersion, _platformService);
2929
}
3030
}

src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -68,11 +68,13 @@ private void AssignVariables(FromImportStatement node, IImportSearchResult impor
6868
if (!string.IsNullOrEmpty(memberName)) {
6969
var nameExpression = asNames[i] ?? names[i];
7070
var variableName = nameExpression?.Name ?? memberName;
71-
var variable = variableModule.Analysis?.GlobalScope?.Variables[memberName];
72-
var exported = variable ?? variableModule.GetMember(memberName);
73-
var value = exported ?? GetValueFromImports(variableModule, imports as IImportChildrenSource, memberName);
74-
// Do not allow imported variables to override local declarations
75-
Eval.DeclareVariable(variableName, value, VariableSource.Import, nameExpression, CanOverwriteVariable(variableName, node.StartIndex));
71+
if (!string.IsNullOrEmpty(variableName)) {
72+
var variable = variableModule.Analysis?.GlobalScope?.Variables[memberName];
73+
var exported = variable ?? variableModule.GetMember(memberName);
74+
var value = exported ?? GetValueFromImports(variableModule, imports as IImportChildrenSource, memberName);
75+
// Do not allow imported variables to override local declarations
76+
Eval.DeclareVariable(variableName, value, VariableSource.Import, nameExpression, CanOverwriteVariable(variableName, node.StartIndex));
77+
}
7678
}
7779
}
7880
}
@@ -84,7 +86,7 @@ private void HandleModuleImportStar(PythonVariableModule variableModule, bool is
8486
}
8587

8688
// If __all__ is present, take it, otherwise declare all members from the module that do not begin with an underscore.
87-
var memberNames = isImplicitPackage
89+
var memberNames = isImplicitPackage
8890
? variableModule.GetMemberNames()
8991
: variableModule.Analysis.StarImportMemberNames ?? variableModule.GetMemberNames().Where(s => !s.StartsWithOrdinal("_"));
9092

@@ -109,7 +111,7 @@ private void HandleModuleImportStar(PythonVariableModule variableModule, bool is
109111

110112
private bool CanOverwriteVariable(string name, int importPosition) {
111113
var v = Eval.CurrentScope.Variables[name];
112-
if(v == null) {
114+
if (v == null) {
113115
return true; // Variable does not exist
114116
}
115117
// Allow overwrite if import is below the variable. Consider
@@ -118,10 +120,10 @@ private bool CanOverwriteVariable(string name, int importPosition) {
118120
// from A import * # brings another x
119121
// x = 3
120122
var references = v.References.Where(r => r.DocumentUri == Module.Uri).ToArray();
121-
if(references.Length == 0) {
123+
if (references.Length == 0) {
122124
// No references to the variable in this file - the variable
123125
// is imported from another module. OK to overwrite.
124-
return true;
126+
return true;
125127
}
126128
var firstAssignmentPosition = references.Min(r => r.Span.ToIndexSpan(Ast).Start);
127129
return firstAssignmentPosition < importPosition;
@@ -151,7 +153,7 @@ private void SpecializeFuture(FromImportStatement node) {
151153
var printNameExpression = node.Names.FirstOrDefault(n => n?.Name == "print_function");
152154
if (printNameExpression != null) {
153155
var fn = new PythonFunctionType("print", new Location(Module), null, string.Empty);
154-
var o = new PythonFunctionOverload(fn.Name, new Location(Module));
156+
var o = new PythonFunctionOverload(fn, new Location(Module));
155157
var parameters = new List<ParameterInfo> {
156158
new ParameterInfo("*values", Interpreter.GetBuiltinType(BuiltinTypeId.Object), ParameterKind.List, null),
157159
new ParameterInfo("sep", Interpreter.GetBuiltinType(BuiltinTypeId.Str), ParameterKind.KeywordOnly, null),

src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ private PythonVariableModule GetOrCreateVariableModule(in string fullName, in Py
180180
return variableModule;
181181
}
182182

183-
variableModule = new PythonVariableModule(fullName, Interpreter);
183+
variableModule = new PythonVariableModule(fullName, Eval.Interpreter);
184184
_variableModules[fullName] = variableModule;
185185
parentModule?.AddChildModule(memberName, variableModule);
186186
return variableModule;

src/Analysis/Ast/Impl/Analyzer/LibraryAnalysis.cs

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,9 @@
1515

1616
using System.Collections.Generic;
1717
using System.Linq;
18-
using Microsoft.Python.Analysis.Analyzer.Evaluation;
1918
using Microsoft.Python.Analysis.Diagnostics;
2019
using Microsoft.Python.Analysis.Documents;
21-
using Microsoft.Python.Analysis.Modules;
2220
using Microsoft.Python.Analysis.Values;
23-
using Microsoft.Python.Core;
2421
using Microsoft.Python.Core.Diagnostics;
2522
using Microsoft.Python.Parsing.Ast;
2623

0 commit comments

Comments
 (0)