Skip to content
This repository was archived by the owner on Apr 14, 2022. It is now read-only.

Commit 2143719

Browse files
author
Mikhail Arkhipov
authored
Multiple fixes and additional tests for persistence (#1372)
* Remove old qualified name * Node storage * Class and scope to use AST map * Library analysis * Fix SO * Keep small AST with imports * AST reduction * Final field * Initial * Reload * Ignore post-final requests * Drop AST * Remove local variables * Test fixes * Fix overload match * Tests * Add locks * Remove local variables * Drop file content to save memory * Cache PEP hints * Recreate AST * Fix specialization * Fix locations * usings * Test fixes * Add options to keep data in memory * Fix test * Fix lambda parameters * Fix argument set Fix global scope node * Fix overload doc * Fix stub merge errors * Fix async issues * Undo some changes * Fix test * Fix race condition * Partial * Models and views * Restore log null checks * Fix merge conflict * Fix merge issue * Null check * Partial * Partial * Partial * Fix test * Partial * Partial * First test * Baseline comparison * Builtins * Partial * Type fixes * Fix type names, part I * Qualified name * Properly write variables * Partial * Construct module from model * Test * Variable creations * Factories * Factories * Split construction * Restore * Save builtins * Test passes * Qualified name * Better export detection * Test fixes * More consistent qualified names * Sys test * Demo * Complete sys write/read * Partial * Partial * Test staility * Perf bug * Baseline, remove debug code, deactivate db * Test fixes * Test fix * Simplify a bit * Baselines and use : separator * Baselines * PR feedback * Merge master * Remove registry reference * PR feedback * PR feedback * Restore persistence + update test * Better handle persistent module in dependencies * Undo * Add location converter abstraction * Store member location * Fix merge issue * Basic locations test * Navigation * Add test * Update baselines * Type restore - initial * Remove debug code * Partial * Fix stub merge * Various model fixes * Improve module handling * Qualified name improvements * Fix unbound case * Improve stub merge * Fix qualified names of typing * Handle stub-only modules * Add tests for io, re and sys * Better handle named tuple * Handle module circular references * Handle else in platform and version clauses + handle it in symbol collector * Formatting * Fix walk of multi-level if statement * Unify package search in imports * Fix tests * Undo change * Port changes from dbtype * Partial * Named tuple support * Baseline updates * Debug code * Support types * Properly compare class member declaring type * Nested classes and functions persistence * Undo debug * Fix numpy restore * Baselines * Fix tests * Update AnyStr test reflecting changes to AnyStr behavior * Exclude baselines from git diff * Fix gitattr * Move git setting to root * Try no path * Add RDT count to the analysis_complete event (#1396) * Add RDT count to the analysis_complete event * Use property instead of method * Test fixes * Undo change * Additional stub merge fixes * Baseline updates * Fix goto def * Protect specific type creation * Track documentaton source * More reliable tests + simplification * Prevent crashes when the AST happens to be null (#1405) * Remove all references to dropped modules and force GC on reload (#1402) * First attempt at ensuring moduled dropped during reload are GC'd * Make sure AST is set after content is cleared * More tests passing * ModuleWalker updates * Also don't merge against specialized modules, do builtin _astMap reset before reanalyzing instead of after any analysis * Don't call dispose directly, it's done elsewhere and has no effect * Force GC via normal session instead of waiting for a hardcoded time * Un-refactor function since it's not being called externally anymore * Formatting/usings * PR feedback * Undo IsBuiltin, some things continued to be referenced with that changed * Move reload logic down into PythonAnalyzer, remove added public inferface items * Mode AddReference to PythonType as override * Remove dead code, make all ResetAnalyzer calls full * Typo * Cleanup * Basic classification * Fix merge error * Module unique id fixes * Stricted check to save analysis * Revert "Fix tests" This reverts commit 247a8c3. * Revert "Unify package search in imports" This reverts commit 67fed10. * Don't clear scope variables with inner classes * Fix typo
1 parent a502cb5 commit 2143719

File tree

106 files changed

+1955
-74710
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

106 files changed

+1955
-74710
lines changed

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.json -diff

‎src/Analysis/Ast/Impl/Analyzer/Definitions/IPythonAnalyzer.cs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,11 @@ public interface IPythonAnalyzer {
5656
/// </summary>
5757
IReadOnlyList<DiagnosticsEntry> LintModule(IPythonModule module);
5858

59+
5960
/// <summary>
60-
/// Removes all the modules from the analysis, except Typeshed and builtin
61+
/// Removes all the modules from the analysis and restarts it, including stubs.
6162
/// </summary>
62-
void ResetAnalyzer();
63+
Task ResetAnalyzer();
6364

6465
/// <summary>
6566
/// Returns list of currently loaded modules.

‎src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Constants.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323

2424
namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
2525
internal sealed partial class ExpressionEval {
26-
public IPythonInstance GetConstantFromLiteral(Expression expr, LookupOptions options) {
26+
public IPythonInstance GetConstantFromLiteral(Expression expr) {
2727
if (expr is ConstantExpression ce) {
2828
switch (ce.Value) {
2929
case string s:

‎src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs‎

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@ public IMember GetValueFromExpression(Expression expr, LookupOptions options = L
189189
m = null;
190190
break;
191191
default:
192-
m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr, options);
192+
m = GetValueFromBinaryOp(expr) ?? GetConstantFromLiteral(expr);
193193
break;
194194
}
195195
if (m == null) {
@@ -237,29 +237,32 @@ private IMember GetValueFromMember(MemberExpression expr) {
237237
return null;
238238
}
239239

240-
IPythonInstance instance = null;
241240
var m = GetValueFromExpression(expr.Target);
242-
if (m is IPythonType typeInfo) {
243-
var member = typeInfo.GetMember(expr.Name);
244-
// If container is class/type info rather than the instance, then the method is an unbound function.
245-
// Example: C.f where f is a method of C. Compare to C().f where f is bound to the instance of C.
246-
if (member is PythonFunctionType f && !f.IsStatic && !f.IsClassMethod) {
247-
f.AddReference(GetLocationOfName(expr));
248-
return f.ToUnbound();
249-
}
250-
instance = new PythonInstance(typeInfo);
241+
if (m == null) {
242+
return UnknownType;
251243
}
252244

253-
instance = instance ?? m as IPythonInstance;
254-
var type = m?.GetPythonType(); // Try inner type
255-
245+
var type = m.GetPythonType();
256246
var value = type?.GetMember(expr.Name);
257247
type?.AddMemberReference(expr.Name, this, GetLocationOfName(expr));
258248

259249
if (type is IPythonModule) {
260250
return value;
261251
}
262252

253+
IPythonInstance instance = null;
254+
if (m == type) {
255+
// If container is class/type info rather than the instance, then the method is an unbound function.
256+
// Example: C.f where f is a method of C. Compare to C().f where f is bound to the instance of C.
257+
if (value is PythonFunctionType f && f.DeclaringType != null && !f.IsStatic && !f.IsClassMethod) {
258+
f.AddReference(GetLocationOfName(expr));
259+
return f.ToUnbound();
260+
}
261+
instance = new PythonInstance(type);
262+
}
263+
264+
instance = instance ?? m as IPythonInstance;
265+
263266
// Class type GetMember returns a type. However, class members are
264267
// mostly instances (consider self.x = 1, x is an instance of int).
265268
// However, it is indeed possible to have them as types, like in

‎src/Analysis/Ast/Impl/Analyzer/Handlers/ConditionalHandler.cs‎

Lines changed: 2 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -13,123 +13,18 @@
1313
// See the Apache Version 2.0 License for specific language governing
1414
// permissions and limitations under the License.
1515

16-
using System;
17-
using System.Linq;
1816
using Microsoft.Python.Core.OS;
19-
using Microsoft.Python.Parsing;
2017
using Microsoft.Python.Parsing.Ast;
2118

2219
namespace Microsoft.Python.Analysis.Analyzer.Handlers {
2320
internal sealed class ConditionalHandler : StatementHandler {
2421
private readonly IOSPlatform _platformService;
2522

26-
private enum ConditionTestResult {
27-
Unrecognized,
28-
DontWalkBody,
29-
WalkBody
30-
}
31-
3223
public ConditionalHandler(AnalysisWalker walker) : base(walker) {
3324
_platformService = Eval.Services.GetService<IOSPlatform>();
3425
}
3526

36-
public bool HandleIf(IfStatement node) {
37-
// System version, platform and os.path specializations
38-
var someRecognized = false;
39-
foreach (var test in node.Tests) {
40-
var result = TryHandleSysVersionInfo(test);
41-
if (result != ConditionTestResult.Unrecognized) {
42-
if (result == ConditionTestResult.WalkBody) {
43-
test.Walk(Walker);
44-
}
45-
someRecognized = true;
46-
continue;
47-
}
48-
49-
result = TryHandleSysPlatform(test);
50-
if (result != ConditionTestResult.Unrecognized) {
51-
if (result == ConditionTestResult.WalkBody) {
52-
test.Walk(Walker);
53-
}
54-
someRecognized = true;
55-
continue;
56-
}
57-
58-
result = TryHandleOsPath(test);
59-
if (result != ConditionTestResult.Unrecognized) {
60-
if (result == ConditionTestResult.WalkBody) {
61-
test.Walk(Walker);
62-
return false; // Execute only one condition.
63-
}
64-
someRecognized = true;
65-
}
66-
}
67-
return !someRecognized;
68-
}
69-
70-
private ConditionTestResult TryHandleSysVersionInfo(IfStatementTest test) {
71-
if (test.Test is BinaryExpression cmp &&
72-
cmp.Left is MemberExpression me && (me.Target as NameExpression)?.Name == "sys" && me.Name == "version_info" &&
73-
cmp.Right is TupleExpression te && te.Items.All(i => (i as ConstantExpression)?.Value is int)) {
74-
Version v;
75-
try {
76-
v = new Version(
77-
(int)((te.Items.ElementAtOrDefault(0) as ConstantExpression)?.Value ?? 0),
78-
(int)((te.Items.ElementAtOrDefault(1) as ConstantExpression)?.Value ?? 0)
79-
);
80-
} catch (ArgumentException) {
81-
// Unsupported comparison, so walk all children
82-
return ConditionTestResult.WalkBody;
83-
}
84-
85-
var shouldWalk = false;
86-
switch (cmp.Operator) {
87-
case PythonOperator.LessThan:
88-
shouldWalk = Ast.LanguageVersion.ToVersion() < v;
89-
break;
90-
case PythonOperator.LessThanOrEqual:
91-
shouldWalk = Ast.LanguageVersion.ToVersion() <= v;
92-
break;
93-
case PythonOperator.GreaterThan:
94-
shouldWalk = Ast.LanguageVersion.ToVersion() > v;
95-
break;
96-
case PythonOperator.GreaterThanOrEqual:
97-
shouldWalk = Ast.LanguageVersion.ToVersion() >= v;
98-
break;
99-
}
100-
return shouldWalk ? ConditionTestResult.WalkBody : ConditionTestResult.DontWalkBody;
101-
}
102-
return ConditionTestResult.Unrecognized;
103-
}
104-
105-
private ConditionTestResult TryHandleSysPlatform(IfStatementTest test) {
106-
if (test.Test is BinaryExpression cmp &&
107-
cmp.Left is MemberExpression me && (me.Target as NameExpression)?.Name == "sys" && me.Name == "platform" &&
108-
cmp.Right is ConstantExpression cex && cex.GetStringValue() is string s) {
109-
switch (cmp.Operator) {
110-
case PythonOperator.Equals:
111-
return s == "win32" && _platformService.IsWindows ? ConditionTestResult.WalkBody : ConditionTestResult.DontWalkBody;
112-
case PythonOperator.NotEquals:
113-
return s == "win32" && _platformService.IsWindows ? ConditionTestResult.DontWalkBody : ConditionTestResult.WalkBody;
114-
}
115-
return ConditionTestResult.DontWalkBody;
116-
}
117-
return ConditionTestResult.Unrecognized;
118-
}
119-
120-
private ConditionTestResult TryHandleOsPath(IfStatementTest test) {
121-
if (test.Test is BinaryExpression cmp &&
122-
cmp.Left is ConstantExpression cex && cex.GetStringValue() is string s &&
123-
cmp.Right is NameExpression nex && nex.Name == "_names") {
124-
switch (cmp.Operator) {
125-
case PythonOperator.In when s == "nt":
126-
return _platformService.IsWindows ? ConditionTestResult.WalkBody : ConditionTestResult.DontWalkBody;
127-
case PythonOperator.In when s == "posix":
128-
return _platformService.IsWindows ? ConditionTestResult.DontWalkBody : ConditionTestResult.WalkBody;
129-
}
130-
return ConditionTestResult.DontWalkBody;
131-
}
132-
return ConditionTestResult.Unrecognized;
133-
}
27+
public bool HandleIf(IfStatement node)
28+
=> node.WalkIfWithSystemConditions(Walker, Ast.LanguageVersion, _platformService.IsWindows);
13429
}
13530
}

‎src/Analysis/Ast/Impl/Analyzer/LibraryAnalysis.cs‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,9 @@
1515

1616
using System.Collections.Generic;
1717
using System.Linq;
18-
using Microsoft.Python.Analysis.Analyzer.Evaluation;
1918
using Microsoft.Python.Analysis.Diagnostics;
2019
using Microsoft.Python.Analysis.Documents;
21-
using Microsoft.Python.Analysis.Modules;
2220
using Microsoft.Python.Analysis.Values;
23-
using Microsoft.Python.Core;
2421
using Microsoft.Python.Core.Diagnostics;
2522
using Microsoft.Python.Parsing.Ast;
2623

0 commit comments

Comments
 (0)