Linter for undefined variables

[MikhailArkhipov]

Fixes #639
Fixes #395
Fixes #461
Fixes #475
Fixes #476
Fixes #513
Fixes #682
Fixes #734

• Walk AST post analysis and use analysis to find out declared variables. Special filtering and handling of comprehensions. Includes Implement storing of variable definition locations #711
• Adds support for analysis of comprehensions. Fixes issue when comprehension variables were not declared in scope. Adds handling of tuple expressions in for in loops.
• Adds support for lambdas. Enables related tests.

[MikhailArkhipov]

Added settings plumbing to turn linter on/off via existing extension setting.

[jakebailey]

There are a few false positives, which can be seen in least_angle.py in the scikit-learn repo

• Builtin types (str, DeprecationWarning)
• Uses of function parameters.
• Some builtin functions like print, isinstance, abs.

[MikhailArkhipov]

Any specific for function arguments?

[jakebailey]

lars_path:

[jakebailey]

But:

def lars_path(X, y, Xy=None, Gram=None, max_iter=500,
              alpha_min=0, method='lar', copy_X=True,
              eps=np.finfo(np.float).eps,
              copy_Gram=True, verbose=0, return_path=True,
              return_n_iter=False, positive=False):

[jakebailey]

That seemed to fix many of the cases, but some still seem to be reported, like:

method is still reported (it's also a param), but X and y aren't anymore with that commit. alpha_min, copy_Gram are reported, and drop is too though it's a local variable and not a parameter, defined as drop = False at the topmost level of that function.

[MikhailArkhipov]

Yes, analysis was missing handling of comprehension (list) statements, handling of tuple expressions in for loops as well as handling of lambdas. Basically analysis issues, less of that of the linter. Related #734 which also requires the above.

[jakebailey]

Working a lot better now, one more thing in scikit-learn's k_means_.py:

Which is caused by a pyx import:

However if I try to make a minimal example where I do from idontexist import foo and use foo, it doesn't warn.

[jakebailey]

This one seems to be tuple/iterator related as well (hierarchical.py next to k_means_.py):

[MikhailArkhipov]

Speaking of imports @AlexanderSher can comment. I think there are different cases when imported variable was declared as unknown (and hence IS defined technically) vs code didn't even reach state of declarations as, I guess, happens with pyx.

I'll look into the iterator

[AlexanderSher]

Which is caused by a pyx import:

Yes, we don't support CPython.

[jakebailey]

Right, but I think what is weird is that in one case, if the imported module doesn't exist like from idontexist import foo, foo is declared, but when it's idontexist.pyx or similar, then doing the same thing causes foo to not be declared.