Skip to content
This repository was archived by the owner on Apr 14, 2022. It is now read-only.

Scraper produces incorrect output for the Anaconda 2X builtin #719

Closed
CMLL opened this issue Mar 7, 2019 · 26 comments · Fixed by #723
Closed

Scraper produces incorrect output for the Anaconda 2X builtin #719

CMLL opened this issue Mar 7, 2019 · 26 comments · Fixed by #723
Assignees
@jakebailey
Labels
bug Something isn't working
Milestone
March 2019.1

Comments

@CMLL
Copy link

CMLL commented Mar 7, 2019

Testing the beta channel of the language server I ran into an issue starting the server. So far I've seen it happen with multiple projects a couple of seconds after a windows reload.

VSCode Info
Version: 1.32.0
Commit: 507312a3e3b34b084b467dfd983263bc7c9d87e0
Date: 2019-03-06T18:53:01.021Z
Electron: 3.1.3
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Linux x64 4.18.0-16-generic

Python 2.7.15
Virtualenv

[Error - 10:14:51 PM] Server initialization failed.
  Message: Object reference not set to an instance of an object.
  Code: -32000 
   at Microsoft.Python.Analysis.Modules.Resolution.MainModuleResolution.CreateModule(String name)
   at Microsoft.Python.Analysis.Modules.Resolution.ModuleResolutionBase.ModuleRef.GetOrCreate(String name, ModuleResolutionBase mrb)
   at Microsoft.Python.Analysis.Modules.Resolution.ModuleResolutionBase.GetOrLoadModule(String name)
   at Microsoft.Python.Analysis.Analyzer.PythonAnalyzer.LoadMissingDocuments(IPythonInterpreter interpreter, ImmutableArray`1 missingKeys)
   at Microsoft.Python.Analysis.Analyzer.PythonAnalyzer.AnalyzeDocumentAsync(ModuleKey key, PythonAnalyzerEntry entry, CancellationToken cancellationToken)
   at Microsoft.Python.Core.TaskExtensions.DoNotWait(Task task)
   at Microsoft.Python.Analysis.Analyzer.PythonAnalyzer.EnqueueDocumentForAnalysis(IPythonModule module, PythonAst ast, Int32 bufferVersion, CancellationToken cancellationToken)
   at Microsoft.Python.Analysis.Modules.PythonModule.Parse(CancellationToken cancellationToken)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Python.Analysis.Modules.PythonModule.GetAstAsync(CancellationToken cancellationToken)
   at Microsoft.Python.Analysis.Modules.PythonModule.LoadAndAnalyzeAsync(CancellationToken cancellationToken)
   at Microsoft.Python.Analysis.Modules.Resolution.MainModuleResolution.LoadBuiltinTypesAsync(CancellationToken cancellationToken)
   at Microsoft.Python.Analysis.Analyzer.PythonInterpreter.LoadBuiltinTypesAsync(String root, IServiceManager sm, CancellationToken cancellationToken)
   at Microsoft.Python.Analysis.Analyzer.PythonInterpreter.CreateAsync(InterpreterConfiguration configuration, String root, IServiceManager sm, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.InitializeAsync(InitializeParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.LanguageServer.Initialize(JToken token, CancellationToken cancellationToken)
@jakebailey
Copy link
Member

Do you have files open which are not saved to disk but are marked in VS Code as Python? (i.e. "Untitled")

Setting "python.analysis.logLevel": "Trace" may also provide some more details.

@CMLL
Copy link
Author

CMLL commented Mar 7, 2019

No. Closed all files and reloaded the window. After that closed the folder project and reopened and then the whole code instance. All times opening a python file would trigger the error.

I added Trace as the log level but doesn't seem to give more information apart from the search paths and the above stack trace.

@jakebailey
Copy link
Member

Can you share your search paths?

@CMLL
Copy link
Author

CMLL commented Mar 7, 2019

Sure.

Starting Microsoft Python language server.
[Info  - 10:34:49 PM] GetCurrentSearchPaths /home/cllamach/.virtualenvs/panopta_env/bin/python 
[Info  - 10:34:49 PM] SearchPaths:
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/_pdbpp_path_hack
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-dynload
[Info  - 10:34:49 PM]     /home/cllamach/.pyenv/versions/2.7.15/lib/python2.7
[Info  - 10:34:49 PM]     /home/cllamach/.pyenv/versions/2.7.15/lib/python2.7/plat-linux2
[Info  - 10:34:49 PM]     /home/cllamach/.pyenv/versions/2.7.15/lib/python2.7/lib-tk
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/TurboJinja-0.0.0-py2.7.egg
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/_pdbpp_path_hack;/home/cllamach/.virtualenvs/panopta_env/lib/python27.zip;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/plat-linux2;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-tk;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-old;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-dynload;/home/cllamach/.pyenv/versions/2.7.15/lib/python2.7;/home/cllamach/.pyenv/versions/2.7.15/lib/python2.7/plat-linux2;/home/cllamach/.pyenv/versions/2.7.15/lib/python2.7/lib-tk;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/Extremes-1.1.1-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/AddOns-0.7-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/DecoratorTools-1.8-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/BytecodeAssembler-0.6.1-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/SymbolType-1.0-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/PEAK_Rules-0.5a1.dev_r2700-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/TurboJinja-0.0.0-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages;
[Info  - 10:34:49 PM]     /home/cllamach/Panopta/classic/src/
[Info  - 10:34:49 PM]     /home/cllamach/Panopta/agent/src/linux
[Info  - 10:34:49 PM]     /home/cllamach/.virtualenvs/panopta_env/bin
[Error - 10:34:50 PM] Server initialization failed.

@CMLL
Copy link
Author

CMLL commented Mar 7, 2019

I tested with another smaller git repository I have and it there it didn't gave the above error so it must be related to a dependency in that one specific repository.

@jakebailey jakebailey mentioned this issue Mar 8, 2019
Closed
@CMLL
Copy link
Author

CMLL commented Mar 8, 2019 •
edited
Loading

Update on this. Switching from the virtualenv to the system python or another virtualenv doesn't trigger the error, which leads me to believe that a library in that virtualenv could be the cause. Sadly the virtualenv in question has 296 libraries installed, which it will make tracking the offending one hard.

One thing I'm noticing, is that when I switch to another virtualenv or system python, I get messages in the status bar, Analyzing files in the background, x remaining which I don't notice when I'm using the offending virtualenv, both in version 0.1.80 and 0.2.16.

Is there a cache that can be cleared, or something that could be analyzing old versions of the files? That's a long shot since most likely this is caused by a specific library.

@jakebailey
Copy link
Member

If initialization fails, it'll never get to the point where we send messages about the ongoing analysis, so that makes sense.

The only caching we have is of scraped files (for example, info extracted from compiled files), which you can delete by deleting the languageServer directory in the VS Code extension. However, if you look at your original stack trace, the error happens in the call chain for LoadBuiltinTypesAsync, which I believe happens before anything is scraped. There are also logging messages which would appear about scraping, which aren't there either. I don't think deleting anything that's cached will help in this case.

From looking through the code, I think this may be happening because _stubResolution is set in PythonInterpreter after LoadBuiltinTypesAsync is called; the latter references the former by calling CreateModule.

@jakebailey jakebailey added the bug Something isn't working label Mar 8, 2019
@jakebailey jakebailey added this to the March 2019.1 milestone Mar 8, 2019
@jakebailey jakebailey self-assigned this Mar 8, 2019
@jakebailey jakebailey mentioned this issue Mar 8, 2019
Merged
@AlexanderSher
Copy link
Contributor

@CMLL , can you look up for a python.pyi file and check if it accidentally has any imports inside it? It is a cache created for builtin types.

@MikhailArkhipov
Copy link 

 /home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/_pdbpp_path_hack;/home/cllamach/.virtualenvs/panopta_env/lib/python27.zip;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/plat-linux2;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-tk;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-old;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/lib-dynload;/home/cllamach/.pyenv/versions/2.7.15/lib/python2.7;/home/cllamach/.pyenv/versions/2.7.15/lib/python2.7/plat-linux2;/home/cllamach/.pyenv/versions/2.7.15/lib/python2.7/lib-tk;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/Extremes-1.1.1-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/AddOns-0.7-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/DecoratorTools-1.8-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/BytecodeAssembler-0.6.1-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/SymbolType-1.0-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/PEAK_Rules-0.5a1.dev_r2700-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/TurboJinja-0.0.0-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages/Jinja2-2.10-py2.7.egg;/home/cllamach/.virtualenvs/panopta_env/lib/python2.7/site-packages;

as in #720 and #726

jakebailey added a commit that referenced this issue Mar 11, 2019
@jakebailey
Create stub resolver before builtins (#723)
63b6e30 
Fixes #719.

Looking at the stack trace on that issue, `Initialize` occurs, which eventually gets an AST, which then spawns a task which eventually calls `CreateModule`, which refers back to `PythonInterpreter`, whose stub resolver has not yet been instantiated. I don't believe instantiating it earlier has any effect other than that it's there earlier so won't be `null`, as `InitializeAsync` only sets up an internal path resolver.

I'm not really sure if this is the "true" fix, since it seems weird for there to be this sort of cycle, but I'm not the most familiar with this specific bit.
@jakebailey
Copy link
Member

@CMLL This is theoretically fixed with that merged PR, so the issue was closed (we'll need to do a build), but knowing the answer to Alex's question is probably a good thing. You can find it somewhere in the Python extension directory (in one of the subdirectories with long names).

@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

@jakebailey That's great to hear. I'll be checking the beta channel for that new build to test it out.

@AlexanderSher I found several *.pyi files in /home/cllamach/.vscode/extensions/ms-python.python-2019.2.5558/languageServer.0.2.16/M2KcDmVDYL_Yq1CJMCom4n3OUETWJLIvUfy8ZvHg0Zw=
and in /home/cllamach/.vscode/extensions/ms-python.python-2019.2.5558/languageServer.0.2.16/bin/Release/Typeshed/third_party

Doing a grep for import shows several ones in those files. Is there a specific one I should be looking for? All of these files shouldn't have any import statement inside?

Example of the imports found in those files....

# These are found in the first path.
python._weakref.pyi
1:import __builtin__ as _mod___builtin__

python.time.pyi
1:import __builtin__ as _mod___builtin__

python._collections.pyi
1:import collections as _mod_collections

python._locale.pyi
1:import locale as _mod_locale

python.zlib.pyi
1:import exceptions as _mod_exceptions

python.binascii.pyi
1:import exceptions as _mod_exceptions

python._random.pyi
1:import __builtin__ as _mod___builtin__

python.errno.pyi
1:import __builtin__ as _mod___builtin__

python.select.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.datetime.pyi
1:import __builtin__ as _mod___builtin__

python.thread.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.array.pyi
1:import __builtin__ as _mod___builtin__

python.gc.pyi
1:import __builtin__ as _mod___builtin__

python.itertools.pyi
1:import __builtin__ as _mod___builtin__

python.operator.pyi
1:import __builtin__ as _mod___builtin__

python.pwd.pyi
1:import __builtin__ as _mod___builtin__

python.posix.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.cPickle.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.__main__.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python._io.pyi
1:import __builtin__ as _mod___builtin__
2:import io as _mod_io

python.signal.pyi
1:import exceptions as _mod_exceptions

python.exceptions.pyi
1:import __builtin__ as _mod___builtin__
281:    'Base class for warnings about probable mistakes in module imports'

python._md5.pyi
1:import __builtin__ as _mod___builtin__

python._functools.pyi
1:import functools as _mod_functools

python._warnings.pyi
1:import __builtin__ as _mod___builtin__

python._struct.pyi
1:import __builtin__ as _mod___builtin__
2:import struct as _mod_struct

# These are in second path.

2/six/moves/urllib/request.pyi
2:from urllib2 import urlopen as urlopen
3:from urllib2 import install_opener as install_opener
4:from urllib2 import build_opener as build_opener
5:from urllib import pathname2url as pathname2url
6:from urllib import url2pathname as url2pathname
7:from urllib import getproxies as getproxies
8:from urllib2 import Request as Request
9:from urllib2 import OpenerDirector as OpenerDirector
10:from urllib2 import HTTPDefaultErrorHandler as HTTPDefaultErrorHandler
11:from urllib2 import HTTPRedirectHandler as HTTPRedirectHandler
12:from urllib2 import HTTPCookieProcessor as HTTPCookieProcessor
13:from urllib2 import ProxyHandler as ProxyHandler
14:from urllib2 import BaseHandler as BaseHandler
15:from urllib2 import HTTPPasswordMgr as HTTPPasswordMgr
16:from urllib2 import HTTPPasswordMgrWithDefaultRealm as HTTPPasswordMgrWithDefaultRealm
17:from urllib2 import AbstractBasicAuthHandler as AbstractBasicAuthHandler
18:from urllib2 import HTTPBasicAuthHandler as HTTPBasicAuthHandler
19:from urllib2 import ProxyBasicAuthHandler as ProxyBasicAuthHandler
20:from urllib2 import AbstractDigestAuthHandler as AbstractDigestAuthHandler
21:from urllib2 import HTTPDigestAuthHandler as HTTPDigestAuthHandler
22:from urllib2 import ProxyDigestAuthHandler as ProxyDigestAuthHandler
23:from urllib2 import HTTPHandler as HTTPHandler
24:from urllib2 import HTTPSHandler as HTTPSHandler
25:from urllib2 import FileHandler as FileHandler
26:from urllib2 import FTPHandler as FTPHandler
27:from urllib2 import CacheFTPHandler as CacheFTPHandler
28:from urllib2 import UnknownHandler as UnknownHandler
29:from urllib2 import HTTPErrorProcessor as HTTPErrorProcessor
30:from urllib import urlretrieve as urlretrieve
31:from urllib import urlcleanup as urlcleanup
32:from urllib import URLopener as URLopener
33:from urllib import FancyURLopener as FancyURLopener
34:from urllib import proxy_bypass as proxy_bypass

2/six/moves/urllib/error.pyi
1:from urllib2 import URLError as URLError
2:from urllib2 import HTTPError as HTTPError
3:from urllib import ContentTooShortError as ContentTooShortError

@jakebailey
Copy link
Member

Ignore the typeshed directory, those are included from: https://github.com/python/typeshed

It's the one in the garbled directory (which is the hash of some info).

@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

Well, ignoring the ones in Typeshed then, these are the ones that have an import...

./M2KcDmVDYL_Yq1CJMCom4n3OUETWJLIvUfy8ZvHg0Zw=/
python.cStringIO.pyi
1:import __builtin__ as _mod___builtin__

python.binascii.pyi
1:import exceptions as _mod_exceptions

python.datetime.pyi
1:import __builtin__ as _mod___builtin__

python.errno.pyi
1:import __builtin__ as _mod___builtin__

python.thread.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.sys.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.time.pyi
1:import __builtin__ as _mod___builtin__

python._collections.pyi
1:import collections as _mod_collections

python._locale.pyi
1:import locale as _mod_locale

./CFsERjFzDo05GXUpq057csNnGePs5VnfNl-QPbOR97M=
cStringIO.pyi
1:import __builtin__ as _mod___builtin__

binascii.pyi
1:import exceptions as _mod_exceptions

_hashlib.pyi
1:import __builtin__ as _mod___builtin__

_random.pyi
1:import __builtin__ as _mod___builtin__

_io.pyi
1:import __builtin__ as _mod___builtin__
2:import io as _mod_io

time.pyi
1:import __builtin__ as _mod___builtin__

termios.pyi
1:import exceptions as _mod_exceptions

_ctypes.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

operator.pyi
1:import __builtin__ as _mod___builtin__

datetime.pyi
1:import __builtin__ as _mod___builtin__

_locale.pyi
1:import locale as _mod_locale

_collections.pyi
1:import collections as _mod_collections

itertools.pyi
1:import __builtin__ as _mod___builtin__

cPickle.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

select.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

_json.pyi
1:import __builtin__ as _mod___builtin__

_functools.pyi
1:import functools as _mod_functools

pyexpat.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

array.pyi
1:import __builtin__ as _mod___builtin__

_struct.pyi
1:import __builtin__ as _mod___builtin__
2:import struct as _mod_struct

./Ltpq9soHp56dLc3QeIhp5X5aMWCO1jH0o7dJycP0Nck=
python.pyi
1:import __builtin__ as _mod___builtin__
2:import future.types.newbytes as _mod_future_types_newbytes
3:import future.types.newrange as _mod_future_types_newrange
4:import itertools as _mod_itertools
75:__builtin_module_names__ = "__builtin_

./q6wyHsQIqEtTFeGHT3oVuUREcdVOADZ6Eb6wnIKIxV8=
python.signal.pyi
1:import exceptions as _mod_exceptions

python._warnings.pyi
1:import __builtin__ as _mod___builtin__

python.exceptions.pyi
1:import __builtin__ as _mod___builtin__
281:    'Base class for warnings about probable mistakes in module imports'

python.sys.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.errno.pyi
1:import __builtin__ as _mod___builtin__

python.posix.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.thread.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions

python.gc.pyi
1:import __builtin__ as _mod___builtin__

python._weakref.pyi
1:import __builtin__ as _mod___builtin__

python.__main__.pyi
1:import __builtin__ as _mod___builtin__
2:import exceptions as _mod_exceptions


python.imp.pyi
1:import __builtin__ as _mod___builtin__


python.pwd.pyi
1:import __builtin__ as _mod___builtin__

@AlexanderSher
Copy link
Contributor 

./Ltpq9soHp56dLc3QeIhp5X5aMWCO1jH0o7dJycP0Nck=
python.pyi
1:import __builtin__ as _mod___builtin__
2:import future.types.newbytes as _mod_future_types_newbytes
3:import future.types.newrange as _mod_future_types_newrange
4:import itertools as _mod_itertools
75:__builtin_module_names__ = "__builtin_

So this one is python.pyi in a Ltpq9soHp56dLc3QeIhp5X5aMWCO1jH0o7dJycP0Nck= folder?

@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

Correct, this is the start of that file...

  14 import __builtin__ as _mod___builtin__                                         
  13 import future.types.newbytes as _mod_future_types_newbytes                     
  12 import future.types.newrange as _mod_future_types_newrange                     
  11 import itertools as _mod_itertools                                             
  10                                                                                
   9 class __Unknown__:                                                             
   8     '<unknown>'                                                                
   7     __name__ = "<unknown>"                                                     
   6                                                                                
   5 class __NoneType__:                                                            
   4     'the type of the None object'                                              
   3                                                                                
   2 object = _mod___builtin__.object                                               
   1 __Object__ = object                                                            
15   type = _mod___builtin__.type

@AlexanderSher
Copy link
Contributor

Can you rename/remove this file from the cache (without deleting it) and try to run VSCode again? I want to compare with the newly created python.pyi.

@AlexanderSher AlexanderSher reopened this Mar 11, 2019
@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

Renamed the file and launched vscode again. Seems like the new file was named python2.pyi

python_pyi_files.zip

@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

Seems that the file only gets generated when I'm using the virtualenv that has the error. If I use a system python the python.pyi doesn't get generated.

@AlexanderSher
Copy link
Contributor

It is probably generated in another folder

@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

Yeah. Seems like it is. I checked this folder Z5B_gR6M7UL6d5wW-RhQYxwGMB-wEDy1So0V96O6Uh0= and indeed has a python.pyi file, that doesn't have any import statements at the top of the file, so that must be the new one that's being generated with system python.

~/.v/e/m/l/Z5B_gR6M7UL6d5wW-RhQYxwGMB-wEDy1So0V96O6Uh0= ➜ head python.pyi 
class NotImplementedType(object):
    __class__ = NotImplementedType
    def __init__(self):
        'x.__init__(...) initializes x; see help(type(x)) for signature'
        pass
    
    def __repr__(self):
        'x.__repr__() <==> repr(x)'
        return ''

@AlexanderSher
Copy link
Contributor

AlexanderSher commented Mar 11, 2019 •
edited
Loading

Looks like the problem is in the scraper. For the normal case, scraper should've created something like this:

class __Unknown__:
    '<unknown>'
    __name__ = "<unknown>"

class __NoneType__:
    'the type of the None object'

class object:
    'The most base type'
    __class__ = object
    def __delattr__(self):
        "x.__delattr__('name') <==> del x.name"
        return None
    
    def __format__(self, format_spec):
        'default object formatter'
        return ''
    
    def __getattribute__(self):
        "x.__getattribute__('name') <==> x.name"
        pass

In case of your virtualenv, it creates this:

class __Unknown__:
    '<unknown>'
    __name__ = "<unknown>"

class __NoneType__:
    'the type of the None object'

object = _mod___builtin__.object

@AlexanderSher
Copy link
Contributor

@CMLL , in the languageServer.0.2.16/bin/Release there should be scrape_module.py. Can you execute that script with -B -E into some output.py from your virtualenv? I want to check if the problem is with the script itself.

@AlexanderSher
Copy link
Contributor

Ok, I can reproduce it in anaconda 2.7

@CMLL
Copy link
Author

CMLL commented Mar 11, 2019

What are the arguments for scrape_module.py? Passing -B returns

scrape_module.py:872: InspectWarning: Working around No module named -B
  warnings.warn("Working around " + ex_msg, InspectWarning)
Traceback (most recent call last):
  File "scrape_module.py", line 1486, in <module>
    state.initial_import(sys.argv[2])
  File "scrape_module.py", line 869, in initial_import
    mod = __import__(self.module_name)
ImportError: No module named -B

And directly passing a python file or a folder containing python files returns

scrape_module.py:872: InspectWarning: Working around Import by filename is not supported.
  warnings.warn("Working around " + ex_msg, InspectWarning)
Traceback (most recent call last):
  File "scrape_module.py", line 1488, in <module>
    state.initial_import()
  File "scrape_module.py", line 869, in initial_import
    mod = __import__(self.module_name)
ImportError: Import by filename is not supported.

If I call the script without arguments I get the output, starting with

import __builtin__ as _mod___builtin__
import future.types.newbytes as _mod_future_types_newbytes
import future.types.newrange as _mod_future_types_newrange
import itertools as _mod_itertools

class __Unknown__:
    '<unknown>'
    __name__ = "<unknown>"

class __NoneType__:
    'the type of the None object'

object = _mod___builtin__.object
__Object__ = object
type = _mod___builtin__.type
__Type__ = type
int = _mod___builtin__.int
__Int__ = int
bool = _mod___builtin__.bool
__Bool__ = bool

@AlexanderSher AlexanderSher changed the title Language Server 0.2.16 Initialization failed. Scraper produces incorrect output for the Anaconda 2X builtin Mar 11, 2019
AlexanderSher added a commit to AlexanderSher/python-language-server that referenced this issue Mar 11, 2019
@AlexanderSher
Fix microsoft#719: Scraper produces incorrect output for the Anaconda…
043835a 
… 2X builtin
@CMLL
Copy link
Author

CMLL commented Mar 12, 2019 •
edited
Loading

Great that there is a fix for this.I downloaded the beta build this morning and noticed that it got bumped to 0.2.17. Does that include this commit? I could spin it around and test it with my virtualenv.

Is there a way to know up to what commit the beta branch is built?

Thanks for the fast fix.

Edit:
Fix seems to have worked,operating inside the virtualenv no longer fails when starting up the LS.

@jakebailey
Copy link
Member

It includes my PR, but not Alex's. His is a broader fix which helps in another way.

jakebailey added a commit to jakebailey/python-language-server that referenced this issue Nov 1, 2019
@jakebailey
Create stub resolver before builtins (microsoft#723)
cd6f824 
Fixes microsoft#719.

Looking at the stack trace on that issue, `Initialize` occurs, which eventually gets an AST, which then spawns a task which eventually calls `CreateModule`, which refers back to `PythonInterpreter`, whose stub resolver has not yet been instantiated. I don't believe instantiating it earlier has any effect other than that it's there earlier so won't be `null`, as `InitializeAsync` only sets up an internal path resolver.

I'm not really sure if this is the "true" fix, since it seems weird for there to be this sort of cycle, but I'm not the most familiar with this specific bit.
jakebailey pushed a commit to jakebailey/python-language-server that referenced this issue Nov 1, 2019
@AlexanderSher
Fix microsoft#719: Scraper produces incorrect output for the Anaconda…
7e01446 
… 2X builtin (microsoft#733)

* Fix microsoft#719: Scraper produces incorrect output for the Anaconda 2X builtin

* Add Anaconda 3.7 to the tests interpreters
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jakebailey jakebailey

Labels
bug Something isn't working
Projects
None yet
Milestone
March 2019.1
Development

Successfully merging a pull request may close this issue.

Create stub resolver before builtins jakebailey/python-language-server
4 participants
@CMLL @jakebailey @AlexanderSher @MikhailArkhipov