Enabling pylint creates excessive amounts of python instances

[james-womack]

Environment data

• VS Code version: 1.30.2
• Extension version (available under the Extensions sidebar): 2019.1.0
• OS and version: Linux Mint 19.1
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pylint 2.2.2

Expected behaviour

To only have one python process running pylint for each file open.

Actual behaviour

Hundreds of pylint processes open.

Steps to reproduce:

Not exactly sure, only noticed when computer started swapping to disk and check memory usage, but general steps to start having the processes spawn.

1. Open and close python files.
2. Enable pylint.

More python processes running pylint will open as time goes on and files get closed/opened. The processes don't end when the file is closed.

[DonJayamanne]

How large is your demo_multi_write.py file?

[james-womack]

@DonJayamanne Currently, it is 265 Lines. The largest file I'm working on currently is 1200 LoC

[james-womack]

Another way to reproduce:

VSCode creates another lint process every time I save (since I have lint on save enabled) that doesn't end after it finishes linting the file:

After saving a new process(s) is created that doesn't go away, even after closing VSCode:

I can disable lint on save to alleviate the issue for now

[DonJayamanne]

Thanks for the info, helps us nail it.
Please do use the work around for now. Will be fixed soon.

[alexzorin]

I have also encountered this just now in VSCode 1.36.1 and VSCode-Python 2019.6.24221. The pylint processes survive even when VSCode is fully terminated.

Performing an strace on the orphaned processes, they are stuck on either read or accept syscalls against a unix socket fd in /tmp:

$ sudo strace -fff -s 1024 -p 21308
[pid 21309] accept4(7,  <unfinished ...>
$ sudo ls -lah /proc/21309/fd/7
lrwx------ 1 alex alex 64 Jul 22 08:43 /proc/21309/fd/7 -> 'socket:[20515340]'
$ grep 20515340  /proc/net/unix
0000000000000000: 00000002 00000000 00010000 0001 01 20515340 /tmp/pymp-owqzjfgd/listener-f61o8njk

[DonJayamanne]

Please could you provide some sample code that would replicate this issue.
Please try linting the same file in the terminal. Basically all we do is run the linter via the cli, if it doesn't complete then it shouldn't complete even via the cli.
We could probably add a timeout and kill these processes.

Again please try linting the file from the terminal, and provide the results (does it complete or not) and provide the comments if this file do we can replicate it at our end.

Here's the cli python -m pylint <file path>

[alexzorin]

Sure, happy to try and help you reproduce it. I am working on https://github.com/certbot/certbot with a Python 3.7 virtual environment. The developer setup is documented here: https://certbot.eff.org/docs/contributing.html

I don't have a smaller repro case at the moment.

I noticed that it doesn't happen reliably - I think that there is either a timing or concurrency element to it. Most of the time the processes are cleaned up properly, but other times they will hang forever.

The most reliable way I know to reproduce it quickly is to rapidly indent/outdent a random line and save the document after every change. After 5-10 seconds of doing that, a bunch of pylint processes will stack up and hang.

Here are the stuck processes generated by VSCode:

alex     27641  0.0  0.2 283824 36076 ?        Sl   11:22   0:00 /home/alex/devel/certbot/venv/bin/python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /home/alex/devel/certbot/certbot/client.py
alex     27650  2.4  0.6 120040 101864 ?       S    11:22   0:03 /home/alex/devel/certbot/venv/bin/python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /home/alex/devel/certbot/certbot/client.py

If I run the command that you suggest, it is kind of slow but works okay and does not leave any processes behind:

$ time ./venv/bin/python -m pylint /home/alex/devel/certbot/certbot/client.py 1>/dev/null
./venv/bin/python -m pylint /home/alex/devel/certbot/certbot/client.py >   3.15s user 0.12s system 100% cpu 3.264 total

(Click for full output)

$ ./venv/bin/python -m pylint /home/alex/devel/certbot/certbot/client.py
************* Module certbot.client
certbot/client.py:410:0: W0311: Bad indentation. Found 16 spaces, expected 12 (bad-indentation)
certbot/client.py:100:0: R0205: Class 'DummyConfig' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
certbot/client.py:225:12: R1720: Unnecessary "else" after "raise" (no-else-raise)
certbot/client.py:237:0: R0205: Class 'Client' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
certbot/client.py:359:8: R1705: Unnecessary "else" after "return" (no-else-return)
certbot/client.py:437:8: R1705: Unnecessary "elif" after "return" (no-else-return)


Report
======
300 statements analysed.

Statistics by type
------------------

+---------+-------+-----------+-----------+------------+---------+
|type     |number |old number |difference |%documented |%badname |
+=========+=======+===========+===========+============+=========+
|module   |1      |1          |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|class    |2      |2          |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|method   |14     |14         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|function |11     |11         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+



External dependencies
---------------------
::

    OpenSSL (certbot.client)
    acme 
      \-client (certbot.client)
      \-crypto_util (certbot.client)
      \-errors (certbot.client)
      \-magic_typing (certbot.client)
      \-messages (certbot.client)
    certbot (certbot.client)
      \-account (certbot.client)
      \-auth_handler (certbot.client)
      \-cli (certbot.client)
      \-compat 
      | \-misc (certbot.client)
      | \-os (certbot.client)
      \-constants (certbot.client)
      \-crypto_util (certbot.client)
      \-display 
      | \-enhancements (certbot.client)
      | \-ops (certbot.client)
      \-eff (certbot.client)
      \-error_handler (certbot.client)
      \-errors (certbot.client)
      \-interfaces (certbot.client)
      \-plugins 
      | \-selection (certbot.client)
      \-reverter (certbot.client)
      \-storage (certbot.client)
      \-util (certbot.client)
    cryptography 
      \-hazmat 
        \-backends (certbot.client)
        \-primitives 
          \-asymmetric 
            \-rsa (certbot.client)
    josepy (certbot.client)
    zope 
      \-component (certbot.client)



Raw metrics
-----------

+----------+-------+------+---------+-----------+
|type      |number |%     |previous |difference |
+==========+=======+======+=========+===========+
|code      |360    |48.00 |360      |=          |
+----------+-------+------+---------+-----------+
|docstring |265    |35.33 |265      |=          |
+----------+-------+------+---------+-----------+
|comment   |43     |5.73  |43       |=          |
+----------+-------+------+---------+-----------+
|empty     |82     |10.93 |82       |=          |
+----------+-------+------+---------+-----------+



Duplication
-----------

+-------------------------+------+---------+-----------+
|                         |now   |previous |difference |
+=========================+======+=========+===========+
|nb duplicated lines      |0     |0        |=          |
+-------------------------+------+---------+-----------+
|percent duplicated lines |0.000 |0.000    |=          |
+-------------------------+------+---------+-----------+



Messages by category
--------------------

+-----------+-------+---------+-----------+
|type       |number |previous |difference |
+===========+=======+=========+===========+
|convention |0      |0        |=          |
+-----------+-------+---------+-----------+
|refactor   |5      |5        |=          |
+-----------+-------+---------+-----------+
|warning    |1      |1        |=          |
+-----------+-------+---------+-----------+
|error      |0      |0        |=          |
+-----------+-------+---------+-----------+



Messages
--------

+---------------------------+------------+
|message id                 |occurrences |
+===========================+============+
|useless-object-inheritance |2           |
+---------------------------+------------+
|no-else-return             |2           |
+---------------------------+------------+
|no-else-raise              |1           |
+---------------------------+------------+
|bad-indentation            |1           |
+---------------------------+------------+




------------------------------------------------------------------
Your code has been rated at 9.80/10 (previous run: 9.80/10, +0.00)

If I run the command that is verbatim generated by VS Code, it also works fine:

$ /home/alex/devel/certbot/venv/bin/python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /home/alex/devel/certbot/certbot/client.py
************* Module certbot.client
410,0,warning,bad-indentation:Bad indentation. Found 16 spaces, expected 12
100,0,refactor,useless-object-inheritance:Class 'DummyConfig' inherits from object, can be safely removed from bases in python3
225,12,refactor,no-else-raise:Unnecessary "else" after "raise"
237,0,refactor,useless-object-inheritance:Class 'Client' inherits from object, can be safely removed from bases in python3
359,8,refactor,no-else-return:Unnecessary "else" after "return"
437,8,refactor,no-else-return:Unnecessary "elif" after "return"

------------------------------------------------------------------
Your code has been rated at 9.80/10 (previous run: 9.80/10, +0.00)

(venv)

There might be a concurrency issue in pylint itself (if concurrency is indeed the issue), but perhaps VSCode could kill try and serialize the use of pylint to avoid running into the stuck processes?

[deitry]

Confirm issue.
My case is that processes are spawned for files that I work on hardly and therefore save them very often. And sometimes there are syntax errors. I suppose, pylint cannot parse file properly and stuck somehow.