Enabling pylint creates excessive amounts of python instances

[james-womack]

Environment data

• VS Code version: 1.30.2
• Extension version (available under the Extensions sidebar): 2019.1.0
• OS and version: Linux Mint 19.1
• Python version (& distribution if applicable, e.g. Anaconda): 3.6.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pylint 2.2.2

Expected behaviour

To only have one python process running pylint for each file open.

Actual behaviour

Hundreds of pylint processes open.

Steps to reproduce:

Not exactly sure, only noticed when computer started swapping to disk and check memory usage, but general steps to start having the processes spawn.

1. Open and close python files.
2. Enable pylint.

More python processes running pylint will open as time goes on and files get closed/opened. The processes don't end when the file is closed.

[DonJayamanne]

How large is your demo_multi_write.py file?

[james-womack]

@DonJayamanne Currently, it is 265 Lines. The largest file I'm working on currently is 1200 LoC

[james-womack]

Another way to reproduce:

VSCode creates another lint process every time I save (since I have lint on save enabled) that doesn't end after it finishes linting the file:

After saving a new process(s) is created that doesn't go away, even after closing VSCode:

I can disable lint on save to alleviate the issue for now

[DonJayamanne]

Thanks for the info, helps us nail it.
Please do use the work around for now. Will be fixed soon.

[alexzorin]

I have also encountered this just now in VSCode 1.36.1 and VSCode-Python 2019.6.24221. The pylint processes survive even when VSCode is fully terminated.

Performing an strace on the orphaned processes, they are stuck on either read or accept syscalls against a unix socket fd in /tmp:

$ sudo strace -fff -s 1024 -p 21308
[pid 21309] accept4(7,  <unfinished ...>
$ sudo ls -lah /proc/21309/fd/7
lrwx------ 1 alex alex 64 Jul 22 08:43 /proc/21309/fd/7 -> 'socket:[20515340]'
$ grep 20515340  /proc/net/unix
0000000000000000: 00000002 00000000 00010000 0001 01 20515340 /tmp/pymp-owqzjfgd/listener-f61o8njk

[DonJayamanne]

Please could you provide some sample code that would replicate this issue.
Please try linting the same file in the terminal. Basically all we do is run the linter via the cli, if it doesn't complete then it shouldn't complete even via the cli.
We could probably add a timeout and kill these processes.

Again please try linting the file from the terminal, and provide the results (does it complete or not) and provide the comments if this file do we can replicate it at our end.

Here's the cli python -m pylint <file path>

[alexzorin]

Sure, happy to try and help you reproduce it. I am working on https://github.com/certbot/certbot with a Python 3.7 virtual environment. The developer setup is documented here: https://certbot.eff.org/docs/contributing.html

I don't have a smaller repro case at the moment.

I noticed that it doesn't happen reliably - I think that there is either a timing or concurrency element to it. Most of the time the processes are cleaned up properly, but other times they will hang forever.

The most reliable way I know to reproduce it quickly is to rapidly indent/outdent a random line and save the document after every change. After 5-10 seconds of doing that, a bunch of pylint processes will stack up and hang.

Here are the stuck processes generated by VSCode:

alex     27641  0.0  0.2 283824 36076 ?        Sl   11:22   0:00 /home/alex/devel/certbot/venv/bin/python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /home/alex/devel/certbot/certbot/client.py
alex     27650  2.4  0.6 120040 101864 ?       S    11:22   0:03 /home/alex/devel/certbot/venv/bin/python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /home/alex/devel/certbot/certbot/client.py

If I run the command that you suggest, it is kind of slow but works okay and does not leave any processes behind:

$ time ./venv/bin/python -m pylint /home/alex/devel/certbot/certbot/client.py 1>/dev/null
./venv/bin/python -m pylint /home/alex/devel/certbot/certbot/client.py >   3.15s user 0.12s system 100% cpu 3.264 total

(Click for full output)

$ ./venv/bin/python -m pylint /home/alex/devel/certbot/certbot/client.py
************* Module certbot.client
certbot/client.py:410:0: W0311: Bad indentation. Found 16 spaces, expected 12 (bad-indentation)
certbot/client.py:100:0: R0205: Class 'DummyConfig' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
certbot/client.py:225:12: R1720: Unnecessary "else" after "raise" (no-else-raise)
certbot/client.py:237:0: R0205: Class 'Client' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
certbot/client.py:359:8: R1705: Unnecessary "else" after "return" (no-else-return)
certbot/client.py:437:8: R1705: Unnecessary "elif" after "return" (no-else-return)


Report
======
300 statements analysed.

Statistics by type
------------------

+---------+-------+-----------+-----------+------------+---------+
|type     |number |old number |difference |%documented |%badname |
+=========+=======+===========+===========+============+=========+
|module   |1      |1          |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|class    |2      |2          |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|method   |14     |14         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+
|function |11     |11         |=          |100.00      |0.00     |
+---------+-------+-----------+-----------+------------+---------+



External dependencies
---------------------
::

    OpenSSL (certbot.client)
    acme 
      \-client (certbot.client)
      \-crypto_util (certbot.client)
      \-errors (certbot.client)
      \-magic_typing (certbot.client)
      \-messages (certbot.client)
    certbot (certbot.client)
      \-account (certbot.client)
      \-auth_handler (certbot.client)
      \-cli (certbot.client)
      \-compat 
      | \-misc (certbot.client)
      | \-os (certbot.client)
      \-constants (certbot.client)
      \-crypto_util (certbot.client)
      \-display 
      | \-enhancements (certbot.client)
      | \-ops (certbot.client)
      \-eff (certbot.client)
      \-error_handler (certbot.client)
      \-errors (certbot.client)
      \-interfaces (certbot.client)
      \-plugins 
      | \-selection (certbot.client)
      \-reverter (certbot.client)
      \-storage (certbot.client)
      \-util (certbot.client)
    cryptography 
      \-hazmat 
        \-backends (certbot.client)
        \-primitives 
          \-asymmetric 
            \-rsa (certbot.client)
    josepy (certbot.client)
    zope 
      \-component (certbot.client)



Raw metrics
-----------

+----------+-------+------+---------+-----------+
|type      |number |%     |previous |difference |
+==========+=======+======+=========+===========+
|code      |360    |48.00 |360      |=          |
+----------+-------+------+---------+-----------+
|docstring |265    |35.33 |265      |=          |
+----------+-------+------+---------+-----------+
|comment   |43     |5.73  |43       |=          |
+----------+-------+------+---------+-----------+
|empty     |82     |10.93 |82       |=          |
+----------+-------+------+---------+-----------+



Duplication
-----------

+-------------------------+------+---------+-----------+
|                         |now   |previous |difference |
+=========================+======+=========+===========+
|nb duplicated lines      |0     |0        |=          |
+-------------------------+------+---------+-----------+
|percent duplicated lines |0.000 |0.000    |=          |
+-------------------------+------+---------+-----------+



Messages by category
--------------------

+-----------+-------+---------+-----------+
|type       |number |previous |difference |
+===========+=======+=========+===========+
|convention |0      |0        |=          |
+-----------+-------+---------+-----------+
|refactor   |5      |5        |=          |
+-----------+-------+---------+-----------+
|warning    |1      |1        |=          |
+-----------+-------+---------+-----------+
|error      |0      |0        |=          |
+-----------+-------+---------+-----------+



Messages
--------

+---------------------------+------------+
|message id                 |occurrences |
+===========================+============+
|useless-object-inheritance |2           |
+---------------------------+------------+
|no-else-return             |2           |
+---------------------------+------------+
|no-else-raise              |1           |
+---------------------------+------------+
|bad-indentation            |1           |
+---------------------------+------------+




------------------------------------------------------------------
Your code has been rated at 9.80/10 (previous run: 9.80/10, +0.00)

If I run the command that is verbatim generated by VS Code, it also works fine:

$ /home/alex/devel/certbot/venv/bin/python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /home/alex/devel/certbot/certbot/client.py
************* Module certbot.client
410,0,warning,bad-indentation:Bad indentation. Found 16 spaces, expected 12
100,0,refactor,useless-object-inheritance:Class 'DummyConfig' inherits from object, can be safely removed from bases in python3
225,12,refactor,no-else-raise:Unnecessary "else" after "raise"
237,0,refactor,useless-object-inheritance:Class 'Client' inherits from object, can be safely removed from bases in python3
359,8,refactor,no-else-return:Unnecessary "else" after "return"
437,8,refactor,no-else-return:Unnecessary "elif" after "return"

------------------------------------------------------------------
Your code has been rated at 9.80/10 (previous run: 9.80/10, +0.00)

(venv)

There might be a concurrency issue in pylint itself (if concurrency is indeed the issue), but perhaps VSCode could kill try and serialize the use of pylint to avoid running into the stuck processes?

[deitry]

Confirm issue.
My case is that processes are spawned for files that I work on hardly and therefore save them very often. And sometimes there are syntax errors. I suppose, pylint cannot parse file properly and stuck somehow.

[deitry]

I discovered a way to reproduce this issue.

1. Open literally any Python file (I tested approach on https://github.com/ros2/launch_ros/blob/master/launch_ros/launch_ros/default_launch_description.py)
2. Add any non-valid Python code (like dd));;;) anywhere. Save file (assuming you have 'lint on save' turned on).
3. Delete that line, save again.
4. Repeat 2-3 as mad, saving & linting again any number of times.
5. Now you got N spawned pylint processes for that file that are not going to be stopped.

[stsdc]

That's still an issue for 2019.11.50794 (22 November 2019) version. Run out of memory when Autosave option enabled.

[brettcannon]

One possible solution to this is #3836 as we could probably implement that feature as a single execution after each edit, which would mean a single instance of any linter per edit.

[gatopeich]

But the root issue here is that pylint instances are left open forever...
As I explained on #8503, this leads to full system OOM even on a high-end system.
Just invoking pylint with a timeout could improve the situation...

[gatopeich]

A workaround using timeout:
Create this shell script at ~/bin/pylint_with_timeout.sh

#!/bin/sh
# Timeout wrapper for pylint to work around VS code issues
timeout 30s pylint $@

Then make "pylintPath point at it:

"python.linting.pylintPath": "~/bin/pylint_with_timeout.sh"

[dviljoen]

Same problem. Running on a MacBook Pro. System started hanging. VS Code stopped responding. ps showed hundreds(?) of pylint instances.

Examples:

  502  8423     1   0  1:13PM ??         0:00.05 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  8424     1   0  1:13PM ??         0:02.68 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  8875     1   0  1:14PM ??         0:00.05 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  8876     1   0  1:14PM ??         0:02.52 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  9421     1   0  1:15PM ??         0:00.04 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  9422     1   0  1:15PM ??         0:02.53 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  9867     1   0  1:18PM ??         0:00.03 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

  502  9868     1   0  1:18PM ??         0:02.62 /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python -m pylint --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /{snip}/migrate.py

Running manually:

$ python3 -m pylint /{snip}/migrate.py
************* Module migrate
migrate.py:45:0: R0902: Too many instance attributes (9/7) (too-many-instance-attributes)
migrate.py:346:0: R0914: Too many local variables (17/15) (too-many-locals)
migrate.py:420:4: C0415: Import outside toplevel (errno) (import-outside-toplevel)

------------------------------------------------------------------
Your code has been rated at 9.90/10 (previous run: 9.90/10, +0.00)

This file has 629 lines.
Running 1.42.1 on MacOS
$ pylint --version
pylint 2.4.4
astroid 2.3.3
Python 3.7.5 (default, Nov 1 2019, 02:16:23)
[Clang 11.0.0 (clang-1100.0.33.8)]

[simmol]

I tried reproducing similar behavior using pylint in the terminal(zsh) with for loops and even several terminals and I can not get one pylint stuck.

The only place I can reproduce this is inside VS code when I make several small changes and save 5-10 times in a row.
In the end, at least 2-3 processes are stuck.

Thanks to @gatopeich workaround ( which works great :) Thanks). I am able to work without having to kill a bunch of processes every 1-2 hours so it would not OOM the machine.

Adding some timeout or some other mechanism to cleanup these processes is needed.
Hope it would be fixed soon.

[karrtikr]

Hi all, please try out https://marketplace.visualstudio.com/items?itemName=ms-python.pylint extension which shouldn't have this issue. cc/ @karthiknadig

[karthiknadig]

Closing this s we have started the process of migrating users to the new pylint extension.