Improve startup performance (use require) #819
Conversation
mattseddon
force-pushed
the
use-require-for-telemetry-module
branch
from
16f6458 to
2b96e5e
Compare
| const getExtension = <T>(id: string): Extension<T & PackageJSON> | undefined => | ||
| extensions.getExtension<T & PackageJSON>(id) | ||
| const getExtension = <T>(id: string) => { | ||
| const { extensions } = require('vscode') as typeof import('vscode') |
There was a problem hiding this comment.
[F] This seems to be the culprit.
There was a problem hiding this comment.
@karthiknadig (from microsoft/vscode-python#17432) I have had some success with not pulling in extensions until I get to this point at runtime. The question for me now is - could ms-python.python be so resource heavy at startup that even importing extensions can affect the performance of another extension?
There was a problem hiding this comment.
Does changing this line make an immeidate difference? Can you try launching extension with a custom set --extension-dir where Python extension is not loaded?
There was a problem hiding this comment.
Python extension is resource heavy currently (we are working on start-up improvement for the extension), but it should not affect other extensions in this way (i.e, cause #656). This is something that VS Code should look into.
There was a problem hiding this comment.
@karthiknadig @karrtikr I have upgrade to the latest ms-python.python version and I am still seeing the same issue. I am running the extension through the extension debugger, could this be effecting things?
There are a couple of things from my globalState file that I wanted to point out:
The list in "PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS" contains largely duplicate information (and lot of it - there are 1752 entries for WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo, 2692 for INSIDERS_PROMPT_STATE_KEY, 2638 for JoinMailingListPrompt, etc). Here is the top of the list:
"PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS": [
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{ "key": "SWITCH_LS" },
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{ "key": "isGlobalSettingCopiedKey", "defaultValue": false },
There are a lot of envs displayed that are no longer on my machine. I have taken the time to go through and delete them. Do you know where these are being cached from?
There was a problem hiding this comment.
Profiling with current master and current release of python extension: CPU-20210921T001856.360Z.cpuprofile.txt
There was a problem hiding this comment.
From the table above it looks like even if the python extension loads faster (the 77ms case), the decoration issue seems to occur. I feel the global storage issue might be independent of this (which we need to investigate why it repeats so many times). I am still trying to make sense of why load time should cause this problem.
There was a problem hiding this comment.
@karthiknadig my globalState being out of whack seems like a cache issue, I went through the following steps:
- Uninstall
ms-python.python. rmms-python.python-2021.9.1246542782,ms-python.vscode-pylance-2021.9.2,ms-toolsai.jupyter-2021.8.2041215044&ms-toolsai.jupyter-keymap-1.0.0underneath~/L/Application Su/Code/CachedExtensionVSIXsdelete from ItemTable where key = 'ms-python.python';in~/Library/Application\ Support/Code/User/globalStorage/state.vscdb- Completely close VS Code
- Reopen VS Code
- Install
ms-python.python - Check
globalStatedb (still empty) - Start debugger
- Check
globalStatedb show below - decorations appear as expected - Reload window - decorations appear as expected
- Check db -
PYTHON_EXTENSION_GLOBAL_STORAGE_KEYShas grown - Reload window - decorations appear as expected
- Check db -
PYTHON_EXTENSION_GLOBAL_STORAGE_KEYShas grown again
For the time being this seems to have fixed the decoration issue. As you can see below I now have significantly less environments in my globalState. My first guess would be something unexpected happening in your CacheableLocatorService. I am slightly concerned that as I continue to load the window that PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS will again get out of control and I'll have to go through the above again.
new globalState after initial load (all dead envs have finally been removed):
{
"VSCode.ABExp.FeatureData": {
"features": [
"livesharecontinuousaacf",
"reusableLinks",
"pythonaacf",
"RemoveKernelToolbarInInteractiveWindowcf",
"portForwardingServiceEnabled-development",
"portForwardingServiceEnabled-staging",
"portForwardingServiceEnabled-production",
"pythonDiscoveryModuleWithoutWatcher",
"pythonTensorboardExperiment",
"testing-appver4cf",
"useImportHeuristic",
"mindaroBinariesVersion",
"mindaroBinariesVersion-1.0.20210702",
"gettingStarted.overrideCategory.ms-python.python.pythonWelcome.when",
"mindaroBinariesVersion-1.0.20210723",
"gettingStarted.overrideCategory.ms-python.python.pythonDataScienceWelcome.when",
"embeddedsurvey"
],
"assignmentContext": "vsliv368cf:30146710;vsreu685:30147344;vspyt267cf:30161958;python383cf:30185419;vspor879:30202332;vspor708:30202333;vspor363:30204092;pythonvspyt639:30300192;pythontb:30283811;vstes263cf:30335440;pythonvsuse255:30340121;binariesv615:30325510;bridge0708:30335490;pygetstartedt2:30360495;bridge0723:30353136;pydsgst2:30361792;vssur157:30367808;",
"configs": [
{
"Id": "vscode",
"Parameters": {
"livesharecontinuousaa": false,
"reusableLinks": true,
"pythonaa": false,
"RemoveKernelToolbarInInteractiveWindow": false,
"portForwardingServiceEnabled-development": true,
"portForwardingServiceEnabled-staging": true,
"portForwardingServiceEnabled-production": true,
"pythonDiscoveryModuleWithoutWatcher": true,
"pythonTensorboardExperiment": true,
"testing-appver4": false,
"useImportHeuristic": true,
"mindaroBinariesVersion": "1.0.20210615.1",
"mindaroBinariesVersion-1.0.20210702": "1.0.20210708.15",
"gettingStarted.overrideCategory.ms-python.python.pythonWelcome.when": "true",
"mindaroBinariesVersion-1.0.20210723": "1.0.20210723.6",
"gettingStarted.overrideCategory.ms-python.python.pythonDataScienceWelcome.when": "true",
"embeddedsurvey": true
}
}
]
},
"PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS": [
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "SWITCH_LS" },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" }
],
"JoinMailingListPrompt": true,
"PYTHON_ENV_INFO_CACHE": [
{
"name": "3.9.5",
"location": "/Users/mattseddon/.pyenv/versions/3.9.5",
"kind": "global-pyenv",
"executable": {
"filename": "/Users/mattseddon/.pyenv/versions/3.9.5/bin/python",
"sysPrefix": "/Users/mattseddon/.pyenv/versions/3.9.5",
"ctime": -1,
"mtime": -1
},
"display": "Python 3.9.5 64-bit ('3.9.5': pyenv)",
"version": {
"major": 3,
"minor": 9,
"micro": 5,
"release": { "level": "final", "serial": 0 },
"sysVersion": "3.9.5 (default, Jun 20 2021, 19:30:53) \n[Clang 12.0.5 (clang-1205.0.22.9)]"
},
"arch": 3,
"distro": { "org": "" },
"source": []
},
{
"name": ".env",
"location": "/Users/mattseddon/PP/vscode-dvc/demo/.env",
"kind": "virt-venv",
"executable": {
"filename": "/Users/mattseddon/PP/vscode-dvc/demo/.env/bin/python",
"sysPrefix": "/Users/mattseddon/PP/vscode-dvc/demo/.env",
"ctime": -1,
"mtime": -1
},
"searchLocation": {
"$mid": 1,
"path": "/Users/mattseddon/PP/vscode-dvc/demo",
"scheme": "file"
},
"display": "Python 3.9.5 64-bit ('.env': venv)",
"version": {
"major": 3,
"minor": 9,
"micro": 5,
"release": { "level": "final", "serial": 0 },
"sysVersion": "3.9.5 (default, Jun 20 2021, 19:30:53) \n[Clang 12.0.5 (clang-1205.0.22.9)]"
},
"arch": 3,
"distro": { "org": "" },
"source": []
},
{
"name": "billy_cart-DlY1dT5Z",
"location": "/Users/mattseddon/.local/share/virtualenvs/billy_cart-DlY1dT5Z",
"kind": "virt-pipenv",
"executable": {
"filename": "/Users/mattseddon/.local/share/virtualenvs/billy_cart-DlY1dT5Z/bin/python",
"sysPrefix": "/Users/mattseddon/.local/share/virtualenvs/billy_cart-DlY1dT5Z/bin/..",
"ctime": -1,
"mtime": -1
},
"display": "Python 3.8.1 64-bit ('billy_cart-DlY1dT5Z': pipenv)",
"version": {
"major": 3,
"minor": 8,
"micro": 1,
"release": { "level": "final", "serial": 0 },
"sysVersion": "3.8.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53) \n[Clang 6.0 (clang-600.0.57)]"
},
"arch": 3,
"distro": { "org": "" },
"source": []
},
{
"name": "",
"location": "",
"kind": "global-other",
"executable": {
"filename": "/usr/bin/python",
"sysPrefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
"ctime": -1,
"mtime": -1
},
"display": "Python 2.7.16 64-bit",
"version": {
"major": 2,
"minor": 7,
"micro": 16,
"release": { "level": "final", "serial": 0 },
"sysVersion": "2.7.16 (default, Jun 18 2021, 03:23:53) \n[GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy"
},
"arch": 3,
"distro": { "org": "" },
"source": ["path env var"]
},
{
"name": "",
"location": "",
"kind": "global-other",
"executable": {
"filename": "/usr/bin/python3",
"sysPrefix": "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8",
"ctime": -1,
"mtime": -1
},
"display": "Python 3.8.9 64-bit",
"version": {
"major": 3,
"minor": 8,
"micro": 9,
"release": { "level": "final", "serial": 0 },
"sysVersion": "3.8.9 (default, Aug 21 2021, 15:53:23) \n[Clang 13.0.0 (clang-1300.0.29.3)]"
},
"arch": 3,
"distro": { "org": "" },
"source": ["path env var"]
},
{
"name": "",
"location": "",
"kind": "global-other",
"executable": {
"filename": "/usr/local/bin/python3.8",
"sysPrefix": "/Library/Frameworks/Python.framework/Versions/3.8",
"ctime": -1,
"mtime": -1
},
"display": "Python 3.8.1 64-bit",
"version": {
"major": 3,
"minor": 8,
"micro": 1,
"release": { "level": "final", "serial": 0 },
"sysVersion": "3.8.1 (v3.8.1:1b293b6006, Dec 18 2019, 14:08:53) \n[Clang 6.0 (clang-600.0.57)]"
},
"arch": 3,
"distro": { "org": "" },
"source": ["path env var"]
}
],
"SWITCH_LS": "Pylance",
"pylanceDefaultPromptMemento": true
}
There was a problem hiding this comment.
After opening a second project PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS grows to:
"PYTHON_EXTENSION_GLOBAL_STORAGE_KEYS": [
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{ "key": "isGlobalSettingCopiedKey", "defaultValue": false },
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "workspaceFolderKeysForWhichTheCopyIsDone_Key",
"defaultValue": []
},
{ "key": "ShowBanner", "defaultValue": true },
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{
"key": "WORKSPACE_FOLDER_INTERPRETER_PATH_/Users/mattseddon/PP/vscode-dvc/demo"
},
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "SWITCH_LS" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" },
{ "key": "MESSAGE_KEY_FOR_27_SUPPORT_PROMPT", "defaultValue": false },
{ "key": "SWITCH_LS" },
{ "key": "ShowBanner", "defaultValue": true },
{ "key": "JoinMailingListPrompt", "defaultValue": false },
{ "key": "PythonTensorBoardWebviewPreferredViewGroup", "defaultValue": -1 },
{ "key": "INSIDERS_PROMPT_STATE_KEY", "defaultValue": false },
{ "key": "preferredGlobalPyInterpreter" },
{ "key": "PYTHON_ENV_INFO_CACHE" }
]
… use-require-for-telemetry-module
… use-require-for-telemetry-module
|
Code Climate has analyzed commit 8a4bc0b and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 94.4% (85% is the threshold). This pull request will bring the total coverage in the repository to 96.0% (0.0% change). View more on Code Climate. |
|
Closing this as the issue appears to have been caused by a massive amount of data underneath the @karthiknadig would you like me to raise a separate bug in |
|
Yes, please raise the issue, since we have to address the repeated keys problem. But the fix FileDecoration itself will come from VS Code (Johannes has already added some changes to help with this microsoft/vscode#133210). Essentially, if the first registered wins then there can always be another extension that just does file decoration as the first thing in it's activation with no dependencies and it will cause this. Thanks for all the investigation, this might have uncovered some interesting bugs. |
|
@karthiknadig I raised microsoft/vscode-python#17488, LMK if I missed anything. Thanks again for looking into this. |
Trying to mitigate issues causing #656