Use A/B testing on the language server caching support

[brettcannon]

Dependent on #7475 .

[karrtikr]

This experiment will only be active for users using LS, and will have 3 components

• Caching - System : When setting value is "System"
• Caching - Library : When setting value is "Library"
• Caching - control : When setting is not set, i.e "None"

[brettcannon]

Do we want to turn on "System" caching now? I thought that was still being tested/worked out?

[brettcannon]

Or are we going to put in "System" for now but start with a group size of 0?

[karrtikr]

Do we want to turn on "System" caching now? I thought that was still being tested/worked out?

I am just guessing by the title, I don't know what was discussed. We can do group size of 0, or not include "System" at all.

[brettcannon]

@karrtikr probably a group size of 0 to make sure we can control it as soon as a new version of the language server goes out to stable, but we should talk with Tyreke and the MPLS team about if/when they want us to turn that on.

[karrtikr]

If user belongs to say 'Caching - Library' experiment, we have to update settings.json to include python.analysis.cachingLevel setting set to "Library", for the users to be using "Library" caching.

It seems dirty and it has its problems, for eg.

• When we turn off the experiments, we need to revert the settings back to normal, i.e remove the setting from settings.json. How do we know if it's us who added it in the settings.json, or the user itself?

Ideally we should be A/B testing this setting on the language server side. It will allow us to change the default value of the setting without modifying settings.json.

[kimadeline]

@luabud is it still relevant, with Pylance and JediLSP available?

[luabud]

I don't think so, let's close it!