Importing SynapseML doesn't work

[vishalovercome]

I'm working on a pyspark project and trying to leverage Microsoft's SynapseML library. There are two ways of installing synapseml:

1. Dowloading as a JAR when running Spark submit - This is how it works in production but after downloading it locally, I'm unable to import the python libraries dues to which none of the IDE features work
2. Installing it via pip - This won't work in production (spark-submit, Jupyter, etc.) but at least lets us leverage IDE features.

Is there a way to get all IDE features without installing it via pip?

[debonte]

There's documentation on how we resolve imports here. Try setting python.analysis.extraPaths in your VS Code settings or extraPaths (in pyrightconfig.json/pyproject.toml) to point to the install location.

[vishalovercome]

I've tried that before creating this issue. Problem is that SynapseML downloads JARs when downloaded via spark shell and I'm not sure how to make it PyLance work with it

[debonte]

Looks like extraPaths will work if you:

1. Change the extensions of the .jar files to .zip
2. Add the .zip file paths (not the parent directory path) to extraPaths (ex. "extraPaths": [ "C:/Users/erikd/.ivy2/jars/com.microsoft.azure_synapseml-core_2.12-0.11.2.zip" ])

After doing that with com.microsoft.azure_synapseml-core_2.12-0.11.2.jar, Pylance was happy with the following import:

from synapse.ml.core.platform.Platform import running_on_synapse

We could in theory make Pylance treat .jar files like .zip and .egg files, but this is the first request we've gotten along these lines. Also, I saw the following in the Spark documentation, which makes me think that SynapseML may be doing something non-standard even for Spark:

For Python, you can use the --py-files argument of spark-submit to add .py, .zip or .egg files to be distributed with your application. If you depend on multiple Python files we recommend packaging them into a .zip or .egg. For third-party Python dependencies, see Python Package Management.

[vishalovercome]

Having to rename .jar to .zip is a weird work-around. I'll do it now but given that pyspark is such a widely used framework and python itself being often times used as a glue language, I would request you to enhance the extension itself.

[debonte]

Ok. I dug into this some more and found that Python's zip import support doesn't care about the extension of zipped files, so ideally we wouldn't either.

Resources:

• Python docs
• CPython implementation docs
• CPython implementation
• Our issues/PRs related to zip/egg support:
  • Support egg/zip files for IntelliSense vscode-python#2532
  • Support egg and zip files python-language-server#196
  • https://github.com/microsoft/pyrx/issues/1340