Flit's dynamic description is not working with pip 22.2 along with python 3.8, 3.9

[furiosamg]

Minimal reproduction: https://github.com/furiosamg/flit-dynamic-pip-22

I think flit's dynamic description generation is broken in pip 22.2.
When trying to install above package with pip 22.2, following error occurred.

$ pip install pip==22.2 && pip install .
Looking in indexes: https://pypi.org/simple, https://internal-pypi.furiosa.dev/simple
Requirement already satisfied: pip==22.2 in /root/miniconda3/envs/env-3.9/lib/python3.9/site-packages (22.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://internal-pypi.furiosa.dev/simple
Processing /root/flit-dynamic-pip-22
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [45 lines of output]
      Looking in indexes: https://pypi.org/simple, https://internal-pypi.furiosa.dev/simple, https://internal-pypi.furiosa.dev/simple
      Collecting flit_core<4,>=3.2
        Using cached flit_core-3.7.1-py3-none-any.whl (60 kB)
      Installing collected packages: flit_core
      /root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py:103: _ExperimentalProjectMetadata: Support for project metadata in `pyproject.toml` is still experimental and may be removed (or change) in future releases.
        warnings.warn(msg, _ExperimentalProjectMetadata)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
          status = run_func(*args)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
          return func(self, options, args)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 461, in run
          installed = install_given_reqs(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
          requirement.install(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 752, in install
          scheme = get_scheme(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/locations/__init__.py", line 246, in get_scheme
          old = _distutils.get_scheme(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 130, in get_scheme
          scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 46, in distutils_scheme
          d.parse_config_files()
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/dist.py", line 854, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 55, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 135, in read_configuration
          return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 190, in expand_configuration
          return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 237, in expand
          self._expand_all_dynamic(dist, package_dir)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 272, in _expand_all_dynamic
          obtained_dynamic = {
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 273, in <dictcomp>
          field: self._obtain(dist, field, package_dir)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 310, in _obtain
          self._ensure_previously_set(dist, field)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 296, in _ensure_previously_set
          raise OptionError(msg)
      distutils.errors.DistutilsOptionError: No configuration found for dynamic 'description'.
      Some dynamic fields need to be specified via `tool.setuptools.dynamic`
      others must be specified via the equivalent attribute in `setup.py`.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip

[takluyver]

Your package builds and installs fine for me (see below). 😕

It looks like pip is using some distutils/setuptools code to work out destination paths, and setuptools ends up trying to parse the metadata from pyproject.toml assuming that it's built with setuptools. What version of setuptools do you have? I've just tried updating to the latest and I still don't see the issue.

Do you have any pip config in files or environment variables?

$ pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/takluyver/Code/flit-dynamic-pip-22
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: flitdynamicpip22
  Building wheel for flitdynamicpip22 (pyproject.toml) ... done
  Created wheel for flitdynamicpip22: filename=flitdynamicpip22-0.0.1-py2.py3-none-any.whl size=1101 sha256=0a1efe3e78bd38c63c5ce540cc616d72da889be62233e2e61033f5b67ade985e
  Stored in directory: /home/takluyver/.cache/pip/wheels/df/08/86/74c3e5aa6010da77b6ecae2b31a34efd35815ddc26e6b382b3
Successfully built flitdynamicpip22
Installing collected packages: flitdynamicpip22
Successfully installed flitdynamicpip22-0.0.1

$ pip --version
pip 22.2 from /home/takluyver/.local/lib/python3.10/site-packages/pip (python 3.10)

[furiosamg]

Thanks for the comment.

When I try to install my package with Python 3.10, the installation succeeds. 🤔 But not with 3.8, 3.9.
Could you please try with Python 3.8 or 3.9?

P.S. I think I haven't set any pip configuration.

[takluyver]

Thanks, when I make a Python 3.9 environment, I do see the same error.

I still don't think this is a Flit bug - I think the issue is somewhere between pip & setuptools, because it looks like setuptools is trying to get the package metadata as if it was a setuptools package, which is wrong. But I don't know when I'll have time to dig into it to work out exactly what is going wrong.

Other than digging through the traceback to figure out what is going on, you could try using git bisect to identify which change in pip introduced or revealed the problem.

[furiosamg]

I agree with that flit is innocent 😇
The reason I raised the issue here was because I thought dynamic description generation was a flit's functionality.

Thanks for gentle and quick response.

[pradyunsg]

Pinging @abravalheri since setuptools' pyproject.toml support seems to be implicated in this.

[abravalheri]

Any chance setuptools got imported before distutils? It seems that pip is using the monkey patched version of Distribution.

[pradyunsg]

I can confirm why this isn't failing on 3.10 -- on 3.10+, pip's using the sysconfig paths and avoid importing/using distutils entirely. That's why the code never reaches the locations/_distutils.py file, which means it never ends up using that.

Any chance setuptools got imported before distutils?

I don't think that should happen? The only reason that I can think of is if setuptools' distutils hack has some logic to detect pip that's broken in 22.2.

---

FWIW, this only happens for local installations and not when installing from a zipfile.

❯ python --version
Python 3.9.6

❯ pip --version
pip 22.2 from /Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip (python 3.9)

❯ pip install https://github.com/furiosamg/flit-dynamic-pip-22/archive/refs/heads/main.zip
Collecting https://github.com/furiosamg/flit-dynamic-pip-22/archive/refs/heads/main.zip
  Downloading https://github.com/furiosamg/flit-dynamic-pip-22/archive/refs/heads/main.zip
     - 1.2 kB 8.5 MB/s 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: flitdynamicpip22
  Building wheel for flitdynamicpip22 (pyproject.toml) ... done
  Created wheel for flitdynamicpip22: filename=flitdynamicpip22-0.0.1-py2.py3-none-any.whl size=1101 sha256=178bba12298d9ad6341796ffee219b2348817d481367f0501ed4bd8f436c4489
  Stored in directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-ephem-wheel-cache-0h1nj302/wheels/cf/e8/77/17cc0fd5e981a5997f17e4fa054ed3cd7d06eb9522308f4722
Successfully built flitdynamicpip22
Installing collected packages: flitdynamicpip22
Successfully installed flitdynamicpip22-0.0.1

❯ g cl https://github.com/furiosamg/flit-dynamic-pip-22
Cloning into 'flit-dynamic-pip-22'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 12 (delta 2), reused 12 (delta 2), pack-reused 0
Receiving objects: 100% (12/12), done.
Resolving deltas: 100% (2/2), done.

❯ cd flit-dynamic-pip-22

❯ pip install .
Processing /Users/pradyunsg/Developer/github/flit-dynamic-pip-22
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [42 lines of output]
      Collecting flit_core<4,>=3.2
        Using cached flit_core-3.7.1-py3-none-any.whl (60 kB)
      Installing collected packages: flit_core
      ERROR: Exception:
      Traceback (most recent call last):
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
          status = run_func(*args)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
          return func(self, options, args)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 461, in run
          installed = install_given_reqs(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
          requirement.install(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 752, in install
          scheme = get_scheme(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/locations/__init__.py", line 246, in get_scheme
          old = _distutils.get_scheme(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 130, in get_scheme
          scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 46, in distutils_scheme
          d.parse_config_files()
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 58, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 136, in read_configuration
          return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 191, in expand_configuration
          return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 238, in expand
          self._expand_all_dynamic(dist, package_dir)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 275, in _expand_all_dynamic
          obtained_dynamic = {
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 276, in <dictcomp>
          field: self._obtain(dist, field, package_dir)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 322, in _obtain
          self._ensure_previously_set(dist, field)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 301, in _ensure_previously_set
          raise OptionError(msg)
      distutils.errors.DistutilsOptionError: No configuration found for dynamic 'description'.
      Some dynamic fields need to be specified via `tool.setuptools.dynamic`
      others must be specified via the equivalent attribute in `setup.py`.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[abravalheri]

The only reason that I can think of is if setuptools' distutils hack has some logic to detect pip that's broken in 22.2.

I can confirm that the custom import logic don't seem to be working for pip 22.2: in pip/_internal/locations/_distutils.py, distutils is loaded from setuptools/_distutils, which is related to #8761.

Maybe we can avoid this error if PipImportRedirectingFinder is added after DistutilsMetaFinder?

Something like:

- sys.meta_path.insert(0, PipImportRedirectingFinder())
+ def _install():
+    for i, finder in enumerate(sys.meta_path):
+        if "DistutilsMetaFinder" in str(type(finder)):
+            sys.meta_path.insert(i + 1, PipImportRedirectingFinder())
+            return
+
+    sys.meta_path.insert(0, PipImportRedirectingFinder())
+
+
+ _install()

[pradyunsg]

Ah, well, that's interesting and would fix the immediate bleeding. I think it's reasonable for us to do a sys.meta_path.append(...) instead of the sys.meta_path.insert(0, ...).

We actually changed this to the start on the rationale that no one else should try modifying pip before us (#11257 (comment)) but setuptools use case seems like a reasonable one.

[furiosamg]

Thank you all for your support and interest. I'd be happy to wait.

[abravalheri]

Thank you very much for the feedback @pradyunsg.

There might be other ways of handling this in the setuptools side, but I think that this is the most straight forward... Without the custom finder intercepting pip, we would need some kind of signalling from pip to avoid replacing distutils (e.g. the approach described by Jason). (I am more than happy to discuss other alternatives if any of you guys has a different idea).

@furiosamg, meanwhile you can try exporting SETUPTOOLS_USE_DISTUTILS=stdlib as an environment variable.

[furiosamg]

@abravalheri Thanks, I checked that workaround works. Great!