[BUG] StaticModule not choosing the Python file encoding

[JulienPalard]

setuptools version

setuptools==65.4.0

Python version

3.7, 3.8, 3.9, 3.10, 3.11

OS

Windows mainly, but reproducible on Linux

Additional environment information

No response

Description

The StaticModule opens Python file without looking at -*- encoding, and without specifying the encoding:

    with open(spec.origin) as strm:

On Linux as most Python files are UTF-8 and as Python defaults to UTF-8 to open files, it goes right.

But on Windows, Python will default to CP1252 at file opening, possibly leading to a UnicodeDecodeError.

When the module don't have an import on an external dependency the fallback to evaluate module hides the bug.

Expected behavior

I expect StaticModule to first look at the encoding marker (# -*- coding: encoding -*-) of the file before trying to decode it, and defaulting to UTF-8 if no maker is found.

How to Reproduce

I first found the bug on a Windows CI runner on github so it was reading UTF-8 file as CP1252. I'm on Debian so I'll repro it the other way around: writing a latin1 file and I let setuptools fail to decode it.

mkdir repro
cd repro
echo $'# -*- coding: latin1 -*-\nimport regex\n__version__ = "0.1"\nprint("\xe9")' > repro.py 
cat <<EOF >pyproject.toml 
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "my_package"
dynamic = ["version"]
dependencies = [
    "regex",
]

[tool.setuptool]
py-modules = ['repro.py']

[tool.setuptools.dynamic.version]
attr = "repro.__version__"
EOF
pip install .

Output

output

Processing /home/mdk/clones/setuptools/repro
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [58 lines of output]
      /tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
        warnings.warn(msg, _BetaConfiguration)
      Traceback (most recent call last):
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 191, in read_attr
          return getattr(StaticModule(module_name, spec), attr_name)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 66, in __init__
          src = strm.read()
        File "/usr/lib/python3.10/codecs.py", line 322, in decode
          (result, consumed) = self._buffer_decode(data, self.errors, final)
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 65: invalid continuation byte
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/mdk/clones/setuptools/repro/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/mdk/clones/setuptools/repro/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/mdk/clones/setuptools/repro/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 1, in <module>
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 159, in setup
          dist.parse_config_files()
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 62, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 140, in read_configuration
          return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 195, in expand_configuration
          return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 242, in expand
          self._expand_all_dynamic(dist, package_dir)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 286, in _expand_all_dynamic
          version=self._obtain_version(dist, package_dir),
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 332, in _obtain_version
          return _expand.version(self._obtain(dist, "version", package_dir))
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 321, in _obtain
          return self._expand_directive(
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 315, in _expand_directive
          return _expand.read_attr(directive["attr"], package_dir, root_dir)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 194, in read_attr
          module = _load_spec(spec, module_name)
        File "/tmp/pip-build-env-by2xrl8v/overlay/lib/python3.10/site-packages/setuptools/config/expand.py", line 214, in _load_spec
          spec.loader.exec_module(module)  # type: ignore
        File "<frozen importlib._bootstrap_external>", line 883, in exec_module
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "/home/mdk/clones/setuptools/repro/repro.py", line 2, in <module>
          import regex
      ModuleNotFoundError: No module named 'regex'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[JulienPalard]

Ohh, looks like ast.parse handle it if given bytes!