Unhandled python exception when installing numpy with pypy on Windows

[schmidtfrk]

Describe the bug
Meson has an unhandled python exception.
The problem occurs on Windows 10 as well on Windows 11. However, I'm not sure whether it is really the same bug.

Windows 10:

Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  Ôöé exit code: 2
  Ôò░ÔöÇ> [48 lines of output]
      + C:\Users\Frank01\Desktop\python lernen\pypy\venv\Scripts\pypy.exe C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\meson.py setup C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2 C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\.mesonpy-e9p05mif -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\.mesonpy-e9p05mif\meson-python-native-file.ini
      Traceback (most recent call last):
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\mesonmain.py", line 194, in run
          return options.run_func(options)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\msetup.py", line 358, in run
          app.generate()
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\msetup.py", line 183, in generate
          return self._generate(env, capture, vslite_ctx)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\msetup.py", line 205, in _generate
          intr = interpreter.Interpreter(b, user_defined_options=user_defined_options)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 331, in __init__
          self.parse_project()
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 134, in parse_project
          self.evaluate_codeblock(self.ast, end=1)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 190, in evaluate_codeblock
          raise e
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 182, in evaluate_codeblock
          self.evaluate_statement(cur)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 196, in evaluate_statement
          return self.function_call(cur)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 521, in function_call
          res = func(node, func_args, kwargs)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\decorators.py", line 277, in wrapper
          return f(*nargs, **wrapped_kwargs)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreterbase\decorators.py", line 596, in wrapper
          return f(*wrapped_args, **wrapped_kwargs)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 1313, in func_project
          mesonlib.setup_vsenv(force_vsenv)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\utils\vsenv.py", line 120, in setup_vsenv
          return _setup_vsenv(force)
        File "C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\vendored-meson\meson\mesonbuild\utils\vsenv.py", line 58, in _setup_vsenv
          bat_json = subprocess.check_output(
        File "C:\Users\Frank01\Downloads\pypy3.10-v7.3.15-win64\pypy3.10-v7.3.15-win64\Lib\subprocess.py", line 421, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
        File "C:\Users\Frank01\Downloads\pypy3.10-v7.3.15-win64\pypy3.10-v7.3.15-win64\Lib\subprocess.py", line 526, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\vswhere.exe', '-latest', '-prerelease', '-requiresAny', '-requires', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', '-requires', 'Microsoft.VisualStudio.Workload.WDExpress', '-products', '*', '-utf8', '-format', 'json']' returned non-zero exit status 87.
      The Meson build system
      Version: 1.2.99
      Source dir: C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2
      Build dir: C:\Users\Frank01\AppData\Local\Temp\pip-install-2rcdrvwj\numpy_75556b21df8442f9aaa88cfa74c021b2\.mesonpy-e9p05mif
      Build type: native build
      Project name: NumPy
      Project version: 1.26.4

      ..\meson.build:1:0: ERROR: Unhandled python exception

          This is a Meson bug and should be reported!

On Windows 11:

 Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  Ôöé exit code: 2
  Ôò░ÔöÇ> [77 lines of output]
      + C:\Program Files (x86)\PyPy\pypy.exe C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\meson.py setup C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4 C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\.mesonpy-ih5pd93w -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\.mesonpy-ih5pd93w\meson-python-native-file.ini
      Exception in thread Thread-4 (_readerthread):
      Exception in thread Thread-3 (_readerthread):
      Traceback (most recent call last):
        File "C:\Program Files (x86)\PyPy\Lib\threading.py", line 1016, in _bootstrap_inner
          self.run()
        File "C:\Program Files (x86)\PyPy\Lib\threading.py", line 953, in run
          self._target(*self._args, **self._kwargs)
        File "C:\Program Files (x86)\PyPy\Lib\subprocess.py", line 1543, in _readerthread
          buffer.append(fh.read())
        File "C:\Program Files (x86)\PyPy\Lib\codecs.py", line 322, in decode
          (result, consumed) = self._buffer_decode(data, self.errors, final)
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0x84 in position 546: invalid start byte
      Traceback (most recent call last):
        File "C:\Program Files (x86)\PyPy\Lib\threading.py", line 1016, in _bootstrap_inner
          self.run()
        File "C:\Program Files (x86)\PyPy\Lib\threading.py", line 953, in run
          self._target(*self._args, **self._kwargs)
        File "C:\Program Files (x86)\PyPy\Lib\subprocess.py", line 1543, in _readerthread
          buffer.append(fh.read())
        File "C:\Program Files (x86)\PyPy\Lib\codecs.py", line 322, in decode
          (result, consumed) = self._buffer_decode(data, self.errors, final)
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 62: invalid start byte
      Traceback (most recent call last):
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\mesonmain.py", line 194, in run
          return options.run_func(options)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\msetup.py", line 358, in run
          app.generate()
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\msetup.py", line 183, in generate
          return self._generate(env, capture, vslite_ctx)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\msetup.py", line 205, in _generate
          intr = interpreter.Interpreter(b, user_defined_options=user_defined_options)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 331, in __init__
          self.parse_project()
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 134, in parse_project
          self.evaluate_codeblock(self.ast, end=1)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 190, in evaluate_codeblock
          raise e
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 182, in evaluate_codeblock
          self.evaluate_statement(cur)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 196, in evaluate_statement
          return self.function_call(cur)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\interpreterbase.py", line 521, in function_call
          res = func(node, func_args, kwargs)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\decorators.py", line 277, in wrapper
          return f(*nargs, **wrapped_kwargs)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreterbase\decorators.py", line 596, in wrapper
          return f(*wrapped_args, **wrapped_kwargs)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 1315, in func_project
          self.add_languages(proj_langs, True, MachineChoice.HOST)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 1472, in add_languages
          success = self.add_languages_for(args, required, for_machine)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 1515, in add_languages_for
          comp = compilers.detect_compiler_for(self.environment, lang, for_machine, skip_sanity_check)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\compilers\detect.py", line 115, in detect_compiler_for
          comp = compiler_from_language(env, lang, for_machine)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\compilers\detect.py", line 112, in compiler_from_language
          return lang_map[lang](env, for_machine) if lang in lang_map else None
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\compilers\detect.py", line 614, in detect_c_compiler
          return _detect_c_or_cpp_compiler(env, 'c', for_machine)
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\compilers\detect.py", line 330, in _detect_c_or_cpp_compiler
          p, out, err = Popen_safe_logged(cmd, msg='Detecting compiler via')
        File "C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\vendored-meson\meson\mesonbuild\utils\universal.py", line 1530, in Popen_safe_logged
          rc, out, err = p.returncode, o.strip(), e.strip()
      AttributeError: 'NoneType' object has no attribute 'strip'
      The Meson build system
      Version: 1.2.99
      Source dir: C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4
      Build dir: C:\Users\schmi\AppData\Local\Temp\pip-install-aj8h1ob1\numpy_a4266b0d35484906a32b87ff2ba42be4\.mesonpy-ih5pd93w
      Build type: native build
      Project name: NumPy
      Project version: 1.26.4
      Activating VS 17.9.3

      ..\meson.build:1:0: ERROR: Unhandled python exception

          This is a Meson bug and should be reported!

To Reproduce

1. Use PyPy3.10
2. pypy -m pip install numpy

Expected behavior
Installing numy.

system parameters

• native build (for the same computer)
• Windows 11 as well as Windows 10
• Python 3.10
• meson 1.2.99

Especially the error for Windows 11 looks very similar to Unhandled python exception when configuring project with MSVC backend
#.

[rgommers]

Hmm, we only test with PyPy 3.9 in NumPy CI, and that works fine - also on Windows. In case you have a PyPy 3.9 install locally, can you check if you get the same errors or not?

[schmidtfrk]

I just checked PyPy3.9 on both of my windows machines and it worked fine.

[schmidtfrk]

I accidentally closed the issue. My problem is solved by simply using PyPy3.9, but I think it would be good if the problem is also solved for PyPy3.10.

[rgommers]

@mattip does this look familiar to you perhaps, or are you aware of a change or regression in PyPy 3.10 that could be related?

[mattip]

PyPy on windows has various problems with unicode and its terminal (sys.out and sys.err). The likely chain of errors is a compiler or command error, which causes a OS-level error with a unicode path or error message, which the PyPy console handling cannot decode. See the tracking issue pypy/pypy#3890.

meson is trying to run "C:\\Program Files (x86)\\Microsoft Visual Studio\\Installer\\vswhere.exe" -latest -prerelease -requiresAny -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -requires Microsoft.VisualStudio.Workload.WDExpress -products * -utf8 -format json, what does that print when you run it from a cmd prompt?

[eli-schwartz]

I think the actual issue may be 58c768a

At least, that's the reason it is raising an unhandled exception. It should just log that the compiler detection went nowhere.

Does it reproduce with a package other than numpy? If so, it should be easy to test -- run:

pip install git+https://github.com/eli-schwartz/meson@compiler-logged-none

and try to build that other package (since numpy includes a custom copy of meson with draft changes).