Issue with NumPy, macOS 11 Big Sur, Python 3.9.1 Does pipenv not use the latest pip? #4564

shanewilkins · 2020-12-12T19:33:10Z

People are probably already aware of this issue, but adding my $0.02 as to what I think may be going on.

In a nutshell: NumPy fails to build on OS 11 and python 3.9.1. The error message is attached below, but NumPy is installed and works. (you can import numpy as np from a python interpreter) However, if you try to import numpy in a notebook, you get an error message saying that numpy is using a buggy "accelerate" backend.

The error about the backend occurs when numpy is being built from source incorrectly instead of using the provided wheel. That known issue was supposedly fixed in a pip upgrade (20.3). However, I get the same error below, even when I have updated pip to 20.3.

However, when i create a new pipenv environment, it defaults to pip 20.2. So my guess is that even if I subsequently update pip to 20.3, that pipenv install numpy is using the old version of pip.

Can anybody confirm and/or offer a workaround?

(blog)  shane@Olympus  ~/github/blog   dev ●  pipenv install numpy
Installing numpy...
Adding numpy to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock (16c839) out of date, updating to (49bf85)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/tmpk03mt0_f
     cwd: /private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy
Complete output (209 lines):
Running from numpy source directory.
setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
  run_build = parse_setuppy_commands()
Cythonizing sources
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_philox.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/_sfc64.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/bit_generator.pyx
Processing numpy/random/mtrand.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_common.pyx
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries tatlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_info:
  libraries satlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

accelerate_info:
  libraries accelerate not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
Library accelerate was not found. Ignoring
  libraries veclib not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
Library veclib was not found. Ignoring
  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

non-existing path in 'numpy/distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
  libraries mkl_rt not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_lapack_info:
  libraries openblas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_clapack_info:
  libraries openblas,lapack not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

flame_info:
  libraries flame not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries tatlas,tatlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries tatlas,tatlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries tatlas,tatlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries satlas,satlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries satlas,satlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries satlas,satlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries ptf77blas,ptcblas,atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries f77blas,cblas,atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries f77blas,cblas,atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running dist_info
running build_src
build_src
building py_modules sources
creating build
creating build/src.macosx-11-x86_64-3.9
creating build/src.macosx-11-x86_64-3.9/numpy
creating build/src.macosx-11-x86_64-3.9/numpy/distutils
building library "npymath" sources
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 508, in <module>
    setup_package()
  File "setup.py", line 500, in setup_package
    setup(**metadata)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 31, in run
    egg_info.run()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/egg_info.py", line 24, in run
    self.run_command("build_src")
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 155, in build_sources
    self.build_library_sources(*libname_info)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 288, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 658, in get_mathlib_info
    st = config_cmd.try_link('int main(void) { return 0;}')
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/config.py", line 241, in try_link
    self._check_compiler()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/config.py", line 80, in _check_compiler
    self.fcompiler = new_fcompiler(compiler=self.fcompiler,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 880, in new_fcompiler
    compiler = get_default_fcompiler(plat, requiref90=requiref90,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 851, in get_default_fcompiler
    compiler_type =  _find_existing_fcompiler(matching_compiler_types,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 802, in _find_existing_fcompiler
    c.customize(dist)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 526, in customize
    linker_so_flags = self.flag_vars.linker_so
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/environment.py", line 37, in __getattr__
    return self._get_var(name, conf_desc)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/environment.py", line 53, in _get_var
    var = self._hook_handler(name, hook)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 705, in _environment_hook
    return hook()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/gnu.py", line 346, in get_flags_linker_so
    flags = GnuFCompiler.get_flags_linker_so(self)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/gnu.py", line 136, in get_flags_linker_so
    os.environ['MACOSX_DEPLOYMENT_TARGET'] = target
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 684, in __setitem__
    value = self.encodevalue(value)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 756, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not int
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 758, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
    return resolve_deps(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1395, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 823, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 229, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 350, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 306, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
    self._subprocess_runner(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/tmpk03mt0_f Check the logs for full command output.

The text was updated successfully, but these errors were encountered:

shanewilkins · 2020-12-13T18:36:00Z

Seem related to the bug reported at NumPy here: #17784

jaymegordo · 2020-12-13T20:21:57Z

+1 exact same issue and error trace for me

@shanewilkins thank you for pointing me in the right direction. The issue was in fact that pipenv was using pip 2.2 instead of pip 2.3.

You can just manually update pipenv's pip with pipenv run python -m pip install -U pip and confirm it worked with pipenv run pip --version which should output pip 20.3.1 from ......

After this the numpy install works for me.

**Edit - Actually, the numpy install only works with pipenv install numpy --skip-lock, while pipenv lock still fails...

MCFreddie777 · 2020-12-15T17:08:34Z

+1 exact same issue and error trace for me

@shanewilkins thank you for pointing me in the right direction. The issue was in fact that pipenv was using pip 2.2 instead of pip 2.3.

You can just manually update pipenv's pip with pipenv run python -m pip install -U pip and confirm it worked with pipenv run pip --version which should output pip 20.3.1 from ......

After this the numpy install works for me.

**Edit - Actually, the numpy install only works with pipenv install numpy --skip-lock, while pipenv lock still fails...

@jaymegordo Upgrading pipenv's pip did not work.

mzagaja · 2020-12-30T23:27:39Z

I can also confirm that upgrading pipenv pip did not work for me.

jo3p · 2021-01-01T15:55:50Z

+1 exact same issue and error trace for me

@shanewilkins thank you for pointing me in the right direction. The issue was in fact that pipenv was using pip 2.2 instead of pip 2.3.

You can just manually update pipenv's pip with pipenv run python -m pip install -U pip and confirm it worked with pipenv run pip --version which should output pip 20.3.1 from ......

After this the numpy install works for me.

**Edit - Actually, the numpy install only works with pipenv install numpy --skip-lock, while pipenv lock still fails...

Any clue why pipenv lock fails?

jaymegordo · 2021-01-04T15:16:30Z

@jo3p I think it has something to do with how Big Sur changed how some system libraries are stored now. FWIW I'm still having the same issue, here's the start of my tb:

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip install --ignore-installed --no-user --prefix /private/var/folders/h3/zjctchbd6jb6vsgm_nq8fwnw0000gn/T/pip-build-env-z40x6oqf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy; python_version>='"'"'3.9'"'"''
     cwd: None
Complete output (233 lines):
Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and platform_system != "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and platform_system == "AIX"' don't match your environment
Collecting setuptools
  Using cached setuptools-51.1.1-py3-none-any.whl (2.0 MB)
Collecting wheel
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting Cython<3,>=0.29.21
  Using cached Cython-0.29.21-py2.py3-none-any.whl (974 kB)
Collecting numpy
  Using cached numpy-1.19.4.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/h3/zjctchbd6jb6vsgm_nq8fwnw0000gn/T/tmpto9f1e3i
         cwd: /private/var/folders/h3/zjctchbd6jb6vsgm_nq8fwnw0000gn/T/pip-install-qba16rxe/numpy
    Complete output (209 lines):
    Running from numpy source directory.
    setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      run_build = parse_setuppy_commands()
    Cythonizing sources
    Processing numpy/random/_bounded_integers.pxd.in
    Processing numpy/random/_philox.pyx
    Processing numpy/random/_bounded_integers.pyx.in
    Processing numpy/random/_sfc64.pyx
    Processing numpy/random/_mt19937.pyx
    Processing numpy/random/bit_generator.pyx
    Processing numpy/random/mtrand.pyx
    Processing numpy/random/_generator.pyx
    Processing numpy/random/_pcg64.pyx
    Processing numpy/random/_common.pyx
    blas_opt_info:
    blas_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/lib', '/usr/local/lib', '/usr/lib']
      NOT AVAILABLE

    blis_info:
      libraries blis not found in ['/Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/lib', '/usr/local/lib', '/usr/lib']
      NOT AVAILABLE

Then a bunch more lib errors after that, exact same error trace as @shanewilkins's original post

jonblum · 2021-01-04T17:00:22Z

I think pipenv's vendored packaging within patched/notpip needs to be updated with the same Big Sur wheel support (accepting 10_9 wheels even when the OS reports 11_0) that went out in packaging 20.7 and pip 20.3:
https://github.com/pypa/pip/pull/9170/files

That will at least allow use of the precompiled wheel for numpy, kicking the separate compiling-from-source issue down the road a bit.

radekerben · 2021-01-08T08:36:20Z

Hi,
this helped me:
export SYSTEM_VERSION_COMPAT=1

Source: https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target

jaymegordo · 2021-01-08T15:04:44Z

@radekerben thank you! That actually worked for me too, I can now successfully run pipenv lock without issue.

lashomb · 2021-01-08T22:13:36Z

This also worked for me as well. Thanks @radekerben 🥇

henryiii · 2021-03-29T19:19:46Z

You need packaging 20.9 for full Big Sur (Intel and Apple Silicon) support. 20.7+ for just Intel Big Sur. Or, of course, you can tell macOS to report a version number of 10.16 instead of 11 using SYSTEM_VERSION_COMPAT=1.

Any chance we can get a bump (master has 20.8) and version release?

lqflqf · 2021-04-29T12:29:42Z

Hi,
this helped me:
export SYSTEM_VERSION_COMPAT=1

Source: https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target

Thanks for the investigation. It works for me and I just upgraded to the latest Big Sur 11.3

henryiii · 2021-04-29T12:32:24Z

This is because packaging expects 10.x, so SYSTEM_VERSION_COMPAT=1 causes macOS to call itself 10.16 instead of 11.0, which temporarily fixes older versions of packaging. This should be properly fixed and updated in pipenv, though. It should just be an update to 20.9 and then a patch release, I would expect.

frostming · 2021-04-29T12:48:39Z

It is already done. I am seeking for people help testing the master branch before i can make a release

mzagaja · 2021-05-05T02:28:13Z

How can we help test?

lqflqf · 2021-05-05T02:57:29Z

Let me know how can I help test

lqflqf · 2021-05-07T12:15:09Z

This issue seems fixed in the latest python release 3.9.5

zedrdave · 2021-05-11T14:34:45Z

Just tested with python 3.9.5 and still having the problem (setting export SYSTEM_VERSION_COMPAT=1 fixes it).

henryiii · 2021-05-21T15:52:05Z

Does anyone have a good Universal2 wheel to test? I have a couple, like boost-histogram, but they require NumPy, and NumPy doesn't have a universal2 or arm64 for macOS wheel yet.

It looks like packaging is vendored at 20.8; 20.9 is required to support Apple Silicon property. But I need a good non-NumPy requiring package to test.

pipx run --spec git+https://github.com/pypa/pipenv.git pipenv install numpy -v

Starts building from source, which of course it should, as there are no supported wheels.

henryiii · 2021-05-21T16:26:58Z

@frostming could you please bump packaging to 20.9? Also, are you on the PyPA discord server?

Would it make sense to use the vendoring package from @pradyunsg like modern Pip does instead of using an old copy of the vendoring tooling from Pip? I tried on my M1 but couldn't install cffi, can try again on Intel later.

pradyunsg · 2021-05-21T23:36:09Z

If this is an issue on M1 + MacOS, that's possibly also because of numpy/numpy#18143.

henryiii · 2021-05-29T04:36:54Z

pipenv was just released without bumping packaging to 20.9. The docs clearly said all dependencies would be bumped before releasing, so bump PRs were discouraged. 😭

I'm trying to help with Universal2 wheel rollout, and packaging 20.9 (and pip 21.0.1) are needed to properly handle some universal2 wheels.

HWiese1980 · 2021-06-21T14:22:15Z

If this is an issue on M1 + MacOS, that's possibly also because of numpy/numpy#18143.

It's not exclusively. I have an older MacBook with an Intel chip and run into the same issue.

henryiii · 2021-06-21T14:26:58Z

A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the "bump all dependencies" step described in its docs, so sadly, it still doesn't understand that "11.0" is a valid macOS version. If you set export SYSTEM_VERSION_COMPAT=1, then macOS will report itself as "10.16" instead, which temporarily fixes the problem (only on Intel machines, I believe).

igor-cheb · 2021-08-22T12:26:39Z

A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the "bump all dependencies" step described in its docs, so sadly, it still doesn't understand that "11.0" is a valid macOS version. If you set export SYSTEM_VERSION_COMPAT=1, then macOS will report itself as "10.16" instead, which temporarily fixes the problem (only on Intel machines, I believe).

works on apple silicon with terminal started under rosetta simulation too

HWiese1980 · 2021-08-23T07:57:26Z

A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the "bump all dependencies" step described in its docs, so sadly, it still doesn't understand that "11.0" is a valid macOS version. If you set export SYSTEM_VERSION_COMPAT=1, then macOS will report itself as "10.16" instead, which temporarily fixes the problem (only on Intel machines, I believe).

It works. However, it's not exactly the ideal solution in my humble opinion.

ghost · 2022-01-28T10:53:14Z

I have faced the same problem but, it seems works when I am using latest version of numpy. In other words, whenever tried to install older version like 1.15.2 it gives me installation failure. So technically it should work when you are using the latest one.

matteius · 2022-08-21T11:42:58Z

This should be resolvable on pipenv==2022.8.19 because we rely exclusively on the vendor'd pip which is the latest version 22.2.2 at the time of writing.

chasedenecke · 2022-09-25T03:06:41Z

In my case the issue was that I hadn't agreed to the xcode licensing agreement.

I fixed the issue by typing sudo xcodebuild -license and agreeing to the license.

I was then prompted to update my xcode CLI tools, which was also necessary since further runs of pipenv install produced errors like xcode-select: Failed to locate 'clang', requesting installation of command line developer tools.

I tried repeatedly to run the software installation, but the same error kept popping up until I found this Stack Overflow thread which suggested running the following command:

sudo xcode-select -switch /Library/Developer/CommandLineTools

After that it finally worked.

…> 10.16 References: - https://eclecticlight.co/2020/07/21/big-sur-is-both-10-16-and-11-0-its-official/ - pypa/packaging#497 (comment) - pypa/pipenv#4564 (comment)

Marto32 mentioned this issue Mar 8, 2021

Error while locking the dependencies #4647

Closed

rdb mentioned this issue Apr 10, 2021

Unable to install panda3d with pipenv on macOS 11 panda3d/panda3d#1145

Closed

boblannon mentioned this issue May 13, 2021

Pipenv failed to build wheels (PEP 517) #4576

Closed

matteius added the OS: MacOS This issue affects MacOS. label Mar 13, 2022

matteius closed this as completed Aug 21, 2022

joshuacwnewton mentioned this issue Feb 2, 2024

Fix broken pip installer by forcing it to detect macOS 11 as 11.0 and not 10.16 spinalcordtoolbox/spinalcordtoolbox#4359

Merged

Uh oh!

Issue with NumPy, macOS 11 Big Sur, Python 3.9.1 Does pipenv not use the latest pip? #4564

Issue with NumPy, macOS 11 Big Sur, Python 3.9.1 Does pipenv not use the latest pip? #4564

Comments

shanewilkins commented Dec 12, 2020 • edited by pradyunsg Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

shanewilkins commented Dec 13, 2020

Uh oh!

jaymegordo commented Dec 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MCFreddie777 commented Dec 15, 2020

Uh oh!

mzagaja commented Dec 30, 2020

Uh oh!

jo3p commented Jan 1, 2021

Uh oh!

jaymegordo commented Jan 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonblum commented Jan 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radekerben commented Jan 8, 2021

Uh oh!

jaymegordo commented Jan 8, 2021

Uh oh!

lashomb commented Jan 8, 2021

Uh oh!

henryiii commented Mar 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lqflqf commented Apr 29, 2021

Uh oh!

henryiii commented Apr 29, 2021

Uh oh!

frostming commented Apr 29, 2021

Uh oh!

mzagaja commented May 5, 2021

Uh oh!

lqflqf commented May 5, 2021

Uh oh!

lqflqf commented May 7, 2021

Uh oh!

zedrdave commented May 11, 2021

Uh oh!

henryiii commented May 21, 2021

Uh oh!

henryiii commented May 21, 2021

Uh oh!

pradyunsg commented May 21, 2021

Uh oh!

henryiii commented May 29, 2021

Uh oh!

HWiese1980 commented Jun 21, 2021

Uh oh!

henryiii commented Jun 21, 2021

Uh oh!

igor-cheb commented Aug 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HWiese1980 commented Aug 23, 2021

Uh oh!

ghost commented Jan 28, 2022

Uh oh!

matteius commented Aug 21, 2022

Uh oh!

chasedenecke commented Sep 25, 2022

Uh oh!

shanewilkins commented Dec 12, 2020 •

edited by pradyunsg

Loading

jaymegordo commented Dec 13, 2020 •

edited

Loading

jaymegordo commented Jan 4, 2021 •

edited

Loading

jonblum commented Jan 4, 2021 •

edited

Loading

henryiii commented Mar 29, 2021 •

edited

Loading

igor-cheb commented Aug 22, 2021 •

edited

Loading