Trouble following packaging libraries tutorial

[jrowellfx]

1. What is your operating system and version?

$ uname -a
Linux xxxxxx 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

2. What is your Python version?

Python 3.6.8

3. What version of pip do you have?

$ python3 -m pip --version
pip 21.1.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

4. If following an online tutorial or guide, please provide a link to the page or section giving you trouble:

https://packaging.python.org/tutorials/packaging-projects/
Now run this command from the same directory where pyproject.toml is located:
python3 -m build

5. Could you describe your issue in as much detail as possible?

$ python3 -m build
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "/usr/lib64/python3.6/venv/__init__.py", line 67, in create
    self._setup_pip(context)
  File "/usr/lib64/python3.6/venv/__init__.py", line 245, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/build-env-ksi9nbzq/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

THANK you for your kind attention!

[jrowellfx]

Further information for what it's worth.

The aforementioned system (i.e. centos 7) is freshly installed/created since yesterday. So it's pretty default-clean.

From root I executed:
# pip3 install --upgrade pip
Which I believe took me from the default v9.something to the v21.1.1 listed above.

Then, I installed/upgraded wheel and setuptools and build from my user account like so:

$  python3 -m pip install --upgrade build
$  python3 -m pip install --upgrade setuptools wheel

...whereupon running the python3 -m build command in my packaging_tutorial directory above lead to the error listed above.

As an experiment, I uninstalled build, setuptools and wheel locally, and reinstalled them as root as so:

#  python3 -m pipinstall --upgrade wheel
#  python3 -m pip install --upgrade setuptools
#  python3 -m pip install --upgrade build

Then, rerunning the python3 -m build command in my packaging_tutorial directory I get a very similar stack trace as follows.

$ python3 -m build
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/build/__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/usr/local/lib/python3.6/site-packages/build/__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/usr/local/lib/python3.6/site-packages/build/__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/usr/local/lib/python3.6/site-packages/build/__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/usr/local/lib/python3.6/site-packages/build/env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/usr/local/lib/python3.6/site-packages/build/env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "/usr/lib64/python3.6/venv/__init__.py", line 67, in create
    self._setup_pip(context)
  File "/usr/lib64/python3.6/venv/__init__.py", line 245, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/build-env-ee_lkhv9/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Not sure how helpful that is, I suspect not very. Thanks again!

[jrowellfx]

Sorry, one last thing that might be helpful. This is the 'packaging_tutorial" as I've created it, and trying to do a build of (following the tutorial): packaging_tutorial.

[henryiii]

I think this is a known bug that has been fixed, but not released yet. It might be worth trying master? If you install build with the virtualenv extra, python3 -m pip install build[virtualenv] (or just install virtualenv), that will work for now (plus it's faster ;) ), and a fix should ship soon(ish). If you could try the master version of build, it would be nice to see if it's fixed:

python3 -m pip install git+https://github.com/pypa/build@main

(It bothers me slightly that it's ensurepip that's failing out here)

[jrowellfx]

Hopefully I did this correctly. (Not as root) - I ran the following two commands (from my packaging_tutorial directory), included below with output. If I didn't do anything wrong, then it seems that the update didn't fix it. :-(

$ python3 -m pip install git+https://github.com/pypa/build@main
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/pypa/build@main
  Cloning https://github.com/pypa/build (to revision main) to /tmp/pip-req-build-pl4k9fxo
  Running command git clone -q https://github.com/pypa/build /tmp/pip-req-build-pl4k9fxo
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: pep517>=0.9.1 in /usr/local/lib/python3.6/site-packages (from build==0.3.1.post1) (0.10.0)
Requirement already satisfied: toml>=0.10.0 in /usr/local/lib/python3.6/site-packages (from build==0.3.1.post1) (0.10.2)
Requirement already satisfied: packaging>=19.0 in /usr/local/lib/python3.6/site-packages (from build==0.3.1.post1) (20.9)
Requirement already satisfied: importlib-metadata>=0.22 in /usr/local/lib/python3.6/site-packages (from build==0.3.1.post1) (4.0.1)
Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.6/site-packages (from importlib-metadata>=0.22->build==0.3.1.post1) (3.10.0.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata>=0.22->build==0.3.1.post1) (3.4.1)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/site-packages (from packaging>=19.0->build==0.3.1.post1) (2.4.7)
Building wheels for collected packages: build
  Building wheel for build (PEP 517) ... done
  Created wheel for build: filename=build-0.3.1.post1-py2.py3-none-any.whl size=13772 sha256=c90302e066daee88aa5e0408b1ec674be5d7544be8b51653a70577253add9159
  Stored in directory: /tmp/pip-ephem-wheel-cache-bx_hsfcx/wheels/8c/a1/31/24f18cf0a40148c76d9df1855a51a71b970070ecc949e16450
Successfully built build
Installing collected packages: build
Successfully installed build-0.3.1.post1
$ python3 -m build
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/env.py", line 92, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/home/jrowell/.local/lib/python3.6/site-packages/build/env.py", line 217, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True, symlinks=_fs_supports_symlink()).create(path)
  File "/usr/lib64/python3.6/venv/__init__.py", line 67, in create
    self._setup_pip(context)
  File "/usr/lib64/python3.6/venv/__init__.py", line 245, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/build-env-2sexpd0q/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Thanks Henry the Third, for your speedy reply!

[henryiii]

Is there anything unusual about your setup? A docker-based test works:

$ docker run --rm -it centos:7 bash
# yum upgrade && yum install -y python3-pip git
# pip3 install --upgrade pip
# python3 -m pip install build
# git clone https://github.com/jrowellfx/packaging_tutorial
# cd packaging_tutorial/
# python3 -m build

What is the output of:

python3 -m venv venv
source venv/bin/activate
python3 -Im ensurepip --upgrade --default-pip

? I'm curious, since it actually does nothing if pip is already present, but seems to be failing above. (This is also "venv" breaking, not really build, venv.EnvBuilder is not working)

[henryiii]

PS: I rather expect the first line to fail, as it seems your system cannot make a venv with pip, python3 -m venv venv will do the same thing as that venv.EnvBuilder command with pip=True.

[jrowellfx]

You predicted it right!

$ python3 -m venv venv
Error: Command '['/eno/media/src/packaging_tutorial/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

[layday]

We can't really tell what's going on here since venv swallows the output of ensurepip. Try running them in separate steps:

$ python3 -m venv --without-pip venv
$ venv/bin/python3 -Im ensurepip --upgrade --default-pip

[jrowellfx]

$ ls
LICENSE  README.md  pyproject.toml  setup.cfg  src  tests  venv
$ python3 -m venv --without-pip venv
$ venv/bin/python3 -Im ensurepip --upgrade --default-pip
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib64/python3.6/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 232, in _main
    default_pip=args.default_pip,
  File "/usr/lib64/python3.6/ensurepip/__init__.py", line 129, in _bootstrap
    shutil.copy(whl, tmpdir)
  File "/usr/lib64/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib64/python3.6/ensurepip/_bundled/setuptools-40.6.2-py2.py3-none-any.whl'
[jrowell@warmjets packaging_tutorial]$ 

PS - Thx layday

[layday]

I don't know what CentOS gets up to with ensurepip but I'd do a clean install of Python with yum and see if that helps. You should generally avoid changing things in the global prefix with pip (including running it as root).

[henryiii]

Distribution package managers often muddle with ensurepip (Debian especially, but RHEL is likely not free from blame here); it's also quite officially unsupported to do "sudo pip install -U pip` and replace the distribution provided pip package. However, the usual way to get around that is to make a venv 🤦. It is working in Docker for me, but everything is ROOT so might be slightly different. Also, my image of CentOS is much newer than the one used here, so it might be an issue (I see a thread about CentOS 7.7 breaking ensurepip; latest version is 7.9)

[jrowellfx]

@layday - couple of quick questions: What do you mean "avoid changing things in the global prefix with pip" - sorry, what's the "global prefix". Please forgive any dumb questions, I'm no expert in this particular area of using python, making packages, distributing code etc. Just trying to get my head around this by following the tutorials as best I can and they don't really talk about these details (that I've yet stumbled across).

Also, @layday, as I mentioned above I had a clean installation of python yesterday (it's a completely newly installed CentOS 7 OS) and I couldn't follow the tutorial as far as I did without having installed the packages as I did above. So how do I back out all my pip updates, including to pip itself? Then, how do I do a clean installation of python?

@henryiii - I'm willing to do anything as long as I can reach my end goal which is: To distribute simple module I created (years ago) that get's used by a couple of different command-line utilities that I wrote (which I also want to distribute to the world). These utilities need to be installed such that they can be used by anyone on a system (be it MacOS or Linux, possibly windows). Think installing a new "ls" command for everyone to use.

Anyway, my foray into pypa was to learn how to properly distribute the above mentioned packages that I'd written years ago because the method that I had used (using a setup.py to make a tarball, then having users install with that as root) doesn't seem to be working properly in all cases. Please see here.

I don't really understand virtual envs very well, and I was hoping that my cloudy understanding of them would clear up as I did the tutorials (which brought me here). So, the question is, if I can get this problem solved by using venv (which I'm still unclear how to do - sorry), then does that help me get to my goal of distributing my package so that it can be installed by "root" such that every user on a system (think CG post-production studio with 100 users) get's my new command-line commands (and access to the one library if they want it too.)

Hope that isn't too much info - I'm just trying to solve this one problem I have, so I want to make sure I'm going down a good path.

Thanks tons to you both for the help so far!

[henryiii]

If you want a quick fix, installing pip install virtualenv (or pip install build[virtualenv] should fix your problem. I'm hoping we could understand more about what is going wrong so it an be fixed in the future for non-virtualenv, but if python -m venv venv breaks, I'm not sure we can do much.

In more detail, there are two ways to make a virtual environment; the venv package has been built in since Python 3(.3ish?), so is always present. It uses the bundled copy of pip (via the built-in ensurepip module) to install pip into the virtual environment it creates. The virtualenv package is faster, kept up to date with a nice fresh pip install, and most importantly, is a separate package so package managers can't get their hands on it. The downside is that it's a package you have to install. They are very similar, venv was based on virtualenv if memory serves.

build makes a virtual environment for you when it builds your package, using virtualenv if available, and falling back to venv if not. The contents of your pyproject.toml requires are installed in a virtual environment for building, then discarded.

Virtual environments are basically just stand-alone Python installs (usually symlinking in your system Python), so that you can have a completely isolated "environment" of packages that do not interfere with your system Python. They are crucial if you develop code, since I don't want package A to work just because package B somewhere else happened to include a dependency I'm using in package A, and the two packages may even have conflicting requirements. Etc.

Once you've made a wheel, using any of these methods, it can be installed anywhere with no extra dependencies. See the benefits here. Of course, you should provide an SDist too, which can be built directly by pip if required.

PS: If you are distributing an "app", a great way to install it is with pipx; that will secretly make a virtual environment and install your app in that virtual environment, just exposing the app but not affecting your Python libraries at all. You can even do it temporarily, such as pipx run <app-name> <app-options>, which will download and run it from PyPI all in one step! Personally I use pipx run build to run build. :)

[henryiii]

PS: the version of pip coming from ensurepip is not the same as the one you have installed, it's bundled in to the stdlib as a tarball or something like that)

[jrowellfx]

@henryiii - Great info! Thanks. I'll dig into it a little later today if I can. Much obliged!