CommandError: To modify pip, please run the following command

[edp1096]

Description

When install pip 22.1 or higher using wheel into python embeddable, error occured.

Is there something I losted?

Expected behavior

Install latest pip to fresh python embeddable

pip version

22.1 or higher

Python version

3.10.6

OS

Windows 11 Pro 21H2 22000.856

How to Reproduce

Create and run below batch files in each empty folder

• work.cmd - Installing pip 22.0.4

@echo off

set PYTHON_VERSION=3.10.6

set PATH=%cd%\bin;C:\Windows;C:\Windows\System;C:\Windows\System32
set PATH=%cd%\python3;%cd%\python3\Scripts;%PATH%

set PIP_NO_CACHE_DIR=off
@REM set PIP_TARGET=%cd%\pip\cache

curl --progress-bar -Lo python3.zip https://www.python.org/ftp/python/%PYTHON_VERSION%/python-%PYTHON_VERSION%-embed-amd64.zip
if not exist python3 ( mkdir python3 )
tar -xf python3.zip --directory=python3

md python3\Scripts

curl --progress-bar -Lo get-pip.py https://bootstrap.pypa.io/get-pip.py
@REM python get-pip.py


echo import site>> python3\python310._pth
echo .\DLLs>> python3\python310._pth
echo .\lib>> python3\python310._pth
echo .\lib\plat-win>> python3\python310._pth
echo .\lib\site-packages>> python3\python310._pth

@REM del get-pip.py
@REM del python3.zip


@REM Work
set PIP_WHEEL_NAME=pip-22.0.4-py3-none-any.whl
set PIP_WHEEL_URL=https://files.pythonhosted.org/packages/4d/16/0a14ca596f30316efd412a60bdfac02a7259bf8673d4d917dc60b9a21812/%PIP_WHEEL_NAME%
curl --progress-bar -Lo %PIP_WHEEL_NAME% %PIP_WHEEL_URL%
python %PIP_WHEEL_NAME%/pip install --no-index %PIP_WHEEL_NAME%

• not_work.cmd - Installing pip 22.2.2

@echo off

set PYTHON_VERSION=3.10.6

set PATH=%cd%\bin;C:\Windows;C:\Windows\System;C:\Windows\System32
set PATH=%cd%\python3;%cd%\python3\Scripts;%PATH%

set PIP_NO_CACHE_DIR=off
@REM set PIP_TARGET=%cd%\pip\cache

curl --progress-bar -Lo python3.zip https://www.python.org/ftp/python/%PYTHON_VERSION%/python-%PYTHON_VERSION%-embed-amd64.zip
if not exist python3 ( mkdir python3 )
tar -xf python3.zip --directory=python3

md python3\Scripts

curl --progress-bar -Lo get-pip.py https://bootstrap.pypa.io/get-pip.py
@REM python get-pip.py


echo import site>> python3\python310._pth
echo .\DLLs>> python3\python310._pth
echo .\lib>> python3\python310._pth
echo .\lib\plat-win>> python3\python310._pth
echo .\lib\site-packages>> python3\python310._pth

@REM del get-pip.py
@REM del python3.zip

@REM Not work
set PIP_WHEEL_NAME=pip-22.2.2-py3-none-any.whl
set PIP_WHEEL_URL=https://files.pythonhosted.org/packages/1f/2c/d9626f045e7b49a6225c6b09257861f24da78f4e5f23af2ddbdf852c99b8/%PIP_WHEEL_NAME%
curl --progress-bar -Lo %PIP_WHEEL_NAME% %PIP_WHEEL_URL%
python %PIP_WHEEL_NAME%/pip install --no-index %PIP_WHEEL_NAME%

Output

• Expected - work 22.0 or before

Microsoft Windows [Version 10.0.22000.856]
(c) Microsoft Corporation. All rights reserved.

D:\devs\pip_work>install_python.cmd
################################################################################################################ 100.0%
################################################################################################################ 100.0%
Processing d:\devs\pip_work\pip-22.0.4-py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-22.0.4

D:\devs\pip_work>

• Output - not work 22.1 or higher

Microsoft Windows [Version 10.0.22000.856]
(c) Microsoft Corporation. All rights reserved.

D:\devs\pip_notwork>install_python.cmd
################################################################################################################ 100.0%
################################################################################################################ 100.0%
Processing d:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl
┌─────────────────────────────── Traceback (most recent call last) ────────────────────────────────┐
│ D:\devs\pip_notwork\runpy.py:196 in _run_module_as_main                                          │
│                                                                                                  │
│ [Errno 2] No such file or directory: 'D:\\devs\\pip_notwork\\runpy.py'                           │
│                                                                                                  │
│ D:\devs\pip_notwork\runpy.py:86 in _run_code                                                     │
│                                                                                                  │
│ [Errno 2] No such file or directory: 'D:\\devs\\pip_notwork\\runpy.py'                           │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\__main__.py:31 in <module>                   │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\__main__.py'                           │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\_internal\cli\main.py:70 in main             │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\_internal\\cli\\main.py'               │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\_internal\cli\base_command.py:101 in main    │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\_internal\\cli\\base_command.py'       │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\_internal\cli\base_command.py:221 in _main   │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\_internal\\cli\\base_command.py'       │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\_internal\cli\req_command.py:247 in wrapper  │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\_internal\\cli\\req_command.py'        │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\_internal\commands\install.py:407 in run     │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\_internal\\commands\\install.py'       │
│                                                                                                  │
│ D:\devs\pip_notwork\pip-22.2.2-py3-none-any.whl\pip\_internal\utils\misc.py:577 in               │
│ protect_pip_from_modification_on_windows                                                         │
│                                                                                                  │
│ [Errno 2] No such file or directory:                                                             │
│ 'D:\\devs\\pip_notwork\\pip-22.2.2-py3-none-any.whl\\pip\\_internal\\utils\\misc.py'             │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
CommandError: To modify pip, please run the following command:
D:\devs\pip_notwork\python3\python.exe -m pip install --debug --no-index pip-22.2.2-py3-none-any.whl

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[hvbtup]

Same or similar problem here.

I am using the embeddable zip distribution on Windows.
Today I wanted to replace 3rdParty software with newer releases.
Before: Python 3.10.2, pip 21.3.1 - this worked.
Now: Python 3.10.9, pip 22.3.1 - does not work.

The way I set it up is:

First, unzip the Embeddable ZIP into a directory python310.
Then

pushd python310

rem this does not help:
echo python310.zip>    python310._pth
echo .>>               python310._pth
echo import site>>     python310._pth

.\python ..\3rdParty\pure_python\pip-22.3.1-py3-none-any.whl/pip install --no-warn-script-location --no-index -f ..\3rdParty\pure_python\ pip

echo setuptools und wheel offline installieren...
.\python -m pip install --no-warn-script-location --no-index -f ..\3rdParty\pure_python\ setuptools wheel

echo pywin32 installieren...
.\python -m pip install --no-warn-script-location --no-index -f ..\3rdParty\win32-AMD64\ pywin32

I downloaded the files from pypi because when installing on some servers, they are not connected to the internet. So this is supposed to work offline.

The output is:

Looking in links: c:\xxx\pure_python\
Processing c:\xxx\3rdparty\pure_python\pip-22.3.1-py3-none-any.whl
ERROR: To modify pip, please run the following command:
C:\xxx\python310\python.exe -m pip install --no-warn-script-location --no-index -f C:\xxx\3rdParty\pure_python\ pip
setuptools und wheel offline installieren...
C:\xxx\python310\python.exe: No module named pip
pywin32 installieren...
C:\xxx\python310\python.exe: No module named pip

[pfmoore]

Running pip direct from the wheel is unsupported. You may have better luck using the zipapp distribution of pip to bootstrap things.

But this looks like it's probably also not a supported way of using the embeddable distribution of Python, so you may have problems even then.

[hvbtup]

I switched to get-pip.py - that worked.

[hvbtup]

This would be a workaround for the OP as well. But nevertheless, the frequent behavior changes in pip are causing me headache.

[pradyunsg]

Using pip with embedded Python is not supported.

[hvbtup]

Using pip with embedded Python is not supported.

I think this makes using embedded Python pointless for most use cases.
Thus this should be discussed with the CPython maintainers.

Let me explain a little bit about the background:

I've been using Python for several background programs in commercial applications for ~ 20 years.
These programs are running on servers "on-premise" in our customers' environments.

In many cases, several different variations of such programs are running on the machine, a typical case may be running the same kind of programs for test, integration and production environment.

Independence requirement:
It must be assured changes in one environment do not change the behavior of a different environment, e.g. if I install a new release of program A in the test environment, this must not effect the production environment.

The classic Python installation required admin rights and installed a pythonXX.dll inside the windows\system32 directory.
So it was not possible with this kind of installation to install e.g. Python 2.4.1 and Python 2.4.3 in parallel while satisfying the independence requirement. It doesn't matter if "install for all users" or "install just for me" is used.

So we tried a different kind of installation. Newer Python releases allowed the installer program to run headless and to specify an installation directory. This worked fine with Python 2.6 and Python 2.7. We could install different Python releases independently in different directories, without modifying the Windows registry or creating files inside windows\system32. We just had to set some environment variables for the program and everything was fine.

When we switched to Python 3.1 (IIRC) this didn't work anymore.
I don't remember what exactly the reason was, but I think it was something about the py launcher. Anyway, it seemed impossible to use different python releases really independently.

We considered using virtual environments, but at that time, this didn't work well on Windows (everything was changing constantly, virtualenv vs venv, ...).

We then found a solution by choosing the embeddable ZIP distribution.

Our programs require ~ 10 3rdParty packages, not all of them are pure Python.
While it is certainly possible to install these packages without pip, that is a very error-prone process.
But once pip is installed, installing other packages is a snap. So the "fiddling around" was only needed once to install pip, instead of once for each package.

This has become more and more important during the last years, because updating packages to newer releases is needed much more frequently nowadays, with everyone being concerned about security,

A bit provocantly, one could say:

Python without pip is useless nowadays.

With pip being available, our clients can install fixes for known security issues in the 3rd Party packages themselves, if they like.
And we can create little helper tools for this task.

Without pip, as soon as a security issue is found in one of the 3rdParty packages and a client wants it to be fixed, they can only tell us about it and ask for a new program release.
Then we would have to build and test such a new release and provide it for download; the client would have to install it.

This results in unnecessary work for us and for our clients and - more important - in a much longer reaction time before a security issue can be fixed in our clients' environment.

Another important problem arises then: Our clients do not want a new release of our software, because that might change behavior. According to their compliance rule, they'd have to test the whole new software then, or even to run an expensive manual software certification process. They just want the security fix, nothing more and nothing less.

If we have to supply a new release of our software just for security fixes in 3rdParty packages, then in fact this is going to be a mess of releases, because the bundled software would be unique for every customer, resulting in several releases per day.

Thus I think, Python Embeddable Zip without pip is useless for real-world development for Python Applications as described in "Using Python for Windows".

The only remaining use-case for Embeddable Python would be for using Python as a scripting language (such as in Blender).

[edp1096]

I found some alternative ways like using get-pip.py or pip.pyz with python embeddable.
Those are work well so, I understand and agree to closing the issue but I cannot understand why pip not support python embeddable.

[pfmoore]

@hvbtup Did you read the documentation for the embedded Python distribution, which clearly says

It is intended for acting as part of another application, rather than being directly accessed by end-users.

If you're trying to use it in a way that contradicts its intended use, I don't see how you can complain if things don't work. If your use case isn't supported well by any of the available Python distribution mechanisms, and you're not willing to "roll your own" solution, then that's something you'd need to raise with the core Python project.

The intended use of pip with the embedded distribution, in line with its designed use as part of another application, is for the libraries required by the application to be preinstalled using something like pip install --target.

The only remaining use-case for Embeddable Python would be for using Python as a scripting language (such as in Blender).

Yes, that sounds to me like exactly what the documentation states it's designed for...

I cannot understand why pip not support python embeddable.

To be clear, we don't do anything that specifically stops pip working with the embedded distribution. However:

1. We don't test pip with that distribution.
2. We don't have the means to replicate issues with that distribution (there are too many possible ways users can "embed" Python, and we can't be expected to replicate a user's precise configuration).
3. If you can replicate a problem with a normal Python build, we will fix it.
4. Conversely, if you can't replicate the problem with a normal Python build, the issue is likely to be with your configureation of the embedded distribution rather than a problem with pip (the embedded distribution is identical as far as language features is concerned).

@edp1096 note that your reported issue is because you are trying to run pip from the wheel, without unpacking. That is not supported, and that has nothing to do with the embedded distribution. We provide get-pip.py and the zipapp distribution for use cases where you need to bootstrap pip. I provided this information in the very first comment I made on the issue, and both you and @hvbtup confirmed it works. So regardless of any question around whether pip supports the embedded distribution, you got a fix for your issue. Is that not enough?

[edp1096]

@pfmoore Sure, my issue certainly solved from your guide.
Because of my usage, I just wondered about embeddable supporting. :)

Thank you.

[hvbtup]

@pfmoore I absolutely understand that you say it's not supported.
And I don't say it's your fault or anything.

But you misunderstood some of my points.

It is intended for acting as part of another application, rather than being directly accessed by end-users.
If you're trying to use it in a way that contradicts its intended use, ...

I didn't say that the embedded Python is directly accessed by end-users.
What I said is that it's important to be able to apply security hotfixes and ideally a server administrator (not an end-user) should be able to do this.

The only remaining use-case for Embeddable Python would be for using Python as a scripting language (such as in Blender).
Yes, that sounds to me like exactly what the documentation states it's designed for...

No. Note that the documentation clearly says that there are two use-cases (4.4.1 and 4.4.2).
Or application is a typical example for 4.4.1 (we are using Python internally almost for the entire program, but this is invisible for the end-users), Blender is an example for 4.4.2 (using Python as a scripting language, end users can write their own scripts).

We don't have the means to replicate issues with that distribution (there are too many possible ways users can "embed" Python, and we can't be expected to replicate a user's precise configuration).
If you can replicate a problem with a normal Python build, we will fix it.

Well, if we agree for a moment that Windows Embeddable ZIP is a normal Python build and put aside that pip is not officially supported here: This problem is reproducible with a normal Python build. Just unzip the embeddable Python and then try install pip.

It seems we used an undefined edge case and the behavior in this case has changed, which is perfectly valid, but bad luck for us.

We found a workaround now by using get-pip.py, so it's ok as long as that works.
Would it be possible for the pip team to automatically test get-pip in conjunction with the Embeddable Zip build?
Even though this isnt' supported, it is working now and it would be nice to know and document when this situation changes.

BTW the embeddable Python does not contain the ensurepip module. I didn't find this documented anywhere.