MINOR: upgrade pip from 20.2.2 to 21.1.1 #10661

chia7712 · 2021-05-10T11:05:02Z

The following error happens on my mac m1 when building docker image for system tests.

Collecting pynacl
  Using cached PyNaCl-1.4.0.tar.gz (3.4 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/local/lib/python3.8/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k867aac0/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"''
       cwd: None
  Complete output (14 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.8/dist-packages/pip/__main__.py", line 23, in <module>
      from pip._internal.cli.main import main as _main  # isort:skip # noqa
    File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/main.py", line 5, in <module>
      import locale
    File "/usr/lib/python3.8/locale.py", line 16, in <module>
      import re
    File "/usr/lib/python3.8/re.py", line 145, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/local/lib/python3.8/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k867aac0/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel 'cffi>=1.4.1; python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.

There was a related issue: pypa/pip#9689 and it is already fixed by pypa/pip#9689 (included by pip 21.1.1). I test the pip 21.1.1 and it works well on mac m1.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

ijuma

LGTM, thanks!

lbradstreet · 2021-07-08T14:39:59Z

This is failing for me with:

Step 10/68 : RUN python3 -m pip install -U pip==21.1.1;                                                                                                                                                                                             
 ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.                                                                                                                     
 ---> Running in 23f5e4d95ed4                                                                                                                                                                                                                       
Collecting pip==21.1.1                                                                                                                                                                                                                              
  Could not find a version that satisfies the requirement pip==21.1.1 (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1, 1.3, 1.3.1
, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0
.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 20.1, 20.1.1, 20.2b1, 20.2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 
20.3b1, 20.3, 20.3.1, 20.3.2, 20.3.3, 20.3.4)                                                                                                                                                                                                       
No matching distribution found for pip==21.1.1

@chia7712 are you using a different jdk base docker image version on your m1?

chia7712 · 2021-07-08T14:44:55Z

are you using a different jdk base docker image version on your m1?

IIRC, I was using openjdk:8. Will retry it later :(

chia7712 · 2021-07-08T15:02:07Z

build image on MacBook Air M1

build image on iMac 27 intel

@lbradstreet Please take a look at above screenshot. I don't observe the error you mentioned. BTW, the step Step 10/68 is different from repo's docker file. Did you build image by other docker file?

lbradstreet · 2021-07-08T15:14:13Z

@chia7712 thanks for checking. Yes, my dockerfile was a bit modified. The unmodified run is:

Sending build context to Docker daemon  43.52kB
Step 1/54 : ARG jdk_version=openjdk:8
Step 2/54 : FROM $jdk_version
 ---> 09df0563bdfc
Step 3/54 : MAINTAINER Apache Kafka [email protected]
 ---> Using cache
 ---> abfbf85c6575
Step 4/54 : VOLUME ["/opt/kafka-dev"]
 ---> Using cache
 ---> d9178c554c16
Step 5/54 : ENV TZ="/usr/share/zoneinfo/America/Los_Angeles"
 ---> Using cache
 ---> 8245a50f89b4
Step 6/54 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 6e41f5a58d78
Step 7/54 : ARG ducker_creator=default
 ---> Using cache
 ---> 9d39a84cd18d
Step 8/54 : LABEL ducker.creator=$ducker_creator
 ---> Using cache
 ---> 25fe89abca0c
Step 9/54 : RUN apt update && apt install -y sudo git netcat iptables rsync unzip wget curl jq coreutils openssh-server net-tools vim python3-pip python3-dev libffi-dev libssl-dev cmake pkg-config libfuse-dev iperf traceroute && apt-get -y clean
 ---> Using cache
 ---> 0b5597132e07
Step 10/54 : RUN python3 -m pip install -U pip==21.1.1;
 ---> [Warning] Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
 ---> Running in 0bae66304b96
Collecting pip==21.1.1
  Could not find a version that satisfies the requirement pip==21.1.1 (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 20.1, 20.1.1, 20.2b1, 20.2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 20.3b1, 20.3, 20.3.1, 20.3.2, 20.3.3, 20.3.4)
No matching distribution found for pip==21.1.1
The command '/bin/sh -c python3 -m pip install -U pip==21.1.1;' returned a non-zero code: 1

chia7712 · 2021-07-08T15:21:31Z

@lbradstreet Could you share the ID of openjdk:8 to me? the image id of openjdk:8 in my env is d61c96e2d100

lbradstreet · 2021-07-08T15:25:25Z

@lbradstreet Could you share the ID of openjdk:8 to me? the image id of openjdk:8 in my env is d61c96e2d100

I ran docker system prune before reporting this in case something was happening there.

       "Id": "sha256:09df0563bdfc760be1ce6f153f4fdb1950e8ccc9dbb298f8c76da38dc0cecfd6",                                  
       "RepoTags": [                                                                                                                                                                                                                               
           "openjdk:8"                                                                                                   
       ],                   
       "RepoDigests": [                                                                                                                                                                                                                            
           "openjdk@sha256:c58264860b32c8603ac125862d17b1d73896bceaa33c843c9a7163904786eb82"                                                                                                                                                       
       ],                                                                                                                                                                                                                                          
       "Parent": "",                                                                                                     
       "Comment": "",                                 
       "Created": "2019-11-23T14:34:16.005709367Z",                                                                      
       "Container": "1f98649c600b636393cc3d3cd8b594134fe979b85d122dc5fa41d4d8311de8b3",
       "ContainerConfig": {                               
           "Hostname": "",                                                                                               
           "Domainname": "",          
           "User": "",                                                                                                   
           "AttachStdin": false,                                                                                                                                                                                                                               "AttachStdout": false,                                                                                                                                                                                                                              "AttachStderr": false,                                                                                        
           "Tty": false,                                                                                                                                                                                                                           
           "OpenStdin": false,                                                                                                                                                                                                                     
           "StdinOnce": false,                                                                                                                                                                                                                     
           "Env": [                                                                                                      
               "PATH=/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",                                                                                                                                       
               "LANG=C.UTF-8",                                                                                                                                                                                                                     
               "JAVA_HOME=/usr/local/openjdk-8",                                                                                                                                                                                                   
               "JAVA_VERSION=8u232",                                                                                                                                                                                                               
               "JAVA_BASE_URL=https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jdk_",                                                                                                           
               "JAVA_URL_VERSION=8u232b09"                                                                                                                                                                                                         
           ],

lbradstreet · 2021-07-08T15:26:36Z

@lbradstreet Could you share the ID of openjdk:8 to me? the image id of openjdk:8 in my env is d61c96e2d100

I ran docker system prune before reporting this in case something was happening there.

       "Id": "sha256:09df0563bdfc760be1ce6f153f4fdb1950e8ccc9dbb298f8c76da38dc0cecfd6",                                  
       "RepoTags": [                                                                                                                                                                                                                               
           "openjdk:8"                                                                                                   
       ],                   
       "RepoDigests": [                                                                                                                                                                                                                            
           "openjdk@sha256:c58264860b32c8603ac125862d17b1d73896bceaa33c843c9a7163904786eb82"                                                                                                                                                       
       ],                                                                                                                                                                                                                                          
       "Parent": "",                                                                                                     
       "Comment": "",                                 
       "Created": "2019-11-23T14:34:16.005709367Z",                                                                      
       "Container": "1f98649c600b636393cc3d3cd8b594134fe979b85d122dc5fa41d4d8311de8b3",
       "ContainerConfig": {                               
           "Hostname": "",                                                                                               
           "Domainname": "",          
           "User": "",                                                                                                   
           "AttachStdin": false,                                                                                                                                                                                                                               "AttachStdout": false,                                                                                                                                                                                                                              "AttachStderr": false,                                                                                        
           "Tty": false,                                                                                                                                                                                                                           
           "OpenStdin": false,                                                                                                                                                                                                                     
           "StdinOnce": false,                                                                                                                                                                                                                     
           "Env": [                                                                                                      
               "PATH=/usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",                                                                                                                                       
               "LANG=C.UTF-8",                                                                                                                                                                                                                     
               "JAVA_HOME=/usr/local/openjdk-8",                                                                                                                                                                                                   
               "JAVA_VERSION=8u232",                                                                                                                                                                                                               
               "JAVA_BASE_URL=https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jdk_",                                                                                                           
               "JAVA_URL_VERSION=8u232b09"                                                                                                                                                                                                         
           ],

Maybe I didn't fully clear all images. I'll see if I can delete them in another way.

lbradstreet · 2021-07-08T15:29:47Z

@chia7712 sorry for the false alarm and thanks for checking that it worked on your side. The problem was that I should have run docker system prune -a rather than docker system prune. It's now working.

vvcephei · 2021-07-08T22:21:00Z

Hmm, I also hit that same error today.

It seems to have been caused because the image was running Debian 9, and the latest version of python3 available in Debian 9 is 3.5, which only supports pip up to version 20. I confirmed this was the issue by getting a terminal on top of the errored image, downloading and compiling python 3.6, and running python3.6 -m pip install -U pip=21.1.1.

After running docker system prune -a, I see it's running Debian 10:

root@826557ee9546:/# cat /etc/issue
Debian GNU/Linux 10 \n \l

Building on the recent discussion here, maybe what's happening is that we had an older cached version of the openjdk:8 image, which limited the python version to 3.5, which in turn make it impossible to upgrade to pip 21.

Just wanted to provide more context. I assume others will hit the same thing. The solution is to run docker system prune -a.

vvcephei · 2021-07-08T22:23:09Z

Ah, and now I noticed that we've added a message to this effect in the script today: #10995

Carry on :)

MINOR: upgrade pip from 20.2.2 to 21.1.1

01a363b

chia7712 requested a review from ijuma May 10, 2021 11:05

ijuma approved these changes May 28, 2021

View reviewed changes

chia7712 merged commit 0bf4b47 into apache:trunk May 29, 2021

lbradstreet mentioned this pull request Jul 8, 2021

MINOR: Hint about "docker system prune" when ducker-ak build fails #10995

Merged

chia7712 deleted the MINOR-10661 branch March 25, 2024 15:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MINOR: upgrade pip from 20.2.2 to 21.1.1 #10661

MINOR: upgrade pip from 20.2.2 to 21.1.1 #10661

Uh oh!

chia7712 commented May 10, 2021

Uh oh!

ijuma left a comment

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

chia7712 commented Jul 8, 2021

Uh oh!

chia7712 commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

chia7712 commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

vvcephei commented Jul 8, 2021

Uh oh!

vvcephei commented Jul 8, 2021

Uh oh!

Uh oh!

MINOR: upgrade pip from 20.2.2 to 21.1.1 #10661

MINOR: upgrade pip from 20.2.2 to 21.1.1 #10661

Uh oh!

Conversation

chia7712 commented May 10, 2021

Committer Checklist (excluded from commit message)

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

chia7712 commented Jul 8, 2021

Uh oh!

chia7712 commented Jul 8, 2021

build image on MacBook Air M1

build image on iMac 27 intel

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

chia7712 commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

lbradstreet commented Jul 8, 2021

Uh oh!

vvcephei commented Jul 8, 2021

Uh oh!

vvcephei commented Jul 8, 2021

Uh oh!

Uh oh!