Alternate Python version error

[RajasGujarathi]

Using the docker image apache/airflow:slim-2.9.1-python3.12.

I installed an alternate python version 3.9.

.
.
for ADD_PYTHON_VERSION in $ALTERNATE_PYTHON_VERSIONS; do \
          wget --quiet "https://www.python.org/ftp/python/${ADD_PYTHON_VERSION}/Python-${ADD_PYTHON_VERSION}.tgz" \
          && tar xzf "Python-${ADD_PYTHON_VERSION}.tgz" \
          && cd "Python-${ADD_PYTHON_VERSION}" && ./configure --prefix=$AIRFLOW_USER_HOME_DIR/.local --enable-optimizations && make altinstall \
          && cd .. && rm -rf "Python-${ADD_PYTHON_VERSION}*" ; \
          done 
.
.

When using the PythonVirtualenvOperator in a DAG

.
.
python_3_9 = PythonVirtualenvOperator(task_id='python_3_9',
                                      requirements=["colorama==0.4.0"],
                                      system_site_packages=False,
                                      python_callable=func,
                                      python_version='3.9',
                                      dag=dag)
.
.

The DAG fails with an error

{process_utils.py:183} INFO - Executing cmd: /home/***/.local/bin/python -m virtualenv /tmp/venv_4wv1u08 --python=python3.9
{process_utils.py:187} INFO - Output:
{process_utils.py:191} INFO - PermissionError: [Errno 13] Permission denied: '/root/bin'

Would someone be able to help?

Suspect
The user airflow has a path /root/bin on its $PATH but does not have access to this directory, can this be the cause of the error?

[potiuk]

You must have somewhat kip pip location cashed to (/root/bin) from switching to root user when you installed your python. Not sure how because you have not published the whole dockerfile of yours. The hash -r somewhere in your Dockerfile should clean the cache. Generally you should use USER directive in the dockerfile to switch users (see examples here for inspiration https://airflow.apache.org/docs/docker-stack/build.html)

[RajasGujarathi]

Thanks. @potiuk

Trying to check the pip cache location.

Attaching a screenshot from the commands executed inside the container.

DAG Logs

[RajasGujarathi]

Non-optimised Dockerfile created only for simulating the behaviour

FROM apache/airflow:slim-2.9.1-python3.12

ARG ALTERNATE_PYTHON_VERSIONS="3.9.0"

WORKDIR /tmp

USER root 

RUN apt-get update \
    && apt-get install -y --fix-missing --no-install-recommends \
        zlib1g-dev \
        wget \
        libsasl2-dev \
        libldap2-dev \
        libssl-dev \
        gcc \
        make

USER airflow

WORKDIR $AIRFLOW_USER_HOME_DIR

RUN pip install virtualenv \
    && for ADD_PYTHON_VERSION in $ALTERNATE_PYTHON_VERSIONS; do \
    wget --quiet "https://www.python.org/ftp/python/${ADD_PYTHON_VERSION}/Python-${ADD_PYTHON_VERSION}.tgz" \
    && tar xzf "Python-${ADD_PYTHON_VERSION}.tgz" \
    && cd "Python-${ADD_PYTHON_VERSION}" && ./configure --prefix=$AIRFLOW_USER_HOME_DIR/.local --enable-optimizations && make altinstall \
    && cd .. && rm -rf "Python-${ADD_PYTHON_VERSION}*" ; \
    done

---

Steps to reproduce:

• Build the docker image
• Run the image - docker run -it <image_name:tag> bash
• Command to create virtualenv - python -m virtualenv /tmp/venviplvcl3b --python=python3.9
• Modify the PATH environment variable, remove /root/bin
• Try re-creating the virtualenv

[potiuk]

• have you tried hash -r ?
• Can you propose another fix?

[RajasGujarathi]

have you tried hash -r ?

Yes, please find the findings below

Can you propose another fix?

Cannot think of any for now, will share as soon as I can find out something.

[RajasGujarathi]

Raised #39953