All kubernetes tests use the same host python version (#12374)

For Kubernetes tests all tests can be executed in the same python
version - default one - no matter which PYTHON_MAJOR_MINOR is
used. This is because we are testing Airflow which is deployed
via production image. Thanks to that we can fix the python version
to be default and avoid any python version problems (this is
especially important for cherry-picking to 1.10 where we have
python 2.7 and 3.5.

(cherry-picked from commit cbd6daf)

Author: potiuk

.github/workflows/ci.yml

@@ -620,7 +620,7 @@ jobs:
       - name: "Setup python"
         uses: actions/setup-python@v2
         with:
-          python-version: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
+          python-version: ${{ needs.build-info.outputs.defaultPythonVersion }}
       - name: "Free space"
         run: ./scripts/ci/tools/ci_free_space_on_ci.sh
       - name: "Prepare PROD Image"
@@ -634,17 +634,19 @@ jobs:
       - name: "Cache virtualenv for kubernetes testing"
         uses: actions/cache@v2
         env:
-          cache-name: cache-kubernetes-tests-virtualenv-v6-v1-10
+          cache-name: cache-kubernetes-tests-v7-v1-10
         with:
-          path: .build/.kubernetes_venv
-          key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}"
+          path: ".build/.kubernetes_venv*"
+          key: "venv-${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}\
+-${{ needs.build-info.outputs.defaultPythonVersion }}"
       - name: "Cache bin folder with tools for kubernetes testing"
         uses: actions/cache@v2
         env:
-          cache-name: cache-kubernetes-tests-bin-v6-v1-10
+          cache-name: cache-kubernetes-tests-v7-v1-10
         with:
-          path: .build/bin
-          key: "${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}\
+          path: ".build/bin"
+          key: "bin-${{ env.cache-name }}-${{ github.job }}-${{ hashFiles('setup.py') }}\
+-${{ needs.build-info.outputs.defaultPythonVersion }}\
 -${{ needs.build-info.outputs.defaultKindVersion }}\
 -${{ needs.build-info.outputs.defaultHelmVersion }}\
 -${{ matrix.kubernetes-version }}"

TESTING.rst

@@ -623,7 +623,7 @@ The virtualenv required will be created automatically when the scripts are run.
 
 4b) You can enter an interactive shell to run tests one-by-one
 
-This prepares and enters the virtualenv in ``.build/.kubernetes_venv`` folder:
+This prepares and enters the virtualenv in ``.build/.kubernetes_venv_<YOUR_CURRENT_PYTHON_VERSION>`` folder:
 
 .. code-block:: bash
 
@@ -666,7 +666,8 @@ a) Add the virtualenv as interpreter for the project:
     :align: center
     :alt: Kubernetes testing virtualenv
 
-The virtualenv is created in your "Airflow" source directory in ``.build/.kubernetes_venv/`` folder and you
+The virtualenv is created in your "Airflow" source directory in the
+``.build/.kubernetes_venv_<YOUR_CURRENT_PYTHON_VERSION>`` folder and you
 have to find ``python`` binary and choose it when selecting interpreter.
 
 b) Choose pytest as test runner:

scripts/ci/kubernetes/ci_run_kubernetes_tests.sh

@@ -61,7 +61,7 @@ else
         "--durations=100"
         "--cov=airflow/"
         "--cov-config=.coveragerc"
-        "--cov-report=html:airflow/www/static/coverage/"
+        "--cov-report=xml:files/coverage.xml"
         "--color=yes"
         "--maxfail=50"
         "--pythonwarnings=ignore::DeprecationWarning"
@@ -72,29 +72,27 @@ fi
 
 cd "${AIRFLOW_SOURCES}" || exit 1
 
-virtualenv_path="${BUILD_CACHE_DIR}/.kubernetes_venv"
+HOST_PYTHON_VERSION=$(python3 -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')
+readonly HOST_PYTHON_VERSION
+
+virtualenv_path="${BUILD_CACHE_DIR}/.kubernetes_venv_${HOST_PYTHON_VERSION}"
 
 if [[ ! -d ${virtualenv_path} ]]; then
     echo
     echo "Creating virtualenv at ${virtualenv_path}"
     echo
-    python -m venv "${virtualenv_path}"
+    python3 -m venv "${virtualenv_path}"
 fi
 
-# In Python 3.5 activating virtualenv hits undefined variable
-set +u
-
 . "${virtualenv_path}/bin/activate"
 
-set -u
-
-pip install --upgrade pip==20.2.3
+pip install --upgrade pip==20.2.3 wheel==0.35.1
 
 pip install pytest freezegun pytest-cov \
-  --constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+  --constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${HOST_PYTHON_VERSION}.txt"
 
 pip install -e ".[kubernetes]" \
-  --constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${PYTHON_MAJOR_MINOR_VERSION}.txt"
+  --constraint "https://raw.githubusercontent.com/apache/airflow/${DEFAULT_CONSTRAINTS_BRANCH}/constraints-${HOST_PYTHON_VERSION}.txt"
 
 if [[ ${interactive} == "true" ]]; then
     echo