TOX_OVERRIDES for testenv.pass_env are processed inconsistently

[jaraco]

Issue

After adding TOX_OVERRIDEs to my projects, releases have started failing when the TWINE_PASSWORD is missing from passenv.

Environment

Provide at least:

• OS: macOS, Linux

Output of pip list of the host Python, where tox is installed

 draft @ pipx runpip tox freeze
cachetools==5.3.1
chardet==5.2.0
colorama==0.4.6
distlib==0.3.7
filelock==3.12.3
packaging==23.1
platformdirs==3.10.0
pluggy==1.3.0
pyproject-api==1.6.1
tox==4.11.3
virtualenv==20.24.5

Output of running tox

Output of tox -rvv

 draft @ tox -rvv
py: 96 I find interpreter for spec PythonSpec(path=/Users/jaraco/.local/pipx/venvs/tox/bin/python) [virtualenv/discovery/builtin.py:58]
py: 97 D got python info of %s from (PosixPath('/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/bin/python3.11'), PosixPath('/Users/jaraco/Library/Application Support/virtualenv/py_info/1/0722d1d654d36a08896c2c727f3d426ef2212e71e059d909d7a685204d5b0d1d.json')) [virtualenv/app_data/via_disk_folder.py:131]
py: 98 D got python info of %s from (PosixPath('/opt/homebrew/opt/python@3.11/bin/python3.11'), PosixPath('/Users/jaraco/Library/Application Support/virtualenv/py_info/1/573546c1eada8c60b27f5300df4435af9ba2007194c80719d45c24c6ea4a493c.json')) [virtualenv/app_data/via_disk_folder.py:131]
py: 98 I proposed PythonInfo(spec=CPython3.11.5.final.0-64, system=/opt/homebrew/opt/python@3.11/bin/python3.11, exe=/Users/jaraco/.local/pipx/venvs/tox/bin/python, platform=darwin, version='3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
py: 98 D accepted PythonInfo(spec=CPython3.11.5.final.0-64, system=/opt/homebrew/opt/python@3.11/bin/python3.11, exe=/Users/jaraco/.local/pipx/venvs/tox/bin/python, platform=darwin, version='3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:67]
py: 99 D filesystem is not case-sensitive [virtualenv/info.py:26]
py: 114 I create virtual environment via CPython3macOsBrew(dest=/Users/jaraco/draft/.tox/py, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:50]
py: 114 D create folder /Users/jaraco/draft/.tox/py/bin [virtualenv/util/path/_sync.py:12]
py: 114 D create folder /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages [virtualenv/util/path/_sync.py:12]
py: 114 D write /Users/jaraco/draft/.tox/py/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:32]
py: 114 D       home = /opt/homebrew/opt/python@3.11/bin [virtualenv/create/pyenv_cfg.py:36]
py: 114 D       implementation = CPython [virtualenv/create/pyenv_cfg.py:36]
py: 114 D       version_info = 3.11.5.final.0 [virtualenv/create/pyenv_cfg.py:36]
py: 114 D       virtualenv = 20.24.5 [virtualenv/create/pyenv_cfg.py:36]
py: 115 D       include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:36]
py: 115 D       base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 [virtualenv/create/pyenv_cfg.py:36]
py: 115 D       base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 [virtualenv/create/pyenv_cfg.py:36]
py: 115 D       base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11 [virtualenv/create/pyenv_cfg.py:36]
py: 115 D symlink /opt/homebrew/opt/python@3.11/bin/python3.11 to /Users/jaraco/draft/.tox/py/bin/python [virtualenv/util/path/_sync.py:32]
py: 115 D create virtualenv import hook file /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:91]
py: 115 D create /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:94]
py: 116 D ============================== target debug ============================== [virtualenv/run/session.py:52]
py: 116 D debug via /Users/jaraco/draft/.tox/py/bin/python /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:200]
py: 116 D {
  "sys": {
    "executable": "/Users/jaraco/draft/.tox/py/bin/python",
    "_base_executable": "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/bin/python3.11",
    "prefix": "/Users/jaraco/draft/.tox/py",
    "base_prefix": "/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11",
    "real_prefix": null,
    "exec_prefix": "/Users/jaraco/draft/.tox/py",
    "base_exec_prefix": "/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11",
    "path": [
      "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python311.zip",
      "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11",
      "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload",
      "/Users/jaraco/draft/.tox/py/lib/python3.11/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)]",
  "makefile_filename": "/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/lib/python3.11/config-3.11-darwin/Makefile",
  "os": "<module 'os' (frozen)>",
  "site": "<module 'site' (frozen)>",
  "datetime": "<module 'datetime' from '/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/datetime.py'>",
  "math": "<module 'math' from '/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload/math.cpython-311-darwin.so'>",
  "json": "<module 'json' from '/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py'>"
} [virtualenv/run/session.py:53]
py: 140 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/jaraco/Library/Application Support/virtualenv) [virtualenv/run/session.py:57]
py: 142 D got embed update of distribution %s from ('setuptools', PosixPath('/Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/embed/3/setuptools.json')) [virtualenv/app_data/via_disk_folder.py:131]
py: 142 D got embed update of distribution %s from ('wheel', PosixPath('/Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/embed/3/wheel.json')) [virtualenv/app_data/via_disk_folder.py:131]
py: 142 D got embed update of distribution %s from ('pip', PosixPath('/Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/embed/3/pip.json')) [virtualenv/app_data/via_disk_folder.py:131]
py: 144 D using periodically updated wheel /Users/jaraco/Library/Application Support/virtualenv/wheel/house/wheel-0.41.0-py3-none-any.whl [virtualenv/seed/wheels/periodic_update.py:49]
py: 144 D using periodically updated wheel /Users/jaraco/Library/Application Support/virtualenv/wheel/house/setuptools-68.0.0-py3-none-any.whl [virtualenv/seed/wheels/periodic_update.py:49]
py: 144 D install pip from wheel /Users/jaraco/.local/pipx/venvs/tox/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py: 144 D install wheel from wheel /Users/jaraco/Library/Application Support/virtualenv/wheel/house/wheel-0.41.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py: 145 D install setuptools from wheel /Users/jaraco/Library/Application Support/virtualenv/wheel/house/setuptools-68.0.0-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:49]
py: 146 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/setuptools-68.0.0.dist-info to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/setuptools-68.0.0.dist-info [virtualenv/util/path/_sync.py:40]
py: 146 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/wheel-0.41.0-py3-none-any/wheel to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/wheel [virtualenv/util/path/_sync.py:40]
py: 146 D copy /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-23.2.1-py3-none-any/pip-23.2.1.virtualenv to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/pip-23.2.1.virtualenv [virtualenv/util/path/_sync.py:40]
py: 147 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-23.2.1-py3-none-any/pip to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/pip [virtualenv/util/path/_sync.py:40]
py: 149 D copy /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/distutils-precedence.pth to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:40]
py: 150 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/setuptools to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/setuptools [virtualenv/util/path/_sync.py:40]
py: 158 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/wheel-0.41.0-py3-none-any/wheel-0.41.0.dist-info to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/wheel-0.41.0.dist-info [virtualenv/util/path/_sync.py:40]
py: 161 D copy /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/wheel-0.41.0-py3-none-any/wheel-0.41.0.virtualenv to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/wheel-0.41.0.virtualenv [virtualenv/util/path/_sync.py:40]
py: 162 D generated console scripts wheel wheel3.11 wheel3 wheel-3.11 [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
py: 214 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/pkg_resources to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/pkg_resources [virtualenv/util/path/_sync.py:40]
py: 228 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/_distutils_hack to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:40]
py: 230 D copy /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/setuptools-68.0.0-py3-none-any/setuptools-68.0.0.virtualenv to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/setuptools-68.0.0.virtualenv [virtualenv/util/path/_sync.py:40]
py: 230 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
py: 295 D copy directory /Users/jaraco/Library/Application Support/virtualenv/wheel/3.11/image/1/CopyPipInstall/pip-23.2.1-py3-none-any/pip-23.2.1.dist-info to /Users/jaraco/draft/.tox/py/lib/python3.11/site-packages/pip-23.2.1.dist-info [virtualenv/util/path/_sync.py:40]
py: 297 D generated console scripts pip3 pip3.11 pip-3.11 pip [virtualenv/seed/embed/via_app_data/pip_install/base.py:43]
py: 298 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:63]
py: 300 D write /Users/jaraco/draft/.tox/py/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:32]
py: 300 D       home = /opt/homebrew/opt/python@3.11/bin [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       implementation = CPython [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       version_info = 3.11.5.final.0 [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       virtualenv = 20.24.5 [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 [virtualenv/create/pyenv_cfg.py:36]
py: 300 D       base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11 [virtualenv/create/pyenv_cfg.py:36]
  py: OK (0.21 seconds)
  congratulations :) (0.23 seconds)

Minimal example

 draft @ cat tox.ini
[testenv]

[testenv:release]
passenv=
        TWINE_PASSWORD
commands=
        py -c "import os; print(os.environ.get('TWINE_PASSWORD'))"

When passing pass_env to the config using overrides, the overrides supersede the explict value in the config:

 draft @ env TOX_OVERRIDE=testenv.pass_env+=FOO,BAR tox config -k passenv -e release
[testenv:release]
pass_env =
  BAR
  CC
  CCSHARED
  CFLAGS
  CPPFLAGS
  CURL_CA_BUNDLE
  CXX
  FOO
  HOME
  LANG
  LANGUAGE
  LDFLAGS
  LD_LIBRARY_PATH
  PIP_*
  PKG_CONFIG
  PKG_CONFIG_PATH
  PKG_CONFIG_SYSROOT_DIR
  REQUESTS_CA_BUNDLE
  SSL_CERT_FILE
  TERM
  TMPDIR
  VIRTUALENV_*
  http_proxy
  https_proxy
  no_proxy

Note that FOO and BAR are present, but TWINE_PASSWORD is lost.

If however, one changes passenv= to pass_env in the config,

 draft @ cat tox.ini
[testenv]

[testenv:release]
pass_env=
        TWINE_PASSWORD
commands=
        py -c "import os; print(os.environ.get('TWINE_PASSWORD'))"

Now TWINE_PASSWORD appears, but FOO and BAR are missing:

 draft @ env TOX_OVERRIDE=testenv.pass_env+=FOO,BAR tox config -k passenv -e release
[testenv:release]
pass_env =
  CC
  CCSHARED
  CFLAGS
  CPPFLAGS
  CURL_CA_BUNDLE
  CXX
  HOME
  LANG
  LANGUAGE
  LDFLAGS
  LD_LIBRARY_PATH
  PIP_*
  PKG_CONFIG
  PKG_CONFIG_PATH
  PKG_CONFIG_SYSROOT_DIR
  REQUESTS_CA_BUNDLE
  SSL_CERT_FILE
  TERM
  TMPDIR
  TWINE_PASSWORD
  VIRTUALENV_*
  http_proxy
  https_proxy
  no_proxy

What is the preferred configuration key for passenv/pass_env? I presume the latter.

I've tried other combinations of passenv in TOX_OVERRIDES and in the config, but I haven't yet found a combination that allows the pass_env to be applied at both the plain [testenv] and also extend the [testenv:release].pass_env. Is that possible? At the very least, I wouldn't expect a pass_env+= to ever mask an existing definition, but it does.

[hashar]

[testenv]

[testenv:release]
pass_env=
        TWINE_PASSWORD
commands=
        py -c "import os; print(os.environ.get('TWINE_PASSWORD'))"

Now TWINE_PASSWORD appears, but FOO and BAR are missing:

TOX_OVERRIDE=testenv.pass_env+=FOO,BAR tox config -k passenv -e release
[testenv:release]
pass_env =
 CC
 CCSHARED
 ...
 TWINE_PASSWORD
 ...

Since I have encountered the same issue, my explanation is the override does append to the generic [testenv]:

$ TOX_OVERRIDE=testenv.pass_env+=FOO,BAR tox config -k pass_env
[testenv:py]
pass_env =
  BAR
...
  FOO

But since:
A) the override applies to the testenv generic section, it is not appended to the other sections
B) In release testenv, pass_env is assigned which takes over whatever got set previously

If you set the override on the release environment, that works:

TOX_OVERRIDE=testenv:release.pass_env+=FOO,BAR tox config -k pass_env -e release
...
BAR
FOO
TWINE_PASSWORD

My guess is the override should be applied after the expansion of sections.