doc: Difference between PYTEST_PLUGINS and -p unclear

[hseg]

What's the problem this feature will solve?

The documentation for -p and PYTEST_PLUGINS differs and appears incomplete. In particular (in all the below, PYTEST_DISABLE_PLUGIN_AUTOLOAD=1):

• Their relation is unclear -- do they conflict? My experiments (see below) suggest they do -- mixing both of them appears to only load the plugins specified with -p.

• Their differences are unclear -- the same experiments suggest that for testing pytest plugins, only PYTEST_PLUGINS is suitable, whereas for general use -p can be used to specify plugins, even if PYTHONPATH is weird.

• Their descriptions are inconsistent (and subtly different?):

  • PYTEST_PLUGINS: (in pytest --help) "plugins to load during startup"
  • PYTEST_PLUGINS: (in reference) "list of modules that should be loaded as plugins"
  • -p "Early-load given plugin module name or entry point"

Describe the solution you'd like

The documentation at these points should be clarified. Also, technically-specific terms such as "early loading" should be documented somewhere in the reference documents and linked to when used (I discount this entry in the how-to guides -- it explains how to early-load a plugin, not what it does or how this differs from other ways of specifying plugins)

Additional context

My core motivation for going against pytest's grain and taking control of the specific plugins invoked is #13369, in the context of which the most likely workaround of "just use a venv" is discouraged.

The following experiments are how I reached my suppositions regarding pytest's behaviours. In all of them, I set PYTEST_DISABLE_PLUGIN_AUTOLOAD=1.

• pytest_recording requires itself, httpbin, and mock to be enabled to run tests successfully. Due to this, in my devenv I create a test installation of pytest_recording in a tempdir, and add it to PYTHONPATH for testing.

  • Passing all the plugins via PYTEST_PLUGINS successfully loads them all
  • Passing them all via -p doesn't load pytest_recording
  • Trying to correct this by mixing the two invocation styles and passing pytest_recording via PYTEST_PLUGINS doesn't load it either
  • Accidentally passing httpbin and mock by both methods yields a ValueError in register complaining the plugin has already been registered under a different name

• To check whether the dev installation of pytest_recording was the issue, I also checked papis. It contains an internal module that it needs registered with pytest for its tests to run, and thus also needs a dev installation. The tests are also configured to run with pytest_cov. Running the same permutations, I saw successful test runs iff both were specified precisely once, with no regard for where they were specified.

  That is, in particular, both PYTEST_PLUGINS=papis.testing and -p papis_testing worked.