pytest (doctests) unable to recognize the root package

[jaraco]

Another challenge I've encountered pertains to running tests, namely doctests. The problem with running doctests in a shallow repo is that relative references are no longer viable. Consider in coherent.build (0.4.0):

# __init__.py

from . import discovery

Although the discovery module exists alongside __init__.py and both will be/are found in coherent/build after installation, pytest "discovers" __init__.py as a plain module and not in the context of coherent.build, so fails thus:

__________________________________________________________ ERROR collecting __init__.py __________________________________________________________
ImportError while importing test module '/Users/jaraco/code/coherent-oss/coherent.build/__init__.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
__init__.py:32: in <module>
    from . import discovery
E   ImportError: attempted relative import with no known parent package

Originally posted by @jaraco in coherent-oss/system#1

[jaraco]

I have a few ideas about how to handle this challenge:

separate test sources

For every test run, create a separate "installation" with the root packages modeled (e.g. manifest coherent and build directories and symlink the contents into there) and run the tests from that temporary directory.

This approach would put a heavy burden on startup and interactive testing (it won't be possible for a user to simply run pytest from a suitable environment).

Native pytest support

Perhaps pytest could natively support the coherent system layout.

I doubt I could just throw this issue at pytest, especially considering how much effort they've put into making doctests and namespaces work in the importlib mode transition, but perhaps with some contributions, the project could honor the vision.

Pytest plugin

Perhaps it would be possible to write a plugin that would either alter the import behavior or doctest collection to allow the doctests to be run from the installed code rather than from the source code.

This approach could get messy, especially since pytest is already altering the import system to support assert rewrites and other behaviors.

A separate doctests plugin

Perhaps this project could supply its own doctests plugin and leave pytest's doctests behavior disabled.

This approach would likely involve some of the same work as the above approaches, but at least it would decouple the concerns from the pytest project. It also would likely involve copying or re-writing much of what pytest already does for doctests.

[jaraco]

I did some investigation into how pytest discovers doctests and it does seem like it may be feasible to replace or manipulate DoctestModule objects to prefer the installed code instead of the source code.

[jaraco]

I've started a conversation with the pytest project to discuss options and concerns.

I've found that jaraco/pytest@48cebfa8f approximates the behavior I'm seeking. After making that change, I'm able to run the tests for coherent.build:

 coherent.build main @ py -m venv .venv
 coherent.build main @ py -m pip install -q git+https://github.com/jaraco/pytest@feature/coherent '.[test]'
WARNING: coherent-build 0.12.1.dev1+g9451603 does not provide the extra 'test'
 coherent.build main @ py -m pytest --doctest-modules
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.3, pytest-7.1.0.dev1844+g48cebfa8, pluggy-1.5.0
rootdir: /Users/jaraco/code/coherent-oss/coherent.build
configfile: pyproject.toml
plugins: typeguard-4.3.0
collected 11 items                                                                                                                               

backend.py ....                                                                                                                            [ 36%]
discovery.py .......                                                                                                                       [100%]

=============================================================== 11 passed in 1.10s ===============================================================

[jaraco]

I started working on an interim solution for this issue, but it's likely going to require a pytest plugin and thus a solution for coherent-oss/coherent.build#10.