Cannot get `recorder_mock` fixture running

[DarkC35]

Hi sorry to bug you with an issue but I cannot find any references online about this problem:
I'm finally trying to add some unit tests to my custom_component and basically need to make something like this for it: https://github.com/home-assistant/core/blob/dev/tests/components/tibber/test_statistics.py

Unfortunately whenever I use the recorder_mock fixture I get this error:

recorder_db_url = 'sqlite://', hass_fixture_setup = [True], enable_nightly_purge = False, enable_statistics = False, enable_statistics_table_validation = False

    @pytest.fixture
    async def async_setup_recorder_instance(
        recorder_db_url,
        hass_fixture_setup,
        enable_nightly_purge,
        enable_statistics,
        enable_statistics_table_validation,
    ) -> AsyncGenerator[SetupRecorderInstanceT, None]:
        """Yield callable to setup recorder instance."""
>       assert not hass_fixture_setup
E       assert not [True]

venv/lib/python3.10/site-packages/pytest_homeassistant_custom_component/plugins.py:1020: AssertionError

FAILED tests/test_sensor.py::test_sensor_service - assert not [True]

I cannot find any setting I might need to set online. You can reproduce this error with a simple test like this:

async def test_sensor_service(recorder_mock, hass):
    """Test sensor service."""

    assert True

The only "config" I set was asyncio_mode in the pyproject.toml to get rid of the async errors:

[tool.pytest.ini_options]
asyncio_mode = "auto"

conftest.py is basically unaltered from the template: https://github.com/custom-components/integration_blueprint/blob/master/tests/conftest.py (if this should make any difference)

Do you know anything I might miss by chance as it seems that nobody who uses recorder functions inside a custom_component seem to use unit tests.

Additional infos:
pytest-homeassistant-custom-component==0.12.29
custom_componet uses the custom-components/integration_blueprint template

Full log:

(venv) root ➜ /workspaces/ha_linznetz (add-tests ✗) $ pytest tests/test_sensor.py 
Test session starts (platform: linux, Python 3.10.2, pytest 7.2.0, pytest-sugar 0.9.5)
rootdir: /workspaces/ha_linznetz, configfile: pyproject.toml
plugins: anyio-3.6.2, aiohttp-1.0.4, asyncio-0.20.2, cov-3.0.0, forked-1.4.0, freezegun-0.4.2, homeassistant-custom-component-0.12.29, socket-0.5.1, sugar-0.9.5, test-groups-1.0.3, timeout-2.1.0, xdist-2.5.0, requests-mock-1.10.0, respx-0.20.1
asyncio: mode=auto
collecting ... 

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR at setup of test_sensor_service ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

event_loop = <_UnixSelectorEventLoop running=False closed=False debug=False>, request = <SubRequest 'async_setup_recorder_instance' for <Function test_sensor_service>>
kwargs = {'enable_nightly_purge': False, 'enable_statistics': False, 'enable_statistics_table_validation': False, 'hass_fixture_setup': [True], ...}
func = <function async_setup_recorder_instance at 0x7fdfd430d5a0>
setup = <function _wrap_asyncgen_fixture.<locals>._asyncgen_fixture_wrapper.<locals>.setup at 0x7fdfd3fe2cb0>
finalizer = <function _wrap_asyncgen_fixture.<locals>._asyncgen_fixture_wrapper.<locals>.finalizer at 0x7fdfd3fe2d40>

    @functools.wraps(fixture)
    def _asyncgen_fixture_wrapper(
        event_loop: asyncio.AbstractEventLoop, request: SubRequest, **kwargs: Any
    ):
        func = _perhaps_rebind_fixture_func(
            fixture, request.instance, fixturedef.unittest
        )
        gen_obj = func(**_add_kwargs(func, kwargs, event_loop, request))
    
        async def setup():
            res = await gen_obj.__anext__()
            return res
    
        def finalizer() -> None:
            """Yield again, to finalize."""
    
            async def async_finalizer() -> None:
                try:
                    await gen_obj.__anext__()
                except StopAsyncIteration:
                    pass
                else:
                    msg = "Async generator fixture didn't stop."
                    msg += "Yield only once."
                    raise ValueError(msg)
    
            event_loop.run_until_complete(async_finalizer())
    
>       result = event_loop.run_until_complete(setup())

venv/lib/python3.10/site-packages/pytest_asyncio/plugin.py:301: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/python/lib/python3.10/asyncio/base_events.py:641: in run_until_complete
    return future.result()
venv/lib/python3.10/site-packages/pytest_asyncio/plugin.py:283: in setup
    res = await gen_obj.__anext__()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

recorder_db_url = 'sqlite://', hass_fixture_setup = [True], enable_nightly_purge = False, enable_statistics = False, enable_statistics_table_validation = False

    @pytest.fixture
    async def async_setup_recorder_instance(
        recorder_db_url,
        hass_fixture_setup,
        enable_nightly_purge,
        enable_statistics,
        enable_statistics_table_validation,
    ) -> AsyncGenerator[SetupRecorderInstanceT, None]:
        """Yield callable to setup recorder instance."""
>       assert not hass_fixture_setup
E       assert not [True]

venv/lib/python3.10/site-packages/pytest_homeassistant_custom_component/plugins.py:1020: AssertionError
------------------------------------------------------------------------- Captured stderr setup --------------------------------------------------------------------------
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:asyncio:Using selector: EpollSelector
--------------------------------------------------------------------------- Captured log setup ---------------------------------------------------------------------------
DEBUG    asyncio:selector_events.py:59 Using selector: EpollSelector
DEBUG    asyncio:selector_events.py:59 Using selector: EpollSelector
                                                                                                                                                           100% ██████████
======================================================================== short test summary info =========================================================================
FAILED tests/test_sensor.py::test_sensor_service - assert not [True]

Results (0.88s):

[MatthewFlamm]

I wonder if you cannot use the autouse=True fixture enable_custom_integrations in your conftest.py. It initializes the hass fixture earlier.

[DarkC35]

Nice catch! Yes I can confirm that the code above works when disabling the enable_custom_integration fixture but sadly it's not a valid solution since I need this fixture to test my custom_integration I guess?

I just tried some other (combinations of) fixtures like hass_recorder or enable_statistics but without success (either the same error or "KeyError: 'recorder_instance'"). Could it be that some kind of fixture combination is need since the recorder_mock does not work?

I use the following homeassistant.components.recorder.statistics functions if this is any helpful:
get_last_statistics
statistics_during_period
async_import_statistics

[DarkC35]

Oh the order of the fixtures seems to be the solution! (Sorry it's my first time writing unit tests in python.)

Just tried it with the autouse=False and with this function definiton and it seems to work!

async def test_sensor_service(recorder_mock, enable_custom_integrations, hass):

Thank you for helping to understand this behaviour!

[MatthewFlamm]

No problem, your detailed information was really helpful. I debated for a long time whether to make enable_custom_integrations automatically use autouse=True from this package. This issue clarifies that this would have been a bad choice as sometimes we need this fixture in a specific order.

[DarkC35]

If I may ask another question: Is there any way to disable the auto_enable_custom_integrations fixture for specific tests only (with some annotation or something like this)? Google is not very helpful in this case.
Or do I have to add the enable_custom_integrations in every test manually?

[MatthewFlamm]

If all your tests need recorder_mock, you could create your own fixture in contest.py with the correct ordering of fixtures.

@pytest.fixture(autouse=True)
def my_fixture(recorder_mock, enable_custom_integrations):
    pass

If you don't need recorder_mock for all tests. You could create two test modules and scope specific fixtures for each module. I recommend looking at the pytest documentation here as there are several possibilities here.

[krahabb]

I've come across the very same issue and I've resolved this way.

# Test initialization must ensure custom_components are enabled
# but we can't autouse a simple fixture for that since the recorder
# need to be initialized first
@pytest.fixture(autouse=True)
def auto_enable(request: pytest.FixtureRequest):
    if "recorder_mock" in request.fixturenames:
        request.getfixturevalue("recorder_mock")
    hass = request.getfixturevalue("hass")
    hass.data.pop("custom_components")
    yield

You'll have to remove the 'enable_custom_integrations' auto_use of course since this new fixture takes care of that.
I've copied over the code from 'enable_custom_integrations' since I didn't find a solution for how to invoke the library 'fixture' in the wild (probably you can't) but this auto_use is basically doing the same, except it checks if you need the 'recorder_mock' and instantiates it in the correct order. This could also work for other fixtures you might need to use and which need to be invoked before 'hass'

As @MatthewFlamm correctly pointed out (this was one of the steps I went through) you could define your own auto_use with the correct ordering of fixtures call but then you would have to manage differentiating the tests as per his suggestion.
This solution, less elegant though, is easier to just drop in