We've been bitten by this several times now. I think the benefit of catching inconsistencies between test runner and this repository is worth the additional time spent with CI runs. If it turns out to slow us down too much, we could also make it something optional and manually triggered. Currently we have to rely on manually running things in the test runner locally, which is cumbersome and error-prone.