Collection-error abort check lives in `pytest_runtestloop` instead of `pytest_collection`

[SemTiOne]

What's the problem this feature will solve?

The collection-error abort check lives in the default pytest_runtestloop (src/_pytest/main.py), so the runtest phase inspects leftovers from collection instead of collection guarding itself.

Describe the solution you'd like

Move the testsfailed check into the default pytest_collection, keeping the exception and exit code. Both hooks are firstresult hooks, so audit which implementations bypass the check before moving it.

Alternative Solutions

Leave as is (behavior is correct).

Additional context

Deferred from #15012 (#15012 (comment)). This can be closed as wontfix if the compatibility impact outweighs the benefit.

[bluetech]

Thanks for opening the issue.

IMO it makes more sense to have this check in pytest_collection than in pytest_runtestloop. I also think it's unlikely to break any plugin. So I think we should do it.

Since we document the current behavior, changing it is technically a breaking change. This doesn't mean we shouldn't have a PR for it, we will just defer merging it until pytest 10.0 enters development (probably after 9.2 is branched).

---

BTW, another thing I find annoying is that collectonly checking is delegated to pytest_runtestloop implementations. IMO it would have been better if _main would have skipped calling pytest_runtestloop directly.

This would mean that plugins won't have control of the flow here -- if --collect-only is passed then runtestloop phase would be skipped unconditionally. But it just doesn't make sense to me that handling of a flag which means "skip the runtestloop phase" is delegated to... the runtestloop phase.

One problem is that the terminal reporter plugin pytest_runtestloop wrapper impl runs some code even when collectonly. I added this code 50d1e81; I don't remember the details but I think I probably picked the wrong hook for this. Might be better done in pytest_sessionfinish or maybe even pytest_cmdline_main.

Anyway this would be a bigger breaking change and not really related to this issue, so sorry for rambling :)

[SemTiOne]

Ahh that makes sense, thanks.

The collectonly point is interesting too. That could be a candidate issue?