Generalize PYTEST_CURRENT_TEST for threading

[Liam-DeVoe]

Part of #13768.

I'm starting to work earnestly on #13768, and figured I would start with a relatively simple fix + some general work for threading tests.

If we really cared about making PYTEST_CURRENT_TEST as good as it could be, then we could track a stack of pushes/pops to PYTEST_CURRENT_TEST, and when a pop occurs, we set it to the most-recent value on the stack instead of setting it to None. This would ensure that PYTEST_CURRENT_TEST always has a value when pytest is running, and is always "a test that is currently running", though there may be multiple currently-running tests. I am happy to implement this if you guys would prefer!

[Liam-DeVoe]

this ignore rule is duplicated farther down the file

[Liam-DeVoe]

Ah hm, test_concurrent is going to fail until Session is thread-safe. I have this hacked around locally, but not something I would want to PR yet. I'll drop the tests for now I guess

(Current very-WIP work is here: https://github.com/pytest-dev/pytest/compare/main...Liam-DeVoe:pytest:threading?expand=1)

[RonnyPfannschmidt]

This is a unacceptable wrong prediction

The current test env var itself doesn't translate to threads to begin with

[Liam-DeVoe]

The current test env var itself doesn't translate to threads to begin with

Yeah, agreed. If this PR is no good, how would you prefer PYTEST_CURRENT_TEST is handled under threading?

[RonnyPfannschmidt]

We need one env var per thread

So some type of indication would be needed

Id says the existing reserved for the pytest session on the main thread

Other threads need a curret test of thread style variables

Existing of the per thread values should be the indicator of multithread pytest as far ad observing processes are concerned

Details should be expermented with

[Liam-DeVoe]

OK nice. Wasn't sure if pytest wanted to support threads as such a first class citizen with special envvars and such.

We'll need some way for pytest to know when a "runner" thread is executing a test, vs when a helper thread has been spawned by a test that happens to interact with pytest in some way. In other words I don't think it's sufficient to check threading.get_ident() at any point to get "the current thread which is responsible for running this test as part of a thread pool". Currently my ideas are to track the thread id that pytest_runtest_protocol is executed from and call that a runner thread, or to have some explicit way to tell pytest "run with a thread pool" and pytest spawns and tracks the threads itself

[RonnyPfannschmidt]

threading.current_thread() is threading.main_thread() only on the main thread, else its another one

names and idents will be tricky

but one thing is for clear - just hiding key errors on a env var that shouldn't be thread-shared is not gona cut it

[RonnyPfannschmidt]

as for thread pools - thats not a consideration as of now as setupstate is not separated

[Liam-DeVoe]

Something like this perhaps? PYTEST_CURRENT_TEST remains on the main thread, and PYTEST_CURRENT_TEST_THREAD_{n} for other threads. runtestprotocol tracks what thread is executing each test. I imagine we'll use the _item_to_thread mapping in the future as well, for example mapping captured thread output back to the corresponding item.

Two tests are failing because they invoke pytest_runtest_call directly and bypass runtestprotocol. I initially put the setting of _item_to_thread in runner.py::pytest_runtest_setup, but skipping.py::pytest_runtest_setup is a tryfirst and so if a test has a skip, it raises and never invokes runner.py::pytest_runtest_setup and then the teardown errors because _item_to_thread doesn't have that item.

[RonnyPfannschmidt]

We should not invent own thread numbering schemes

[Liam-DeVoe]

should the envvars be named PYTEST_CURRENT_TEST_THREAD_{thread.ident}instead, eg. PYTEST_CURRENT_TEST_THREAD_8422586432? My thinking was that a more human readable approach would be nice, and also that way someone could tell how many threads are active. Happy to just use thread.ident though.

[RonnyPfannschmidt]

we should require a "pytest thread" name for a thread run to be picked no guessing, no random long integers - the env names should be something people can deal with

[RonnyPfannschmidt]

(i considered picking the threading thread name but their default is not ux friendly for env vars)

[bluetech]

@Liam-DeVoe would you mind opening a specific issue about PYTEST_CURRENT_TEST and the problem with threads?