test_concurrent_futures.test_wait.ThreadPoolWaitTests.test_timeout flakes frequently on free-threaded Windows x64

[zanieb]

Bug report

Bug description:

The following test fails frequently, but not consistently:

test_timeout (test.test_concurrent_futures.test_wait.ThreadPoolWaitTests.test_timeout) ... FAIL
0.86s 
======================================================================
FAIL: test_timeout (test.test_concurrent_futures.test_wait.ThreadPoolWaitTests.test_timeout)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures\test_wait.py", line 129, in test_timeout
    self.assertEqual(set([CANCELLED_AND_NOTIFIED_FUTURE,
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          EXCEPTION_FUTURE,
                          ^^^^^^^^^^^^^^^^^
                          SUCCESSFUL_FUTURE]),
                          ^^^^^^^^^^^^^^^^^^^^
                     finished)
                     ^^^^^^^^^
AssertionError: Items in the second set but not the first:
<Future at 0x20000554620 state=finished returned NoneType>

As seen in the following CI runs for recent commits to main

• https://github.com/python/cpython/actions/runs/12549107170/job/34989573917
• https://github.com/python/cpython/actions/runs/12527558956/job/34941261499
• https://github.com/python/cpython/actions/runs/12533132642/job/34952530854
• https://github.com/python/cpython/actions/runs/12529744428/job/34945648292
• https://github.com/python/cpython/actions/runs/12505440015/job/34888819142

A brief analysis of the code at

cpython/Lib/test/test_concurrent_futures/test_wait.py

Lines 115 to 133 in 1d3225a

	def test_timeout(self):
	short_timeout = 0.050
	long_timeout = short_timeout * 10
	future = self.executor.submit(time.sleep, long_timeout)
	finished, pending = futures.wait(
	[CANCELLED_AND_NOTIFIED_FUTURE,
	EXCEPTION_FUTURE,
	SUCCESSFUL_FUTURE,
	future],
	timeout=short_timeout,
	return_when=futures.ALL_COMPLETED)
	self.assertEqual(set([CANCELLED_AND_NOTIFIED_FUTURE,
	EXCEPTION_FUTURE,
	SUCCESSFUL_FUTURE]),
	finished)
	self.assertEqual(set([future]), pending)

looks like the sleeping future is present in the finished set instead of pending

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-128364: Fix flaky test_timeout test #130724
• [3.13] gh-128364: Fix flaky test_timeout test (gh-130724) #130728
• [3.12] gh-128364: Fix flaky test_timeout test (gh-130724) #130729
• Revert "gh-128364: Fix flaky test_timeout test (gh-130724)" #130732
• [3.13] Revert "gh-128364: Fix flaky test_timeout test (gh-130724)" (gh-130732) #130734
• [3.12] Revert "gh-128364: Fix flaky test_timeout test (gh-130724)" (gh-130732) #130735
• gh-128364: Fix flaky test_concurrent_futures.test_wait tests #130742
• [3.13] gh-128364: Fix flaky test_concurrent_futures.test_wait tests (gh-130742) #130922

[zanieb]

We could try bumping the sleep time (e.g., 2a150fc) but I don't think that would help. It's interesting it's consistently on this platform.

[colesbury]

This might be a radical opinion, but I think we should delete the test. These sorts of timeout based tests that can spuriously fail if a process takes too long to start are a frequent source of flaky tests. I don't think they catch enough bugs or prevent enough regressions to justify the time we spend tweaking the tests.

[zanieb]

Should be fixed by #130724 — can reopen if not.

[picnixz]

We have build bot failures on android and others now

[colesbury]

Should be fixed now