bpo-45403: Fix test_sys.test_stdlib_dir() #28785

vstinner · 2021-10-07T11:29:04Z

Fix test_sys.test_stdlib_dir() when Python is built outside the
source tree: compare normalized paths.

https://bugs.python.org/issue45403

Fix test_sys.test_stdlib_dir() when Python is built outside the source tree: compare normalized paths.

vstinner · 2021-10-07T11:30:48Z

@ericsnowcurrently: In a perfect world, we would be able ot normalize paths in C, but for now, I think that it's ok to have build/../ in sys._stdlib_dir. There is the same issue with sys.executable.

I propose to fix the test, we can revisit normalization during Python startup later.

ericsnowcurrently · 2021-10-07T14:50:07Z

I propose to fix the test, we can revisit normalization during Python startup later.

You may want to have a chat with @zooba, who has been threatening to make getpath.c pure Python (frozen). 🙂

ericsnowcurrently

LGTM

Thanks for doing this.

ericsnowcurrently · 2021-10-07T14:53:05Z

@vstinner, do we have a buildbot for out-of-tree builds? Clearly I've missed cases where I broke that with some of my recent work.

vstinner · 2021-10-07T19:21:19Z

You may want to have a chat with @zooba, who has been threatening to make getpath.c pure Python (frozen). slightly_smiling_face

I rewrote getpath.c in pure Python: #23169

But it changes the behavior, Nick Coghlan requested a PEP. I wasn't very motivated. I abandonned my PR.

vstinner · 2021-10-07T19:22:15Z

@vstinner, do we have a buildbot for out-of-tree builds? Clearly I've missed cases where I broke that with some of my recent work.

It seems like we don't. The issue was catched by my colleague Miro @hroncok when he built the Fedora Python 3.11.0a1 package.

bedevere-bot · 2021-10-07T19:42:34Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 macOS 3.x has failed when building commit 768aaf6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/725/builds/98) and take a look at the build logs.
Check if the failure is related to this commit (768aaf6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/725/builds/98

Failed tests:

test_os
test_importlib
test_pathlib

Failed subtests:

test_multiprocessing_pool_circular_import - test.test_importlib.test_threaded_import.ThreadedImportTests
test_glob_many_open_files - test.test_pathlib.PathTest
test_walk_many_open_files - test.test_os.BytesWalkTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

408 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 32 sec
test_multiprocessing_spawn: 2 min 32 sec
test_multiprocessing_forkserver: 1 min 51 sec
test_unparse: 1 min 26 sec
test_asyncio: 1 min 18 sec
test_tokenize: 1 min 13 sec
test_logging: 1 min 4 sec
test_lib2to3: 1 min 3 sec
test_nntplib: 1 min 1 sec
test_capi: 57.7 sec

1 test failed:
test_importlib

16 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_spwd
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

3 re-run tests:
test_importlib test_os test_pathlib

Total duration: 9 min 25 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_6f8039ac'


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 191, in __init__
    self._setup_queues()
    ^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 344, in _setup_queues
    self._outqueue = self._ctx.SimpleQueue()
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/queues.py", line 347, in __init__
    self._wlock = ctx.Lock()
                  ^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
                            ^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 36 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-8kl1ifwv': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-qevixc4u': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2t8wdz38': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
---


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
    script_helper.assert_python_ok(fn)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/partial/pool_in_threads.py']


Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
    ^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
    with multiprocessing.Pool(1):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 119, in Pool
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 212, in __init__
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/popen_spawn_posix.py", line 54, in _launch
    child_r, parent_w = os.pipe()
                        ^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/process.py", line 121, in start
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/context.py", line 284, in _Popen
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/util.py", line 450, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
                                  ^^^^^^^^^
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 90 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
---


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_pathlib.py", line 1689, in test_glob_many_open_files
    self.assertEqual(next(it), p)
                     ^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/pathlib.py", line 1045, in rglob
    for p in selector.select_from(self):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/pathlib.py", line 492, in _select_from
    for starting_point in self._iterate_directories(parent_path, is_dir, scandir):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/pathlib.py", line 482, in _iterate_directories
    for p in self._iterate_directories(path, is_dir, scandir):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/pathlib.py", line 482, in _iterate_directories
    for p in self._iterate_directories(path, is_dir, scandir):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/pathlib.py", line 482, in _iterate_directories
    for p in self._iterate_directories(path, is_dir, scandir):
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 21 more times]
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/pathlib.py", line 471, in _iterate_directories
    with scandir(parent_path) as scandir_it:
         ^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 23] Too many open files in system: '/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/build/test_python_6821æ/test_python_worker_10716æ/@test_10716_tmpæ/deep/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d/d'


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_os.py", line 1458, in test_walk_many_open_files
    self.assertEqual(next(it), expected)
                     ^^^^^^^^
StopIteration


Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '/psm_058c185c'

bpo-45403: Fix test_sys.test_stdlib_dir()

4537c29

Fix test_sys.test_stdlib_dir() when Python is built outside the source tree: compare normalized paths.

vstinner requested a review from ericsnowcurrently October 7, 2021 11:29

bedevere-bot added the awaiting core review label Oct 7, 2021

the-knights-who-say-ni added the CLA signed label Oct 7, 2021

bedevere-bot added the tests Tests in the Lib/test dir label Oct 7, 2021

ericsnowcurrently approved these changes Oct 7, 2021

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 7, 2021

vstinner merged commit 768aaf6 into python:main Oct 7, 2021

bedevere-bot removed the awaiting merge label Oct 7, 2021

vstinner deleted the test_stdlib_dir branch October 7, 2021 19:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-45403: Fix test_sys.test_stdlib_dir() #28785

bpo-45403: Fix test_sys.test_stdlib_dir() #28785

Uh oh!

vstinner commented Oct 7, 2021 •

edited by bedevere-bot

Loading

Uh oh!

vstinner commented Oct 7, 2021

Uh oh!

ericsnowcurrently commented Oct 7, 2021

Uh oh!

ericsnowcurrently left a comment

Uh oh!

ericsnowcurrently commented Oct 7, 2021

Uh oh!

vstinner commented Oct 7, 2021

Uh oh!

vstinner commented Oct 7, 2021

Uh oh!

bedevere-bot commented Oct 7, 2021

Uh oh!

Uh oh!

Uh oh!

bpo-45403: Fix test_sys.test_stdlib_dir() #28785

bpo-45403: Fix test_sys.test_stdlib_dir() #28785

Uh oh!

Conversation

vstinner commented Oct 7, 2021 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Oct 7, 2021

Uh oh!

ericsnowcurrently commented Oct 7, 2021

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently commented Oct 7, 2021

Uh oh!

vstinner commented Oct 7, 2021

Uh oh!

vstinner commented Oct 7, 2021

Uh oh!

bedevere-bot commented Oct 7, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

vstinner commented Oct 7, 2021 •

edited by bedevere-bot

Loading