Skip to content

test_asyncio and test_compileall leak temporary files/directories #109295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vstinner opened this issue Sep 12, 2023 · 2 comments
Closed

test_asyncio and test_compileall leak temporary files/directories #109295

vstinner opened this issue Sep 12, 2023 · 2 comments
Labels
tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

vstinner commented Sep 12, 2023

Using PR #109290 fix, I found that the two following tests leak temporary files/directories.

test_compileall:

$ ./python -m test -j1 test_compileall --fail-env-changed -m test.test_compileall.CompileallTestsWithSourceEpoch.test_ddir_empty_multiple_workers
0:00:00 load avg: 0.81 Run tests in parallel using 1 child processes
0:00:01 load avg: 0.81 [1/1/1] test_compileall failed (env changed)
Warning -- files was modified by test_compileall
Warning --   Before: []
Warning --   After:  ['pymp-kmfq1ivj/'] 
Warning -- files was modified by test_compileall
Warning --   Before: []
Warning --   After:  ['pymp-kmfq1ivj/']

== Tests result: ENV CHANGED ==

1 test altered the execution environment:
    test_compileall

Total duration: 1.2 sec
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) env_changed=1
Result: ENV CHANGED

test.test_asyncio.test_events:

$ ./python -m test -j1 test.test_asyncio.test_events --fail-env-changed -m test.test_asyncio.test_events.TestPyGetEventLoop.test_get_event_loop_new_process 
0:00:00 load avg: 0.65 Run tests in parallel using 1 child processes
0:00:01 load avg: 0.65 [1/1/1] test.test_asyncio.test_events failed (env changed)
Warning -- files was modified by test.test_asyncio.test_events
Warning --   Before: []
Warning --   After:  ['pymp-tvmenicy/'] 
Warning -- files was modified by test.test_asyncio.test_events
Warning --   Before: []
Warning --   After:  ['pymp-tvmenicy/']

== Tests result: ENV CHANGED ==

1 test altered the execution environment:
    test.test_asyncio.test_events

Total duration: 1.4 sec
Total tests: run=1 (filtered)
Total test files: run=1/1 (filtered) env_changed=1
Result: ENV CHANGED

Linked PRs

@vstinner vstinner added the tests Tests in the Lib/test dir label Sep 12, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
test_asyncio and test_compileall now clean up multiprocessing by
calling _cleanup_tests() method: explicitly clean up resources and
stop background processes like the resource tracker.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename since self.fname already exists.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
@vstinner
Copy link
Member Author

Another problem: test_os fails if the TEMP environment variable is equal to the current working directory.

ERROR: test_access_denied (test.test_os.StatAttributeTests.test_access_denied)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\test_os.py", line 742, in test_access_denied
    create_file(fname, b'ABC')
  File "D:\a\cpython\cpython\Lib\test\test_os.py", line 94, in create_file
    with open(filename, "xb", 0) as fp:
         ^^^^^^^^^^^^^^^^^^^^^^^
FileExistsError: [Errno 17] File exists: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\test_python_worker_a5djsgg5\\@test_5340_tmp�'

vstinner added a commit that referenced this issue Sep 12, 2023
…ll (#109298)

test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 12, 2023
…mpileall (pythonGH-109298)

test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
(cherry picked from commit 09ea4b8)

Co-authored-by: Victor Stinner <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 12, 2023
…mpileall (pythonGH-109298)

test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
(cherry picked from commit 09ea4b8)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Sep 12, 2023
Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 12, 2023
…nGH-109299)

Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
(cherry picked from commit 7dedfd3)

Co-authored-by: Victor Stinner <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 12, 2023
…nGH-109299)

Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
(cherry picked from commit 7dedfd3)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Sep 12, 2023
…ompileall (GH-109298) (#109302)

gh-109295: Clean up multiprocessing in test_asyncio and test_compileall (GH-109298)

test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
(cherry picked from commit 09ea4b8)

Co-authored-by: Victor Stinner <[email protected]>
@vstinner
Copy link
Member Author

Last issue with test_generated_cases:

0:19:14 load avg: 6.45 [2/3/2] test_generated_cases failed (uncaught exception)
Re-running test_generated_cases in verbose mode
test test_generated_cases crashed -- Traceback (most recent call last):
  File "D:\a\cpython\cpython\Lib\test\libregrtest\single.py", line 139, in _runtest_env_changed_exc
    _load_run_test(result, runtests)
  File "D:\a\cpython\cpython\Lib\test\libregrtest\single.py", line 87, in _load_run_test
    test_mod = importlib.import_module(module_name)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\importlib\__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1006, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "D:\a\cpython\cpython\Lib\test\test_generated_cases.py", line 11, in <module>
    import generate_cases
  File "D:\a\cpython\cpython\Tools\cases_generator\generate_cases.py", line 41, in <module>
    DEFAULT_INPUT = os.path.relpath(os.path.join(ROOT, "Python/bytecodes.c"))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\cpython\cpython\Lib\ntpath.py", line 791, in relpath
    raise ValueError("path is on mount %r, start on mount %r" % (
ValueError: path is on mount 'D:', start on mount 'C:'

See PR #109290 for the background.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
On Windows, skip the test if the current working directory and the
Python source code directory have different mount drives. It happens
if the temporary directory is on a different mount drive than the
Python source code.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 12, 2023
On Windows, skip the test if the current working directory and the
Python source code directory have different mount drives. It happens
if the temporary directory is on a different mount drive than the
Python source code.
vstinner added a commit that referenced this issue Sep 12, 2023
…09299) (#109303)

gh-109295: Fix test_os.test_access_denied() for TEMP=cwd (GH-109299)

Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
(cherry picked from commit 7dedfd3)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit that referenced this issue Sep 12, 2023
On Windows, skip the test if the current working directory and the
Python source code directory have different mount drives. It happens
if the temporary directory is on a different mount drive than the
Python source code.
Yhg1s pushed a commit that referenced this issue Sep 12, 2023
…ompileall (GH-109298) (#109301)

gh-109295: Clean up multiprocessing in test_asyncio and test_compileall (GH-109298)

test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
(cherry picked from commit 09ea4b8)

Co-authored-by: Victor Stinner <[email protected]>
Yhg1s pushed a commit that referenced this issue Sep 12, 2023
…09299) (#109304)

gh-109295: Fix test_os.test_access_denied() for TEMP=cwd (GH-109299)

Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
(cherry picked from commit 7dedfd3)

Co-authored-by: Victor Stinner <[email protected]>
vstinner added a commit to vstinner/cpython that referenced this issue Sep 13, 2023
…mpileall (python#109298)

test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 13, 2023
…n#109299)

Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 13, 2023
…ython#109308)

On Windows, skip the test if the current working directory and the
Python source code directory have different mount drives. It happens
if the temporary directory is on a different mount drive than the
Python source code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Tests in the Lib/test dir
Projects
None yet
Development

No branches or pull requests

1 participant