Skip to content

Commit 4c3f8c9

Browse files
committed
tests: garbage collect sqlite connections on Windows + Python 3.11
There seems to be a regression in Python 3.11, where the sqlite connections are not deallocated, due to some internal changes in Python 3.11, where they are now using LRU cache. They are not deallocated until `gc.collect()` is not called. See python/cpython#97641. This affects only Windows, because when we try to remove the tempdir for the exp run, the sqlite connection is open which prevents us from deleting that folder. Although this may happen in real scenario in `exp run`, I am only fixing the tests by mocking `dvc.close()` and extending it by calling `gc.collect()` after it. We could also mock `State.close()` but didnot want to mock something that is not in dvc itself. The `diskcache` uses threadlocal for connections, so they are expected to be garbage collected, and therefore does not provide a good way to close the connections. The only API it offers is `self.close()` and that only closes main thread's connection. If we had access to connection, an easier way would have been to explicitly call `conn.close()`. But we don''t have such option at the moment. Related: iterative#8404 (comment) GHA Failure: https://github.com/iterative/dvc/actions/runs/3437324559/jobs/5731929385#step:5:57
1 parent 929de7c commit 4c3f8c9

File tree

1 file changed

+17
-0
lines changed

1 file changed

+17
-0
lines changed

tests/conftest.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import gc
12
import json
23
import os
34
import sys
@@ -259,3 +260,19 @@ def mock_hydra_conf(mocker):
259260
# NOTE: using sentinel here so that any imports from `hydra.conf`
260261
# return a mock.
261262
sys.modules["hydra.conf"] = mocker.sentinel
263+
264+
265+
@pytest.fixture(autouse=True)
266+
def gc_collect_on_dvc_close_on_win_311(mocker):
267+
if sys.version_info < (3, 11) and os.name != "nt":
268+
return
269+
270+
from dvc.repo import Repo
271+
272+
close = Repo.close
273+
274+
def wrapped(repo):
275+
close(repo)
276+
gc.collect()
277+
278+
mocker.patch("dvc.repo.Repo.close", wrapped)

0 commit comments

Comments
 (0)