-
-
Notifications
You must be signed in to change notification settings - Fork 31.9k
Re-entering pairwise.__next__() leaks references #109786
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Let's get the other issue fixed first. This one is somewhat exotic and the PR makes a mess of the code. |
I don't see your leak reproducer, wrote one based on your test: Leak reproducerWhole code at Attempt This Online!
def incref(x):
class I:
count = 0
def __iter__(self):
return self
def __next__(self):
self.count += 1
if self.count == 1:
return next(pairs)
if self.count == 3:
return x
return None
pairs = pairwise(I())
next(pairs) Demo increasing an object's refcount by 10000: x = object()
print(f'refcount before:', sys.getrefcount(x))
for _ in range(10000):
incref(x)
gc.collect()
print('refcount after:', sys.getrefcount(x)) Output:
Demo leaking 10000 objects of 10 kB each: tracemalloc.start()
for _ in range(10000):
incref(bytes(10000))
gc.collect()
print('leaked memory:', tracemalloc.get_traced_memory()[0] // 10**6, 'MB') Output:
|
I added them as tests. You can run tests with the |
….__next__() (pythonGH-109788) (cherry picked from commit 6ca9d3e) Co-authored-by: Serhiy Storchaka <[email protected]>
….__next__() (pythonGH-109788) (cherry picked from commit 6ca9d3e) Co-authored-by: Serhiy Storchaka <[email protected]>
…e.__next__() (GH-109788) (GH-112699) (cherry picked from commit 6ca9d3e) Co-authored-by: Serhiy Storchaka <[email protected]>
…e.__next__() (GH-109788) (GH-112700) (cherry picked from commit 6ca9d3e) Co-authored-by: Serhiy Storchaka <[email protected]>
Re-entering the
__next__()
method of itertools.pairwise leaks references, because old hold a borrowed reference when the__next__()
method of the underlying iterator is called. It may potentially lead to the use of a freed memory and a crash, but I only have reproducer for leaks.Even if the re-entrant call is not very meaningful, it should correctly count references. There are several ways to fix this issue. I choose the simplest one which matches the current results (only without leaks) in most of simple tests, even if there is no other reasons to prefer these results.
cc @rhettinger
Linked PRs
The text was updated successfully, but these errors were encountered: