Skip to content

gh-118218: Reuse return tuple in itertools.pairwise #118219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 30, 2024
Merged

gh-118218: Reuse return tuple in itertools.pairwise #118219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3bd6a58
gh-118218: Reuse return tuple in itertools.pairwise
hauntsaninja Apr 24, 2024
f653f22
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 24, 2024
ea0cfd6
add bpo-42536 style test for gc tracking
hauntsaninja Apr 24, 2024
68b5d27
nit
hauntsaninja Apr 28, 2024
1bfddd6
Update Misc/NEWS.d/next/Library/2024-04-24-07-45-08.gh-issue-118218.m…
hauntsaninja Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Lib/test/test_itertools.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1821,6 +1821,13 @@ def test_zip_longest_result_gc(self):
gc.collect()
self.assertTrue(gc.is_tracked(next(it)))

@support.cpython_only
def test_pairwise_result_gc(self):
# Ditto for pairwise.
it = pairwise([None, None])
gc.collect()
Comment on lines +1827 to +1828
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In all tests above except for zip_longest there is a next(it) before gc.collect(). All these tests were added in 226a012 (bpo-42536).

@brandtbucher, why is such difference? Is there a bug in test_zip_longest_result_gc? Do we need next(it) there and here?

self.assertTrue(gc.is_tracked(next(it)))

@support.cpython_only
def test_immutable_types(self):
from itertools import _grouper, _tee, _tee_dataobject
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2024-04-24-07-45-08.gh-issue-118218.m1OHbN.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Speed up :func:`itertools.pairwise` in the common case by up to 1.8x.
34 changes: 32 additions & 2 deletions Modules/itertoolsmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,7 @@ typedef struct {
PyObject_HEAD
PyObject *it;
PyObject *old;
PyObject *result;
} pairwiseobject;

/*[clinic input]
Expand Down Expand Up @@ -301,6 +302,11 @@ pairwise_new_impl(PyTypeObject *type, PyObject *iterable)
}
po->it = it;
po->old = NULL;
po->result = PyTuple_Pack(2, Py_None, Py_None);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyTuple_Pack is slow. I created an issue #118222 to discuss this.

An alternative that might be faster:

Suggested change
po->result = PyTuple_Pack(2, Py_None, Py_None);
PyObject tmp[2] = {Py_None, Py_None};
po->result = _PyTuple_FromArraySteal(tmp, 2),;

(note: untested!)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this locally. Since this line is once called once for each call to pairwise the performance difference is insignificant. It does matter for the other call to PyTuple_Pack (the one in pairwise_next), but that is best handled in a different PR.

if (po->result == NULL) {
Py_DECREF(po);
return NULL;
}
return (PyObject *)po;
}

Expand All @@ -311,6 +317,7 @@ pairwise_dealloc(pairwiseobject *po)
PyObject_GC_UnTrack(po);
Py_XDECREF(po->it);
Py_XDECREF(po->old);
Py_XDECREF(po->result);
tp->tp_free(po);
Py_DECREF(tp);
}
Expand All @@ -321,6 +328,7 @@ pairwise_traverse(pairwiseobject *po, visitproc visit, void *arg)
Py_VISIT(Py_TYPE(po));
Py_VISIT(po->it);
Py_VISIT(po->old);
Py_VISIT(po->result);
return 0;
}

Expand Down Expand Up @@ -355,8 +363,30 @@ pairwise_next(pairwiseobject *po)
Py_DECREF(old);
return NULL;
}
/* Future optimization: Reuse the result tuple as we do in enumerate() */
result = PyTuple_Pack(2, old, new);

result = po->result;
if (Py_REFCNT(result) == 1) {
Py_INCREF(result);
PyObject *last_old = PyTuple_GET_ITEM(result, 0);
PyObject *last_new = PyTuple_GET_ITEM(result, 1);
PyTuple_SET_ITEM(result, 0, Py_NewRef(old));
PyTuple_SET_ITEM(result, 1, Py_NewRef(new));
Py_DECREF(last_old);
Py_DECREF(last_new);
Comment on lines +372 to +375
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since last_new is equal to old we could do:

Suggested change
PyTuple_SET_ITEM(result, 0, Py_NewRef(old));
PyTuple_SET_ITEM(result, 1, Py_NewRef(new));
Py_DECREF(last_old);
Py_DECREF(last_new);
PyTuple_SET_ITEM(result, 0, old);
PyTuple_SET_ITEM(result, 1, Py_NewRef(new));
Py_DECREF(last_old);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not necessarily true that last_new is old. Really up to the whims of when the user lets go of the tuple

// bpo-42536: The GC may have untracked this result tuple. Since we're
// recycling it, make sure it's tracked again:
if (!_PyObject_GC_IS_TRACKED(result)) {
_PyObject_GC_TRACK(result);
}
}
else {
result = PyTuple_New(2);
if (result != NULL) {
PyTuple_SET_ITEM(result, 0, Py_NewRef(old));
PyTuple_SET_ITEM(result, 1, Py_NewRef(new));
}
}

Py_XSETREF(po->old, new);
Py_DECREF(old);
return result;
Expand Down
Loading