Skip to content

gh-112075: _Py_dict_lookup needs to lock shared keys #117528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Apr 25, 2024

Conversation

DinoV
Copy link
Contributor

@DinoV DinoV commented Apr 3, 2024

_Py_dict_lookup needs to lock the shared keys if we have a split dictionary. If we're looking up with a non-exact unicode we need to also incref the keys as the lookup could mutate the keys and we could lose the last reference.

insertdict is updated to avoid contention on the shared dict lookup by calling the threadsafe unicode lookup directly and only falling back to _Py_dict_lookup if the thread safe lookup can't succeed.

@DinoV DinoV requested a review from colesbury April 3, 2024 23:53
@DinoV DinoV force-pushed the nogil_dict_shared_key_insert branch 2 times, most recently from c51437d to f3e3db3 Compare April 4, 2024 00:04
@DinoV DinoV marked this pull request as ready for review April 4, 2024 00:35
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks right, but I'm still seeing issues with the test that uncovered the problem. I'll investigate more today.

@DinoV DinoV force-pushed the nogil_dict_shared_key_insert branch from 8942e4d to 2f5f5a1 Compare April 5, 2024 02:23
@DinoV DinoV force-pushed the nogil_dict_shared_key_insert branch 10 times, most recently from 35cf56e to d580aa5 Compare April 16, 2024 01:31
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still looking at _Py_dict_lookup, but I think insertdict and dict_setdefault_ref_lock_held can be simpler if they handle split dictionaries early on and separate from the main code path.

@DinoV DinoV force-pushed the nogil_dict_shared_key_insert branch 2 times, most recently from cfd659c to 19d1446 Compare April 24, 2024 20:30
@DinoV DinoV force-pushed the nogil_dict_shared_key_insert branch from 19d1446 to a74c0ee Compare April 25, 2024 16:19
Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than the unused code warning in the default build

@@ -1171,6 +1188,10 @@ _PyDictKeys_StringLookup(PyDictKeysObject* dk, PyObject *key)
return unicodekeys_lookup_unicode(dk, key, hash);
}

static Py_ssize_t
unicodekeys_lookup_unicode_threadsafe(PyDictKeysObject* dk, PyObject *key,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused code warning

@DinoV DinoV merged commit d5df252 into python:main Apr 25, 2024
34 checks passed
@DinoV DinoV deleted the nogil_dict_shared_key_insert branch May 31, 2024 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants