-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Adjusting type_caster<std::reference_wrapper<T>>
to support const/non-const propagation in cast_op
.
#2705
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…tive reference. Before, both std::reference_wrapper<T> and std::reference_wrapper<const T> would invoke cast_op<type>. This doesn't allow the type_caster<> specialization for T to distinguish reference_wrapper types from value types. After, the type_caster<> specialization invokes cast_op<type&>, which allows reference_wrapper to behave in the same way as a native reference type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Laramie, I tried out your change locally, with small changes applied on top:
That commit adds back the original code with #if #else #endif
, and uncomments one of your additional tests.
I found that your additional tests work with and without your change to type_caster<std::reference_wrapper<type>>
. Is that expected? Would it be possible to add tests that fail without your change?
For completeness: I was using clang version 9.0.1 (gLinux rodette) with -std=c++17
.
... except you changed the test. The behavior below fails:
The problem with your uncommented test is that pybind11 doesn't really have a mechanism to note mutable vs. immutable value types that correspond to c++ references vs c++ const references. It's possible to construct them by implementing a custom type_caster that manages a lot of the complexity itself. A skeleton might look like this:
But, well, putting that into the test is a bit much. Anyway, yes, the tests (other than the uncommented one) are expected to pass in both cases. |
My take:
I'll add two other maintainers as reviewers, maybe they have different takes. Two approvals from maintainers are needed to merge. |
This test is a chimera; it blends the pybind11 casters with a custom pytype implementation that supports immutable and mutable calls. In order to detect the immutable/mutable state, the cast_op needs to propagate it, even through e.g. std::reference<const T> Note: This is still a work in progress; some things are crashing, which likely means that I have a refcounting bug or something else missing.
Fixes the bugs in my custom python type implementation, demonstrate test that requires const& and reference_wrapper<const T> being treated differently from Non-const.
Ok, this test_chimera demonstrates the requirement by creating a custom type and using pybind11 type_caster to convert between it and the c++ type. The custom type holds a pointer to an c++ object and a flag that indicates mutability. |
Apply formatting presubmit check.
... And I've pushed the other change that demonstrates that const-references are made non-const by pybind11. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The suggestions are purely for the tests and mostly even just about naming and wording of comments. The core change is definitely fine. I ran it through the Google-global testing and found no issues.
include/pybind11/cast.h
Outdated
subcaster_cast_op_type>::value || | ||
std::is_same<reference_t, subcaster_cast_op_type>::value), | ||
"std::reference_wrapper<T> caster requires T to have a caster " | ||
"with an `T &` operator or `const T&` operator"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"with operator `T &` or `const T&`");
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
tests/CMakeLists.txt
Outdated
@@ -96,6 +96,7 @@ set(PYBIND11_TEST_FILES | |||
test_constants_and_functions.cpp | |||
test_copy_move.cpp | |||
test_custom_type_casters.cpp | |||
test_chimera.cpp |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This name stands out as the only one that is non-descriptive of what is being tested. The name is fitting if the context of the test is known first, but doesn't help people understand what the test is doing without looking inside. Could you please find a descriptive name, in line with the existing test names?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
tests/test_builtin_casters.py
Outdated
m.refwrap_lvalue().value = 4 | ||
assert m.refwrap_lvalue().value == 4 | ||
|
||
# const-ness is not propagated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At second thought, I think it's best to only exercise
assert m.refwrap_lvalue_const().value == 1
and to delete the comment and the other two lines.
Rationale: You want to exercise that the caster changed in this PR is fully covered (load
and cast
for T&
and const T&
). The more general issue/feature that const
is lost isn't relevant to this purpose.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
tests/test_chimera.cpp
Outdated
@@ -0,0 +1,408 @@ | |||
/* | |||
tests/test_chimera.cpp -- This demonstrates a hybrid usage of pybind11, where the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Casters for types not wrapped with py::class_
are perfectly normal: chrono.h
, complex.h
, eigen.h
, functional.h
, numpy.h
, stl.h
and probably many more in client code.
I think it's nice to have this test, even though it's more an end-to-end test than a unit test, but it's important to explain here, for example starting with:
This original purpose of this test is to exercise that type_caster<std::reference_wrapper<T>>
works for sub-casters that only have a const T&
operator, but not a T&
operator. ... continue with explanation why this is not a more focused unit test ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
tests/test_chimera.cpp
Outdated
#pragma GCC diagnostic pop | ||
#endif | ||
|
||
static std::unordered_map<Chimera*, void*>* mapping = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The shorter
static std::unordered_map<Chimera*, void*> mapping;
(with mapping->
below replaced by mapping
) also works. Is there a strong reason for allocating mapping
on the heap? (Its size is only 56 bytes in a 64-bit build.)
Similarly for the shared
and shared_const
variables below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
tests/test_chimera.cpp
Outdated
#include "pybind11_tests.h" | ||
|
||
/// C++ type | ||
class Chimera { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
class FreezableInt
?
"Chimara" fits for the situation as a whole, but not for this type by itself.
Meaningful naming has a huge impact on readability, especially if this is meant to be a demo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@henryiii or @bstaletic, is there a chance you could help giving this a second look? The core change (in cast.h
) is very small. I double-checked that the new test fails without the core change, i.e. we certainly have full coverage for the change.
tests/test_freezable_type_caster.py
Outdated
|
||
# by reference, so it's not converted. | ||
with pytest.raises(TypeError): | ||
assert m.roundtrip_ref(m.get_const_ref()).x == 4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The assert
isn't actually needed, this by itself raises the TypeError
:
m.roundtrip_ref(m.get_const_ref())
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
I saw the CI / 🐍 pypy2 • ubuntu-latest • x64 failure we're getting here also on current pure master. It is definitely unrelated to this PR. |
This test verifies that cast_op may be used to correctly detect const reference types when used with std::reference_wrapper.
tests/test_const_ref_caster.cpp
Outdated
@@ -0,0 +1,88 @@ | |||
/* | |||
tests/test_const_ref_caster.cpp |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests seem more appropriate in test_builtin_casters.cpp
/test_builtin_casters.py
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps, but I think that since they are intended to exercise different aspects, putting it into a separate file makes the intent a bit more clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, pybind11's tests are a mess. But I think the idea it to keep a minimal amount of files, because they all take quite some time to compile. I also don't think this one test is enough to create a separate file for.
Meanwhile, type_caster<std::reference_wrapper<T>>
is already tested in test_builtin_casters.cpp
and other test files already have custom casters as well (test_custom_type_casters.cpp
, test_copy_move.cpp
, and even one in pybind11_tests.h
). So if we can keep the scope of the caster & test focused enough, I think it's fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other counter point is that this requires a custom type caster; so maybe move them to custom_type_casters.cc? I think that the current test is about as minimal as it can be while exercising the various paths we want to test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just logging my view, not a request: it makes me sad to see this test lumped in with other things. I know the existing tests do that a lot, but I think it really isn't a good approach, because it convolutes unrelated things, makes them harder to understand/follow, and obviously is counter the the idea of testing small self-contained units. Sticking with the established pattern only has so much value. Nudging things in a better direction has value, too. My preference is definitely to keep this test separate, if Boris or Yannick can agree.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Anyway, Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @rwgk however I will change it to the reviewers consensus.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other counter point is that this requires a custom type caster; so maybe move them to custom_type_casters.cc? I think that the current test is about as minimal as it can be while exercising the various paths we want to test.
Well, coming from this PR, it's testing the built-in caster for std::reference_wrapper
, so I thought it would be best to keep this as close to the other tests for reference_wrapper
, and to not create yet another file is "something with casters" (which takes time to compile, each time!).
move them to custom_type_casters.cc
It's not really about the custom caster, though. It's about testing reference_wrapper
, currently, I though? If this maybe gets expanded further in the future, the caster could be shared in a header, though, when it's necessary.
it makes me sad to see this test lumped in with other things.
Why exactly? Because I had explicitly tried to design/shrink this to be as minimal, unit-testy as possible, such that it could fit in with the rest of the std::reference_wrapper
tests (that were even extended) in builtin_casters.cpp
.
And to me it's more confusing to break convention than to add a few lines to the existing tests.
I do agree the tests are messy, btw, but I don't see the urgency of splitting things up. Build times are (partially) to blame, I think. So maybe once #2445 gets merged, it will be more feasible to split up tests, in a separate PR that brings order to the tests in a less ad-hoc fashion?
I agree with @rwgk however I will change it to the reviewers consensus.
Not sure there's consensus, though. If you want, we can give @henryiii a decisive vote, but ... well, yeah, I've explained my reasoning :-)
1. std::add_lvalue_reference<type> -> type& 2. Simplify the test a little more; we're never returning the ConstRefCaster type so the class_ definition can be removed.
The new test is an awesome demo for how For myself I changed it around like this:
I think that makes it even easier to understand. |
I like this version better as it moves the assertion that matters back into python.
Huh, OK. This was my first idea, but then I thought "let's make things more descriptive". Wasn't
I'm just not convinced by this. That name describes the internals, not what's being tested by it? |
I think that it's easier to see that each cast path has a unique result, and the numeric is easier to map back than a pair would be. |
That's true, yes, I like that argument! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I forgot to approve, it seems.
Thanks for the PR, for holding on during the review, and for fixing all the bikeshedding comments, @laramiel!
using subcaster_cast_op_type = typename caster_t::template cast_op_type<type>; | ||
static_assert(std::is_same<typename std::remove_const<type>::type &, subcaster_cast_op_type>::value, | ||
"std::reference_wrapper<T> caster requires T to have a caster with an `T &` operator"); | ||
using reference_t = type&; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could still be collapsed where it's used, to be shorter and more transparent (which was why I suggested type &
instead of add_lvalue_reference<type>::type
). But if that's the only thing left to change, I'm fine with this as well, and you don't need to add yet another commits, AFAIC.
Regarding CI: I restarted a few times, but that didn't help. No clue why https://developer.download.nvidia.com/hpc-sdk/nvhpc-20-7-20.7-1.x86_64.rpm is unreachable, and PyPy 2 has been failing for a week or so now (also ignore; #2724 is dealing with disabling PyPy 2 in CI for now). |
type_caster<std::reference_wrapper<T>>
to support const/non-const propagation in cast_op
.
Please assign a milestone, either 2.6.2 or 2.7.0. Thanks. :) |
Is this a fix? I guess it could be considered one. I'm happy to add this into 2.6.2; this shouldn't really be affecting existing code, I would hope. (If not, something went wrong?) |
I'll take care of the change log. |
Do we need a changelog? In principle these should be internal changes, that don't immediately concern users. @henryiii? |
* CI: Intel icc/icpc via oneAPI Add testing for Intel icc/icpc via the oneAPI images. Intel oneAPI is in a late beta stage, currently shipping oneAPI beta09 with ICC 20.2. CI: Skip Interpreter Tests for Intel Cannot find how to add this, neiter the package `libc6-dev` nor `intel-oneapi-mkl-devel` help when installed to solve this: ``` -- Looking for C++ include pthread.h -- Looking for C++ include pthread.h - not found CMake Error at /__t/cmake/3.18.4/x64/cmake-3.18.4-Linux-x86_64/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message): Could NOT find Threads (missing: Threads_FOUND) Call Stack (most recent call first): /__t/cmake/3.18.4/x64/cmake-3.18.4-Linux-x86_64/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE) /__t/cmake/3.18.4/x64/cmake-3.18.4-Linux-x86_64/share/cmake-3.18/Modules/FindThreads.cmake:234 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) tests/test_embed/CMakeLists.txt:17 (find_package) ``` CI: libc6-dev from GCC for ICC CI: Run bare metal for oneAPI CI: Ubuntu 18.04 for oneAPI CI: Intel +Catch -Eigen CI: CMake from Apt (ICC tests) CI: Replace Intel Py with GCC Py CI: Intel w/o GCC's Eigen CI: ICC with verbose make [Debug] Find core dump tests: use arg{} instead of arg() for Intel tests: adding a few more missing {} fix: sync with @tobiasleibner's branch fix: try ubuntu 20-04 fix: drop exit 1 docs: Apply suggestions from code review Co-authored-by: Tobias Leibner <[email protected]> Workaround for ICC enable_if issues Another workaround for ICC's enable_if issues fix error in previous commit Disable one test for the Intel compiler in C++17 mode Add back one instance of py::arg().noconvert() Add NOLINT to fix clang-tidy check Work around for ICC internal error in PYBIND11_EXPAND_SIDE_EFFECTS in C++17 mode CI: Intel ICC with C++17 docs: pybind11/numpy.h does not require numpy at build time. (#2720) This is nice enough to be mentioned explicitly in the docs. docs: Update warning about Python 3.9.0 UB, now that 3.9.1 has been released (#2719) Adjusting `type_caster<std::reference_wrapper<T>>` to support const/non-const propagation in `cast_op`. (#2705) * Allow type_caster of std::reference_wrapper<T> to be the same as a native reference. Before, both std::reference_wrapper<T> and std::reference_wrapper<const T> would invoke cast_op<type>. This doesn't allow the type_caster<> specialization for T to distinguish reference_wrapper types from value types. After, the type_caster<> specialization invokes cast_op<type&>, which allows reference_wrapper to behave in the same way as a native reference type. * Add tests/examples for std::reference_wrapper<const T> * Add tests which use mutable/immutable variants This test is a chimera; it blends the pybind11 casters with a custom pytype implementation that supports immutable and mutable calls. In order to detect the immutable/mutable state, the cast_op needs to propagate it, even through e.g. std::reference<const T> Note: This is still a work in progress; some things are crashing, which likely means that I have a refcounting bug or something else missing. * Add/finish tests that distinguish const& from & Fixes the bugs in my custom python type implementation, demonstrate test that requires const& and reference_wrapper<const T> being treated differently from Non-const. * Add passing a const to non-const method. * Demonstrate non-const conversion of reference_wrapper in tests. Apply formatting presubmit check. * Fix build errors from presubmit checks. * Try and fix a few more CI errors * More CI fixes. * More CI fixups. * Try and get PyPy to work. * Additional minor fixups. Getting close to CI green. * More ci fixes? * fix clang-tidy warnings from presubmit * fix more clang-tidy warnings * minor comment and consistency cleanups * PyDECREF -> Py_DECREF * copy/move constructors * Resolve codereview comments * more review comment fixes * review comments: remove spurious & * Make the test fail even when the static_assert is commented out. This expands the test_freezable_type_caster a bit by: 1/ adding accessors .is_immutable and .addr to compare identity from python. 2/ Changing the default cast_op of the type_caster<> specialization to return a non-const value. In normal codepaths this is a reasonable default. 3/ adding roundtrip variants to exercise the by reference, by pointer and by reference_wrapper in all call paths. In conjunction with 2/, this demonstrates the failure case of the existing std::reference_wrpper conversion, which now loses const in a similar way that happens when using the default cast_op_type<>. * apply presubmit formatting * Revert inclusion of test_freezable_type_caster There's some concern that this test is a bit unwieldly because of the use of the raw <Python.h> functions. Removing for now. * Add a test that validates const references propagation. This test verifies that cast_op may be used to correctly detect const reference types when used with std::reference_wrapper. * mend * Review comments based changes. 1. std::add_lvalue_reference<type> -> type& 2. Simplify the test a little more; we're never returning the ConstRefCaster type so the class_ definition can be removed. * formatted files again. * Move const_ref_caster test to builtin_casters * Review comments: use cast_op and adjust some comments. * Simplify ConstRefCasted test I like this version better as it moves the assertion that matters back into python. ci: drop pypy2 linux, PGI 20.7, add Python 10 dev (#2724) * ci: drop pypy2 linux, add Python 10 dev * ci: fix mistake * ci: commented-out PGI 20.11, drop 20.7 fix: regression with installed pybind11 overriding local one (#2716) * fix: regression with installed pybind11 overriding discovered one Closes #2709 * docs: wording incorrect style: remove redundant instance->owned = true (#2723) which was just before set to True in instance->allocate_layout() fix: also throw in the move-constructor added by the PYBIND11_OBJECT macro, after the argument has been moved-out (if necessary) (#2701) Make args_are_all_* ICC workarounds unconditional Disable test_aligned on Intel ICC Fix test_aligned on Intel ICC Skip test_python_alreadyset_in_destructor on Intel ICC Fix test_aligned again ICC CI: Downgrade pytest pytest 6 does not capture the `discard_as_unraisable` stderr and just writes a warning with its content instead. * refactor: simpler Intel workaround, suggested by @laramiel * fix: try version with impl to see if it is easier to compile * docs: update README for ICC Co-authored-by: Axel Huebl <[email protected]> Co-authored-by: Henry Schreiner <[email protected]>
Description
Before
std::reference_wrapper<T>
would invokecast_op<type>
rather thancast_op<T&>
to construct the conversion operator type. This introduces different behavior for reference wrapped types when the underlyingtype_caster<>::cast_op<T>&
differs fromtype_caster<>::cast_op<T&>
.There is additional enforcement from a
static_assert
that requires the type resulting fromtype_caster<>::cast_op<T>
to be equivalent tointrinsic_t<T>
(essentiallystd:decay<T>
).After, the
type_caster<>
specialization invokescast_op<type&>
, which allowsstd::reference_wrapper<>
to behave in the same way as a native reference type.Also, the
static_assert
is changed to allow the resulting type to be eitherT&
orconst T&
, which gives slightly more flexibility totype_caster
overloads.This change is expected to be low impact for existing code and tests. Existing tests use a use the default implementation of cast_op which applies intrinsic_t to the result, effectively ignoring const, and making
The added test case, test_freezable_type_caster, demonstrates custom casting to either mutable or immutable types, and is a cleanroom test based somewhat on ideas from https://github.com/protocolbuffers/protobuf/tree/master/python/google/protobuf/pyext