Skip to content

Add py::potentially_slicing_weak_ptr(handle) function #5624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
May 13, 2025

Conversation

nsoblath
Copy link
Contributor

@nsoblath nsoblath commented Apr 17, 2025

Description

Closes #5623.

Please see the added documentation and source code comments for more information.

For quick access, this is the new section in the documentation:

virtual ~WpBase() = default;
};

struct PyWpBase : WpBase {
Copy link
Collaborator

@rwgk rwgk Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

struct PyWpBase : WpBase, py::trampoline_self_life_support {

Does that help?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I just fixed an oversight in my previous comment.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
struct PyWpBase : WpBase {
struct PyWpBase : WpBase, py::trampoline_self_life_support {

@rwgk
Copy link
Collaborator

rwgk commented Apr 19, 2025

I simplified the reproducer and changed the variable names to make them more intuitive (to me).

I'm guessing we're building an intermediate shared_ptr<VirtBase> that has a different control block than the shared_ptr held by the smart_holder.

I wrote that code four years ago. Quite likely this will take me several hours to re-learn what exactly is happening. Not sure when I'll get to it.

@nsoblath How did you arrive at that test? Did you have code that worked with std::shared_ptr as holder?

…s holder). Tweak test_class_sh_trampoline_weak_ptr.py to pass, with `# THIS NEEDS FIXING` comment.
@rwgk
Copy link
Collaborator

rwgk commented Apr 19, 2025

@nsoblath How did you arrive at that test? Did you have code that worked with std::shared_ptr as holder?

On that suspicion, I added test_class_sp_trampoline_weak_ptr.cpp,py, and it seems to work: it certainly works locally, but I'm waiting for GitHub Actions to finish.

Assuming the new test works reliably on all platforms, I'd guess that it's feasible to fix the smart_holder equivalent.

rwgk added 3 commits April 19, 2025 13:39
```
/__w/pybind11/pybind11/tests/test_class_sh_trampoline_weak_ptr.cpp:23:43: error: the parameter 'sp' is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param,-warnings-as-errors]
   23 |     void set_wp(std::shared_ptr<VirtBase> sp) { wp = sp; }
      |                                           ^
      |                 const                    &
4443 warnings generated.
```

```
/__w/pybind11/pybind11/tests/test_class_sp_trampoline_weak_ptr.cpp:23:43: error: the parameter 'sp' is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param,-warnings-as-errors]
   23 |     void set_wp(std::shared_ptr<VirtBase> sp) { wp = sp; }
      |                                           ^
      |                 const                    &
4430 warnings generated.
```
@rwgk
Copy link
Collaborator

rwgk commented Apr 19, 2025

Before I wrote:

I'm guessing we're building an intermediate shared_ptr<VirtBase> that has a different control block than the shared_ptr held by the smart_holder.

That's here (conclusive):

Note that the smart_holder instance is readily available in the same context:

hld.vptr is the shared_ptr instance with the control block that we need in order to fix #5623.

So in line 782 we need to return a shared_ptr that

  • has the same control block as hld.vptr
  • but also owns a Python reference count for loaded_v_h.inst

Any ideas, please let me know. (I haven't thought about it much yet; the penny didn't want to drop immediately.)

@virtuald for visibility

@nsoblath
Copy link
Contributor Author

@nsoblath How did you arrive at that test? Did you have code that worked with std::shared_ptr as holder?

I copied the test_class_sh_trampoline_shared_ptr_cpp_arg test, modified it to store a std::weak_ptr and stripped out everything that wasn't needed. It replicates the essential parts of the application I have that's not working. I didn't try using std::shared_ptr because the application I have also needs the benefits of py::smart_holder.

@rwgk
Copy link
Collaborator

rwgk commented Apr 20, 2025

I've now turned this over in my mind for a couple hours, and consulted ChatGPT, which I believe got very confused/confusing, but please see for yourself here.

Stepping out of the box:

  • The code I pointed out before is pulling a trick to avoid inheritance slicing ([BUG] Problem when creating derived Python objects from C++ (inheritance slicing) #1333), by keeping the derived Python object alive as long as the returned shared_ptr exists.

  • But you don't actually want a shared_ptr, you actually want a weak_ptr, which isn't meant to keep anything alive.

  • So in your particular case, the trick to avoid inheritance slicing backfires. It fixes one problem, but creates another one.

Solving this is a real brain teaser. I'm not sure I can help. I'm not sure if there is a viable solution. I have a couple ideas, but each of them might take hours of work only to find out it leads to a dead end.

Could you try to work around this limitation? For example, instead of owning the C++ weak_ptr, own a Python weakref instead (in WpOwner). Dereference the Python weakref to see if the Python object is still alive, then you can safely extract the C++ pointer if it is (in WpOwner::get_code).

rwgk added 3 commits April 20, 2025 10:54
…s does not inherit from this class, preserve established Inheritance Slicing behavior.

rwgk reached this point with the help of ChatGPT:

* https://chatgpt.com/share/68056498-7d94-8008-8ff0-232e2aba451c

The only production code change in this commit is:

```
diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h
index d4f9a41..f3d4530 100644
--- a/include/pybind11/detail/type_caster_base.h
+++ b/include/pybind11/detail/type_caster_base.h
@@ -776,6 +776,14 @@ struct load_helper : value_and_holder_helper {
                 if (released_ptr) {
                     return std::shared_ptr<T>(released_ptr, type_raw_ptr);
                 }
+                auto *self_life_support
+                    = dynamic_raw_ptr_cast_if_possible<trampoline_self_life_support>(type_raw_ptr);
+                if (self_life_support == nullptr) {
+                    std::shared_ptr<void> void_shd_ptr = hld.template as_shared_ptr<void>();
+                    std::shared_ptr<T> to_be_released(void_shd_ptr, type_raw_ptr);
+                    vptr_gd_ptr->released_ptr = to_be_released;
+                    return to_be_released;
+                }
                 std::shared_ptr<T> to_be_released(
                     type_raw_ptr, shared_ptr_trampoline_self_life_support(loaded_v_h.inst));
                 vptr_gd_ptr->released_ptr = to_be_released;
```
@rwgk
Copy link
Collaborator

rwgk commented Apr 20, 2025

I'm testing commit 4638e01 to see if that works on all platforms.

See https://chatgpt.com/share/68056498-7d94-8008-8ff0-232e2aba451c for how I got there / rationale.

I'm not sure if 4638e01 is a wise twist to add.

Fundamentally, it'd be better to support std::weak_ptr conversions more directly, but that's certainly a lot more work than I'm able to invest into pybind11.

```
    11>D:\a\pybind11\pybind11\tests\test_class_sp_trampoline_weak_ptr.cpp(44,50): error C2220: the following warning is treated as an error [D:\a\pybind11\pybind11\build\tests\pybind11_tests.vcxproj]
    11>D:\a\pybind11\pybind11\tests\test_class_sp_trampoline_weak_ptr.cpp(44,50): warning C4458: declaration of 'sp' hides class member [D:\a\pybind11\pybind11\build\tests\pybind11_tests.vcxproj]
             D:\a\pybind11\pybind11\tests\test_class_sp_trampoline_weak_ptr.cpp(54,31):
             see declaration of 'pybind11_tests::class_sp_trampoline_weak_ptr::SpOwner::sp'
```
@rwgk
Copy link
Collaborator

rwgk commented Apr 21, 2025

@nsoblath I feel I'm pretty clear now about the situation. I think it's best not to change the smart_holder code, but to go with the approach outlined before:

instead of owning the C++ weak_ptr, own a Python weakref instead (in WpOwner). Dereference the Python weakref to see if the Python object is still alive, then you can safely extract the C++ pointer if it is (in WpOwner::get_code).

I'll explain more when I get to it, probably next weekend.

(The only thing I want to change, probably, is to add a static_assert, to enforce that py::trampoline_self_life_support is used if the holder is smart_holder, and not used otherwise. This doesn't change anything at runtime, but is to avoid leaving users unclear.)

@nsoblath
Copy link
Contributor Author

@rwgk Thank you very much for your time and attention looking into this. I really appreciate it.

Though my understanding of the internals of pybind11 is quite incomplete, my thoughts about what might be going wrong were similar to what you suggested: that the control block for the shared_ptr<VirtBase> that's extracted is different from the one held by the smart_holder. Do I understand correctly that the change you added in 4638e01 fixes this problem? That's what it looks like since the test_class_sh_trampoline_weak_ptr tests now pass.

In your later message you said

I think it's best not to change the smart_holder code, . . .

Does that mean you're preferring to not use the change in 4638e01? Is there a downside to making sure that the control block of the returned shared_ptr matches the one held by the smart_holder in this particular situation?

Regarding the alternative solution you proposed:

instead of owning the C++ weak_ptr, own a Python weakref instead (in WpOwner). Dereference the Python weakref to see if the Python object is still alive, then you can safely extract the C++ pointer if it is (in WpOwner::get_code).

I believe I can implement something to this effect. It's not ideal because the library where I'm using this is sometimes used as a pure-C++ library, sometimes used with its Python bindings. WpOwner (which in reality holds a collection of weak_ptrs) can simultaneously hold pointers to pure C++ classes and Python classes. When the library is used without Python bindings, I don't want to require the user to also have Python and Pybind11 installed. However, I can probably make a struct that combines weak_ptr and weakref and handles the source differences between when the Python bindings are used and when they're not.

@rwgk
Copy link
Collaborator

rwgk commented Apr 21, 2025

Does that mean you're preferring to not use the change in 4638e01?

Yes.

Is there a downside to making sure that the control block of the returned shared_ptr matches the one held by the smart_holder in this particular situation?

Yes, it'll be very confusing to many (I think), while only being useful in a few niche cases.

It is not obvious at all that the shared_ptr that works for you is subject to inheritance slicing.

I feel it'll be better to keep the smart_holder feature simpler and the behavior consistent. It will be easier to reason about the bindings.

We could probably add an interface that makes it relatively easy for you to get the inheritance-slicing-shared_ptr from the Python object, which you could use from a lambda function in the pybind11 bindings.

What I didn't explain: I convinced myself that it is impossible to get a shared_ptr that does both:

  • Keeps the PyDrvd alive (in other words, isn't subject to inheritance slicing).
  • Has the same control block as the shared_ptr inside the holder.

Reason: This would introduce a hard reference cycle, because the Python object (PyDrvd) owns the holder.

@rwgk
Copy link
Collaborator

rwgk commented Apr 24, 2025

@nsoblath If you can wait a week or two, I think (although I'm not sure) that I can give you something like this:

    py::classh<WpOwner>(m, "WpOwner")
        ...
        .def("set_wp", [](WpOwner& self, py::handle obj) {
            self.set_wp(py::slicing_shared_ptr_from_smart_holder<VirtBase>(obj));
        })
        ...

@nsoblath
Copy link
Contributor Author

@nsoblath If you can wait a week or two, I think (although I'm not sure) that I can give you something like this:

    py::classh<WpOwner>(m, "WpOwner")
        ...
        .def("set_wp", [](WpOwner& self, py::handle obj) {
            self.set_wp(py::slicing_shared_ptr_from_smart_holder<VirtBase>(obj));
        })
        ...

@rwgk I can certainly wait, and if that works, it'd be great. Thank you!

Incidentally I have been working on the workaround we talked about, but this would be a cleaner solution (at least on the side of my application).

@rwgk
Copy link
Collaborator

rwgk commented Apr 24, 2025

Incidentally I have been working on the workaround we talked about, but this would be a cleaner solution (at least on the side of my application).

I wouldn't call the proposed "slicing" option the cleaner solution. It's only easier. — You might still get a shared_ptr from the weak_ptr, but the virtual overrides in Python might be gone (sliced). I convinced myself that's not UB, but depending on the exact circumstances, it can be very surprising.

rwgk added 2 commits April 26, 2025 12:08
Also undo the corresponding test change in test_class_sh_trampoline_weak_ptr.py

But keep all extra debugging code for now.
…cast to `std::shared_ptr<VirtBase>` for now.
@rwgk
Copy link
Collaborator

rwgk commented Apr 28, 2025

@nsoblath I'm mostly done, if you want to take a look. The only part missing is an addition to the trampoline documentation.

@rwgk
Copy link
Collaborator

rwgk commented Apr 28, 2025

@nsoblath All done now. Could you please review the added documentation?

@virtuald Is there a chance that you could help with a review? — The production code changes are small and nearly trivial.

Most of my effort for this PR went into the new stress-test level unit test, documentation, and source code comments.

@nsoblath
Copy link
Contributor Author

@rwgk The added documentation is excellent. I think it clarifies the situation and makes the options clear. I appreciate both the .rst documentation that will be in the RTD site, and the header-file documentation that includes more technical details.

With the documentation, it was clear that for my application the py::potentially_slicing_shared_ptr option is the best way to go. The risk of slicing in this specific situation is acceptable because the consequences of using the sliced C++ object are not significant.

My simplified test cases and actual application code all now work.

Thank you very much for your time and efforts on this.

Copy link
Contributor

@virtuald virtuald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely an esoteric bug, but I see how you got there. If everyone else is happy with this, I am happy with it too and you can ignore my nitpicky comment.

/// - the std::shared_ptr would own a reference to the derived Python object,
/// completing the cycle
template <typename T>
std::shared_ptr<T> potentially_slicing_shared_ptr(handle obj) {
Copy link
Contributor

@virtuald virtuald May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API is weird to me. If the sole reason for this API to exist is to obtain a weak_ptr that doesn't accidentally expire, why not just return a weak_ptr directly instead? Are there other reasons one might want to get this problematic shared_ptr?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're right. Trying it out right now: 222b4a7 (that's minimal changes only; more will be needed if testing goes well (as expected))

The implementation and the testing is a little more awkward, because we cannot pass std::weak_ptr around between Python and C++ (and I wouldn't want to add that complexity), but I totally agree that the public-facing API is better at nudging people in the right direction. What I mean is, it'll be even more likely that the new public API will only get used if a std::weak_ptr is truly needed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All done. Please see commits 03a8981 and bad0c12. (The last commit 683fff4 is trivial.)

@henryiii henryiii requested a review from Copilot May 1, 2025 05:00
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new function, py::potentially_slicing_shared_ptr, to address issues related to inheritance slicing and weak_ptr behavior when using py::smart_holder as the holder type.

  • Introduces a new overload in load_as_shared_ptr with an extra force flag to control potentially slicing conversions.
  • Adds C++ and Python tests to verify expected behavior and error reporting for both shared_ptr conversion functions.
  • Updates documentation in cast.h to explain the behavior and potential trade-offs (such as reintroduced inheritance slicing).

Reviewed Changes

Copilot reviewed 4 out of 6 changed files in this pull request and generated 1 comment.

File Description
tests/test_potentially_slicing_shared_ptr.py Added comprehensive tests covering shared_ptr conversion behavior.
tests/test_potentially_slicing_shared_ptr.cpp Introduced C++ tests for both obj.cast and potentially slicing path.
include/pybind11/detail/type_caster_base.h Extended load_as_shared_ptr() to include a force flag parameter.
include/pybind11/cast.h Added the potentially_slicing_shared_ptr() function and its docs.
Files not reviewed (2)
  • docs/advanced/classes.rst: Language not supported
  • tests/CMakeLists.txt: Language not supported
Comments suppressed due to low confidence (2)

include/pybind11/detail/type_caster_base.h:763

  • [nitpick] Consider adding a brief comment explaining the purpose of the 'force_potentially_slicing_shared_ptr' parameter to clarify its effect on alias handling.
bool force_potentially_slicing_shared_ptr = false) const {

tests/test_potentially_slicing_shared_ptr.py:163

  • [nitpick] Using exact string equality for error message checks can be brittle; consider verifying that the error message contains an expected substring rather than matching it exactly.
assert str(excinfo.value) == ( ...

@@ -983,6 +983,16 @@ struct copyable_holder_caster<
return shared_ptr_storage;
}

std::shared_ptr<type> &potentially_slicing_shared_ptr() {
if (typeinfo->holder_enum_v == detail::holder_enum_t::smart_holder) {
Copy link
Preview

Copilot AI May 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] A short inline comment explaining why the call to load_as_shared_ptr uses force_potentially_slicing_shared_ptr=true would help maintain clarity for future readers.

Suggested change
if (typeinfo->holder_enum_v == detail::holder_enum_t::smart_holder) {
if (typeinfo->holder_enum_v == detail::holder_enum_t::smart_holder) {
// Use force_potentially_slicing_shared_ptr=true to handle cases where slicing
// might occur, ensuring the shared_ptr is created even if the type is not exact.

Copilot uses AI. Check for mistakes.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: // Reusing shared_ptr code to minimize code complexity.

@henryiii
Copy link
Collaborator

henryiii commented May 1, 2025

By the way, feel free to ignore the co-pilot review, was just curious, and I've had it catch a bug before elsewhere.

…e) as the public API. For CI testing, before changing the names around more widely, and the documentation.
@rwgk rwgk changed the title Add py::potentially_slicing_shared_ptr(handle) function Add py::potentially_slicing_weak_ptr(handle) function May 8, 2025
@henryiii henryiii requested a review from Copilot May 10, 2025 19:17
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new function, py::potentially_slicing_weak_ptr(handle), which provides an alternative way to obtain a std::weak_ptr from a Python object while addressing issues related to inheritance slicing when using py::smart_holder.

  • Added new tests in Python and C++ to validate both correct behavior and error cases associated with potentially slicing weak pointers.
  • Enhanced the underlying type caster implementations in include/pybind11/detail/type_caster_base.h and include/pybind11/cast.h to support the new functionality.
  • Updated the documentation to explain the differences in lifetime management between std::shared_ptr and py::smart_holder, and the trade-offs when using potentially slicing weak pointers.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/test_potentially_slicing_weak_ptr.py New tests to verify shared and weak pointer behavior.
tests/test_potentially_slicing_weak_ptr.cpp Adds C++ test support for the new slicing weak pointer function.
tests/CMakeLists.txt Registers the new test suite.
include/pybind11/detail/type_caster_base.h Modified load_as_shared_ptr() to accept a force flag for slicing.
include/pybind11/cast.h Introduced the potentially_slicing_weak_ptr() API and documentation.
docs/advanced/classes.rst Adds a documentation section to explain the new weak_ptr behavior.

@@ -774,7 +776,7 @@ struct load_helper : value_and_holder_helper {
throw std::runtime_error("Non-owning holder (load_as_shared_ptr).");
}
auto *type_raw_ptr = static_cast<T *>(void_raw_ptr);
Copy link
Preview

Copilot AI May 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding a comment here to explain the purpose of the new parameter 'force_potentially_slicing_shared_ptr' and how it affects the alias check, so future maintainers understand why this conditional branch bypasses the slicing protection.

Suggested change
auto *type_raw_ptr = static_cast<T *>(void_raw_ptr);
auto *type_raw_ptr = static_cast<T *>(void_raw_ptr);
// The `force_potentially_slicing_shared_ptr` parameter allows bypassing the slicing
// protection mechanism when `python_instance_is_alias` is true. This is necessary
// in scenarios where the caller explicitly requests a shared_ptr that may result
// in slicing, typically for advanced use cases or performance optimizations.
// Without this flag, the function enforces strict aliasing rules to prevent
// potential slicing issues.

Copilot uses AI. Check for mistakes.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose this less verbose, and less redundant, alternative: commit b349e84

…ase.h, to make a direct connection py::potentially_slicing_weak_ptr
@rwgk
Copy link
Collaborator

rwgk commented May 13, 2025

Thanks @henryiii and @virtuald! I'll merge this when I see that our GHA queue isn't overloaded anymore.

@henryiii henryiii merged commit 74b5242 into pybind:master May 13, 2025
65 checks passed
@github-actions github-actions bot added the needs changelog Possibly needs a changelog entry label May 13, 2025
@rwgk rwgk removed the needs changelog Possibly needs a changelog entry label May 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG]: std::weak_ptr expires for derived Python objects using py::smart_holder
4 participants