Importing pybind11 smart_holder PR #2886 from GitHub and adjusting PyCLIF smart_ptrs_test accordingly.

pybind/pybind11#2886

High-level summary of the two critical behavior changes:
* Context: handling of smart pointers for instances of Python-derived classes with pybind11-wrapped bases.
* Keywords: trampoline, also known as alias class, virtual functions with Python overrides.
* Minimal example: `class PyDrvd(m.Abase)` in test_class_sh_with_alias.py
* When a `shared_ptr<base>` is passed from Python to C++ (as a C++ function argument), the `shared_ptr` is built specifically for the function call, using a custom deleter tying the lifetime of the Python instance to the lifetime of the `shared_ptr`. This solves what's sometimes referred to as "inheritance slicing" issue in connection with the pybind11 trampoline feature. Note that there are ~10 open pybind11 issues related to this problem, enumerated in the description of [PR #2839](pybind/pybind11#2839).
* Passing a `unique_ptr<base>` is disabled. A `ValueError` is raised, with message: "Ownership of instance with virtual overrides in Python cannot be transferred to C++."

For full details see the description of PR #2886.

  - 97a7fb722a9842cae4d91be82897dd863f6b607d Porting/adapting Dustin's PR #2839 to smart_holder branch... by Ralf W. Grosse-Kunstleve <[email protected]>

PiperOrigin-RevId: 360966763

Author: rwgk

clif/pybind11/staging/smart_ptrs_test.py

@@ -69,13 +69,14 @@ def testSmartPtrs(self):
       smart_ptrs.Func(a)
 
     add = Add(120, 3)
-    # TODO: Temporarily disable the test because pybind11 throws
-    # `RuntimeError: Tried to call pure virtual function` when calling
-    # smart_ptrs.PerformUP.
-    # self.assertEqual(smart_ptrs.PerformUP(add), 123)
-    # # Previous call to Perform invalidated |add|
-    # with self.assertRaises(ValueError):
-    #   smart_ptrs.PerformUP(add)
+    # pybind11 (with smart_holder) deliberatly does not support this
+    # inherently unsafe transfer of ownership.
+    with self.assertRaises(ValueError) as ctx:
+      smart_ptrs.PerformUP(add)
+    self.assertEqual(
+        str(ctx.exception),
+        'Ownership of instance with virtual overrides in Python cannot be'
+        ' transferred to C++.')
 
     add = Add(1230, 4)
     self.assertEqual(smart_ptrs.PerformSP(add), 1234)

clif/testing/python/smart_ptrs_test.py

@@ -99,15 +99,29 @@ def testSmartPtrs(self, wrapper_lib):
 
     add = AddParameterized(wrapper_lib)(120, 3)
     if wrapper_lib is smart_ptrs:
-      # TODO: Temporarily disabling test for pybind11, which
-      # throws `RuntimeError: Tried to call pure virtual function` when calling
-      # smart_ptrs.PerformUP.
+      # This test (and the test for `PerformSP` below) can work safely only
+      # because PyCLIF leaks the reference count for `add` (use
+      # testInfiniteLoopAddVirtualOverride below to reproduce the leak), which
+      # keeps the Python instance alive beyond the time of transferring
+      # ownership from Python to C++.
       self.assertEqual(wrapper_lib.PerformUP(add), 123)
       # Previous call to Perform invalidated |add|
       with self.assertRaises(ValueError):
         wrapper_lib.PerformUP(add)
+    else:
+      # pybind11 (with smart_holder) deliberatly does not support this
+      # inherently unsafe transfer of ownership. (Knowingly leaking the
+      # reference count is not an option.)
+      with self.assertRaises(ValueError) as ctx:
+        wrapper_lib.PerformUP(add)
+      self.assertEqual(
+          str(ctx.exception),
+          'Ownership of instance with virtual overrides in Python cannot be'
+          ' transferred to C++.')
 
     add = AddParameterized(wrapper_lib)(1230, 4)
+    # This works with the C-API code generator only because the reference count
+    # for `add` is leaked. It works cleanly in pybind11 after PR #2886.
     self.assertEqual(wrapper_lib.PerformSP(add), 1234)
     # Calls to PerformSP should not invalidate |add|.
     self.assertEqual(wrapper_lib.PerformSP(add), 1234)