Support ownership transfer between C++ and Python with shared_ptr<T> and unique_ptr<T> for pure C++ instances and single-inheritance instances

Author: EricCousineau-TRI

docs/advanced/cast/overview.rst

@@ -119,6 +119,9 @@ as arguments and return values, refer to the section on binding :ref:`classes`.
 | ``std::string_view``,              | STL C++17 string views    | :file:`pybind11/pybind11.h`   |
 | ``std::u16string_view``, etc.      |                           |                               |
 +------------------------------------+---------------------------+-------------------------------+
+| ``std::unique_ptr<T>``,            | STL (or custom) smart     | :file:`pybind11/cast.h`       |
+| ``std::shared_ptr<T>``, etc.       | pointers.                 |                               |
++------------------------------------+---------------------------+-------------------------------+
 | ``std::pair<T1, T2>``              | Pair of two custom types  | :file:`pybind11/pybind11.h`   |
 +------------------------------------+---------------------------+-------------------------------+
 | ``std::tuple<...>``                | Arbitrary tuple of types  | :file:`pybind11/pybind11.h`   |

docs/advanced/classes.rst

@@ -64,10 +64,10 @@ helper class that is defined as follows:
 
 .. code-block:: cpp
 
-    class PyAnimal : public Animal {
+    class PyAnimal : public py::wrapper<Animal> {
     public:
         /* Inherit the constructors */
-        using Animal::Animal;
+        using py::wrapper<Animal>::wrapper;
 
         /* Trampoline (need one for each virtual function) */
         std::string go(int n_times) override {
@@ -89,6 +89,8 @@ function* slots, which defines the name of function in Python. This is required
 when the C++ and Python versions of the
 function have different names, e.g.  ``operator()`` vs ``__call__``.
 
+The base class ``py::wrapper<>`` is optional, but is recommended as it allows us to attach the lifetime of Python objects directly to C++ objects, explained in :ref:`virtual_inheritance_lifetime`.
+
 The binding code also needs a few minor adaptations (highlighted):
 
 .. code-block:: cpp
@@ -230,15 +232,15 @@ override the ``name()`` method):
 
 .. code-block:: cpp
 
-    class PyAnimal : public Animal {
+    class PyAnimal : public py::wrapper<Animal> {
     public:
-        using Animal::Animal; // Inherit constructors
+        using py::wrapper<Animal>::wrapper; // Inherit constructors
         std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, Animal, go, n_times); }
         std::string name() override { PYBIND11_OVERLOAD(std::string, Animal, name, ); }
     };
-    class PyDog : public Dog {
+    class PyDog : public py::wrapper<Dog> {
     public:
-        using Dog::Dog; // Inherit constructors
+        using py::wrapper<Dog>::wrapper; // Inherit constructors
         std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, Dog, go, n_times); }
         std::string name() override { PYBIND11_OVERLOAD(std::string, Dog, name, ); }
         std::string bark() override { PYBIND11_OVERLOAD(std::string, Dog, bark, ); }
@@ -258,24 +260,24 @@ declare or override any virtual methods itself:
 .. code-block:: cpp
 
     class Husky : public Dog {};
-    class PyHusky : public Husky {
+    class PyHusky : public py::wrapper<Husky> {
     public:
-        using Husky::Husky; // Inherit constructors
+        using py::wrapper<Husky>::wrapper; // Inherit constructors
         std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, Husky, go, n_times); }
         std::string name() override { PYBIND11_OVERLOAD(std::string, Husky, name, ); }
         std::string bark() override { PYBIND11_OVERLOAD(std::string, Husky, bark, ); }
     };
 
 There is, however, a technique that can be used to avoid this duplication
 (which can be especially helpful for a base class with several virtual
-methods).  The technique involves using template trampoline classes, as
+methods).  The technique (the Curiously Recurring Template Pattern) involves using template trampoline classes, as
 follows:
 
 .. code-block:: cpp
 
-    template <class AnimalBase = Animal> class PyAnimal : public AnimalBase {
+    template <class AnimalBase = Animal> class PyAnimal : public py::wrapper<AnimalBase> {
     public:
-        using AnimalBase::AnimalBase; // Inherit constructors
+        using py::wrapper<AnimalBase>::wrapper; // Inherit constructors
         std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, AnimalBase, go, n_times); }
         std::string name() override { PYBIND11_OVERLOAD(std::string, AnimalBase, name, ); }
     };
@@ -1080,3 +1082,120 @@ appropriate derived-class pointer (e.g. using
     more complete example, including a demonstration of how to provide
     automatic downcasting for an entire class hierarchy without
     writing one get() function for each class.
+
+.. _virtual_inheritance_lifetime:
+
+Virtual Inheritance and Lifetime
+================================
+
+When an instance of a Python subclass of a ``pybind11``-bound C++ class is instantiated, there are effectively two "portions": the C++ portion of the base class's alias instance, and the Python portion (``__dict__``) of the derived class instance.
+Generally, the lifetime of an instance of a Python subclass of a ``pybind11``-bound C++ class will not pose an issue as long as the instance is owned in Python - that is, you can call virtual methods from C++ or Python and have the correct behavior.
+
+However, if this Python-constructed instance is passed to C++ such that there are no other Python references, then C++ must keep the Python portion of the instance alive until either (a) the C++ reference is destroyed via ``delete`` or (b) the object is passed back to Python. ``pybind11`` supports both cases, but **only** when (i) the class inherits from :class:`py::wrapper`, (ii) there is only single-inheritance in the bound C++ classes, and (iii) the holder type for the class is either :class:`std::shared_ptr` (suggested) or :class:`std::unique_ptr` (default).
+
+.. seealso::
+
+    :ref:`holders` has more information regaring holders and how general ownership transfer should function.
+
+When ``pybind11`` detects case (a), it will store a reference to the Python object in :class:`py::wrapper` using :class:`py::object`, such that if the instance is deleted by C++, then it will also release the Python object (via :func:`py::wrapper::~wrapper()`). The wrapper will have a unique reference to the Python object (as any other circumstance would trigger case (b)), so the Python object should be destroyed immediately upon the instance's destruction.
+This will be a cyclic reference per Python's memory management, but this is not an issue as the memory is now managed via C++.
+
+For :class:`std::shared_ptr`, this case is detected by placing a shim :func:`__del__` method on the Python subclass when ``pybind11`` detects an instance being created. This shim will check for case (a), and if it holds, will "resurrect" since it created a new reference using :class:`py::object`.
+
+For :class:`std::unique_ptr`, this case is detected when calling `py::cast<unique_ptr<T>>`, which itself implies ownership transfer.
+
+.. seealso::
+
+    See :ref:`unique_ptr_ownership` for information about how ownership can be transferred via a cast or argument involving ``unique_ptr<Type>``.
+
+When ``pybind11`` detects case (b) (e.g. ``py::cast()`` is called to convert a C++ instance to `py::object`) and (a) has previously occurred, such that C++ manages the lifetime of the object, then :class:`py::wrapper` will release the Python reference to allow Python to manage the lifetime of the object.
+
+.. note::
+
+    This mechanism will be generally robust against reference cycles in Python as this couples the two "portions"; however, it does **not** protect against reference cycles with :class:`std::shared_ptr`. You should take care and use :class:`std::weak_ref` or raw pointers (with care) when needed.
+
+.. note::
+
+    There will a slight difference in destructor order if the complete instance is destroyed in C++ or in Python; however, this difference will only be a difference in ordering in when :func:`py::wrapper::~wrapper()` (and your alias destructor) is called in relation to :func:`__del__` for the subclass. For more information, see the documentation comments for :class:`py::wrapper`.
+
+For this example, we will build upon the above code for ``Animal`` with alias ``PyAnimal``, and the Python subclass ``Cat``, but will introduce a situation where C++ may have sole ownership: a container. In this case, it will be ``Cage``, which can contain or release an animal.
+
+.. note::
+
+    For lifetime, it is important to use a more Python-friendly holder, which in this case would be :class:`std::shared_ptr`, permitting an ease to share ownership.
+
+.. code-block:: cpp
+
+    class Animal {
+    public:
+        virtual ~Animal() { }
+        virtual std::string go(int n_times) = 0;
+    };
+
+    class PyAnimal : public py::wrapper<Animal> {
+    public:
+        /* Inherit the constructors */
+        using py::wrapper<Animal>::wrapper;
+        std::string go(int n_times) override {
+            PYBIND11_OVERLOAD_PURE(std::string, Animal, go, n_times);
+        }
+    };
+
+    class Cage {
+    public:
+        void add(std::shared_ptr<Animal> animal) {
+            animal_ = animal;
+        }
+        std::shared_ptr<Animal> release() {
+            return std::move(animal_);
+        }
+    private:
+        std::shared_ptr<Animal> animal_;
+    };
+
+And the following bindings:
+
+.. code-block:: cpp
+
+    PYBIND11_MODULE(example, m) {
+        py::class_<Animal, PyAnimal, std::shared_ptr<Animal>> animal(m, "Animal");
+        animal
+            .def(py::init<>())
+            .def("go", &Animal::go);
+
+        py::class_<Cage, std::shared_ptr<Cage>> cage(m, "Cage")
+            .def(py::init<>())
+            .def("add", &Cage::add)
+            .def("release", &Cage::release);
+    }
+
+With the following Python preface:
+
+.. code-block:: pycon
+
+    >>> from examples import *
+    >>> class Cat(Animal):
+    ...     def go(self, n_times):
+    ...             return "meow! " * n_times
+    ...
+    >>> cage = Cage()
+
+Normally, if you keep the object alive in Python, then no additional instrumentation is necessary:
+
+.. code-block:: pycon
+
+    >>> cat = Cat()
+    >>> c.add(cat)  # This object lives in both Python and C++.
+    >>> c.release().go(2)
+    meow! meow! 
+
+However, if you pass an instance that Python later wishes to destroy, without :class:`py::wrapper`, we would get an error that ``go`` is not implented,
+as the `Cat` portion would have been destroyed and no longer visible for the trampoline. With the wrapper, ``pybind11`` will intercept this event and keep the Python portion alive:
+
+.. code-block:: pycon
+
+    >>> c.add(Cat())
+    >>> c.release().go(2)
+    meow! meow! 
+
+Note that both the C++ and Python portion of ``cat`` will be destroyed once ``cage`` is destroyed.

docs/advanced/smart_ptrs.rst

@@ -1,5 +1,17 @@
-Smart pointers
-##############
+.. _holders:
+
+Smart pointers and holders
+##########################
+
+Holders
+=======
+
+The binding generator for classes, :class:`class_`, can be passed a template
+type that denotes a special *holder* type that is used to manage references to
+the object.  If no such holder type template argument is given, the default for
+a type named ``Type`` is ``std::unique_ptr<Type>``, which means that the object
+is deallocated when Python's reference count goes to zero. It is possible to switch to other types of reference counting wrappers or smart
+pointers, which is useful in codebases that rely on them, such as ``std::shared_ptr<Type>``, or even a custom type.
 
 std::unique_ptr
 ===============
@@ -15,31 +27,100 @@ instances wrapped in C++11 unique pointers, like so
 
     m.def("create_example", &create_example);
 
-In other words, there is nothing special that needs to be done. While returning
-unique pointers in this way is allowed, it is *illegal* to use them as function
-arguments. For instance, the following function signature cannot be processed
-by pybind11.
+In other words, there is nothing special that needs to be done.
+
+.. _unique_ptr_ownership:
+
+Transferring ownership
+----------------------
+
+It is also possible to pass ``std::unique_ptr<Type>`` as a function
+argument, or use ``py::cast<unique_ptr<Type>>(std::move(obj))``. Note that this tells pybind11 to not manage the memory for this object, and delegate that to ``std::unique_ptr<Type>``.
+For instance, the following function signature can be processed by pybind11:
 
 .. code-block:: cpp
 
     void do_something_with_example(std::unique_ptr<Example> ex) { ... }
 
-The above signature would imply that Python needs to give up ownership of an
-object that is passed to this function, which is generally not possible (for
-instance, the object might be referenced elsewhere).
+The above signature does imply that Python needs to give up ownership of an
+object that is passed to this function. There are two ways to do this:
+
+1.  Simply pass the object in. The reference count of the object can be greater than one (non-unique) when passing the object in. **However**, you *must* ensure that the object has only **one** reference when C++ (which owns the C++ object).
+
+    To expand on this, when transferring ownership for ``std::unique_ptr``, this means that Pybind11 no longer owns the reference, which means that if C++ lets the ``std::unique_ptr`` destruct but if there is a dangling reference in Python, then you will encounter undefined behavior.
+
+    Examples situations:
+
+    * The C++ function is terminal (i.e. will destroy the object once it completes). This is generally not an issue unless a Python portion of the object has a non-trivial ``__del__`` method.
+    * The Python object is passed to a C++ container which only tracks ``std::unique_ptr<Type>``. If the container goes out of scope, the Python object reference will be invalid.
+
+    .. note::
+
+        For polymorphic types that inherit from :class:`py::wrapper`, ``pybind11`` *can* warn about these situations.
+        You may enable this behavior with ``#define PYBIND11_WARN_DANGLING_UNIQUE_PYREF``. This will print a warning to ``std::err`` if this case is detected.
+
+2.  Pass a Python "move container" (a mutable object that can "release" the reference to the object). This can be a single-item list, or any Python class / instance that has the field ``_is_move_container = True`` and has a ``release()`` function.
+
+    .. note::
+
+        When using a move container, this expects that the provided object is a **unique** reference, or will throw an error otherwise. This is a little more verbose, but will make debugging *much* easier.
+
+    As an example in C++:
+
+    .. code-block:: cpp
+
+        void terminal_func(std::unique_ptr<Example> obj) {
+            obj.do_something();  // `obj` will be destroyed when the function exits.
+        }
+
+        // Binding
+        py::class_<Example> example(m, "Example");
+        m.def("terminal_func", &terminal_func);
+
+    In Python, say you would normally do this:
+
+    .. code-block:: pycon
+
+        >>> obj = Example()
+        >>> terminal_func(obj)
+
+    As mentioned in the comment, you *must* ensure that `obj` is not used past this invocation, as the underlying data has been destroyed. To be more careful, you may "move" the object. The following will throw an error:
+
+    .. code-block:: pycon
+
+        >>> obj = Example()
+        >>> terminal_func([obj])
+
+    However, this will work, using a "move" container:
+
+    .. code-block:: pycon
+
+        >>> obj = Example()
+        >>> obj_move = [obj]
+        >>> del obj
+        >>> terminal_func(obj_move)
+        >>> print(obj_move)  # Reference will have been removed.
+        [None]
+
+    or even:
+
+    .. code-block:: pycon
+
+        >>> terminal_func([Example()])
+
+    .. note::
+
+        ``terminal_func(Example())`` also works, but still leaves a dangling reference, which is only a problem if it is polymorphic and has a non-trivial ``__del__`` method.
+
+    .. warning::
+
+        This reference counting mechanism is **not** robust aganist cyclic references. If you need some sort of cyclic reference, *please* consider using ``weakref.ref`` in Python.
 
 std::shared_ptr
 ===============
 
-The binding generator for classes, :class:`class_`, can be passed a template
-type that denotes a special *holder* type that is used to manage references to
-the object.  If no such holder type template argument is given, the default for
-a type named ``Type`` is ``std::unique_ptr<Type>``, which means that the object
-is deallocated when Python's reference count goes to zero.
-
-It is possible to switch to other types of reference counting wrappers or smart
-pointers, which is useful in codebases that rely on them. For instance, the
-following snippet causes ``std::shared_ptr`` to be used instead.
+If you have an existing code base with ``std::shared_ptr``, or you wish to enable reference counting in C++ as well, then you may use this type as a holder.
+As an example, the following snippet causes ``std::shared_ptr`` to be used instead.
 
 .. code-block:: cpp
 
@@ -111,6 +192,12 @@ There are two ways to resolve this issue:
 
     class Child : public std::enable_shared_from_this<Child> { };
 
+.. seealso::
+
+    While ownership transfer is generally not an issue with ``std::shared_ptr<Type>``, it becomes an issue when an instance of a Python subclass of a pybind11 class is effectively managed by C++ (e.g. all live references to the object are from C++, and all reference in Python have "died").
+
+    See :ref:`virtual_inheritance_lifetime` for more information.
+
 .. _smart_pointers:
 
 Custom smart pointers
@@ -147,7 +234,7 @@ Please take a look at the :ref:`macro_notes` before using this feature.
 
 By default, pybind11 assumes that your custom smart pointer has a standard
 interface, i.e. provides a ``.get()`` member function to access the underlying
-raw pointer. If this is not the case, pybind11's ``holder_helper`` must be
+raw pointer, and a ``.release()`` member function for move-only holders. If this is not the case, pybind11's ``holder_helper`` must be
 specialized:
 
 .. code-block:: cpp
@@ -171,3 +258,20 @@ provides ``.get()`` functionality via ``.getPointer()``.
     The file :file:`tests/test_smart_ptr.cpp` contains a complete example
     that demonstrates how to work with custom reference-counting holder types
     in more detail.
+
+.. warning::
+
+    Holder type conversion (see :ref:`smart_ptrs_casting`) and advanced ownership transfer (see :ref:`virtual_inheritance_lifetime`) is **not** supported for custom shared pointer types, due to constraints on dynamic type erasure.
+
+.. _smart_ptrs_casting:
+
+Casting smart pointers
+======================
+
+As shown in the :ref:`conversion_table`, you may cast to any of the available holders (e.g. ``py::cast<std::shared_ptr<Type>>(obj)``) that can properly provide access to the underlying holder.
+
+``pybind11`` will raise an error if there is an incompatible cast. You may of course cast to the exact same holder type. You may also move a ``std::unique_ptr<Type>`` into a ``std::shared_ptr<Type>``, as this is allowed. **However**, you may not convert a ``std::shared_ptr<Type>`` to a ``std::unique_ptr<Type>`` as you cannot release an object that is managed by ``std::shared_ptr<Type>``.
+
+Additionally, conversion to ``std::unique_ptr<Type, Deleter>`` is not supported if ``Deleter`` is not ``std::default_deleter<Type>``.
+
+Conversion to a different custom smart pointer is not supported.

include/pybind11/attr.h

@@ -221,11 +221,14 @@ struct type_record {
     void *(*operator_new)(size_t) = ::operator new;
 
     /// Function pointer to class_<..>::init_instance
-    void (*init_instance)(instance *, const void *) = nullptr;
+    void (*init_instance)(instance *, holder_erased) = nullptr;
 
     /// Function pointer to class_<..>::dealloc
     void (*dealloc)(detail::value_and_holder &) = nullptr;
 
+    /// See `type_info::has_cpp_release`.
+    instance::type_release_info_t release_info;
+
     /// List of base classes of the newly created type
     list bases;