WIP: PyPy support (#527)

This commit includes modifications that are needed to get pybind11 to work with PyPy. The full test suite compiles and runs except for a last few functions that are commented out (due to problems in PyPy that were reported on the PyPy bugtracker).

Two somewhat intrusive changes were needed to make it possible: two new tags ``py::buffer_protocol()`` and ``py::metaclass()`` must now be specified to the ``class_`` constructor if the class uses the buffer protocol and/or requires a metaclass (e.g. for static properties).

Note that this is only for the PyPy version based on Python 2.7 for now. When the PyPy 3.x has caught up in terms of cpyext compliance, a PyPy 3.x patch will follow.

Author: wjakob

.travis.yml

@@ -29,6 +29,13 @@ matrix:
   - os: osx
     osx_image: xcode7.3
     env: PYTHON=3.5 CPP=14 CLANG
+  # Test a PyPy 2.7 nightly build
+  - os: linux
+    env: PYPY=1 PYTHON=2.7 CPP=11 GCC=4.8
+    addons:
+      apt:
+        sources: [ubuntu-toolchain-r-test, kubuntu-backports]
+        packages: [g++-4.8, cmake]
   # A barebones build makes sure everything still works without optional deps (numpy/scipy/eigen)
   # and also tests the automatic discovery functions in CMake (Python version, C++ standard).
   - os: linux
@@ -70,9 +77,18 @@ before_install:
   fi
   if [ -n "$CPP" ]; then export CPP=-std=c++$CPP; fi
   if [ "${PYTHON:0:1}" = "3" ]; then export PY=3; fi
+  if [ -n "$PYPY" ]; then
+    curl http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 | tar -xj
+    export PYPY_BINARY=$(echo `pwd`/pypy-c-jit*/bin/pypy)
+    export CMAKE_EXTRA_ARGS="-DPYTHON_EXECUTABLE:FILEPATH=$PYPY_BINARY"
+  fi
   if [ -n "$DEBUG" ]; then export CMAKE_EXTRA_ARGS="-DCMAKE_BUILD_TYPE=Debug"; fi
 - |
-  # Initialize enviornment
+  # Initialize environment
+  if [ -n "$PYPY" ]; then
+    $PYPY_BINARY -m ensurepip
+    $PYPY_BINARY -m pip install pytest
+  fi
   if [ -n "$DOCKER" ]; then
     docker pull $DOCKER
     export containerid=$(docker run --detach --tty \

README.md

@@ -25,7 +25,7 @@ become an excessively large and unnecessary dependency.
 Think of this library as a tiny self-contained version of Boost.Python with
 everything stripped away that isn't relevant for binding generation. Without
 comments, the core header files only require ~2.5K lines of code and depend on
-Python (2.7 or 3.x) and the C++ standard library. This compact implementation
+Python (2.7 or 3.x, or PyPy2.7 >= 5.5) and the C++ standard library. This compact implementation
 was possible thanks to some of the new C++11 language features (specifically:
 tuples, lambda functions and variadic templates). Since its creation, this
 library has grown beyond Boost.Python in many ways, leading to dramatically
@@ -58,12 +58,15 @@ pybind11 can map the following core C++ features to Python
 ## Goodies
 In addition to the core functionality, pybind11 provides some extra goodies:
 
-- pybind11 uses C++11 move constructors and move assignment operators whenever
-  possible to efficiently transfer custom data types.
+- Python 2.7, 3.x, and PyPy (PyPy2.7 >= 5.5) are supported with an
+  implementation-agnostic interface.
 
 - It is possible to bind C++11 lambda functions with captured variables. The
   lambda capture data is stored inside the resulting Python function object.
 
+- pybind11 uses C++11 move constructors and move assignment operators whenever
+  possible to efficiently transfer custom data types.
+
 - It's easy to expose the internal storage of custom data types through
   Pythons' buffer protocols. This is handy e.g. for fast conversion between
   C++ matrix classes like Eigen and NumPy without expensive copy operations.
@@ -100,7 +103,7 @@ In addition to the core functionality, pybind11 provides some extra goodies:
 
 ## About
 
-This project was created by [Wenzel Jakob](https://www.mitsuba-renderer.org/~wenzel/).
+This project was created by [Wenzel Jakob](http://rgl.epfl.ch/people/wjakob).
 Significant features and/or improvements to the code were contributed by
 Jonas Adler,
 Sylvain Corlay,

docs/advanced/classes.rst

@@ -422,15 +422,24 @@ The section on :ref:`properties` discussed the creation of instance properties
 that are implemented in terms of C++ getters and setters.
 
 Static properties can also be created in a similar way to expose getters and
-setters of static class attributes. It is important to note that the implicit
-``self`` argument also exists in this case and is used to pass the Python
-``type`` subclass instance. This parameter will often not be needed by the C++
-side, and the following example illustrates how to instantiate a lambda getter
-function that ignores it:
+setters of static class attributes. Two things are important to note:
+
+1. Static properties are implemented by instrumenting the *metaclass* of the
+   class in question -- however, this requires the class to have a modifiable
+   metaclass in the first place. pybind11 provides a ``py::metaclass()``
+   annotation that must be specified in the ``class_`` constructor, or any
+   later method calls to ``def_{property_,∅}_{readwrite,readonly}_static`` will
+   fail (see the example below).
+
+2. For static properties defined in terms of setter and getter functions, note
+   that the implicit ``self`` argument also exists in this case and is used to
+   pass the Python ``type`` subclass instance. This parameter will often not be
+   needed by the C++ side, and the following example illustrates how to
+   instantiate a lambda getter function that ignores it:
 
 .. code-block:: cpp
 
-    py::class_<Foo>(m, "Foo")
+    py::class_<Foo>(m, "Foo", py::metaclass())
         .def_property_readonly_static("foo", [](py::object /* self */) { return Foo(); });
 
 Operator overloading

docs/advanced/pycpp/numpy.rst

@@ -33,7 +33,7 @@ completely avoid copy operations with Python expressions like
 
 .. code-block:: cpp
 
-    py::class_<Matrix>(m, "Matrix")
+    py::class_<Matrix>(m, "Matrix", py::buffer_protocol())
        .def_buffer([](Matrix &m) -> py::buffer_info {
             return py::buffer_info(
                 m.data(),                               /* Pointer to buffer */
@@ -46,9 +46,12 @@ completely avoid copy operations with Python expressions like
             );
         });
 
-The snippet above binds a lambda function, which can create ``py::buffer_info``
-description records on demand describing a given matrix. The contents of
-``py::buffer_info`` mirror the Python buffer protocol specification.
+Supporting the buffer protocol in a new type involves specifying the special
+``py::buffer_protocol()`` tag in the ``py::class_`` constructor and calling the
+``def_buffer()`` method with a lambda function that creates a
+``py::buffer_info`` description record on demand describing a given matrix
+instance. The contents of ``py::buffer_info`` mirror the Python buffer protocol
+specification.
 
 .. code-block:: cpp
 
@@ -77,7 +80,7 @@ buffer objects (e.g. a NumPy matrix).
     typedef Matrix::Scalar Scalar;
     constexpr bool rowMajor = Matrix::Flags & Eigen::RowMajorBit;
 
-    py::class_<Matrix>(m, "Matrix")
+    py::class_<Matrix>(m, "Matrix", py::buffer_protocol())
         .def("__init__", [](Matrix &m, py::buffer b) {
             typedef Eigen::Stride<Eigen::Dynamic, Eigen::Dynamic> Strides;
 

docs/intro.rst

@@ -51,6 +51,9 @@ Goodies
 *******
 In addition to the core functionality, pybind11 provides some extra goodies:
 
+- Python 2.7, 3.x, and PyPy (PyPy2.7 >= 5.5) are supported with an
+  implementation-agnostic interface.
+
 - It is possible to bind C++11 lambda functions with captured variables. The
   lambda capture data is stored inside the resulting Python function object.
 

include/pybind11/attr.h

@@ -47,6 +47,12 @@ struct multiple_inheritance { };
 /// Annotation which enables dynamic attributes, i.e. adds `__dict__` to a class
 struct dynamic_attr { };
 
+/// Annotation which enables the buffer protocol for a type
+struct buffer_protocol { };
+
+/// Annotation which requests that a special metaclass is created for a type
+struct metaclass { };
+
 /// Annotation to mark enums as an arithmetic type
 struct arithmetic { };
 
@@ -136,7 +142,9 @@ struct function_record {
 
 /// Special data structure which (temporarily) holds metadata about a bound class
 struct type_record {
-    PYBIND11_NOINLINE type_record() { }
+    PYBIND11_NOINLINE type_record()
+        : multiple_inheritance(false), dynamic_attr(false),
+          buffer_protocol(false), metaclass(false) { }
 
     /// Handle to the parent scope
     handle scope;
@@ -166,10 +174,16 @@ struct type_record {
     const char *doc = nullptr;
 
     /// Multiple inheritance marker
-    bool multiple_inheritance = false;
+    bool multiple_inheritance : 1;
 
     /// Does the class manage a __dict__?
-    bool dynamic_attr = false;
+    bool dynamic_attr : 1;
+
+    /// Does the class implement the buffer protocol?
+    bool buffer_protocol : 1;
+
+    /// Does the class require its own metaclass?
+    bool metaclass : 1;
 
     PYBIND11_NOINLINE void add_base(const std::type_info *base, void *(*caster)(void *)) {
         auto base_info = detail::get_type_info(*base, false);
@@ -309,6 +323,16 @@ struct process_attribute<dynamic_attr> : process_attribute_default<dynamic_attr>
     static void init(const dynamic_attr &, type_record *r) { r->dynamic_attr = true; }
 };
 
+template <>
+struct process_attribute<buffer_protocol> : process_attribute_default<buffer_protocol> {
+    static void init(const buffer_protocol &, type_record *r) { r->buffer_protocol = true; }
+};
+
+template <>
+struct process_attribute<metaclass> : process_attribute_default<metaclass> {
+    static void init(const metaclass &, type_record *r) { r->metaclass = true; }
+};
+
 
 /// Process an 'arithmetic' attribute for enums (does nothing here)
 template <>

include/pybind11/cast.h

@@ -108,14 +108,10 @@ PYBIND11_NOINLINE inline handle get_type_handle(const std::type_info &tp, bool t
 }
 
 PYBIND11_NOINLINE inline bool isinstance_generic(handle obj, const std::type_info &tp) {
-    const auto type = detail::get_type_handle(tp, false);
+    handle type = detail::get_type_handle(tp, false);
     if (!type)
         return false;
-
-    const auto result = PyObject_IsInstance(obj.ptr(), type.ptr());
-    if (result == -1)
-        throw error_already_set();
-    return result != 0;
+    return isinstance(obj, type);
 }
 
 PYBIND11_NOINLINE inline std::string error_string() {
@@ -141,6 +137,7 @@ PYBIND11_NOINLINE inline std::string error_string() {
         PyException_SetTraceback(scope.value, scope.trace);
 #endif
 
+#if !defined(PYPY_VERSION)
     if (scope.trace) {
         PyTracebackObject *trace = (PyTracebackObject *) scope.trace;
 
@@ -160,6 +157,7 @@ PYBIND11_NOINLINE inline std::string error_string() {
         }
         trace = trace->tb_next;
     }
+#endif
 
     return errorString;
 }
@@ -176,7 +174,9 @@ PYBIND11_NOINLINE inline handle get_object_handle(const void *ptr, const detail:
 }
 
 inline PyThreadState *get_thread_state_unchecked() {
-#if   PY_VERSION_HEX < 0x03000000
+#if defined(PYPY_VERSION)
+    return PyThreadState_GET();
+#elif PY_VERSION_HEX < 0x03000000
     return _PyThreadState_Current;
 #elif PY_VERSION_HEX < 0x03050000
     return (PyThreadState*) _Py_atomic_load_relaxed(&_PyThreadState_Current);
@@ -224,7 +224,7 @@ class type_caster_generic {
 
             /* If this is a python class, also check the parents recursively */
             auto const &type_dict = get_internals().registered_types_py;
-            bool new_style_class = PyType_Check(tobj);
+            bool new_style_class = PyType_Check((PyObject *) tobj);
             if (type_dict.find(tobj) == type_dict.end() && new_style_class && tobj->tp_bases) {
                 auto parents = reinterpret_borrow<tuple>(tobj->tp_bases);
                 for (handle parent : parents) {
@@ -662,10 +662,10 @@ template <> class type_caster<std::wstring> {
 #if PY_MAJOR_VERSION >= 3
         buffer = PyUnicode_AsWideCharString(load_src.ptr(), &length);
 #else
-        temp = reinterpret_steal<object>(
-            sizeof(wchar_t) == sizeof(short)
-                ? PyUnicode_AsUTF16String(load_src.ptr())
-                : PyUnicode_AsUTF32String(load_src.ptr()));
+        temp = reinterpret_steal<object>(PyUnicode_AsEncodedString(
+            load_src.ptr(), sizeof(wchar_t) == sizeof(short)
+            ? "utf16" : "utf32", nullptr));
+
         if (temp) {
             int err = PYBIND11_BYTES_AS_STRING_AND_SIZE(temp.ptr(), (char **) &buffer, &length);
             if (err == -1) { buffer = nullptr; }  // TypeError
@@ -868,7 +868,7 @@ template <typename type, typename holder_type> class type_caster_holder : public
 
             /* If this is a python class, also check the parents recursively */
             auto const &type_dict = get_internals().registered_types_py;
-            bool new_style_class = PyType_Check(tobj);
+            bool new_style_class = PyType_Check((PyObject *) tobj);
             if (type_dict.find(tobj) == type_dict.end() && new_style_class && tobj->tp_bases) {
                 auto parents = reinterpret_borrow<tuple>(tobj->tp_bases);
                 for (handle parent : parents) {

include/pybind11/eval.h

@@ -95,8 +95,15 @@ object eval_file(str fname, object global = object(), object local = object()) {
         pybind11_fail("File \"" + fname_str + "\" could not be opened!");
     }
 
+#if PY_VERSION_HEX < 0x03000000 && defined(PYPY_VERSION)
+    PyObject *result = PyRun_File(f, fname_str.c_str(), start, global.ptr(),
+                                  local.ptr());
+    (void) closeFile;
+#else
     PyObject *result = PyRun_FileEx(f, fname_str.c_str(), start, global.ptr(),
                                     local.ptr(), closeFile);
+#endif
+
     if (!result)
         throw error_already_set();
     return reinterpret_steal<object>(result);

include/pybind11/functional.h

@@ -39,7 +39,7 @@ struct type_caster<std::function<Return(Args...) PYBIND11_NOEXCEPT_SPECIFIER>> {
            captured variables), in which case the roundtrip can be avoided.
          */
         if (PyCFunction_Check(src_.ptr())) {
-            auto c = reinterpret_borrow<capsule>(PyCFunction_GetSelf(src_.ptr()));
+            auto c = reinterpret_borrow<capsule>(PyCFunction_GET_SELF(src_.ptr()));
             auto rec = (function_record *) c;
 
             if (rec && rec->is_stateless && rec->data[1] == &typeid(function_type)) {