Skip to content

More robust pickling support #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bluescarni opened this issue Apr 29, 2016 · 5 comments
Closed

More robust pickling support #186

bluescarni opened this issue Apr 29, 2016 · 5 comments

Comments

@bluescarni
Copy link
Contributor

The current documentation about pickling support states:

Note that only the cPickle module is supported on Python 2.7. It is also important to request usage of the highest protocol version using the -1 argument to dumps. Failure to follow these two steps will lead to important pybind11 memory allocation routines to be skipped during unpickling, which will likely cause memory corruption and/or segmentation faults.

The cPickle restriction for Python 2.x does not look too problematic, but I believe the requirement about the explicit protocol version is an issue. Forcing the -1 argument in internal use is not problematic if one writes a couple of thin wrappers around loads/dumps for the purpose, but it is fragile with respect to those situations in which one cannot control how loads/dumps are called (e.g., the multiprocessing module, ipyparallel, etc.).

Would it be possible to remove this restriction?
@wjakob
Copy link
Member

wjakob commented Apr 29, 2016

I don't think you absolutely need the latest version. It's just that the default (oldest) one doesn't invoke all the callbacks in a way that is needed for pybind11, which causes a crash.

@bluescarni
Copy link
Contributor Author

@wjakob Thanks for the quick reply!

So do you think it would be ok in general (let's say Python 2.7 and 3.x) to stick with the default protocol version?

@wjakob
Copy link
Member

wjakob commented Apr 29, 2016

Try it yourself -- I think it crashes with the default.

@wjakob wjakob closed this as completed in 81e0975 Apr 30, 2016
@wjakob
Copy link
Member

wjakob commented Apr 30, 2016

So the minimum version turns out to be 2, which was introduced for new-style classes. All of this works fine with multiprocessing by the way, it seems to use >= 2.

@bluescarni
Copy link
Contributor Author

@wjakob Thanks a lot for looking into this!

@supervacuus supervacuus mentioned this issue Jan 4, 2018
Closed
@rwgk rwgk mentioned this issue Feb 9, 2023
Closed
rwgk added a commit to rwgk/pybind11 that referenced this issue Feb 15, 2023
@rwgk
Resolve "pointless comparison of unsigned integer with zero" warnings…
6113211 
… emitted by CUDA and ICC:

CUDA 11.7 Ubuntu 22.04:
```
cast.h(1318): error pybind#186-D: pointless comparison of unsigned integer with zero
```

ICC latest x64 (Intel 2021.8.0.20221119):
```
cast.h(1318): error pybind#186: pointless comparison of unsigned integer with zero
```
rwgk added a commit to rwgk/pybind11 that referenced this issue Feb 17, 2023
@rwgk
Resolve "pointless comparison of unsigned integer with zero" warnings…
60fe930 
… emitted by CUDA and ICC:

CUDA 11.7 Ubuntu 22.04:
```
cast.h(1318): error pybind#186-D: pointless comparison of unsigned integer with zero
```

ICC latest x64 (Intel 2021.8.0.20221119):
```
cast.h(1318): error pybind#186: pointless comparison of unsigned integer with zero
```
rwgk added a commit to rwgk/pybind11 that referenced this issue Mar 16, 2023
@rwgk
Resolve "pointless comparison of unsigned integer with zero" warnings…
9a63d77 
… emitted by CUDA and ICC:

CUDA 11.7 Ubuntu 22.04:
```
cast.h(1318): error pybind#186-D: pointless comparison of unsigned integer with zero
```

ICC latest x64 (Intel 2021.8.0.20221119):
```
cast.h(1318): error pybind#186: pointless comparison of unsigned integer with zero
```
rwgk added a commit to rwgk/pybind11 that referenced this issue Apr 18, 2023
@rwgk @wangxf123456
Introduce return_value_policy_pack (will need follow-on adjustments…
c5bb69e 
…) (#30011)

* [smart_holder] Implement `try_as_void_ptr_capsule` as a free function  (pybind#4539)

* Move try_as_void_ptr_capsule out from modified_type_caster_generic_load_impl.

* Try fixing clangtidy

* Introduce return_value_policy_pack

Currently only for string_caster, tuple_caster, map_caster

* Add return_value_policy_pack::override_policy() helpers.

* PYBIND11_TYPE_CASTER_RVPP return_value_policy_pack

* Systematically use return_value_policy_pack in stl.h

* clang-tidy auto fix

* Fix MSVC warning C4458: declaration of 'policy' hides class member

* Fix oversight (when renaming return_value_policy_opts to return_value_policy_pack).

* Cover all changed casters in stl.h

* WIP callbacks

* Explicitly specify return_value_policy::automatic_reference in functional.h

* WIP proof of concept manipulating the return value policy for callbacks.

* Introduce from_python_policies as generalization of load bool convert

* Change `std::vector<bool> args_convert;` to `std::vector<from_python_policies> args_policies;`

* clang-tidy auto fix

* WIP: tests pass

* Replace `argument_record.convert`,`none` with `from_python_policies`

Core diff:

```
diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h
-    bool Zonvert : 1;  ///< True if the argument is allowed to convert when loading
-    bool Mone : 1;     ///< True if None is allowed when loading
+    from_python_policies policies;
```

* functional.h using fpp.rvpp (instead of fpp.convert) but arg is non-constexpr

-        value = func_wrapper(func_handle(std::move(func)), fpp.convert);
+        value = func_wrapper(func_handle(std::move(func)), fpp.rvpp);

Passes:

scons -j 24 selected_test_cpp=test_exceptions.cpp,test_return_value_policy_pack.cpp && /usr/bin/python3 $HOME/clone/pybind11_scons/run_tests.py ../pybind11 test_return_value_policy_pack -s -vv

But many other tests broken because arg is non-constexpr and therefore this had to be commented out in cast.h:

constexpr arg operator"" _a(const char *name, size_t) { return arg(name); }

* Split out `detail::arg_literal` to make ALL tests work again.

* clang-format auto fixes

* Quick workaround for MSVC private/friend issue.

* Add test_nested_callbacks_rtn_string

* WIP: Add collect_arguments_rvpp() functions (unused).

* Use collect_arguments_rvpp() from object_api<Derived>::operator()

* clang-format auto fixes

* Make test_return_value_policy_pack.cpp compatible with C++11 (skip optional, variant). Also fix oversight: missing Python-side test for variant.

* Add test_return_value_policy_pack to tests/CMakeLists.txt and clang-tidy auto fix.

* Resolve `-Wmissing-braces` emitted by old clang versions (3.6, 3.7, 3.9, 5):

```
test_return_value_policy_pack.cpp:51:70: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
```

* Resolve "pointless comparison of unsigned integer with zero" warnings emitted by CUDA and ICC:

CUDA 11.7 Ubuntu 22.04:
```
cast.h(1318): error pybind#186-D: pointless comparison of unsigned integer with zero
```

ICC latest x64 (Intel 2021.8.0.20221119):
```
cast.h(1318): error pybind#186: pointless comparison of unsigned integer with zero
```

* object_api<Derived>::call_with_policies()

* Use object_api<Derived>::call_with_policies() in functional.h

* test_call_callback_pass_pair_string

* Systematically exercise nested callbacks to level 4.

* make_tuple -> make_tuple_rvpp

* simple_collector -> simple_collector_rvpp

* unpacking_collector -> unpacking_collector_rvpp

* PYBIND11_TYPE_CASTER_IMPL

* Universally pass return_value_policy_pack via const &

* Rename arg_literal to arg_base

* Fix git rebase accident.

* Suppress MINGW `-Wmismatched-new-delete` (seen in mingw32 & mingw64 ci.yml logs).

```
2023-02-17T09:47:45.7114041Z D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:381:30: error: 'void operator delete(void*)' called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete]
2023-02-17T09:47:45.7114692Z   381 |             ::operator delete(p);
2023-02-17T09:47:45.7115055Z       |             ~~~~~~~~~~~~~~~~~^~~
2023-02-17T09:47:45.7115444Z In lambda function,
2023-02-17T09:47:45.7116873Z     inlined from 'pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>' at D:/a/pybind11/pybind11/include/pybind11/detail/init.h:376:33,
2023-02-17T09:47:45.7119232Z     inlined from 'Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, pybind11::detail::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&; long long unsigned int ...Is = {0, 1, 2}; Guard = pybind11::detail::void_type; Args = {pybind11::detail::value_and_holder&, int, int}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1563:37,
2023-02-17T09:47:45.7122014Z     inlined from 'pybind11::detail::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&; Args = {pybind11::detail::value_and_holder&, int, int}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1537:65,
2023-02-17T09:47:45.7125679Z     inlined from 'pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>, void, pybind11::detail::value_and_holder&, int, int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&&, void (*)(pybind11::detail::value_and_holder&, int, int), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:249:71,
2023-02-17T09:47:45.7130204Z     inlined from 'static pybind11::handle pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>, void, pybind11::detail::value_and_holder&, int, int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&&, void (*)(pybind11::detail::value_and_holder&, int, int), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call&)' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:224:21:
2023-02-17T09:47:45.7132694Z D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:399:71: note: returned from 'static void* test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc::operator new(size_t)'
2023-02-17T09:47:45.7133385Z   399 |     pyNoisyAlloc.def(py::init([](int i, int) { return new NoisyAlloc(i); }));
2023-02-17T09:47:45.7133874Z       |                                                                       ^
2023-02-17T09:47:45.7134610Z In static member function 'static void test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc::operator delete(void*, size_t)',
2023-02-17T09:47:45.7135502Z     inlined from 'test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>' at D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:408:74,
2023-02-17T09:47:45.7139172Z     inlined from 'pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>' at D:/a/pybind11/pybind11/include/pybind11/detail/init.h:376:33,
2023-02-17T09:47:45.7141646Z     inlined from 'Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, pybind11::detail::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&; long long unsigned int ...Is = {0, 1, 2}; Guard = pybind11::detail::void_type; Args = {pybind11::detail::value_and_holder&, int, double}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1563:37,
2023-02-17T09:47:45.7144252Z     inlined from 'pybind11::detail::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&; Args = {pybind11::detail::value_and_holder&, int, double}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1537:65,
2023-02-17T09:47:45.7147880Z     inlined from 'pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>, void, pybind11::detail::value_and_holder&, int, double, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&&, void (*)(pybind11::detail::value_and_holder&, int, double), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:249:71,
2023-02-17T09:47:45.7358442Z     inlined from 'static pybind11::handle pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>, void, pybind11::detail::value_and_holder&, int, double, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&&, void (*)(pybind11::detail::value_and_holder&, int, double), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call&)' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:224:21:
```

* Fix copy-paste mishap (thanks @lalaland for catching this).

* ruff auto-fixes

* Undo temporary change to test_gil_scoped.py

---------

Co-authored-by: Xiaofei Wang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bluescarni @wjakob