feat: py::type::of<T>() and py::type::of(h)

[henryiii]

This adds py::type::of<T>(), which provides a way to get the Python type associated with a C++ type. Currently there are two possible ways to do this (pointed out by @YannickJadoul and @bstaletic on Gitter about a year ago, IIRC), but they require usage of internal structures or the detail namespace. This provides a natural, pythonic way to get the type object without reverting to internal details.

This is very useful in templated code; for example, in boost-histogram, it is needed to get the templated storage from a templated histogram.

EDIT (@YannickJadoul):
Closes #1692

[YannickJadoul]

Damn, @henryiii, you beat me to it, again ;-)

+1 for adding this!

When considering this PR over the last year, I realized this only works for class_-registered types. So, given that we'll undoubtedly get lots of questions/issues about "why doesn't py::type<int>() not return the int type object?":

1. Should we extend this? Yes, your documentation says it's for exposed types, so it's not strictly necessary. But I do wonder how many "bug" reports we're going to get. Plus, it would be a nice way of getting hold of the int type object?
2. Is there an easy way to do so? One possibility I can see would be to add this to type_caster as static py::object type() (or static py::handle type()) and add the current implementation to type_caster_base? Not sure if this is a great idea, but it would a) be extensible, and b) be backwards compatible (since you'll get a compile error if the type_caster doesn't have a static type function).

[EricCousineau-TRI]

Thanks!!! I've had some code that, uh, dipped into the internals for a while now 😅
https://github.com/RobotLocomotion/drake/blob/26bf5dca63fe0c63df940ef30c2e3b5196c4339a/bindings/pydrake/common/cpp_param_pybind.cc#L101-L102

@YannickJadoul Unfortunately, certain users may have certain desires for py::type(). I, for one, would like to have the option to direct it towards the closest NumPy / ctype, and I register my own aliases to ensure that I can distinguish types like uint8_t, int64_t, etc., whereas using the type_caster approach may scrub out those details.

EDIT: Thinking on it more, given that I already intercept typeids first to handle my aliases, I guess I wouldn't be hurt by it...

I am totes fine with this only dealing with registered types. Perhaps, then renaming it to something slightly more obtuse, like py::generic_type, or something like that to align with the hierarchy of casters using type_caster_generic?
Perhaps terminology from the docs could help steer this?
https://pybind11.readthedocs.io/en/stable/advanced/cast/index.html#type-conversions

[YannickJadoul]

I am totes fine with this only dealing with registered types. Perhaps, then renaming it to something slightly more obtuse, like py::generic_type, or something like that to align with the hierarchy of casters using type_caster_generic?

I like the py::type name, though, so I'd prefer to keep it. But how hard would it be to make using another type for this fail compiling? Could we have some enable_if or static_assert checking std::is_base_of<py::detail::generic_type_caster<the_same_kind_of_decay_as_in_make_caster<...>>, py::detail::make_caster<...>>::value?

[EricCousineau-TRI]

Yeah, I think a fail fast is achievable. Perhaps with something like this in the function body?

static_assert(
  is_generic_type_v<T>,
  "This only works for registered C++ types. The type here is most likely type converted (using type_caster).")

[YannickJadoul]

is_generic_type_v<T>

Except that we don't have that, I believe? So it'll be something like checking that a subclass of the default type_caster_base or type_caster_generic is used? (Actually, I think it's type_caster_base that does this mapping between T and the Python type, no?)

[henryiii]

If we have it fail for now with non-user registered types, it could always be added later - changing a failure to a success is always allowed. :)

[YannickJadoul]

If we have it fail for now with non-user registered types, it could always be added later - changing a failure to a success is always allowed. :)

Exactly! As long as we do it safely (at compile time), this would be perfect!

[wjakob]

This looks great, but would you mind adding an explicit discussion in the documentation (the auto-extracted docstrings IIRC do end up in the documentation as well, but rather hidden..)

[henryiii]

Waiting pending discussion on #2388

[henryiii]

By the way, I'm open to ideas on how to get something like py::type::of<int>() to work. :) Also, should we use py::type::of(h) for the from handle form? And should this be a handle subclass or object?

[EricCousineau-TRI]

Filed #2486 ;)

[YannickJadoul]

The simplest case would be std::variant - there will be run-time branching of the type, and what they (users) may want may heavily depend on their use case.

Right, but then for std::variant, you can never do that? py::type::of<T> doesn't get an instance, anyway?

Should this return something like Typing.Variant[...], where ... is filled in with the inner type-of expressions, only for Python 3.6+?

That's an interesting suggestion, though, but I'm not sure. It's not really a type, right? Can you create an object using this?

And that's a relatively easy case. What about when users may actually want to distinguish between int32 and uint32?

But these do get mapped to the same Python type? I don't see the problem, here? There's a mismatch between C++ and Python, but you can still say "If you pass a C++ type T, you'll get a Python type py::type::of<T>. At least it's better than nothing.
And it provides a way to get the int type object, through py::type::of<int>()? (Maybe there's also a better API, indeed.)

I mean, having py::type::of<int>() could help, I just dunno if that should be lumped into this PR without more discussion, as I think it's more nuanced.

Agreed! Let's get the easy stuff in, just making sure it doesn't stop us from achieving the more complicated stuff :-)

[EricCousineau-TRI]

Looks awesome!

[henryiii]

Right, but then for std::variant, you can never do that? py::type::of doesn't get an instance, anyway?

Without verifying it could be done, I think you could have py::type::of<variant<...>>(instance) working in that case, but only if all the inner types were supported (note that there is an argument above). But I don't think supporting <int> means you have to support <variant<...>>?

Let's get the easy stuff in

Unless it was extremely easy, was not planning to add to this PR :)

[YannickJadoul]

This isn't necessary, I think? object should be a subclass of handle, and there's no extra refcounting going on?

[YannickJadoul]

Oh, yes, some slicing, since it's not by reference. But there are precedents for that, I think?

[YannickJadoul]

If you/we choose to keep the object overload, better to make it a const object &, then. It avoids an unnecessary refcount.

[YannickJadoul]

Ah, no, PYBIND11_OBJECT already adds that overload.

pybind11/include/pybind11/pytypes.h

Lines 813 to 817 in cc982ac

	#define PYBIND11_OBJECT(Name, Parent, CheckFun) \
	PYBIND11_OBJECT_COMMON(Name, Parent, CheckFun) \
	/* This is deliberately not 'explicit' to allow implicit conversion from object: */ \
	Name(const object &o) : Parent(o) { } \
	Name(object &&o) : Parent(std::move(o)) { }

How do we fix this such that it works in the intended way? When do we want py::type(...) to "cast" a py::object to a subclass, and when do we want it to actually call type(...)? Does this mean that @henryiii's original py::type::of is necessary, and that we can't have the py::type(obj) interface I've been dreaming of? :-(

[henryiii]

We use PYBIND11_OBJECT_COMMON instead of PYBIND11_OBJECT.

[henryiii]

(I'll try dropping this overload soon)

[henryiii]

Can this be a const handle &?

[YannickJadoul]

We use PYBIND11_OBJECT_COMMON instead of PYBIND11_OBJECT.

🤦 How did I miss that?

Can this be a const handle &?

Probably, but I don't think it's necessary. handle really is a fancy and more beautiful way of writing PyObject *.

[YannickJadoul]

Sorry to open that discussion again (after first telling @henryiii to change it), but, I'm now realizing the following:

If py::type(obj) always gets the type of an object; we have no way of actually casting a py::object (being returned from a Python call or so) to be converted? Is that OK, or did I speak too soon, when saying py::type(handle) would be a great API?

[henryiii]

I think I preferred py::type::of(h), even though it's a little further from the Python API, both from symmetry with py::type::of<T>() and because it keeps you from casting a known type into a type. (I had tests for this, the behavior of py::type::of was not the same as py::type if you gave it a valid class).

[henryiii]

It looks like PYBIND11_OBJECT doesn't throw if the constructor doesn't pass the check function. So only manually checking the ->check is the only way to ensure the object really is a type if you do py::type(object)? I take it there's a reason constructers should not throw in this case? I can manually add it instead of using PYBIND11_OBJECT if there's not a reason.

[YannickJadoul]

It looks like PYBIND11_OBJECT doesn't throw if the constructor doesn't pass the check function. So only manually checking the ->check is the only way to ensure the object really is a type if you do py::type(object)? I take it there's a reason constructers should not throw in this case? I can manually add it instead of using PYBIND11_OBJECT if there's not a reason.

I worked on that here in #2349, but it got stalled behind the __qualname__ unification with Python 2 & PyPy.

[henryiii]

Good, then leave as is, then we can take advantage of that if it gets merged?

[YannickJadoul]

I think I preferred py::type::of(h), even though it's a little further from the Python API, both from symmetry with py::type::of<T>() and because it keeps you from casting a known type into a type. (I had tests for this, the behavior of py::type::of was not the same as py::type if you gave it a valid class).

I really liked py::type(...), but I'm now realizing that will be ambiguous and confusing :-( So yes, sorry for having you make that change to then revert it! py::type::of will be better.

[YannickJadoul]

Good, then leave as is, then we can take advantage of that if it gets merged?

Good for me. I'll try to pick that up again, soon'ish...

[wjakob]

Thanks everyone!