Support numpy ufuncs for ExtensionArrays

[jorisvandenbossche]

Currently calling numpy ufuncs such as np.exp on a Series[EA] or EA does not work yet:

In [44]: s = pd.Series([1, 2, 3, 4], dtype='Int64')

In [45]: np.exp(s)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-45-fb7258693ae9> in <module>()
----> 1 np.exp(s)

~/scipy/pandas/pandas/core/series.py in __array_prepare__(self, result, context)
    671                                 obj=type(obj).__name__,
    672                                 dtype=getattr(obj, 'dtype', None),
--> 673                                 op=context[0].__name__))
    674         return result
    675 

TypeError: Series with dtype Int64 cannot perform the numpy op exp

In [46]: np.exp(s.values)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-46-69f0b0471ea8> in <module>()
----> 1 np.exp(s.values)

AttributeError: 'int' object has no attribute 'exp'

In [47]: np.exp(s.astype(int))   # but works for numpy dtyped series
Out[47]: 
0     2.718282
1     7.389056
2    20.085537
3    54.598150
dtype: float64

I think it would be nice to have this working, and without looking in detail into it, I would assume the best way to go is to actually support the __array_ufunc__ protocol on ExtensionArrays itself and to ensure Series then properly uses that?

[TomAugspurger]

I'm bumping into this for PeriodArray. Previously np.add(Series[period], 2) worked since the values were an ndarray of period objects. Now that the values are a real PeriodArray, we raise an exception before trying the op.

I'll do this as a separate PR from PeriodArray.

[TomAugspurger]

This seems difficult for older numpys without __array_ufunc__. There are a few issues:

1. Series.__array__ is called before we have a chance to do anything. This must return an ndarray, which means we have to convert the Series[EA] to an ndarray, which may be wasteful / expensive.
2. __array_prepare__ has to return an ndarray. Ideally we would kind of "break out of" the ufunc on the series, and have the ufunc called on the EA instead, and wrap up that result.

At the moment, I think working around these would be too difficult. Right now, I'll see if we can support numpy with __array_ufunc__ and raise with a nice error message otherwise.

[jorisvandenbossche]

BTW, I also have some branch adding the basics of this to ExtensionArrary / IntegerArray. And also stumbled on the current implementation with __array__ / __array_prepare__.

So I would maybe also start with only supporting it for newer numpy.

Question: is np.add(Series[period], 2) used somewhere in our code, or in a test?

[TomAugspurger]

I think just a test asserting that it worked.

…

________________________________ From: Joris Van den Bossche <[email protected]> Sent: Monday, October 1, 2018 5:11:48 PM To: pandas-dev/pandas Cc: Tom Augspurger; Comment Subject: Re: [pandas-dev/pandas] Support numpy ufuncs for ExtensionArrays (#22798) BTW, I also have some branch adding the basics of this to ExtensionArrary / IntegerArray. And also stumbled on the current implementation with __array__ / __array_prepare__. So I would maybe also start with only supporting it for newer numpy. Question: is np.add(Series[period], 2) used somewhere in our code, or in a test? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#22798 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIqQxJjvcj5TqjdAcjgSfm_yFaoaAks5ugpMkgaJpZM4W0Ct2>.

[TomAugspurger]

This was fixed at the pandas level in #23293. 3rd party arrays just need to implement __array_ufunc__ and pandas will pass the array through correctly.