Skip to content

Fix: Pandas rolling removes imaginary part of complex #47028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 21 commits into from
Closed

Fix: Pandas rolling removes imaginary part of complex #47028

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
d1fd5c3
rolling type
May 15, 2022
2736431
add empty cases
May 15, 2022
cdc4c6e
add test
May 15, 2022
436bc99
resolve conflict
May 15, 2022
186096e
Merge branch 'main' into bug_rolling_complex
May 15, 2022
09e9635
:Merge branch 'main' of https://github.com/weikhor/pandas into bug_ro…
May 16, 2022
ac9fd04
use aggregation
May 16, 2022
7389a95
Merge branch 'main' of https://github.com/weikhor/pandas into bug_rol…
May 21, 2022
098dc16
test rolling
May 22, 2022
25c6700
Merge branch 'main' into bug_rolling_complex
May 22, 2022
dc9cd7e
count
May 22, 2022
1aaa3a7
Merge branch 'bug_rolling_complex' of https://github.com/weikhor/pand…
May 22, 2022
beaa1ae
remove type dependency
May 22, 2022
1dcfa05
add buffer source array is read-only
May 22, 2022
6e4259b
revert
May 23, 2022
115feaf
update complex
May 23, 2022
7bdde24
update complex
May 23, 2022
c2dbf37
Merge branch 'main' into bug_rolling_complex
May 23, 2022
796d2bd
Merge branch 'main' of https://github.com/weikhor/pandas into bug_rol…
Jun 6, 2022
9f275de
add test
Jun 6, 2022
2cae983
Merge branch 'main' into bug_rolling_complex
Jun 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion pandas/core/window/rolling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
from pandas.core.dtypes.common import (
ensure_float64,
is_bool,
is_complex_dtype,
is_integer,
is_list_like,
is_scalar,
Expand Down Expand Up @@ -365,7 +366,10 @@ def _prep_values(self, values: ArrayLike) -> np.ndarray:
if isinstance(values, ExtensionArray):
values = values.to_numpy(np.float64, na_value=np.nan)
else:
values = ensure_float64(values)
if is_complex_dtype(values):
values = values.astype(np.complex64)
else:
values = ensure_float64(values)
except (ValueError, TypeError) as err:
raise TypeError(f"cannot handle this type -> {values.dtype}") from err

Expand Down
15 changes: 15 additions & 0 deletions pandas/tests/window/test_rolling.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1871,3 +1871,18 @@ def test_rolling_skew_kurt_floating_artifacts():
assert (result[-2:] == 0).all()
result = r.kurt()
assert (result[-2:] == -3).all()


def test_rolling_imaginary_part_of_complex(arithmetic_win_operators):
# GH 46619
func_name = arithmetic_win_operators
df = DataFrame([1j, 1 + 2j])
result = getattr(
df.rolling(2).apply(lambda x: print(x) is None),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So func_name should now be tested instead of apply

Copy link
Contributor Author

@weikhor weikhor May 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mroeschke, I not able to understand fully. If I not use apply.

func_name = arithmetic_win_operators
df = DataFrame([1j, 1 + 2j])
result = getattr(
    df.rolling(2),
    func_name,
)()

Will get AssertionError depend on function from arithmetic_win_operators

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, the cython code may need modifying to support complex dtypes in this case if an AssertionError is raised (guessing since no traceback was provided)

Copy link
Contributor Author

@weikhor weikhor May 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke

The code I run

func_name = arithmetic_win_operators
df = DataFrame([1j, 1 + 2j])
result = getattr(
    df.rolling(2),
    func_name,
)()

The 'arithmetic_win_operators' can be sum, mean, median, max, min, var, std, kurt, skew, count, sem (11 possible arithmetic function.

For example, if arithmetic_win_operators='sum', the error is raised that function sum does not support complex dtypes from cython function inside 'pandas/_libs/window/aggregations.pyx'

Traceback (most recent call last):
  File "/home/open_source/pandas_2/development/development_5.py", line 26, in <module>
    result = getattr(
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 1974, in sum
    return super().sum(*args, engine=engine, engine_kwargs=engine_kwargs, **kwargs)
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 1407, in sum
    return self._apply(window_func, name="sum", **kwargs)
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 619, in _apply
    return self._apply_blockwise(homogeneous_func, name)
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 487, in _apply_blockwise
    res = hfunc(arr)
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 477, in hfunc
    return homogeneous_func(values)
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 614, in homogeneous_func
    result = calc(values)
  File "/home/open_source/pandas_2/pandas/pandas/core/window/rolling.py", line 611, in calc
    return func(x, start, end, min_periods, *numba_args)
  File "pandas/_libs/window/aggregations.pyx", line 132, in pandas._libs.window.aggregations.roll_sum
ValueError: Buffer dtype mismatch, expected 'const float64_t' but got 'complex float'

Some question. Thus, to make this work. I should write new roll sum function that support complex dtypes in cython. Is this right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can create fused types (e.g. float + complex) that the current cython functions accept instead which should get this working for the existing functions. You can see the groupby cython code for an example.

Copy link
Contributor Author

@weikhor weikhor May 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke Hi
I not sure whether I do it correctly. I try to update roll_mean to support complex type in this file pandas/_libs/window/aggregations.pyx

ctypedef fused float_complex_t:
    float64_t
    complex64_t

cdef inline void add_mean(float_complex_t val, Py_ssize_t *nobs, float_complex_t *sum_x,
                          Py_ssize_t *neg_ct, float_complex_t *compensation,
                          int64_t *num_consecutive_same_value, float_complex_t *prev_value) nogil:
    """ add a value from the mean calc using Kahan summation """
    cdef:
        float_complex_t y, t

    # Not NaN
    if val == val:
        nobs[0] = nobs[0] + 1
        y = val - compensation[0]
        t = sum_x[0] + y
        compensation[0] = t - sum_x[0] - y
        sum_x[0] = t
        if signbit(val):
            neg_ct[0] = neg_ct[0] + 1

        # GH#42064, record num of same values to remove floating point artifacts
        if val == prev_value[0]:
            num_consecutive_same_value[0] += 1
        else:
            # reset to 1 (include current value itself)
            num_consecutive_same_value[0] = 1
        prev_value[0] = val

The error is raised

Error compiling Cython file:
------------------------------------------------------------
...
        nobs[0] = nobs[0] + 1
        y = val - compensation[0]
        t = sum_x[0] + y
        compensation[0] = t - sum_x[0] - y
        sum_x[0] = t
        if signbit(val):
                  ^
------------------------------------------------------------

pandas/_libs/window/aggregations.pyx:231:19: Cannot assign type 'float complex' to 'long double'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will probably need cast this to float before passing in this C func, or just use a comparison operator to replace sigbit.

Copy link
Contributor Author

@weikhor weikhor May 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modify code in cython pandas/_libs/window/aggregations.pyx file to support complex type. Do casting complex values to float in arithmetic operation

func_name,
)()
expected = getattr(
DataFrame([np.nan, 1.0]),
func_name,
)()
tm.assert_series_equal(result, expected)