Allow rolling API to accept BaseIndexer subclass #2

mroeschke · 2019-07-02T05:54:58Z

This PR just allows the current rolling API to accept BaseIndexer subclasses in all the validation checks.

Some discussion points about BaseIndexer

I was thinking that get_window_bounds should accept additional keyword arguments from the rolling API like win_type, closed, center, and win_type just in case the author wants to utilize those arguments in the bounds calculation.
Since we are instantiating the BaseIndexer before passing it into rolling, I assumed BaseIndexer would be accepting the index, offset, and keys in the init.

Any feedback welcome!

…tom_window_span

jreback · 2019-07-02T12:43:49Z

closing and opening to trigger travis

jreback · 2019-07-02T12:46:48Z

ok looks like travis is connected

jreback · 2019-07-02T12:53:45Z

I also added some branches to prevent force pushing / deletion but no other restrictions: https://github.com/twosigma/pandas/settings/branches

can edit / change if needed.

mroeschke · 2019-07-02T15:08:20Z

Thanks for setting up Travis, Jeff.

jreback

lgtm so far. As mentioned, doing a couple of pre-cursor PRs to master makes sense to add typing & move tests to the appropriate place make sense.

jreback · 2019-07-05T14:30:29Z

pandas/_libs/_window.py

+
+class BaseIndexer(abc.ABC):
+
+    def __init__(self, index, offset, keys):


whatever you can type would be great

jreback · 2019-07-05T14:30:59Z

pandas/_libs/_window.py

+
+        Parameters
+        ----------
+        kwargs


can we explicity set the kwargs?

I think we want to allow indexer authors to pass arbitrary data here.

Scratch that, these should be explicit.

jreback · 2019-07-05T14:32:47Z

pandas/core/window.py

@@ -11,6 +11,7 @@
 import numpy as np

 import pandas._libs.window as libwindow
+import pandas._libs._window as libwindow_refactor


so I would either do this directly in .window or name this something non-private (as its already private). Note that if say refactoring needs to be done to make things cleaner we can do pre-cursors to master.

jreback · 2019-07-05T14:33:18Z

pandas/core/window.py

@@ -451,14 +452,19 @@ class Window(_Window):

    Parameters
    ----------
-    window : int, or offset
+    window : int, offset, or BaseIndexer subclass


ideally type as much as possible (again I know that things are not typed now), maybe makes sense to do a pre-cursor to type what we have now

jreback · 2019-07-05T14:34:05Z

pandas/tests/test_window.py

@@ -54,6 +56,38 @@ def arithmetic_win_operators(request):
    return request.param


+@pytest.fixture(params=['right', 'left', 'both', 'neither'])


some of this seems a good candidate for a pre-cursor PR to master (the fixtures)

jreback · 2019-07-05T14:35:06Z

pandas/tests/test_window.py

@@ -4154,3 +4188,20 @@ def test_rolling_cov_offset(self):

        expected2 = ss.rolling(3, min_periods=1).cov()
        tm.assert_series_equal(result, expected2)
+
+


not averse to actually putting the tests in a separate file (may make review easier).

so I think a pre-cursor PR to move pandas/tests/test_window.py -> pandas/tests/window/test_window.py makes sense (then we can add on things in an easy namespace)

…tom_window_span

cpcloud · 2019-07-10T20:51:08Z

pandas/core/window.py

@@ -2775,6 +2788,8 @@ def _get_center_of_mass(comass, span, halflife, alpha):


 def _offset(window, center):
+    # TODO: (MATT) If the window is a BaseIndexer subclass,
+    #  we need to pass in the materialized window


What is the type of window here? It looks like it can be a sequence or an integer?

Correct. After a light audit, I anticipate the materialized window may be passed here and other times and Indexer class might be passed here.

Overall I think this routine is for label formatting.

cpcloud · 2019-07-10T20:52:19Z

pandas/tests/test_window.py

@@ -4749,3 +4782,20 @@ def test_rolling_cov_offset(self):

        expected2 = ss.rolling(3, min_periods=1).cov()
        tm.assert_series_equal(result, expected2)
+
+
+class TestCustomIndexer:


This class isn't necessary. Since we have pytest there's really no reason not to use top-level-function-style tests.

cpcloud · 2019-07-10T21:03:36Z

pandas/_libs/_window.py

+            window
+
+        """
+        return NotImplemented


This doesn't need to return anything.

…tom_window_span

Matt Roeschke added 4 commits June 25, 2019 15:28

Allow rolling API to accept BaseIndexer subclasses

60f41db

Rename test

f269944

Merge branch 'feature/generalized_window_operations' into feature/cus…

6b45a0d

…tom_window_span

Fix some tests

f2e815b

jreback closed this Jul 2, 2019

jreback reopened this Jul 2, 2019

jreback requested changes Jul 5, 2019

View reviewed changes

Merge branch 'feature/generalized_window_operations' into feature/cus…

266dc4e

…tom_window_span

mroeschke mentioned this pull request Jul 8, 2019

Create test/window directory and conftest file #3

Closed

cpcloud reviewed Jul 10, 2019

View reviewed changes

pandas/_libs/_window.py Outdated

window

"""

return NotImplemented

Copy link

cpcloud Jul 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't need to return anything.

Matt Roeschke added 4 commits July 10, 2019 22:00

Merge branch 'feature/generalized_window_operations' into feature/cus…

6fab256

…tom_window_span

Address comments

010aeb6

Remove duplicate test

97b1386

Fix import

691f72b

mroeschke changed the title ~~WIP: Allow rolling API to accept BaseIndexer subclass~~ Allow rolling API to accept BaseIndexer subclass Jul 12, 2019

Matt Roeschke added 2 commits July 14, 2019 14:01

Merge branch 'feature/generalized_window_operations' into feature/cus…

f94ae89

…tom_window_span

Merge branch 'feature/generalized_window_operations' into feature/cus…

7e34fa0

…tom_window_span

mroeschke merged commit 609433d into feature/generalized_window_operations Jul 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow rolling API to accept BaseIndexer subclass #2

Allow rolling API to accept BaseIndexer subclass #2

mroeschke commented Jul 2, 2019

jreback commented Jul 2, 2019

jreback commented Jul 2, 2019

jreback commented Jul 2, 2019

mroeschke commented Jul 2, 2019

jreback left a comment

jreback Jul 5, 2019

jreback Jul 5, 2019

cpcloud Jul 10, 2019

cpcloud Jul 10, 2019

jreback Jul 5, 2019

jreback Jul 5, 2019

jreback Jul 5, 2019

jreback Jul 5, 2019

cpcloud Jul 10, 2019

mroeschke Jul 12, 2019

cpcloud Jul 10, 2019

cpcloud Jul 10, 2019


		class BaseIndexer(abc.ABC):

		def __init__(self, index, offset, keys):

		@@ -54,6 +56,38 @@ def arithmetic_win_operators(request):
		return request.param


		@pytest.fixture(params=['right', 'left', 'both', 'neither'])

		@@ -4154,3 +4188,20 @@ def test_rolling_cov_offset(self):

		expected2 = ss.rolling(3, min_periods=1).cov()
		tm.assert_series_equal(result, expected2)

Allow rolling API to accept BaseIndexer subclass #2

Allow rolling API to accept BaseIndexer subclass #2

Conversation

mroeschke commented Jul 2, 2019

jreback commented Jul 2, 2019

jreback commented Jul 2, 2019

jreback commented Jul 2, 2019

mroeschke commented Jul 2, 2019

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment