ENH: pd.Series.shift and .diff to accept a collection of numbers

[jona-sassenhagen]

Is your feature request related to a problem?

No

Describe the solution you'd like

pd.DataFrame.shift/pd.Series.shift could alternatively accept a list of ints or a collection of numbers, and the result would be a dataframe with one column per shift. This would make it very convenient to e.g. create multiple lagged versions of a column for time series problems.

E.g.,

time_series_df[my_time_feature].shift([-3, -2, -1, 0, 1, 2, 3])

would give me a dataframe with 7 columns, each shifted versions of time_series_df[my_time_feature].

API breaking implications

The first input to df.shift would be overloaded to also accept a list of ints.

Describe alternatives you've considered

Alternatives are very simple to do, you just iterate over the lags and concatenate or join.
This would just be a handy shortcut.

Additional context

I'd in principle be open to implementing this myself, although I've never contributed to pandas.

[rhshadrach]

+1 - This is pretty easy to accomplish today, but pandas accepts lists in many places like this and it feels like a natural extension of that. A few thoughts if we do go forward with this:

• Should accept any iterable
• Should raise a TypeError if any value is not an integer
• A list of one element should return a DataFrame and not a Series
• Need to handle the case of an empty list (raise a ValueError?)

[rhshadrach]

Would DataFrame.shift also need to accept an iterable?

[jona-sassenhagen]

I would say for the sake of consistency both series and dataframe should accept the iterable.

This raises the question of column order for the Dataframe case.

[jreback]

this should just be a series operation only or if ina frame would have to generate a multi index

[jona-sassenhagen]

this should just be a series operation only or if ina frame would have to generate a multi index

You mean, have either lags or columns on the index?
This would be annoying for the use case of analysing time series data with e.g. a regression, where you want one column per lag and feature. My idea would be to default to having the list of ints case be the same as a concatenation of looping over ints.

[rhshadrach]

For Series, what would the columns be labeled? Is there another Series method that can take multiple inputs and act as a transformer? I am only aware of reducers (e.g. quantile).

[jona-sassenhagen]

Is there another Series method that can take multiple inputs and act as a transformer?

pd.Series.transform right? It returns a series if called with a single function and a df if called with a list of functions.

For Series, what would the columns be labeled?

I would label them as some function of the shift and the original name. E.g. orig_column_shift_0, orig_column_shift_1 etc.

[jbrockmendel]

xref #19298

[GitChein]

take

[rhshadrach]

Thanks @jbrockmendel - did not realize this was a duplicate. A lot more discussion in here, so I think maybe leave this one open and close the other?

@jona-sassenhagen

I would label them as some function of the shift and the original name. E.g. orig_column_shift_0, orig_column_shift_1 etc.

This would work well for string / integer labels, but could produce odd results if other Python objects are used. Perhaps that's not a case we should worry too much about though.

[jbrockmendel]

A lot more discussion in here, so I think maybe leave this one open and close the other?

Fine by me

[rhshadrach]

From #19298: Maybe take an optional prefix, e.g. ser.shift([1, 2, 3], prefix="L_") produces a DataFrame with columns "L_1",

[jona-sassenhagen]

From #19298: Maybe take an optional prefix, e.g. ser.shift([1, 2, 3], prefix="L_") produces a DataFrame with columns "L_1",

That sounds good to me

[jona-sassenhagen]

But need to handle the pd.DataFrame case, where you also have column names.

[GitChein]

A few clarification questions:

• If the list is shorter or longer than the number of columns (In data frame), what is the intended behavior?
• do we also want axis to take an iter?

[rhshadrach]

@GitChein - I'm guessing you're thinking of the list of shifts to be zipped with the columns. My first intuition would be that shifting a DataFrame would instead apply each shift to every column. E.g.

df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
shifts = [0, 1, 2]
result = (
    pd.concat(
        [
            df.shift(shift).add_suffix(f'_{shift}')
            for shift in shifts
        ],
        axis=1,
    )
)
print(result)

produces

   a_0  b_0  a_1  b_1  a_2  b_2
0    1    4  NaN  NaN  NaN  NaN
1    2    5  1.0  4.0  NaN  NaN
2    3    6  2.0  5.0  1.0  4.0

This is similar to quantile: df.quantile([0.1, 0.2, 0.3]) will also produce each quantile for every column.

[jona-sassenhagen]

Yes, each shift should be applied to each column IMO.

[skwirskj]

@jona-sassenhagen @rhshadrach From my and @GitChein 's understanding, we want the function to essentially apply consecutive shifts and then return a resulting DataFrame, correct?

Also, if the input is an empty list, should we keep the default functionality of returning a shift by 1, or should we just return the original DataFrame/Series? I personally think returning the original frame makes more sense since we pass an empty list rather than using the default value for the period parameter.

[GitChein]

Hey all,

We made a potential fix and ran ci/code_checks.sh and test_shift.py for both frame and series and they all pass. Below is what the current output looks like in our branch:

DataFrame

df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
shifts = [0, 1, 2]
df.shift(shifts)
   a_0  b_0  a_1  b_1  a_2  b_2
0  1.0  4.0  NaN  NaN  NaN  NaN
1  2.0  5.0  1.0  4.0  NaN  NaN
2  3.0  6.0  2.0  5.0  1.0  4.0

Series

ser = pd.Series([1,2,3])
ser.shift([0,1,2])
   0_0  0_1  0_2
0  1.0  NaN  NaN
1  2.0  1.0  NaN
2  3.0  2.0  1.0

Some Notes:

• It handles empty lists by returning self (either a dataframe or series respectively)
• It handles non integer iterables by raising a typerror

If this is suitable for the needs of the issue, then we will either create some tests or create a pull request. Whichever is more appropriate. This is my and @skwirskj first time contributing so apologies for any misunderstanding of the process!