ENH: Add window to exponentially weighted moment functions

[matthewgilbert]

Currently there is no way to specify a rolling window for the exponentially weighted functions. From the docs the weighting function is giving by

However I am wondering if it is straightforward to add a window parameter, similar to the rolling window statistics. This would change the formula to look like

[max-sixty]

Can you clarify with a numerical example? i=0 means it starts from the present. It then decays as time passes, as you'd expect from an exponential function.

[matthewgilbert]

In the above I would think of t as present and 0 (or n in my suggestion) as the start. Here is an example

s = pd.Series(range(1,11))
alpha = 0.05
span = 2/alpha - 1
pd.ewma(s, span=span)

0    1.000000
1    1.512821
2    2.034181
3    2.564069
4    3.102470
5    3.649363
6    4.204725
7    4.768525
8    5.340732
9    5.921307
dtype: float64

which for the last element is equivalent to doing

weights = [(1-alpha)**i for i in reversed(range(0,10))]
weights = pd.Series(weights)
sum(weights * s) / sum(weights)
5.921307

What I was wondering was if it is possible to add a window such that there is n exponentially decaying weights and the rest are 0. So for the above data, with n=3 for example, for the last element of the series we would have

[max-sixty]

That's very clear @matthewgilbert, thanks.

Re the calc, I think you could implement that by taking the difference between two series - one lagged by n and multiplied by alpha ^ n. I haven't thought this through enough; there may be better ways.

In terms of the API, I think that could be added, although I'm not sure of the use cases - generally I would use either a rolling OR an exponentially weighted series. Do you have use cases in mind?

[matthewgilbert]

A simple hack to apply this would be

s = pd.Series(range(1,10))
window = 4
halflife = 2
ewma1 = lambda x: pd.ewma(x, halflife=halflife)[-1]
pd.rolling_apply(s, window, ewma1)

so if there is very little interest from others in this added functionality this is a workaround

[matthewgilbert]

As of version 0.18.0 the above workaround is flagged for future deprecation. I attempted to apply something similar using .rolling() and .EWM() however this does not seem to work. Wondering if anyone has any suggestions for a way to incorporate a window into exponentially weighted functions under the new API?

[jreback]

In [8]:  s = pd.Series(range(1,10))
   ...: window = 4
   ...: halflife = 2
   ...: ewma1 = lambda x: Series(x).ewm(halflife=halflife).mean().iloc[-1]
   ...: s.rolling(window).apply(ewma1)
   ...: 
Out[8]: 
0        NaN
1        NaN
2        NaN
3    2.91912
4    3.91912
5    4.91912
6    5.91912
7    6.91912
8    7.91912
dtype: float64

But this would be way more efficient to do it in-line.
Though @MaximilianR method seems much easier.

[matthewgilbert]

Great thanks. Unless I am mistaken, the solution @MaximilianR suggests does not seem that straight forward? My understanding of his suggestion is to solve for C in something like the following, where we want the rolling window to be 3

which seems to lead to a pretty messy solution? e.g.

[max-sixty]

@matthewgilbert gosh, that's certainly not what I envisaged.

I actually think your solution is v good. It's not as fast, but I imagine it's fast enough?

Otherwise I'm happy to go into my suggestion more, it's really not supposed to be anything like that complicated.

[matthewgilbert]

Yes my application is fairly small amount of data and performance with jreback's proposed solution above works fine.

[NirantK]

Hey @jreback @matthewgilbert, consider closing this issue since mentioned that the proposed solution works for your data sizes?

[matthewgilbert]

Okay I'll go ahead and close this, good call.

[lycanthropes]

Hi, How about the solution now ? I also meet the same challenge.

[mcherkassky]

+1 - anything in the pipes for a faster vectorized solution for larger amounts of data?

[alankabisov]

+1 - adding window size to ewm() would be nice

[ostwilkens]

+1 - I'm using the earlier proposed solution, but it's very slow for my use. Is it possible to optimize?

[alankabisov]

Seems like using span do approximately the same thing as setting the window. I am not sure, may be anybody can give more concrete information.