group_nanvar, group_nanstd needs to support `ddof` kwarg (0 by default)

[dcherian]

needs to be count-ddof in the denominator:

numbagg/numbagg/grouped.py

Line 175 in f3599a5

(sums_of_squares[label] - (sums[label] ** 2 / count)) / (count - 1)

import numbagg.grouped
import numpy as np

a = np.array(
    [
        0.67625467,
        np.nan,
        np.nan,
        0.59830875,
        0.18673419,
        0.67275604,
        0.94180287,
        0.24824571,
        0.94888115,
    ],
    dtype=np.float64,
)

np.nanstd(a)  # 0.27862342899274856
numbagg.grouped.group_nanstd(a, np.zeros_like(a, dtype=int)) # array([0.3009477])

[max-sixty]

Does anyone actually use this?

Somewhat a serious question — can we find anyone using something other than (n-1) intentionally?

I would otherwise vote to just raise an error and ask someone to open an issue if they do want it. But if you really want me to, I will implement!

[dcherian]

can we find anyone using something other than (n-1) intentionally?

I think the issue is that its a bad default choice in numpy. But I just raise an error if ddof!=1

[max-sixty]

WDYT about defaulting to 1 in flox? I'm not sure what we do in xarray, I'll check...

[dcherian]

woops it's 0 by default for nanstd, nanvar:

numbagg/numbagg/funcs.py

Line 62 in 1b86d7a

ddof = 0

[max-sixty]

D'oh!

Should we add ddof to std & var? I'm trying to balance:

• We should provide useful things! Maybe ddof is actually useful?¹
  • I had been working on the moving exponential functions, where the values are always a sample, and so I don't think anything but ddof=1 makes sense. But are there practical cases is the normal use of the functions where it is useful?
• Having the functions have consistent signatures massively reduces the cost of their maintenance, and amortizes the infra such as decorators across all our functions. Adding this option would start to balkanize the functions.
• We can use this opportunity to change defaults; seems like a missed opportunity to just inherit all old defaults
• Lower confidence, but arguably there are too many options in some of the numpy/pandas implementations — for example in the moving exponential functions, adjust=False seems like a bad and confusing option, at least for the cases I'm familiar with.

@shoyer as the metaphorical elder here, do you have a view?

Footnotes

1. would be v cool to be able to do a search of GH and see what was actually used ↩

[shoyer]

I think ddof=1 is almost always the right choice, though I do think there might be a couple of cases (maximum likelihood estimation?) where ddof=0 is preferred.

[max-sixty]

OK, so we could at least change the default for now to ddof=1.

Then still not sure whether we should add the option. Maybe we wait to see whether there's demand...

[shoyer]

In any case, the only two cases that matter are definitely ddof=0 and ddof=1. So you could also just write two copies of the function.

[max-sixty]

Closed by #263