BUG: Fix nansum overflow on Windows with bottleneck #15507
Conversation
JaviSorribes
force-pushed
the
overflow_nanops
branch
from
76e3dc1 to
c40f6dc
Compare
Codecov Report
@@ Coverage Diff @@
## master #15507 +/- ##
==========================================
- Coverage 91.1% 91.03% -0.07%
==========================================
Files 136 136
Lines 49102 49103 +1
==========================================
- Hits 44736 44703 -33
- Misses 4366 4400 +34
Continue to review full report at Codecov.
|
cpcloud
added
Bug
Numeric Operations
There was a problem hiding this comment.
The biggest issue here is this semantically changes things. IOW, now since we are always using pandas nansum (rather than bottleneck nansum if installed), we will always yield Series([np.nan]).sum() -> np.nan (rather than 0 if bottleneck is installed). So I think we actually have to change this for all platforms or then this becomes even more confusing.
pandas/core/nanops.py
Outdated
| @@ -130,6 +131,8 @@ def _bn_ok_dtype(dt, name): | |||
| if name == 'nansum': | |||
| if dt.itemsize < 8: | |||
| return False | |||
| elif os.name == 'nt': # bottleneck has overflow issue in Windows | |||
There was a problem hiding this comment.
use pandas.compat.is_platform_windows()
| @@ -327,6 +327,12 @@ def test_nansum(self): | |||
| self.check_funs(nanops.nansum, np.sum, allow_str=False, | |||
| allow_date=False, allow_tdelta=True, check_dtype=False) | |||
|
|
|||
| def test_nansum_overflow(self): | |||
| s = Series([2**31]) | |||
There was a problem hiding this comment.
needs testing both with and w/o bottleneck (to yield the same results), see other tests for how to do this.
|
w.r.t. #9422 I think we need to simply change this (in effect what this PR does), but for all platforms (and not just windows). |
cpcloud
changed the title
|
@JaviSorribes Looking at @jreback's comments, it looks like we should just disable |
JaviSorribes
force-pushed
the
overflow_nanops
branch
from
9e121fe to
9335903
Compare
|
xref #15542 |
|
We indeed just need to take a decision on #9422, as we have left this lingering too long. |
|
The fixes are related to fixing #9422 are a bit bigger in scope than this. That said If you want to rebase / add some additional tests could take this as a precurser. |
|
closing this. as #9422 needs resolution. |
Bottleneck used in nansum produces overflow error on Windows. Temporary fix that could be undone once the error in bottleneck is fixed and released, to avoid platform specificity.
git diff upstream/master | flake8 --diff