BUG: Inconsistent overflow behavior in Series.__setitem__

[wirable23]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> df23 = pd.DataFrame({"Col0": pd.Series([-1291861603, None], dtype="int32[pyarrow]"), "Col1": pd.Series([True, True], dtype="bool[pyarrow]"), "Col10": pd.Series([2.193378806488941e-13, None], dtype="float32[pyarrow]")})
>>> rhs = df23["Col0"]
>>> lhs = df23["Col10"]
>>> lhs[df23["Col1"]] = rhs
>>> lhs
0   -1291861632.0
1            <NA>
Name: Col10, dtype: float[pyarrow]
>>>

Issue Description

An unsafe conversion succeeded. The first value of lhs is rounded when pyarrow should do safe conversions by default. Somehow, when I remove the 2nd row of the df, the expected error about an unsafe conversion is raised. I wouldn't expect the presence of a null value to hide an overflow error.

Expected Behavior

df24 = pd.DataFrame({"Col0": pd.Series([-1291861603], dtype="int32[pyarrow]"), "Col1": pd.Series([True], dtype="bool[pyarrow]"), "Col10": pd.Series([2.193378806488941e-13], dtype="float32[pyarrow]")})
df24
Col0 Col1 Col10
0 -1291861603 True 2.193379e-13
rhs = df24["Col0"]
lhs = df24["Col10"]
lhs[df24["Col1"]] = rhs
Traceback (most recent call last):
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1159, in setitem
self._set_with_engine(key, value)
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1222, in _set_with_engine
loc = self.index.get_loc(key)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\indexes\range.py", line 350, in get_loc
self._check_indexing_error(key)
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\indexes\base.py", line 5736, in _check_indexing_error
raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: 0 True
Name: Col1, dtype: bool[pyarrow]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1209, in setitem
self._where(~key, value, inplace=True)
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\generic.py", line 9730, in _where
new_data = self._mgr.putmask(mask=cond, new=other, align=align)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\internals\managers.py", line 403, in putmask
return self.apply(
^^^^^^^^^^^
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\internals\managers.py", line 352, in apply
applied = getattr(b, f)(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\internals\blocks.py", line 1739, in putmask
values._putmask(mask, new)
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\base.py", line 1528, in _putmask
self[mask] = val
~~~~^^^^^^
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 1383, in setitem
data = self._replace_with_mask(self._data, key, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 1666, in _replace_with_mask
return pa.array(result, type=values.type, from_pandas=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow\array.pxi", line 323, in pyarrow.lib.array
File "pyarrow\array.pxi", line 83, in pyarrow.lib._ndarray_to_array
File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value -1291861603 is outside of the range exactly representable by a IEEE 754 single precision value

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.1
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.13.2
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[wirable23]

>>> df24 = pd.DataFrame({"Col0": pd.Series([-1291861603, 10], dtype="int32[pyarrow]"), "Col1": pd.Series([True, True], dtype="bool[pyarrow]"), "Col10": pd.Series([2.193378806488941e-13, 10], dtype="float32[pyarrow]")})
>>> rhs = df24["Col0"]
>>> lhs = df24["Col10"]
>>> lhs[df24["Col1"]] = rhs
Traceback (most recent call last):
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1159, in __setitem__
    self._set_with_engine(key, value)
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1222, in _set_with_engine
    loc = self.index.get_loc(key)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\indexes\range.py", line 350, in get_loc
    self._check_indexing_error(key)
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\indexes\base.py", line 5736, in _check_indexing_error
    raise InvalidIndexError(key)
pandas.errors.InvalidIndexError: 0    True
1    True
Name: Col1, dtype: bool[pyarrow]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1209, in __setitem__
    self._where(~key, value, inplace=True)
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\generic.py", line 9730, in _where
    new_data = self._mgr.putmask(mask=cond, new=other, align=align)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\internals\managers.py", line 403, in putmask
    return self.apply(
           ^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\internals\managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\internals\blocks.py", line 1739, in putmask
    values._putmask(mask, new)
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\base.py", line 1528, in _putmask
    self[mask] = val
    ~~~~^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 1383, in __setitem__
    data = self._replace_with_mask(self._data, key, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 1666, in _replace_with_mask
    return pa.array(result, type=values.type, from_pandas=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\array.pxi", line 323, in pyarrow.lib.array
  File "pyarrow\array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value -1291861603 is outside of the range exactly representable by a IEEE 754 single precision value
>>>

Interesting, if the null value in df23 is replaced with actual data, the overflow error is raised, so somehow the null is hiding the overflow error and causing an unsafe conversion to succeed.

[topper-123]

I tried running your example in the dev version of Pandas + pyarrow 12 and I got ArrowInvalid: Integer value -1291861603 not in range: -16777216 to 16777216.

I then tried in pandas v.2.0.1 and got the error you describe. So this has been fixed in the dev version, but I don't see any entry on this in the whatsnew.

INSTALLED VERSIONS

commit : 855673f
python : 3.9.16.final.0
python-bits : 64
OS : Darwin
OS-release : 22.3.0
Version : Darwin Kernel Version 22.3.0: Mon Jan 30 20:39:35 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T8103
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 2.1.0.dev0+808.g855673fdc5
numpy : 1.23.4
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.6.3
pip : 22.3.1
Cython : 0.29.33
pytest : 7.1.2
hypothesis : 6.37.0
sphinx : 5.0.2
blosc : 1.10.6
feather : None
xlsxwriter : 3.0.3
lxml.etree : 4.9.2
html5lib : None
pymysql : 1.0.2
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.9.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : 1.3.5
brotli :
fastparquet : 0.8.3
fsspec : 2022.11.0
gcsfs : 2022.11.0
matplotlib : 3.6.2
numba : 0.56.4
numexpr : 2.8.4
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : 1.0.10
s3fs : 2022.11.0
scipy : 1.9.1
snappy : None
sqlalchemy : 1.4.43
tables : 3.7.0
tabulate : 0.9.0
xarray : 2023.4.2.dev3+gc75ac8b7
xlrd : 2.0.1
zstandard : 0.18.0
tzdata : 2023.3
qtpy : 2.2.0
pyqt5 : None

[wirable23]

Thanks @topper-123 for checking. Will this be part of pandas 2.0.2 or 2.1.0?

[topper-123]

v2.1.

[wirable23]

#53171

This is the PR that fixed it for reference.

[mroeschke]

Closing as this will be fixed