BUG: cumsum/cumprod raises ArrowInvalid: overflow on integer ArrowDtypes instead of upcasting

[arunkpe]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

s = pd.Series([100, 100], dtype="int8[pyarrow]")

s.cumsum()                                    # ArrowInvalid: overflow
pd.Series([100, 100], dtype="int8").cumsum()  # [100, 200], dtype int64
pd.Series([100, 100], dtype="Int8").cumsum()  # [100, 200], dtype Int64

# Same dtype, same data — these all upcast correctly:
s.sum()                        # 200
s.prod()                       # 10000
s.groupby([0, 0]).cumsum()     # [100, 200], dtype int64[pyarrow]

Issue Description

I ran into this using int8 for the target variable in a parquet-backed training run. A cumulative bad count over a score-sorted target is how you build a gains or KS table, and with a 0/1 target the running total exceeds int8 on the 128th positive row.

cumsum on an integer ArrowDtype raises instead of widening the accumulator, so any series whose running total exceeds the storage width fails. Three rows of value 50 in an int8[pyarrow] column is enough to trigger it.

Every integer width is affected at its own boundary. The numpy-backed and masked equivalents widen to 64-bit and return the correct value:

dtype	value	Arrow	numpy
int8	127	ArrowInvalid	[127, 254]
int16	32767	ArrowInvalid	[32767, 65534]
int32	2147483647	ArrowInvalid	[2147483647, 4294967294]
uint8	255	ArrowInvalid	[255, 510]
int64	2**63-1	ArrowInvalid	[9223372036854775807, -2]

cumprod, agg("cumsum") and Series + Series fail the same way. cummax and cummin are unaffected, no overflow is possible.

But the same dtype widens correctly everywhere else. So:

s = pd.Series([100, 100], dtype="int8[pyarrow]")
s.sum()                       # 200
s.prod()                      # 10000
s.groupby([0, 0]).cumsum()    # [100, 200], int64[pyarrow]

So Series.cumsum and SeriesGroupBy.cumsum disagree on identical input.

This is present on 2.3.3 with the same pyarrow, so this is not a 3.0 regression.

Root Cause:

ArrowExtensionArray._accumulate widens before accumulating, but only for temporal types
(pandas/core/arrays/arrow/array.py):

convert_to_int = (
    pa.types.is_temporal(pa_dtype) and name in ["cummax", "cummin"]
) or (pa.types.is_duration(pa_dtype) and name == "cumsum")
 
if convert_to_int:
    if pa_dtype.bit_width == 32:
        data_to_accum = data_to_accum.cast(pa.int32())
    else:
        data_to_accum = data_to_accum.cast(pa.int64())
 
try:
    result = pyarrow_meth(data_to_accum, skip_nulls=skipna, **kwargs)
except pa.ArrowNotImplementedError as err:
    msg = f"operation '{name}' not supported for dtype '{self.dtype}'"
    raise TypeError(msg) from err

Integer types skip the cast and go into cumulative_sum_checked at their native width. The existing except translates ArrowNotImplementedError into a TypeError but does not cover ArrowInvalid, so the pyarrow error surfaces raw with no column or dtype context. _reduce already applies the upcast rule, which is why .sum() works on the same data.

Expected Behavior

pd.Series([100, 100], dtype="int8[pyarrow]").cumsum()
# [100, 200], dtype int64[pyarrow]

Widen narrow integer types to int64/uint64 before accumulating, matching _reduce and the other two backends.

Behavior at the 64-bit boundary is a different question. numpy and masked wrap silently there ([2**63-1, -2]), so raising is defensible and I am not asking for it to change. The narrow widths are the part that looks unintended.

Installed Versions

Details

commit : e68db09
python : 3.12.13
python-bits : 64
OS : Darwin
OS-release : 24.6.0
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.5
numpy : 2.5.1
dateutil : 2.9.0.post0
pip : 26.1.2
pyarrow : 25.0.0
fastparquet : None
fsspec : None

[adityaanikam]

Verified against current main — ArrowExtensionArray._accumulate (pandas/core/arrays/arrow/array.py) confirms exactly what you found:

convert_to_int = (
    pa.types.is_temporal(pa_dtype) and name in ["cummax", "cummin"]
) or (pa.types.is_duration(pa_dtype) and name == "cumsum")

Plain integer ArrowDtypes never satisfy either branch, so they skip the upcast and go into pyarrow_meth at native width — and the except pa.ArrowNotImplementedError clause doesn't catch the ArrowInvalid that PyArrow actually raises on overflow, so it surfaces raw with no pandas-level context. Nice catch cross-referencing _reduce, which already has the correct widen-before-operate rule for the same dtype.

I'd like to work on this: widen narrow integer types to int64/uint64 in the same branch, matching _reduce's existing rule, and leave the 64-bit boundary raising as-is per your note. Will add regression tests across the dtype-boundary table you documented (int8/int16/int32/uint8 at their overflow points) plus one confirming Series.cumsum() and SeriesGroupBy.cumsum() now agree on identical input.