BUG: Assigning pd.NA to StringDtype column causes "Unknown error: Wrapping .. failed" after pd.concat with PyArrow

[askalonso]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

I get a pyarrow.lib.ArrowException: Unknown error: Wrapping � failed when running this in a new environment with only pandas and pyarrow installed. Confirmed error on Windows (python 3.13) and Ubuntu (python 3.12). I've included pd.show_versions() of the Ubuntu replication.

import pandas as pd

df = pd.concat([
    pd.DataFrame({"a": ["x"] * 33, "b": ["x"] * 33}),
    pd.DataFrame({"a": ["x"] * 34, "b": ["x"] * 34}),
])

for i in range(1, 50):
    df.loc[[0], "a"] = pd.NA

Traceback

I've replaced part of the full file path with "~"

Details

Traceback (most recent call last):
File "~/bug-repro/repro_fail_single_row_list.py", line 9, in
df.loc[[0], "a"] = pd.NA
~~~~~~^^^^^^^^^^
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 941, in setitem
iloc._setitem_with_indexer(indexer, value, self.name)
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 2480, in _setitem_with_indexer
self._setitem_with_indexer_split_path(indexer, value, name)
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 2571, in _setitem_with_indexer_split_path
self._setitem_single_column(loc, value, pi)
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 2709, in _setitem_single_column
self.obj._mgr.column_setitem(loc, plane_indexer, value)
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 1521, in column_setitem
self.iset(loc, new_mgr._block.values, inplace=True)
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 1299, in iset
return self._iset_single(
^^^^^^^^^^^^^^^^^^
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/internals/managers.py", line 1482, in _iset_single
blk.set_inplace(slice(iloc, iloc + 1), value, copy=copy)
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1965, in set_inplace
self.values[:] = values
~~~~~~~~~~~^^^
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/arrays/arrow/array.py", line 2417, in setitem
value = self._maybe_convert_setitem_value(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/arrays/string_arrow.py", line 289, in _maybe_convert_setitem_value
value = np.array(value, dtype=object, copy=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/arrays/arrow/array.py", line 869, in array
return self.to_numpy(dtype=dtype, copy=copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/bug-repro/.venv/lib/python3.12/site-packages/pandas/core/arrays/arrow/array.py", line 1895, in to_numpy
result[~mask] = data[~mask]._pa_array.to_numpy()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 531, in pyarrow.lib.ChunkedArray.to_numpy
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
failed.lib.ArrowException: Unknown error: Wrapping

Issue Description

When assigning pd.NA (or other None-types) to a string-column in a dataframe with two or more string-columns, it eventually fails with an "Unknown error: Wrapping ... failed" when the dataframes are large enough. If you decrease the size of either dataframe in the reproducible example the error goes away. I've confirmed that it happens both on pandas main-branch (3.1.0.dev0+218.gea4d2c0b27) and 3.0.1.

Only happens with Pandas and PyArrow. It doesn't always happen on the first pd.NA assignment (thus the for loop). Also, it only happens with two or more string columns in the dataframe, if column "b" is dropped, the issue goes away. If you set elements with single indices df.loc[0, "a"] = pd.NA instead, the error also seems to dissapear.

From my investigations, it only seems to happen when the total size in memory of the column in the dataframe after the concat is above a certain threshold. Meaning that if you increase the length of the strings in column "a", the threshold in order to trigger the error decreases. Example of a shorter dataframe that also fails:

df = pd.concat([
    pd.DataFrame({"a": ["xx"] * 18, "b": ["x"] * 18}),
    pd.DataFrame({"a": ["xx"] * 17, "b": ["x"] * 17}),
])

The size of the strings in column "b" doesn't seem to impact if an error occurs or not.

If you set more elements to pd.NA at the same time, the threshold for when the error occurs increases. Example:

df = pd.concat([
    pd.DataFrame({"a": ["x"] * 36, "b": ["x"] * 36}),
    pd.DataFrame({"a": ["x"] * 35, "b": ["x"] * 35}),
])

for i in range(1, 50):
    df.loc[[0, 1, 2], "a"] = pd.NA

Here the size of the dataframe had to be increased to trigger the error.

A workaround is to set the storage type of the dataframes to "python":

df = pd.concat([
    pd.DataFrame({"a": ["x"] * 33, "b": ["x"] * 33}, dtype=pd.StringDtype(storage="python")),
    pd.DataFrame({"a": ["x"] * 34, "b": ["x"] * 34}, dtype=pd.StringDtype(storage="python")),
])

I've never raised an issue on github before so I'd be grateful for some feedback on anything that could be improved :)

Expected Behavior

The expected behavior is that no errors happen, and the first row in column "a" is pd.NA.

Installed Versions

Details

INSTALLED VERSIONS ------------------ commit : ea4d2c0 python : 3.12.3 python-bits : 64 OS : Linux OS-release : 6.14.0-1017-azure Version : #17~24.04.1-Ubuntu SMP Mon Dec 1 20:10:50 UTC 2025 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : C.UTF-8

pandas : 3.1.0.dev0+218.gea4d2c0b27
numpy : 2.4.2
dateutil : 2.9.0.post0
pip : None
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
psycopg2 : None
pymysql : None
pyarrow : 23.0.1
pyiceberg : None
pyreadstat : None
pytest : None
python-calamine : None
pytz : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
qtpy : None
pyqt5 : None

[jorisvandenbossche]

@askalonso thanks for the detailed report!
I can reproduce this on Linux/Ubuntu as well. I assume we are somehow hitting a bug in pyarrow. When I debug the exception, it seems that we have malformed arrow data:

ipdb> p data._pa_array
<pyarrow.lib.ChunkedArray object at 0x7f2468c07b80>
[
  [
    null,
    "x",
    "x",
    "x",
    "x",
    ...
    "x",
    "x",
    "x",
    "x",
    "x"
  ],
<Invalid array: First or last binary offset out of bounds>
]

that shouldn't happen when just using public pyarrow APIs as we are doing. But still have to figure out which exact pyarrow code triggers this.

[kjmin622]

This issue reproduces regardless of the array length. Even with very small inputs, I can observe unexpected data mutations.

>>> df = pd.concat([
...     pd.DataFrame({"a": ["x", "x", "x"], "b": ["x", "x", "x"]}),
...     pd.DataFrame({"a": ["x", "x", "x", "x"], "b": ["x", "x", "x", "x"]}),
... ])
>>> df.loc[[0], "a"] = pd.NA
>>> df
     a  b
0  NaN  x
1    x  x
2    x  x
0  NaN  x
1    x  x
2    x  x
3    x  x
>>> df.loc[[0], "a"] = pd.NA
>>> df
     a  b
0  NaN  x
1    x  x
2    x  x
0  NaN  x
1    x  x
2      x
3      x

With shorter arrays, the Wrapping ... failed exception may not trigger, but the underlying problem is still present. In the example above, repeatedly running df.loc[[0], "a"] = pd.NA results in unintended nulls / value corruption.

Across several variations, I also observed cases where values change unexpectedly (not only to nulls, but also to other characters), which suggests broader data corruption rather than a length-specific failure.

[kjmin622]

Also, I confirmed that the issue starts from the second assignment, regardless of whether the first assignment actually inserted pd.NA. In addition, the problem seems to occur in the non-first chunks of a ChunkedArray (i.e., chunks other than the first one).

>>> df = pd.concat([
...     pd.DataFrame({"a": ["x", "x","x"], "b": ["x", "x", "x"]}),
...     pd.DataFrame({"a": ["c"] * 4, "b": ["c"] * 4}),
...     pd.DataFrame({"a": ["d"] * 4, "b": ["d"] * 4}),
... ])
>>> df.loc[[0],"a"] = pd.NA
>>> df.loc[[0],"a"] = pd.NA
>>> df
     a  b
0  NaN  x
1    x  x
2    x  x
0  NaN  c
1    c  c
2      c
3      c
0  NaN  d
1      d
2      d
3      d
---
df = pd.concat( ...
df.loc[[0], "a"] = "c"
df.loc[[0], "a"] = pd.NA 
# same result

So my current hypothesis is that the underlying issue is in the pc.if_else path (I haven't built & debugged pyarrow from source yet, so this isn't conclusive). If that's correct, a possible pandas-side workaround would be to call combine_chunks() on left and right (when they are ChunkedArrays) before passing them to if_else.

@jorisvandenbossche If you'd like to provide a pandas-side workaround ahead of a pyarrow fix, I'm happy to take this on.

[jorisvandenbossche]

I managed to track down the actual pyarrow bug, and reported it at apache/arrow#49410. It seems to specifically happen when calling if_else with a chunked string array where one of the chunks has an offset.
We will definitely have to workaround that on our side in pandas, since this causes a serious bug with string columns.