BUG: ValueError with loc[] = (Regression 2.1.0)

[rxxg]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame(index=[1, 1, 2, 2], data=["1", "1", "2", "2"])
df.loc[df[0].str.len() > 1, 0] = df[0]
df

Issue Description

Given code fails on the third line with exception given below
Code executes normally with panda versions <2.1.0

Traceback (most recent call last):
File "", line 3, in
File "C:\Users\venv\lib\site-packages\pandas\core\indexing.py", line 885, in setitem
iloc._setitem_with_indexer(indexer, value, self.name)
File "C:\Users\venv\lib\site-packages\pandas\core\indexing.py", line 1888, in _setitem_with_indexer
indexer, value = self._maybe_mask_setitem_value(indexer, value)
File "C:\Users\venv\lib\site-packages\pandas\core\indexing.py", line 789, in _maybe_mask_setitem_value
value = self.obj.iloc._align_series(indexer, value)
File "C:\Users\venv\lib\site-packages\pandas\core\indexing.py", line 2340, in _align_series
return ser.reindex(new_ix)._values
File "C:\Users\venv\lib\site-packages\pandas\core\series.py", line 4982, in reindex
return super().reindex(
File "C:\Users\venv\lib\site-packages\pandas\core\generic.py", line 5521, in reindex
return self._reindex_axes(
File "C:\Users\venv\lib\site-packages\pandas\core\generic.py", line 5544, in _reindex_axes
new_index, indexer = ax.reindex(
File "C:\Users\venv\lib\site-packages\pandas\core\indexes\base.py", line 4433, in reindex
raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels

Expected Behavior

Code should execute normally with result
0
1 1
1 1
2 2
2 2

(No reindexing should be necessary since no rows are selected with code on line 3.)

Installed Versions

INSTALLED VERSIONS

commit : ba1cccd
python : 3.9.0.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_Ireland.1252
pandas : 2.1.0
numpy : 1.24.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.5.1
pip : 22.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report!

(No reindexing should be necessary since no rows are selected with code on line 3.)

To be sure, it's not the left hand side that is reindexing, it's the right. E.g.

df.loc[df[0].str.len() > 1, 0] = 5

works. I believe we raise anytime the RHS has a duplicate value because the result can be ambiguous, even though it won't necessarily be ambiguous. In general we try to avoid values-dependent behavior. In this case, if it just so happens that in one case the mask on the left is all False you may think the code works, but will then fail as soon as it isn't all False. That can be a bad user experience.

[rhshadrach]

Code executes normally with panda versions <2.1.0

Ah, I missed this! Thanks for that detail. We should run a git blame and see where this ended up getting changed.

[20revsined]

take

[matiaslindgren]

take