BUG: DataFrame iloc does not accept Series assignment

[jonasreiher]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

---

Code Sample, a copy-pastable example

df = pd.DataFrame(columns=[0], index=[0])

df.iloc[0,0] = pd.Series([1,2,3])    # this raises a ValueError
df.iloc[0] = [pd.Series([1,2,3])]    # this works, but is ugly

Problem description

When trying to assign a pandas.Series to one cell of a pandas.DataFrame, the following error is raised:
ValueError: Incompatible indexer with Series
Series in DataFrames are a common thing e.g. in sktime, when working with multivariate time series data and (as the working example above shows) it is unproblematically supported by pandas. Just the assignment to one cell via .iloc seems to be erroneous.

INSTALLED VERSIONS

commit : db08276
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : de_DE.cp1252

pandas : 1.1.3
numpy : 1.19.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 50.3.0.post20201006
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[GYHHAHA]

I don't think this is necessary. Could you show a more specific example about its usage in sktime?

[jreback]

This is a very strange thing to do. I suppose this is technically a bug as a scalar could be anything, even a list-like

e.g. this works

In [189]: df = pd.DataFrame(columns=[0], index=[0]) 
     ...:  
     ...:                                                                                                                                                                                       

In [190]: df.iloc[0, 0] = [5, 3, 1]                                                                                                                                                             

so will mark it though embedding complicated structures inside a cell is non-performant and likely has rough edges.

[onshek]

take

[jonasreiher]

@GYHHAHA sktime uses exactly this format when loading multivariate time series data (see documentation) with samples in rows, channels in columns and the time dimension in the series. And this is then also the format expected by different classifiers, even if its performance may not be the best.
So I think you should be able to construct something like this manually without unpleasant workarounds.

[GYHHAHA]

Strange. This looks fine.

>>>df = pd.DataFrame({'dim0':[pd.Series([1,2])], 'dim1':[pd.Series([3,4])]})
>>>df.loc[0, 'dim0']
0    1
1    2
dtype: int64

And also the following assignment raises.

>>>df = pd.DataFrame([[0]])
>>>df.iloc[0,0] = [5, 3, 1] 
ValueError: setting an array element with a sequence.

[jreback]

i was running on an older version actually

can u run on master and show the results

[GYHHAHA]

Now I have checked these two on master. Still the same results.

[phofl]

I think this would be fixed through #37728, even if this was not the intent :)

[onshek]

I think this would be fixed through #37728, even if this was not the intent :)

well, then I will unassign myself

[elisim]

@jreback @GYHHAHA

Following this issue, the same error is raised when using df.loc:

df = pd.DataFrame(columns=[0], index=[0])

df.loc[0,0] = pd.Series([1,2,3])    # this raises a ValueError (note I used "loc")
df.loc[0] = [pd.Series([1,2,3])]    # this works

[cedricdonie]

Workaround for older pandas versions: use df.at instead of df.loc. I tested this with pandas 1.1.5 and don't get the ValueErroranymore.

Perhaps df.iat can also be used a workaround for df.iloc, but I have not tested this.

Unfortunately, sktime-dl restricts me to using this old pandas version since I need Python 3.6.

[jmpalumbo]

df.at worked like a charm. This is the only board online where I saw df.at as a solution to this ValueError. Thank you @cedricdonie