BUG: apply swallows exceptions, shows inconsistent behaviour

[gshaikov]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd

# The problem case is 2.3

# --- Dataframe with 1 row ---

# 1.1 instantiate dataframe with 1 row.
df_full = pd.DataFrame([{"a": 1, "b": 2, "c": 3}], columns=["a", "b", "c"])
print(f"df_full =\n{df_full}\n")

# 1.2 returns Series with 1 element ("Series([6], dtype: float64)"").
df_full_apply_succeeded = df_full.apply(lambda row: row.sum(), axis=1)
print(
    f"df_full_apply_succeeded =\n{df_full_apply_succeeded}\n"
    f"type = {type(df_full_apply_succeeded)}\n"
)

# 1.3 raises exception correctly.
try:
    df_full_apply_raises = df_full.apply(lambda row: row["not_in_df"], axis=1)
except Exception as e:
    print(f"df_full_apply_raises raises with msg: {e}\n")


# --- Dataframe with no rows ---

# 2.1 instantiate "Empty DataFrame".
df_empty = pd.DataFrame([], columns=["a", "b", "c"])
print(f"df_empty =\n{df_empty}\n")

# 2.2 returns Series with 0 elements ("Series([], dtype: float64)").
# behaviour mirrors example 1.2, consistent for both dataframes.
df_empty_apply_succeeded = df_empty.apply(lambda row: row.sum(), axis=1)
print(
    f"df_empty_apply_succeeded =\n{df_empty_apply_succeeded}\n"
    f"type = {type(df_empty_apply_succeeded)}\n"
)

# 2.3 swallows exception and returns "Empty DataFrame".
# ----> behaviour DIFFERENT from example 1.3, not consistent!
df_empty_apply_does_not_raise = df_empty.apply(lambda row: row["not_in_df"], axis=1)
print(
    f"df_empty_apply_does_not_raise =\n{df_empty_apply_does_not_raise}\n"
    f"type = {type(df_empty_apply_does_not_raise)}\n"
)

Issue Description

Inconsistency in semantics

When calling apply on a DataFrame with >1 elements and the func provided to apply raises, the exception is propagated to the caller, which is what one would expect.

When calling apply on a DataFrame with 0 elements and the func provided to apply raises, the exception is swallowed and apply returns an original empty dataframe.

Inconsistency in return types

As the code snippet demonstrates, apply returns Series in all the cases except when it fails on an empty dataframe and swallows the exception - then it returns an empty DataFrame. The downstream code eventually breaks but with an error which is hard to track.

This is where exceptions are swallowed:

pandas/pandas/core/apply.py

Lines 779 to 782 in 22de58e

	try:
	r = self.f(Series([], dtype=np.float64))
	except Exception:
	pass

Because the code doesn't fail right away but returns an unexpected empty DataFrame identical to the original, debugging this issue has been hard and time consuming.

Expected Behavior

It is reasonable to expect that both empty and non-empty dataframes should behave in the same way when it comes to errors raised inside func functions provided to apply - namely to let them propagate to the user, as it's done in case of non-empty dataframes.

Installed Versions

INSTALLED VERSIONS

commit : 73c6825
python : 3.9.0.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Tue Aug 24 20:28:00 PDT 2021; root:xnu-6153.141.40~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.3
numpy : 1.21.2
pytz : 2021.3
dateutil : 2.8.2
pip : 21.0.1
setuptools : 49.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.20.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[rhshadrach]

Thanks for the report! Related: #41997.

[gshaikov]

Thanks @rhshadrach! Will reply on that thread

[gshaikov]

Closing this in favour of #41997