PERF: pd.DataFrame.copy() leaks

[timvantongeren]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this issue exists on the latest version of pandas.

• I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

Leakage is quite slow, but very much noticeable. Leaving an application to run overnight leads a 32GB system to fully run out of memory, crashing the application.

import pandas as pd
import numpy as np
from uuid import uuid4

index_length = 10_000
column_length = 100

index = list(range(index_length))
columns = [uuid4() for _ in range(column_length)]
data = np.random.random((index_length, column_length))
df = pd.DataFrame(data=data, index=index, columns=columns)

while True:
    # This leaks
    df2 = df.copy()

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.11.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 167 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.3
numpy : 1.25.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.5.0
pip : 22.3.1
Cython : None
pytest : 7.4.0
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.6
jinja2 : None
IPython : 8.14.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : 2023.7.0
fsspec : 2023.6.0
gcsfs : None
matplotlib : 3.7.2
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 12.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.1
snappy : None
sqlalchemy : 1.4.41
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
None

Prior Performance

This system seems to not have the leakage issue. Might be related to python 3.10 vs 3.11, but since the problem shows up in both the newer python and newer pandas version it would be a problem either way.

INSTALLED VERSIONS

commit           : 91111fd
python           : 3.10.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19045
machine          : AMD64
processor        : Intel64 Family 6 Model 151 Stepping 2, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_Netherlands.1252

pandas           : 1.5.1
numpy            : 1.23.4
pytz             : 2022.5
dateutil         : 2.8.2
setuptools       : 63.2.0
pip              : 22.2.1
Cython           : 0.29.32
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : 3.0.3
lxml.etree       : 4.9.1
html5lib         : None
pymysql          : None
psycopg2         : 2.9.5
jinja2           : 3.1.2
IPython          : 8.5.0
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
matplotlib       : 3.6.2
numba            : 0.56.3
numexpr          : None
odfpy            : None
openpyxl         : 3.0.10
pandas_gbq       : None
pyarrow          : 10.0.0
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.9.3
snappy           : None
sqlalchemy       : 1.4.47
tables           : None
tabulate         : None
xarray           : None
xlrd             : 2.0.1
xlwt             : None
zstandard        : None
tzdata           : 2022.7

[wangwillian0]

from ctypes import cdll, CDLL
from memory_profiler import profile
import pandas as pd

cdll.LoadLibrary("libc.so.6")
libc = CDLL("libc.so.6")
n_entries = 1

data = [1] * n_entries
df1 = pd.DataFrame(data)
df2 = pd.DataFrame(data=data, columns=["x"])
df3 = pd.DataFrame(data=data, columns=["x"*128])

batch_size = 50000

@profile
def try_leak(original):
    libc.malloc_trim(0)
    for _i in range(batch_size):
    	_copied = original.copy()
    libc.malloc_trim(0)
    for _i in range(batch_size):
    	_copied = original.copy()
    libc.malloc_trim(0)

if __name__ == '__main__':
    libc.malloc_trim(0)

    print("Pandas dataframe with no column label")
    try_leak(df1)

    print("Pandas dataframe with column label")
    try_leak(df2)
    
    print("Pandas dataframe with column label (long string)")
    try_leak(df3)

    print("Pandas dataframe with no column label")
    try_leak(df1)

Pandas dataframe with no column label
Filename: pd_leak.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    16    115.4 MiB    115.4 MiB           1   @profile
    17                                         def try_leak(original):
    18    115.4 MiB      0.0 MiB           1       libc.malloc_trim(0)
    19    115.4 MiB      0.0 MiB       50001       for _i in range(batch_size):
    20    115.4 MiB      0.0 MiB       50000            _copied = original.copy()
    21    115.4 MiB      0.0 MiB           1       libc.malloc_trim(0)
    22    115.4 MiB      0.0 MiB       50001       for _i in range(batch_size):
    23    115.4 MiB      0.0 MiB       50000            _copied = original.copy()
    24    115.4 MiB      0.0 MiB           1       libc.malloc_trim(0)


Pandas dataframe with column label
Filename: pd_leak.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    16    115.4 MiB    115.4 MiB           1   @profile
    17                                         def try_leak(original):
    18    115.4 MiB      0.0 MiB           1       libc.malloc_trim(0)
    19    121.3 MiB      0.0 MiB       50001       for _i in range(batch_size):
    20    121.3 MiB      5.9 MiB       50000            _copied = original.copy()
    21    119.0 MiB     -2.3 MiB           1       libc.malloc_trim(0)
    22    123.6 MiB      0.0 MiB       50001       for _i in range(batch_size):
    23    123.6 MiB      4.6 MiB       50000            _copied = original.copy()
    24    123.2 MiB     -0.5 MiB           1       libc.malloc_trim(0)


Pandas dataframe with column label (long string)
Filename: pd_leak.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    16    123.2 MiB    123.2 MiB           1   @profile
    17                                         def try_leak(original):
    18    123.2 MiB      0.0 MiB           1       libc.malloc_trim(0)
    19    129.7 MiB      0.0 MiB       50001       for _i in range(batch_size):
    20    129.7 MiB      6.5 MiB       50000            _copied = original.copy()
    21    127.4 MiB     -2.2 MiB           1       libc.malloc_trim(0)
    22    131.8 MiB      0.0 MiB       50001       for _i in range(batch_size):
    23    131.8 MiB      4.4 MiB       50000            _copied = original.copy()
    24    131.3 MiB     -0.5 MiB           1       libc.malloc_trim(0)


Pandas dataframe with no column label
Filename: pd_leak.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    16    131.3 MiB    131.3 MiB           1   @profile
    17                                         def try_leak(original):
    18    131.3 MiB      0.0 MiB           1       libc.malloc_trim(0)
    19    131.3 MiB      0.0 MiB       50001       for _i in range(batch_size):
    20    131.3 MiB      0.0 MiB       50000            _copied = original.copy()
    21    131.3 MiB      0.0 MiB           1       libc.malloc_trim(0)
    22    131.3 MiB      0.0 MiB       50001       for _i in range(batch_size):
    23    131.3 MiB      0.0 MiB       50000            _copied = original.copy()
    24    131.3 MiB      0.0 MiB           1       libc.malloc_trim(0)

Hi, trying to contribute to pandas for the first time!

I think the issue may be related to cloning of the column labels, I will continue to investigate.

Update

probably related to pandas.core.indexes.base.Index.view(), which is called when deep=True. The snippet below leaks ~10mb very quickly

def leak():
    tmp = index.view()
for _i in range(100_000):
        leak()