PERF cache find_stack_level

[MarcoGorelli]

This would be a precursor to #47998 and #47828. If repeatedly calling find_stack_level, then performance takes a hit

Demo of impact: if we run the file

import cProfile
import pstats, cProfile

import re
import pandas as pd

data = pd.date_range('1990', '2000').strftime('%d/%m/%Y').tolist()

def fun():
    pd.to_datetime(data, dayfirst=False)

cProfile.runctx("fun()", globals(), locals(), "Profile.prof")

s = pstats.Stats("Profile.prof")
s.strip_dirs().sort_stats("time").print_stats()

On main:

#         1199 function calls (1187 primitive calls) in 0.046 seconds
#
#   Ordered by: internal time
#
#   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#        1    0.034    0.034    0.035    0.035 {built-in method pandas._libs.tslib.array_to_datetime}
#      262    0.001    0.000    0.001    0.000 {built-in method builtins.isinstance}
#      2/1    0.000    0.000    0.006    0.006 series.py:342(__init__)
#        2    0.000    0.000    0.000    0.000 {built-in method numpy.array}
#        1    0.000    0.000    0.046    0.046 {built-in method builtins.exec}
#        1    0.000    0.000    0.001    0.001 sre_parse.py:493(_parse)
#        1    0.000    0.000    0.046    0.046 datetimes.py:702(to_datetime)
#      2/1    0.000    0.000    0.000    0.000 sre_compile.py:71(_compile)
#      2/1    0.000    0.000    0.001    0.001 base.py:431(__new__)
#        2    0.000    0.000    0.000    0.000 {pandas._libs.lib.maybe_convert_objects}

Using find_stack_level in

pandas/pandas/_libs/tslibs/parsing.pyx

Line 217 in 6db95e7

stacklevel=4,

and

pandas/pandas/_libs/tslibs/parsing.pyx

Line 225 in 6db95e7

stacklevel=4,

#         224611 function calls (224599 primitive calls) in 0.582 seconds
#
#   Ordered by: internal time
#
#   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#    11060    0.106    0.000    0.342    0.000 inspect.py:655(getfile)
#    82106    0.081    0.000    0.082    0.000 {built-in method builtins.isinstance}
#     2212    0.063    0.000    0.517    0.000 _exceptions.py:30(find_stack_level)
#        1    0.045    0.045    0.572    0.572 {built-in method pandas._libs.tslib.array_to_datetime}
#    22128    0.027    0.000    0.027    0.000 {method 'startswith' of 'str' objects}
#    11060    0.023    0.000    0.035    0.000 inspect.py:64(ismodule)
#    11060    0.023    0.000    0.034    0.000 inspect.py:81(ismethod)
#    11060    0.023    0.000    0.033    0.000 inspect.py:261(iscode)
#    11060    0.023    0.000    0.034    0.000 inspect.py:237(istraceback)
#    11060    0.023    0.000    0.034    0.000 inspect.py:159(isfunction)

Using find_stack_level in the same places as above, but with caching (as in this branch):

#         7916 function calls (7904 primitive calls) in 0.065 seconds
#
#   Ordered by: internal time
#
#   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#        1    0.031    0.031    0.056    0.056 {built-in method pandas._libs.tslib.array_to_datetime}
#     2212    0.018    0.000    0.024    0.000 inspect.py:1546(currentframe)
#     2231    0.003    0.000    0.003    0.000 {built-in method builtins.hasattr}
#     2212    0.003    0.000    0.003    0.000 {built-in method sys._getframe}
#      292    0.001    0.000    0.001    0.000 {built-in method builtins.isinstance}
#        1    0.000    0.000    0.065    0.065 {built-in method builtins.exec}
#      2/1    0.000    0.000    0.005    0.005 series.py:342(__init__)
#        1    0.000    0.000    0.001    0.001 sre_parse.py:493(_parse)
#        2    0.000    0.000    0.000    0.000 {built-in method numpy.array}
#    49/41    0.000    0.000    0.000    0.000 {built-in method builtins.len}

[alimcmaster1]

Nice - LGTM!

[lithomas1]

Generally LGTM. I'm just a little uneasy at the number of changes the change to the function signature caused.

[mroeschke]

Thanks @MarcoGorelli

[jbinary]

Good day,
This change leads to some memory leaks that are really hard to debug becuase @lru_cache stores stack frames for the sake of look ups and those frames refer all the variables of that frame, so garbage collector will never remove them from memory because of that? Since pandas is very likely to be called from frames with extensive memory use, that may lead to significant memory overheads?
Thanks.

[MarcoGorelli]

Thanks @jbinary for the report

I'd be fine with reverting this - as of PDEP0004, we're no longer going with #47828, so repeatedly calling find_stacklevel is less of an issue

opening a PR soon

[Expost]

Good day, This change leads to some memory leaks that are really hard to debug becuase @lru_cache stores stack frames for the sake of look ups and those frames refer all the variables of that frame, so garbage collector will never remove them from memory because of that? Since pandas is very likely to be called from frames with extensive memory use, that may lead to significant memory overheads? Thanks.

Coincidentally. The day before yesterday I spent nearly two days to locate an OOM problem, and finally found that it was caused by the lru_cache of the find_stack_level method.

[MarcoGorelli]

Thanks for reporting, version 1.5.1 (probably coming out Monday) will fix this