Possible memory leak in Celery integration

[antonpirker]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

1.20.0

Steps to Reproduce

Have a Flask project that has multiple views where one is calling a task (ex called sometask) multiple times using the delay method: sometask.delay(some_msg). (In my test case some_msg was a string with 10,000 random characters.

Now load test this project and generate lots of hits to the view (and thus lots of calls to the task)

The memory usage will increase over time:

This is the test project I created:
https://github.com/antonpirker/test-celery-memory-leak

Expected Result

Memory usage stays roughly the same.

Actual Result

Memory usage increasing.

[antonpirker]

Ok, I did a shorter test run (10minutes) and it seems that Sentry is preventing some garbage collection:

Without Sentry:

With Sentry:

[samuelhwilliams]

Hey 👋 Thanks for investigating this after I reported some weird memory issues a while back. This definitely looks promising :) I'll try to have another dig this week and see if I can get a clearer reproduction case on our setup. Out of curiosity was this configured for just error reporting, just transaction tracing, or both?

[antonpirker]

I had both, error reporting and tracing. (and around 40% of my requests errored).

If you evaluate this again please make sure to use the lastes Sentry SDK version, because we released some fixes in the mean time.

All in all we did find a memory leak related to profiling, and also decreased memory usage in combination with redis.
But we could not find the root cause of the memory usage you reported.

[jparkrr]

I'm having a similar issue, except it's with the arq integration instead of celery. (Added in #1872). It also seems to prevent some garbage collection for me.

Do you think I should create a new issue? Could it be because of the same root cause?

This is happening with the latest sentry version, 1.24.0.

[antonpirker]

Yes please create a new issue for that @jparkrr . And it would be great if you could include a sample app to reproduce the possible memory leak

[williamjamir]

Does this issue still exist, or has it already been addressed in another PR?

asking this because this issue was created one day after this comment: #1980 (comment)