Memory leak with Python 3.11.2

[scotloach]

Describe the bug

I am seeing a memory leak with aiohttp using Python 3.11.2.

When I downgrade to Python 3.11.0, it does not leak.

This is a script that does a large number of client requests to a large number of different servers. The script leaks hundreds of MB over the course of a few hours. I haven't yet gotten this to a form that can be easily reproduced and shared, but I thought I would get this entered in case anyone else is seeing similar symptoms.

To Reproduce

The script I am running creates about 700 https client sessions to different servers. Each session does a simple GET request, which most succeed but some fail in various ways. The script sleeps for 5 seconds and then repeats this.

Expected behavior

This should not leak memory over time. And it doesn't with 3.11.0. (I don't currently have an easy way to run it with 3.11.1).

Logs/tracebacks

n/a

Python Version

Python 3.11.2

aiohttp Version

Name: aiohttp
Version: 3.8.4
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: /usr/lib64/python3.11/site-packages
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl

multidict Version

Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/lib64/python3.11/site-packages
Requires: 
Required-by: aiohttp, yarl

yarl Version

Name: yarl
Version: 1.8.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/local/lib64/python3.11/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

Fedora release 37 (Thirty Seven)

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[Dreamsorcerer]

If the memory leak appears only in a specific patch version of Python, I suspect the bug report will need to go to cpython.
If you can profile the memory to atleast figure out which objects are causing the memory leak, that would certainly help figure out the cause.

[Dreamsorcerer]

Also, sharing the script would be useful.

[scotloach]

tracemalloc profile after running for awhile: (This is python 3.11.3 which also exhibits the leak)
/usr/lib64/python3.11/asyncio/sslproto.py:274: size=349 MiB, count=2788, average=128 KiB /usr/lib64/python3.11/asyncio/sslproto.py:374: size=14.7 MiB, count=241614, average=64 B /usr/lib64/python3.11/asyncio/sslproto.py:343: size=2156 KiB, count=2788, average=792 B /usr/lib64/python3.11/site-packages/aiohttp/connector.py:598: size=1505 KiB, count=112, average=13.4 KiB /usr/lib64/python3.11/asyncio/sslproto.py:305: size=1035 KiB, count=2788, average=380 B /usr/lib64/python3.11/site-packages/aiohttp/connector.py:666: size=1015 KiB, count=40, average=25.4 KiB /usr/lib64/python3.11/site-packages/aiohttp/client_proto.py:213: size=525 KiB, count=7042, average=76 B /usr/lib64/python3.11/site-packages/aiohttp/streams.py:571: size=507 KiB, count=1366, average=380 B /usr/lib64/python3.11/site-packages/aiohttp/connector.py:365: size=467 KiB, count=42, average=11.1 KiB

This doesn't accurately reflect the heap sizes I see, but I suspect that second line with 241614 allocations is the issue.
This is with long running ClientSession objects. If I close the ClientSession objects and re-initialize them after each request, it doesn't leak.

This is the main body of the script I'm using to reproduce it; it's loading a set of around 700 private IP endpoints which respond in a variety of different ways.

def init():
    timeout = aiohttp.ClientTimeout(connect=5, sock_read=5)
    ssl_ctx = ssl._create_unverified_context()
    conn = aiohttp.TCPConnector(ssl_context=ssl_ctx, enable_cleanup_closed=True)
    session = aiohttp.ClientSession(connector=conn, timeout=timeout, cookie_jar=aiohttp.CookieJar(unsafe=True))
    return session
    
async def fetch(client, ip):
    try:
        async with client.request('GET', url=f'https://{ip}:8000/status.cgi') as r:
            msg = await r.text()
    except asyncio.CancelledError:
        raise
    except Exception as err:
        pass

async def main(): 
    ips = [...] # load IPs to poll here
    clients = {ip: init() for ip in ips}
    tracemalloc.start()
    try:
        while True:
            await asyncio.gather(*[fetch(client, ip) for ip, client in clients.items()])
            await asyncio.sleep(5)
    except asyncio.CancelledError:
        pass  # end and clean things up
    finally:
        memory_used = tracemalloc.get_traced_memory()
        snapshot = tracemalloc.take_snapshot()
        stats = snapshot.statistics('lineno')
        for stat in stats[:10]:
            print(stat)
        try:
            await asyncio.gather(*[client.close() for client in clients.values()])
        except:
            pass

asyncio.run(main()) 

[scotloach]

I have an update. It seems that removing the enable_cleanup_closed=True from my test script, has resolved the leak.

I know I had to put that in at some earlier time to prevent a memory leak. I wonder if something has changed in recent Python versions that has fixed the cause of this, and now that flag ends up causing a leak.

[Dreamsorcerer]

I'm not familiar with the code, but at a glance it just appends to a list:
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L314
And create a weakref:
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L363

Maybe check the size of that list? I think your tracealloc might be indicating something with that too. No idea why a cpython change would affect the behaviour though.

[bdraco]

I'm seeing the leak as well with enable_cleanup_closed=True

I see a lot of _SSLProtocolTransport objects in memory with no socket attached to them.

2023-05-12 23:34:58.325 CRITICAL (MainThread) [homeassistant.components.profiler] _SSLProtocolTransport <asyncio.sslproto._SSLProtocolTransport object at 0x7fa1382d6210> ssl_proto=None ssl_object=<ssl.SSLObject object at 0x7fa156521650> peercert={'su
bject': ((('commonName', 'new-home.sensibo.com'),),), 'issuer': ((('countryName', 'US'),), (('organizationName', 'Amazon'),), (('commonName', 'Amazon RSA 2048 M02'),)), 'version': 3, 'serialNumber': '0FFA0B652D4CB4ADA67C967B5136739D', 'notBefore': 'F
eb 23 00:00:00 2023 GMT', 'notAfter': 'Mar 23 23:59:59 2024 GMT', 'subjectAltName': (('DNS', 'new-home.sensibo.com'), ('DNS', 'home.sensibo.com')), 'OCSP': ('http://ocsp.r2m02.amazontrust.com',), 'caIssuers': ('http://crt.r2m02.amazontrust.com/r2m02.
cer',), 'crlDistributionPoints': ('http://crl.r2m02.amazontrust.com/r2m02.crl',)} sock=None

[bdraco]

python/cpython#98539 looks related

[bdraco]

python/cpython#98540

[bdraco]

related elastic/rally#1712

[bdraco]

related elastic/rally#1714

[webknjaz]

related elastic/rally#1712

Looks like this was backported via python/cpython@bd8b32b?
Maybe 3.11 nightly has the fix already?

[Dreamsorcerer]

It was backported the same day that 3.11.0 was released. So, I think the suspicion is that the "fix" may actually be causing the memory leak? Because it must have been in 3.11.1+, but probably just missed the 3.11.0 release, which aligns with the reports.