Use-after-free due to race between SSLContext.set_alpn_protocols and opening a connection

[dgrisby]

Bug report

Bug description:

I am not completely sure whether this should be considered a bug in Python, or urllib3, or the code using them, but I encountered a use-after-free related to the set_alpn_protocols method of SSLContext. I saw it with production code using urllib3 via requests, but here is a reproducer using just standard library modules. It assumes an HTTPS server running on localhost:

import ssl
import threading
import http.client

context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

class Test(threading.Thread):
    def run(self):
        while True:
            context.set_alpn_protocols(['http/1.1'])

            conn = http.client.HTTPSConnection("localhost", context=context)
            conn.request("GET", "/")
            response = conn.getresponse()
            conn.close()


threads = [ Test() for _ in range(8) ]
for t in threads:
    t.start()

Run it under AddressSanitizer. You don't actually need to compile with ASAN to detect the problem, because its normal heap checking catches the problem. On my system it runs for a few minutes before it fails. This is from Python 3.11 where I have somewhat useful debugging symbols, but I've seen it with 3.13, and I don't think anything has changed since then.

$ LD_PRELOAD=/usr/lib64/libasan.so.8 python3 alpn_test.py
=================================================================
==229438==ERROR: AddressSanitizer: heap-use-after-free on address 0x7b9ae6766350 at pc 0x7f7ae7ce4937 bp 0x7b7ad33ea950 sp 0x7b7ad33ea110
READ of size 9 at 0x7b9ae6766350 thread T3
    #0 0x7f7ae7ce4936 in memcpy (/usr/lib64/libasan.so.8+0xe4936) (BuildId: 10b8ccd49f75c21babf1d7abe51bb63589d8471f)
    #1 0x7f7ae674283e in ossl_ssl_connection_new_int (/lib64/libssl.so.3+0x1a83e) (BuildId: 9618498bf75fd2ec21ade1eb4d21b85902c78f54)
    #2 0x7f7ae7be3f08 in newPySSLSocket /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/_ssl.c:830
    #3 0x7f7ae7be45a5 in _ssl__SSLContext__wrap_socket_impl /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/_ssl.c:4236
    #4 0x7f7ae7be45a5 in _ssl__SSLContext__wrap_socket /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/clinic/_ssl.c.h:687
    #5 0x7f7ae77ea4d0 in method_vectorcall_FASTCALL_KEYWORDS Objects/descrobject.c:426
    #6 0x7f7ae77e59a7 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
    #7 0x7f7ae77e59a7 in PyObject_Vectorcall Objects/call.c:299
    #8 0x7f7ae78438c9 in _PyEval_EvalFrameDefault Python/ceval.c:4774
    #9 0x7f7ae7842556 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:73
    #10 0x7f7ae7842556 in _PyEval_Vector Python/ceval.c:6439
    #11 0x7f7ae77e7185 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
    #12 0x7f7ae77e7185 in method_vectorcall Objects/classobject.c:67
    #13 0x7f7ae78f627f in thread_run Modules/_threadmodule.c:1092
    #14 0x7f7ae78d91a3 in pythread_wrapper Python/thread_pthread.h:241
    #15 0x7f7ae7c28ee5 in asan_thread_start(void*) (/usr/lib64/libasan.so.8+0x28ee5) (BuildId: 10b8ccd49f75c21babf1d7abe51bb63589d8471f)
    #16 0x7f7ae747ff53 in start_thread (/lib64/libc.so.6+0x71f53) (BuildId: 48c4b9b1efb1df15da8e787f489128bf31893317)
    #17 0x7f7ae750332b in __clone3 (/lib64/libc.so.6+0xf532b) (BuildId: 48c4b9b1efb1df15da8e787f489128bf31893317)

0x7b9ae6766350 is located 0 bytes inside of 9-byte region [0x7b9ae6766350,0x7b9ae6766359)
freed by thread T7 here:
    #0 0x7f7ae7ce5beb in free.part.0 (/usr/lib64/libasan.so.8+0xe5beb) (BuildId: 10b8ccd49f75c21babf1d7abe51bb63589d8471f)
    #1 0x7f7ae67416f0 in SSL_CTX_set_alpn_protos (/lib64/libssl.so.3+0x196f0) (BuildId: 9618498bf75fd2ec21ade1eb4d21b85902c78f54)
    #2 0x7f7ae7be2109 in _ssl__SSLContext__set_alpn_protocols_impl /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/_ssl.c:3378
    #3 0x7f7ae7be2109 in _ssl__SSLContext__set_alpn_protocols /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/clinic/_ssl.c.h:507
    #4 0x7b7ae4b1d4ef  (<unknown module>)

previously allocated by thread T3 here:
    #0 0x7f7ae7ce6f2b in malloc (/usr/lib64/libasan.so.8+0xe6f2b) (BuildId: 10b8ccd49f75c21babf1d7abe51bb63589d8471f)
    #1 0x7b7ad5b32b4d in CRYPTO_malloc (/lib64/libcrypto.so.3+0x132b4d) (BuildId: bb6058f574b5116601d7bf1eaa40513e2b781e36)
    #2 0x7b7ad5b32e6b in CRYPTO_memdup (/lib64/libcrypto.so.3+0x132e6b) (BuildId: bb6058f574b5116601d7bf1eaa40513e2b781e36)
    #3 0x7f7ae67416cf in SSL_CTX_set_alpn_protos (/lib64/libssl.so.3+0x196cf) (BuildId: 9618498bf75fd2ec21ade1eb4d21b85902c78f54)
    #4 0x7f7ae7be2109 in _ssl__SSLContext__set_alpn_protocols_impl /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/_ssl.c:3378
    #5 0x7f7ae7be2109 in _ssl__SSLContext__set_alpn_protocols /usr/src/debug/tw-python3-3.11.4-4.el9.x86_64/Modules/clinic/_ssl.c.h:507
    #6 0x7b7ae4b1f06f  (<unknown module>)

What is happening is that the threads are sharing the SSLContext object. The call to set_alpn_protocols() replaces the ALPN data in the underlying OpenSSL SSL_CTX. It frees the previously-set value. It does that while holding the GIL. Simultaneously, another thread uses the same SSL_CTX, while not holding the GIL, as it opens a new connection. It can therefore read the data freed by another thread.

This is particularly a problem related to urllib3, because that always calls set_alpn_protocols(), even if it is given an existing SSLContext. Perhaps it should be considered a problem there, rather than a core Python bug.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[kumaraditya303]

Can you try it on latest 3.13 release? I think #134698 was supposed to fix this

cc @ZeroIntensity

[dgrisby]

The problem still occurs with 3.13.9, which I believe includes that fix.

The locking added there does not help in this case, because the call to SSL_CTX_set_alpn_protos is holding the GIL but not the tstate_mutex that was added for the fix: https://github.com/python/cpython/blob/main/Modules/_ssl.c#L3918

It is therefore still possible for one thread to be mutating the ALPN data at the same time another thread is trying to use it.