Memory leak using SslStream in dotnet9 running on Linux

[heppth]

Description

The use of SslStream in dotnet9 leads to a significant increase in memory consumption under Linux and results in OOM.

Reproduction Steps

I have written a small test application that creates and disposes SslStreams. The code leads to following memory increase when executed multiple times.

void StartTest(int testRunCount)
{
    for (int i = 0; i < testRunCount; i++)
    {
        var server = RunServerUnitMessageOrTimeout();
        var client = RunClientAndSendMessageToServer();
        Task.WaitAll(server, client);
    }
}

async Task RunServerUnitMessageOrTimeout()
{
    var listener = new TcpListener(new IPEndPoint(IPAddress.Loopback, 9999));
    listener.Start();

    using var client = await listener.AcceptTcpClientAsync();
    using var networkStream = client.GetStream();
    using var secureStream = new SslStream(networkStream);
    await secureStream.AuthenticateAsServerAsync(cert, false, false);

    using var reader = new StreamReader(secureStream);
    var line = await reader.ReadLineAsync();

    listener.Stop();
}

async Task RunClientAndSendMessageToServer()
{
    using var client = new TcpClient();
    await client.ConnectAsync(new IPEndPoint(IPAddress.Loopback, 9999));

    using var networkStream = client.GetStream();
    using var secureStream = new SslStream(networkStream);
    await secureStream.AuthenticateAsClientAsync(new SslClientAuthenticationOptions
    {
        TargetHost = "localhost",
        RemoteCertificateValidationCallback = (object sender, X509Certificate? certificate, X509Chain? chain, SslPolicyErrors sslPolicyErrors) => true
    });

    using var writer = new StreamWriter(secureStream);
    await writer.WriteLineAsync("This is a test");
}

This is the output of the RAM after every 1000 runs

2024-12-18 07:35:02 Current memory usage: 60.76MB
2024-12-18 07:35:06 Current memory usage: 86.65MB
2024-12-18 07:35:07 Current memory usage: 87.02MB
2024-12-18 07:35:11 Current memory usage: 105.05MB
2024-12-18 07:35:12 Current memory usage: 105.06MB
2024-12-18 07:35:16 Current memory usage: 117.52MB
2024-12-18 07:35:17 Current memory usage: 117.54MB
2024-12-18 07:35:21 Current memory usage: 129.53MB
2024-12-18 07:35:22 Current memory usage: 129.53MB
2024-12-18 07:35:26 Current memory usage: 142.07MB
2024-12-18 07:35:27 Current memory usage: 142.07MB
2024-12-18 07:35:31 Current memory usage: 155.43MB
2024-12-18 07:35:32 Current memory usage: 155.43MB
2024-12-18 07:35:35 Current memory usage: 167.96MB
2024-12-18 07:35:36 Current memory usage: 167.95MB
2024-12-18 07:35:40 Current memory usage: 181.02MB
2024-12-18 07:35:41 Current memory usage: 181.04MB
2024-12-18 07:35:45 Current memory usage: 194.40MB

Expected behavior

Since all resources are cleaned up properly, it is expected that the memory usage does not increase.

Actual behavior

Massive increase in memory usage under Linux

Regression?

With dotnet 6 the memory consumption is stable:

Current memory usage: 45.82MB
Current memory usage: 50.00MB
Current memory usage: 50.00MB
Current memory usage: 50.00MB
Current memory usage: 50.00MB
Current memory usage: 50.58MB
Current memory usage: 50.58MB

With dotnet8, memory consumption increases much more slowly then dotnet9.

Known Workarounds

No response

Configuration

• dotnet version: 9.0.0
• os: linux default docker image: mcr.microsoft.com/dotnet/runtime:9.0
• architecture: x64

The problem is not reproducible under Windows.
Changing the Linux distribution (e.g. alpine) makes no difference

Other information

Entire test application:
SslStreamMemoryLeak.zip

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[janvorli]

cc: @rzikm

[rzikm]

Looks like we are leaking SSL_CTX objects, possibly due to a bug in a caching mechanism?

Disabling TLS Resume (either via EnableTlsResume=false on the ssl options, or by setting DOTNET_SYSTEM_NET_SECURITY_DISABLETLSRESUME=1) should work around the issue until we release a fix

[rzikm]

I see, we regressed the case when the server certificate is not provided via SslStreamCertificateContext (but only using the ServerCertificate).

So better workaround would be refactoring the code to construct a single SslStreamCertificateContext beforehand and use the AuthenticateAsServerAsync overload accepting SslServerAuthenticationOptions and pass it in ServerCertificateContext (or just recycle the options object). This will avoid this bug and improve perf by voiding the cost of building the cert context per connection.

[heppth]

Thank you very much @rzikm! Your first workaround with the environment variable DOTNET_SYSTEM_NET_SECURITY_DISABLETLSRESUME=1 works.

I followed the recommendation and refactored my code so that I now use a shared SslStreamCertificateContext.

Question: How did you find this out? We analyzed memory dumps for days and didn't find anything. How did you discover the leak?

[rzikm]

Question: How did you find this out? We analyzed memory dumps for days and didn't find anything. How did you discover the leak?

While working on this repo I had enough opportunities to learn how to debug memory leaks.

In short, first step is to figure out whether the leak is in native code or managed code, If you see the amount of private bytes of the process increase, while the GC heap size stays more or less the same, then you are most likely looking at native memory leak.

Note: Sometimes, you can find safe handle instances on the GC heap in the (managed) memory dump, which would be keeping the native memory alive (i.e. a few bytes large SafeHandle can reference objects of several kB of native memory), I did not check if this is the case here. SafeHandles are general concept for referencing native memory which makes sure the memory gets properly released (in objects finalizer at the latest).

Anyway, for investigating native leaks, I like to use the heaptrack tool which will hook malloc and free API calls and track if they are paired correctly, and is able to give callstacks of the leaked allocations, for leaks in System.Net.Security space, the native code is generally allocated from OpenSSL, in this case, the SSL_CTX_new() function, which is the factory function for SSL_CTX object.

The rest comes from knowing the implementation details of SslStream