OpenSSL error with Ubuntu 22.04 on Arm32 architecture

[mthalman]

When attempting to run a .NET app that makes an HTTPS connection with the upcoming release of Ubuntu 22.04 on the Arm32 architecture, it results in an OpenSSL error. This issue will block our upcoming support for Ubuntu 22.04 (see dotnet/core#7038).

This only happens in Arm32. Using Arm64 works fine. The easiest way to reproduce this is with a Docker container. But because .NET doesn't work with QEMU, emulation from an x64 machine won't allow you to repro this. You'll need an Arm machine.

I've repro'd this on my Raspberry Pi 4 machine. But it also repros on the .NET Docker team's Jetson build machines which are Arm64.

Repro

1. Get an Arm machine with Docker installed.

2. Save the following contents to a file named Dockerfile in an empty directory:

   FROM mcr.microsoft.com/dotnet/sdk:6.0-focal AS build
   
   WORKDIR dotnetapp
   
   # Create a simple project that makes an HTTPS connection
   RUN dotnet new console --no-restore
   RUN echo 'var response = await new System.Net.Http.HttpClient().GetAsync("https://www.microsoft.com"); response.EnsureSuccessStatusCode(); System.Console.WriteLine("Hello World!");' > Program.cs
   
   RUN dotnet publish -c release -o /app -r linux-arm --self-contained /p:PublishSingleFile=true
   
   
   FROM arm32v7/ubuntu:jammy
   
   # Install .NET dependencies
   RUN apt-get update \
       && apt-get install -y --no-install-recommends \
           ca-certificates \
           \
           libc6 \
           libgcc1 \
           libgssapi-krb5-2 \
           libicu67 \
           libssl1.1 \
           libstdc++6 \
           zlib1g \
       && rm -rf /var/lib/apt/lists/*
   
   WORKDIR /app
   COPY --from=build /app .
   ENTRYPOINT ["./dotnetapp"]

3. Set the current directory to the directory where the Dockerfile is located.

4. $ docker build -t test

5. $ docker run --rm test

Expected Results:

Hello World!

Actual Results:

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000BF:SSL routines::no protocols available
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in /dotnetapp/Program.cs:line 1
   at Program.<Main>(String[] args)

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

When attempting to run a .NET app that makes an HTTPS connection with the upcoming release of Ubuntu 22.04 on the Arm32 architecture, it results in an OpenSSL error. This issue will block our upcoming support for Ubuntu 22.04 (see dotnet/core#7038).

This only happens in Arm32. Using Arm64 works fine. The easiest way to reproduce this is with a Docker container. But because .NET doesn't work with QEMU, emulation from an x64 machine won't allow you to repro this. You'll need an Arm machine.

I've repro'd this on my Raspberry Pi 4 machine. But it also repros on the .NET Docker team's Jetson build machines which are Arm64.

Repro

1. Get an Arm machine with Docker installed.

2. Save the following contents to a file named Dockerfile in an empty directory:

   FROM mcr.microsoft.com/dotnet/sdk:6.0-focal AS build
   
   WORKDIR dotnetapp
   
   # Create a simple project that makes an HTTPS connection
   RUN dotnet new console --no-restore
   RUN echo 'var response = await new System.Net.Http.HttpClient().GetAsync("https://www.microsoft.com"); response.EnsureSuccessStatusCode(); System.Console.WriteLine("Hello World!");' > Program.cs
   
   RUN dotnet publish -c release -o /app -r linux-arm --self-contained /p:PublishSingleFile=true
   
   
   FROM arm32v7/ubuntu:jammy
   
   # Install .NET dependencies
   RUN apt-get update \
       && apt-get install -y --no-install-recommends \
           ca-certificates \
           \
           libc6 \
           libgcc1 \
           libgssapi-krb5-2 \
           libicu67 \
           libssl1.1 \
           libstdc++6 \
           zlib1g \
       && rm -rf /var/lib/apt/lists/*
   
   WORKDIR /app
   COPY --from=build /app .
   ENTRYPOINT ["./dotnetapp"]

3. Set the current directory to the directory where the Dockerfile is located.

4. $ docker build -t test

5. $ docker run --rm test

Expected Results:

Hello World!

Actual Results:

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000BF:SSL routines::no protocols available
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in /dotnetapp/Program.cs:line 1
   at Program.<Main>(String[] args)

Author:	mthalman
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[mthalman]

@dotnet/ncl, @vcsjones - This issue is time-sensitive in order to have support for the upcoming release of Ubuntu 22.04. I'd like to make sure it's triaged soon.

[vcsjones]

@mthalman I can reproduce it using the steps you provided on Graviton2 ARM64 instance.

I can run openssl s_client -connect www.microsoft.com:443 in the container, I end up negotiating TLSv1.3 TLS_AES_256_GCM_SHA384 with an X25519 key exchange, so I don't think it's a SECLEVEL issue.

Attached is a tcpdump (opens in Wireshark just fine). dump.bin.zip

(Learning how to get a tcpdump from a docker container, to a remote VM, to my host was a fun exercise).

[vcsjones]

Possibly worth pointing out: this docker image has OpenSSL 3.0.1 on it.

[karelz]

@mthalman is there a repro machine (the ARM64 one) we could use? I don't think we have arm32 Raspberry Pis handy for investigation :(