STARTTLS issue with outlook.com / TLS session tickets

[l6d-dev]

Hello, I am running chasquid since a couple of months now and it works great.

However, I observed that I am currently not able to receive mail from outlook.com (not sure if this ever worked in the past weeks...). So I signed up, sent some mail - the enabled debug log indeed reveals the issue (name of my mail-server redacted):

. conn.go:198        SMTP.Conn 52.103.10.11:58737: Connected, mode: SMTP
. conn.go:261        SMTP.Conn 52.103.10.11:58737: -> EHLO CO1PR03CU002.outbound.protection.outlook.com
. conn.go:314        SMTP.Conn 52.103.10.11:58737: <- 250  xxx.xxx.xxx - Your hour of destiny has come.\n8BITMIME\nPIPELINING\nSMTPUTF8\nENHANCEDSTATUSCODES\nSIZE 52428800\nSTARTTLS\nHELP\n
. conn.go:261        SMTP.Conn 52.103.10.11:58737: -> STARTTLS 
. conn.go:1102       SMTP.Conn 52.103.10.11:58737: <- 220  You experience a strange sense of peace
. conn.go:314        SMTP.Conn 52.103.10.11:58737: <- 554  5.5.0 Error in TLS handshake: EOF
_ conn.go:318        SMTP.Conn 52.103.10.11:58737: error: STARTTLS failed: 554  5.5.0 Error in TLS handshake: EOF
. conn.go:346        SMTP.Conn 52.103.10.11:58737: client closed the connection

I am not sure what to make of this 🤔 But maybe someone can confirm the issue to begin with or has an idea how to proceed.

Thanks,
Michael

[l6d-dev]

Ok, so I found this post from December '24 on some Microsoft forum where someone describes a very similar issue with a postfix setup:

https://answers.microsoft.com/en-us/outlook_com/forum/all/office-365-mail-servers-disconnecting-after/7bbe494b-029e-42a1-bf45-aa1af14541ed

His solution was to disable TLS session tickets. I replicated this and passed SessionTicketsDisabled: true to the tls.Config when initializing the server and that also solved the issue.

I am running chasquid from the Docker image which is based on debian 12 - not sure what is exactly failing but it looks like this combination has an issue with TLS session tickets.

So what would be the general solution 🤔 Have a flag to opt-in/out or turn this off generally? Performance-wise this might be neglect-able and security-wise I wonder why this enabled as default in go's tls-stack in the first place.

[albertito]

Thanks a lot for the report, and for investigating the cause Michael!

Some questions for you as I investigate this:

1. After the STARTTLS failure, you should see another attempt over plain text. Maybe not immediately, as maybe Microsoft waits to retry, but at some point. Do you see this happening at all?
   Note that if you ever received emails over TLS, chasquid's downgrade attack prevention may reject the delivery on plain text. But if Microsoft is retrying over plain text (as it should), at least that is easier to mitigate from your side, while we figure this out.

2. Which chasquid version from docker are you running? If you can give me the container id or as much detail as possible, so I can figure out things like which Go version was it built with, etc.

Thank you!

[l6d-dev]

Hey Alberto, thanks for your response.

reg. 1)

I saw that Microsoft tried several times over the course of about 24h before returning the failed delivery notification. I haven't noticed any rejections due to "downgrade attempts", however looking at the diagnostic information it reads:

3/28/2025 6:58:25 PM - Server at SJ2PR12MB9191.namprd12.prod.outlook.com returned '550 5.4.316 Message expired, connection refused(Socket error code 10061)'
3/28/2025 6:48:07 PM - Server at xxx.xxx (xxx:xxx:xxx::35a1) returned '450 4.4.316 Connection refused [Message=Socket error code 10061] [LastAttemptedServerName=xxx.xxx] [LastAttemptedIP=xxx:xxx:xxx::35a1:25] [SmtpSecurity=-2;-2] [CO1PEPF000061ED.namprd03.prod.outlook.com 2025-03-28T18:48:25.757Z 08DD6AED6FE9941B](Socket error code 10061)'

Unfortunately I could only observe this in the docker logs not the chasquid.log file and I lost it when re-building the container. I am still running a newer version with the "fixed" tls.Config, but i guess this would be fairly easy to reproduce with the original chasquid binary.

reg. 2)

The container I am running is simply based on registry.gitlab.com/albertito/chasquid:v1.15.0 which reports:

➜  ~ docker run --entrypoint chasquid registry.gitlab.com/albertito/chasquid:v1.15.0 -version
chasquid 20250117-deb2a2f22 (source date: 2025-01-17 18:10:51 +0000 UTC)

The version in which I changed the tls.Config reports as chasquid 20250201-cef7bb079-dirty (source date: 2025-02-01 10:31:06 +0000 UTC).

Regards,
Michael

[albertito]

I created a Hotmail account and could also reproduce this.

Same behaviour as described above: Microsoft starts an SMTP connection, does a STARTTLS, and then disconnects. Server-side, the failure just looks like a client-side disconnection (Error in TLS handshake: EOF). While all MX servers are attempted, Microsoft does not fallback to plain-text.

But if I set SessionTicketsDisabled: true on the server, then the next delivery attempt by Microsoft is successful.

This behaviour matches exactly what the Postfix user described in the previously linked forum article.

Looking more into this, I find more reports for the same problem (some in Postfix, some in other software):

• https://github.com/mjl-/mox/issues/237 seems to match this exactly
• Unable to receive mail from outlook servers foxcpp/maddy#730 also affects maddy
• https://www.mail-archive.com/postfix-users@postfix.org/msg104308.html also affects Postfix/OpenSSL (the whole thread is relevant)

From the Go side, this seems to be extremely well analyzed in golang/go#70232, which is the crux of the problem.

In that Go issue, there is a mention that this was escalated to Microsoft and is being worked on.

[albertito]

I have a preliminary patch to work around this on the server side (commit 85221b6) in the next branch.

I really don't like doing this, as it is enabling Microsoft's bad behaviour of not fixing their problems in a reasonable timeframe, considering they are affecting ~all Go servers and at least many based on OpenSSL too.

But given how large of a provider they are, and how minimal impact a workaround has, it is worth considering.

Tomorrow I will consider doing a point release for it, or alternatively dropping the workaround altogether.

@l6d-dev: I have included a mention thanking you for the report and workaround, but not sure how you want to be credited. Please take a look and let me know. The patch is in the next branch so I can still update it accordingly.

Thank you!

[l6d-dev]

Oh wow this is crazy, thank you for your time and dedication to tie this all together!

This also kind of demonstrates the state of email nowadays where you basically have 2 or 3 providers that define what "works" - funny enough you can receive mail from outlook.com to gmail without any problems...

Anyways, as for chasquid I am happy that you reacted so quickly and carefully documented all of this - even though the solution is just a workaround.

I don't mind being credited but the most important thing is that chasquid keeps me and others able to run our own mailservers and contribute that things stay decentralized for as long as possible - so I stay excited to deploy that release and take my nap :)

[albertito]

chasquid 1.15.1 has been released, which includes the workaround (48fe124).

It's also in the next branch so it will be included in the next major release.

I'm going to close this since the problem is mitigated. Once Microsoft fixes the issue (which is better tracked in golang/go#70232), I'll remove the workaround.

Thanks a lot!