Fedora messaging CA cert is failing OpenSSL strict verification

[mh21]

With Python 3.13, the default SSL context enabled strict verification [1][2]:

Changed in version 3.13: The context now uses VERIFY_X509_PARTIAL_CHAIN and VERIFY_X509_STRICT in its default verify flags.

This can also be checked via openssl directly, eg via openssl verify -x509_strict.

The fedora-messaging CA certificate fails this check, which causes a naive connection to the messaging cluster via pika fail on Python >=3.13 with [3]

Attempt to create the streaming transport failed: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Basic Constraints of CA cert not marked critical (_ssl.c:1028)'); 'rabbitmq.fedoraproject.org'/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('38.145.60.20', 5671)); ssl=True

This can be confirmed with the openssl CLI:

$ openssl verify -x509_strict -CAfile /tmp/fedora-cacert.pem /tmp/fedora-cert.pem 
CN=RabbitMQ PRODUCTION CA
error 89 at 1 depth lookup: Basic Constraints of CA cert not marked critical
error /tmp/fedora-cert.pem: verification failed

Expected outcome: the CA cert is following x509 best practices as embodied in the x509_strict checks.

[1] python/cpython#107361
[2] https://docs.python.org/3/library/ssl.html#ssl.create_default_context
[3] https://gitlab.com/cki-project/cki-tools/-/issues/205

[Zlopez]

It seems that the newer OpenSSL needs critical constraint as well. I'm encountering this issue when trying to setup some project locally.

This should be easily solved by recreating the certs with critical constraint enabled.

[abompard]

Unfortunately, it looks like the CA needs to be entirely regenerated, because the CA cert is also missing the critical extention on Basic Constraints. This means regenerating all the certs too, and re-deploying. When the new certs are in place, any users of the current version will stop working until they upgrade. It would also be a problem for our applications, as they also use certificates to connect.
I don't see how to keep things working. Any idea? Maybe @jeremycline ?

[jeremycline]

Unfortunately, it looks like the CA needs to be entirely regenerated, because the CA cert is also missing the critical extention on Basic Constraints. This means regenerating all the certs too, and re-deploying. When the new certs are in place, any users of the current version will stop working until they upgrade. It would also be a problem for our applications, as they also use certificates to connect. I don't see how to keep things working. Any idea? Maybe @jeremycline ?

Hmm, that is tricky.

It looks like in addition to VERIFY_X509_STRICT, VERIFY_X509_PARTIAL_CHAIN was also enabled, which will cause verification to succeed even if the CA is not self-signed (e.g. an intermediate certificate authority).

If we generate a new CA with the correct settings, but sign it with our old CA (creating an intermediate CA), and then adjust the fedora-messaging TLS context so that instead of just the root CA, it contains the intermediate CA and root CA, that might work for both old and new clients. I've not tested this, so it's speculation at this point.

Also, while it may be that the client certificates are signed by the same CA, I don't think they need to be, or that it will matter; as long as RabbitMQ is still happy clients should still successfully authenticate. That's not to say they shouldn't be rotated out to be signed by a CA that passes -x509_strict, but I believe we don't have to worry about a flag day where everything gets rotated at once.

[abompard]

Good news! According to RabbitMQ's docs, the server can trust mutiple CAs. They just have to be concatenated in the CA file. So we can just create a new CA with the correct parameters, re-issue the certs in the new CA, and keep the old CA trusted by RabbitMQ for a while to cover existing clients.
That should work, right?

[jeremycline]

Much simpler than my work-around, yes!

[nirik]

Thats a file solution as long as our old version of rabbitmq supports it. :)

(But as part of the DC move I am hoping to move us to a newer version)