Get a certificate chain with `crypto.get_certificate` module

[ivanov17]

SUMMARY

Currently, crypto.get_certificate module only returns a server certificate itself, but it is more useful to return all certificates sent by the server, as does the -showcerts option of openssl s_client command.

ISSUE TYPE

• Feature Idea

COMPONENT NAME

crypto.get_certificate

ADDITIONAL INFORMATION

For example, I would like to get CA certificate from a LDAP server and save it as a file in the client's trust store. If I don't already have the certificate saved on disk or in a variable, I should try to parse the command module output.

I don't think any additional options are needed here, just expanding the output.

Sample output:

ok: [example] => {
    "my_retrieved_cert": {
        "cert": "-----BEGIN CERTIFICATE-----\nxSERVERxCERTIFICATEx==\n-----END CERTIFICATE-----\n",
        "intermediate": "-----BEGIN CERTIFICATE-----\nxINTERMEDIATExCERTIFICATEx==\n-----END CERTIFICATE-----\n",
        "root": "-----BEGIN CERTIFICATE-----\nxROOTxCERTIFICATEx==\n-----END CERTIFICATE-----\n",
        "changed": false,
        "expired": false,
...

[felixfontein]

https://docs.python.org/3/library/ssl.html#ssl.SSLSocket.getpeercert (what we are using) only returns the certificate itself, and a quick glance does not show a way to get hold of the whole chain. Maybe taking a look at the source is more helpful, but the first looks I took myself seem to suggest it is just not possible.

[ivanov17]

Is it not possible to use SSLContext.get_ca_certs in the module? It's just a guess, I have not experience with this python module.

[felixfontein]

That function is available in Python 2.7 and Python 3.4+ (though not in Python 2.6, which this collection still supports), though that probably isn't a big problem. What I'm more worried about is whether this function actually does what you expect. get_ca_cert() (essentially https://github.com/python/cpython/blob/3.11/Modules/_ssl.c#L4575-L4603) uses OpenSSL's SSL_CTX_get_cert_store (https://www.openssl.org/docs/man1.1.1/man3/SSL_CTX_get_cert_store.html), which returns a list of certificates that were used while validating the connection's certificates, and adds all certificates from there that X509_check_ca (https://www.openssl.org/docs/man1.1.1/man3/X509_check_ca.html) returns true on. This is not necessarily the list of certificates sent by the server as far as I understand. It probably will work fine in quite a few cases though, but I'm not convinced (yet) that this is good enough.

[felixfontein]

I looked into the openssl s_client code, it uses SSL_get_peer_cert_chain (https://www.openssl.org/docs/man1.1.1/man3/SSL_get_peer_cert_chain.html) to get hold of the chain. Python's _ssl.c module also calls that function (https://github.com/python/cpython/blob/3.11/Modules/_ssl.c#L1868-L1899), which returns a Python list with these certificates. Apparently this function is reachable from Python as SSLSocket.get_unverified_chain (to verify whether it's actually this), but it is not described in the docs (https://docs.python.org/3/library/ssl.html#ssl.SSLSocket). In the CPython sources I can find its only use in Lib/test/test_ssl.py, and there referenced as s._sslobj.get_unverified_chain (where s seems to be a SSLSocket object). So apparently it is there, but private. It was added in python/cpython@666991f, and has a comment:

These are private APIs for now. There is a slim chance to stabilize the approach and provide a public API for 3.10. Otherwise I'll provide a stable API in 3.11.

The tracking issue for this functionality seems to be python/cpython#62433, and python/cpython#17938 is a PR providing a stable interface for this. It hasn't been rebased after python/cpython#25467 was merged though, so no idea when it will progress...

[ivanov17]

Hello, Felix. Sorry, is there any news about the public API that we can use to solve this issue? Python 3.12 was recently released.

Maybe it makes sense to try to implement obtaining information about CA certificates using the currently available library functionality? What do you think?

[felixfontein]

It looks Python 3.13 will have such functionality: python/cpython#109113 (python/cpython#109109)

[felixfontein]

Once #768 is merged there's Python 3.13 in CI that can be used to test a PR for this feature.

[felixfontein]

I've implemented this in #784. The PR has some code for Python 3.10..3.12 that uses the internal APIs, which isn't exactly clean, but works :) Unfortunately there is a bug in the Python 3.13 API right now that hopefully will get fixed soon...