uasyncio-friendly outbound SSL connections?

[davidmcnabnz]

Hi,

One of my projects needs instances of MicroPython on RP2020 (Pico W modules), within uasyncio event loop, to:

• each make a socket connection to a server in the local WLAN
• wrap that connection as SSL (where each Pico presents client certificate, recognised by server)
• have uasyncio tasks running which read lines from, and write lines to, the socket, while at the same time, reading/writing GPIO pins

Is this possible currently? Or will I need to just use unencrypted uasyncio sockets for now?

[peterhinch]

A search for TLS yields many responses which you might like to study. This is something of a hot topic and I gather it can be quite difficult. In the specific case of asynchronous MQTT users have reported success and the doc gives some pointers.

[davidmcnabnz]

As an addendum, I'm now building store/forward clients to sit with the server, to give the PicoW based clients a way to "catch up on what they've missed" during WLAN outages. The idea is that on establishment, the client can publish a message with a timestamp of the last message(s) it received on its subscription(s), and the store/forward daemon(s) can re-publish as needed. Also, the client would publish a less frequent "ack" style message, allowing the store/forward daemon(s) to purge stuff that's known to have been received.

[davidmcnabnz]

@peterhinch I'm delighted to confirm that mqtt_as is working well with SSL/TLS secured sockets.

For me, I had to:

• create a private key for a self-signed CA cert
• generate a CA certificate from this
• create server key and certificate for mosquitto. with common_name set to the IP address mosquitto is reached on
• create a signing request for mosquitto's certificate
• sign mosquitto's certificate with the self-signed CA certificate
• convert the CA certificate to DER format (needed by uPython's cut-down ussl.wrap_socket())
• install server key/certificate, plus CA certificate, into mosquitto's config, and restart mosquitto
• install DER version of CA certificate onto the Pico
• on the Pico, open the DER file as binary read, then load it into a bytes object (caDerBytes)
• set mqtt_as's config['ssl_params'] with server_side=False, cert_reqs=ussl.CERT_REQUIRED, cadata=caDerBytes
• populate the config with updated details for reaching mosquitto (eg, now on port 8883)

After this, my MQTTClient object comes up, connects to Wifi, then to mosquitto, then secures the socket, then Simply Just Works over the encrypted connection.

Resources:

• https://mosquitto.org/man/mosquitto-tls-7.html
• https://docs.micropython.org/en/latest/library/ssl.html
• http://www.steves-internet-guide.com/mosquitto-tls/

Thanks so much again! :)

David

[peterhinch]

Thanks for that. I will link to this post in the docs.

As an addendum, I'm now building store/forward clients to sit with the server, to give the PicoW based clients a way to "catch up on what they've missed" during WLAN outages.

I'm probably not understanding this, but I assume you're aware of MQTT's concept of "clean sessions"? If a client sets clean session False then, after an outage, the broker will send all pending messages. As explained in the mqtt_as docs, you need to consider throughput if outages are long. There is also provision in mqtt_as for handling client power outages differently from WiFi outages since power outages might be very long indeed if a unit is downed for servicing.

[davidmcnabnz]

Thanks for that, @peterhinch . Looks like I just needed more MQTT reading.

That's the store/forward part sorted. Now, for high-availability via redundancy/replication, but that's another scenario, in my day job, well away from my local "broker, server and picos" environment :)

[davidmcnabnz]

@peterhinch there is a fairly serious surprise pitfall with Micropython's SSL, which does affect mqtt-as.

If the device is booted without a connection to Thonny, and runs its turnkey main.py, the SSL certificate verification will fail.

This is because, on boot, the default datetime for machine.RTC is hardwired to (2021, 1, 1, 4, 0, 0, 14, 0).

If the device checks a server's certificate, the verification will fail in cases where the this hardwired RTC datetime lies outside the server certificate's validity start/end dates.

There is a workaround:

from machine import RTC
rtc = RTC()
# set this to whatever datetime is within certificate window
# tuple format is (year, month, day, day_of_week, hour, minute, second, subsecond)
rtc.datetime((2023, 2, 23, 4, 22, 19, 0, 0))

Executing this before any attempt to SSL-wrap a socket will prevent date-related certificate verification failures. If the trusted CA certificate, and the server certificate, are otherwise in order, the certificate validation, and SSL socket wrapping, will succeed.

FWIW I've avoided any reliance on ntptime.settime(), because on principle, an MQTT client device really should (IMHO) be able to connect to their broker on the WLAN without being able to access the broader internet.

Reference:

• TLS/SSL module does not work on its own (Pico w) #9071

Cheers
David

[smurfix]

The basic idea is to use the SSL library as a sans-IO wrapper. In the CPython standard library that's ssl.SSLObject, created by sslcontext.wrap_bio.

In MicroPython you might be able to teach its SSL code to work with io.BytesIO objects on both ends. Then you use it like https://github.com/agronholm/anyio/blob/master/src/anyio/streams/tls.py.

[peterhinch]

FWIW I've avoided any reliance on ntptime.settime(), because on principle, an MQTT client device really should (IMHO) be able to connect to their broker on the WLAN without being able to access the broader internet.

In a previous discussion it emerged that mosquitto running on a local machine runs the NTP daemon, so you can point ntptime at your local server. Please see this doc.

[Carglglz]

@davidmcnabnz you may be interested in #8968

Following up, I've filed a bug with micropython-libs recommending they add more detail to the certificate verify failure exception, and/or update their API docs for ussl.wrap_socket().

It's not possible to know exactly the cause from the OSError code

I've also filed a bug with the underlying mbedtls library project, recommending they create extra error codes for solely date-related certificate validation failures.

They are already there, see:

X509 Verify codes
--
#define | MBEDTLS_X509_BADCERT_EXPIRED   0x01
 
#define | MBEDTLS_X509_BADCERT_REVOKED   0x02
 
#define | MBEDTLS_X509_BADCERT_CN_MISMATCH   0x04
 
#define | MBEDTLS_X509_BADCERT_NOT_TRUSTED   0x08
 
#define | MBEDTLS_X509_BADCRL_NOT_TRUSTED   0x10
 
#define | MBEDTLS_X509_BADCRL_EXPIRED   0x20
 
#define | MBEDTLS_X509_BADCERT_MISSING   0x40
 
#define | MBEDTLS_X509_BADCERT_SKIP_VERIFY   0x80
 
#define | MBEDTLS_X509_BADCERT_OTHER   0x0100
 
#define | MBEDTLS_X509_BADCERT_FUTURE   0x0200
 
#define | MBEDTLS_X509_BADCRL_FUTURE   0x0400
 
#define | MBEDTLS_X509_BADCERT_KEY_USAGE   0x0800
 
#define | MBEDTLS_X509_BADCERT_EXT_KEY_USAGE   0x1000
 
#define | MBEDTLS_X509_BADCERT_NS_CERT_TYPE   0x2000
 
#define | MBEDTLS_X509_BADCERT_BAD_MD   0x4000
 
#define | MBEDTLS_X509_BADCERT_BAD_PK   0x8000
 
#define | MBEDTLS_X509_BADCERT_BAD_KEY   0x010000
 
#define | MBEDTLS_X509_BADCRL_BAD_MD   0x020000
 
#define | MBEDTLS_X509_BADCRL_BAD_PK   0x040000
 
#define | MBEDTLS_X509_BADCRL_BAD_KEY   0x080000

and

lib/mbedtls/library/x509_crt.c
1699:    { MBEDTLS_X509_BADCERT_EXPIRED,       "The certificate validity has expired" },
1700:    { MBEDTLS_X509_BADCERT_REVOKED,       "The certificate has been revoked (is on a CRL)" },
1701:    { MBEDTLS_X509_BADCERT_CN_MISMATCH,   "The certificate Common Name (CN) does not match with the expected CN" },
1702:    { MBEDTLS_X509_BADCERT_NOT_TRUSTED,   "The certificate is not correctly signed by the trusted CA" },
1705:    { MBEDTLS_X509_BADCERT_MISSING,       "Certificate was missing" },
1706:    { MBEDTLS_X509_BADCERT_SKIP_VERIFY,   "Certificate verification was skipped" },
1707:    { MBEDTLS_X509_BADCERT_OTHER,         "Other reason (can be used by verify callback)" },
1708:    { MBEDTLS_X509_BADCERT_FUTURE,        "The certificate validity starts in the future" },
1710:    { MBEDTLS_X509_BADCERT_KEY_USAGE,     "Usage does not match the keyUsage extension" },
1711:    { MBEDTLS_X509_BADCERT_EXT_KEY_USAGE, "Usage does not match the extendedKeyUsage extension" },
1712:    { MBEDTLS_X509_BADCERT_NS_CERT_TYPE,  "Usage does not match the nsCertType extension" },
1713:    { MBEDTLS_X509_BADCERT_BAD_MD,        "The certificate is signed with an unacceptable hash." },
1714:    { MBEDTLS_X509_BADCERT_BAD_PK,        "The certificate is signed with an unacceptable PK alg (eg RSA vs ECDSA)." },
1715:    { MBEDTLS_X509_BADCERT_BAD_KEY,       "The certificate is signed with an unacceptable key (eg bad curve, RSA too short)." },

see
https://github.com/Carglglz/micropython/blob/6f9a282ea5687457ef87b3fb66694e7f9ce97e7a/extmod/modussl_mbedtls.c#L519

Also currently Micropython is using mbedtls 2.16, they are already developing mbedtls 3.x see #8988

[davidmcnabnz]

@Carglglz , I've delved into the OSError exception object and can't find any specifics apart from the VERIFY_FAILED.

Interactive console extract below:

OSError: (-9984, 'MBEDTLS_ERR_X509_CERT_VERIFY_FAILED')
MicroPython v1.19.1-869-gde1f1dd16 on 2023-02-16; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>> e = pubs['e']
>>> dir(e)
['__class__', 'args', 'value', 'errno']
>>> e.args
(-9984, 'MBEDTLS_ERR_X509_CERT_VERIFY_FAILED')
>>> e.value
-9984
>>> e.errno
-9984
>>> 

In the standard MicroPython environment, can you suggest any way for me to get hold of the actual error code as you've mentioned above? Or will I need to fork, grab the PR, build my own MicroPython and go from there?

[Carglglz]

@davidmcnabnz

In the standard MicroPython environment, can you suggest any way for me to get hold of the actual error code as you've mentioned above?

Is not possible, it requires calling mbedtls C methods mbedtls_ssl_get_verify_result() and mbedtls_x509_crt_verify_info(). So

Or will I need to fork, grab the PR, build my own MicroPython and go from there?

This is the way to go if you want to know the exact cause of the error, so far I've tested the cases of expired, starts in the future, Common Name does not match and not correctly signed by the trusted CA, which I think are the most common cases. So I know those at least are working...

Also note that I've not tested this on the Pico W (only esp32 and unix ports), but I don't see any reason why it wouldn't work.
Finally note that you will need to use ssl.wrap_socket not ussl.wrap_socket, so if you are using @peterhinch's mqtt_as library you will need to change this

        if self._ssl:
            import ussl

            self._sock = ussl.wrap_socket(self._sock, **self._ssl_params)

to this

        if self._ssl:
            import ssl

            self._sock = ssl.wrap_socket(self._sock, **self._ssl_params)

Let me know if you decide to test my PR and wether it works or not 👍🏼

Also in tests/net_inet/ there are a some tests you could upload to the pico and see if they run or not before trying mqtt_as ...
e.g. tests/net_inet/ssl_cert.py is the basic one.

[peterhinch]

Finally note that you will need to use ssl.wrap_socket not ussl.wrap_socket

Please can you explain this? Does this apply to all platforms? I thought the names were aliases.

[Carglglz]

@peterhinch, to test my PR yes (ports that use mbedtls), but I could change that, e.g. rename mp_module_ussl to mp_module__ssl so it will appear as _ssl in help("modules") and ssl.py to ussl.py, although I don't know if that would work, do u aliases work with .py modules too? 🤔, also see #9018