Overhaul - Update to build with ESP-IDF 4.4.5

[dlizotte-uwo]

I wasn't able to connect to my work wpa2-enterprise network with version 1.7.4 and I started investigating and one thing led to another and I ported the whole thing from esp-idf 3.3.1 to esp-idf 4.4.5 because it literally refused to build on my M2 air. WPA2 enterprise (and regular) works; I have not tested bluetooth. I'm calling the new version 2.0.0alpha because it's a pretty substantial overhaul - the whole build system changed from 3 to 4 and now uses cmake. I also moved the WiFi and SPIS libraries into a minimal arduino "component," (as they are called in esp-idf) with supporting code copied in from the arduino-esp32 component. I don't know if the risk/reward is worth it for the adafruit/nina-fw maintainers and users to integrate this change; happy to discuss.

I have run all of the esp32spi examples from that repo using an AirLift Featherwing into a PyGamer; output is below.

Edit: I discovered later that the existing nina-fw can't open more than one TLS session at a time, due to memory constraints. Upgrading to esp-idf 4.4.x enables configuration of mbedtls to use dynamic buffers, which lets at least two (probably several I haven't tested for max) sessions to be opened reliably. Could be useful for more advanced users on like a Metro M7 Airlift. (I'm using it on a MatrixPortal M4 to have mqtt open and occasionally fetch data from other places.) I have a fork of Adafruit_CircuitPython_ESP32SPI that enables this.

Test notes

esp32spi_aio_post.py

I forgot to create a 'test' feed in my adafruit IO account but it connects fine.

ESP32 SPI webclient test
Posting data...{'error': 'not found - API documentation can be found at https://io.adafruit.com/api/docs'}
OK
Posting data...{'error': 'not found - API documentation can be found at https://io.adafruit.com/api/docs'}
OK
Posting data...{'error': 'not found - API documentation can be found at https://io.adafruit.com/api/docs'}
OK

esp32spi_cheerlights.py

ESP32 SPI webclient test
Fetching json from https://api.thingspeak.com/channels/1417/feeds.json?results=1
Failed to connect, retrying
 ('Failed to connect to ssid', b'internet')
{'channel': {'id': 1417, 'latitude': '40.5', 'name': 'CheerLights', 'longitude': '-80.22', 'updated_at': '2021-12-10T03:36:08Z', 'description': 'CheerLights is an “Internet of Things” project created by Hans Scharler that allows people’s lights all across the world to synchronize to one color set by Twitter. This is a way to connect physical things with social networking experiences.', 'last_entry_id': 972968, 'field2': 'CheerLights HEX Color', 'field1': 'Last CheerLights Command', 'created_at': '2011-12-01T03:30:04Z'}, 'feeds': [{'created_at': '2023-07-04T01:37:00Z', 'field2': '#ffc0cb', 'field1': 'pink', 'entry_id': 972968}]}
[{'created_at': '2023-07-04T01:37:00Z', 'field2': '#ffc0cb', 'field1': 'pink', 'entry_id': 972968}]
{'created_at': '2023-07-04T01:37:00Z', 'field2': '#ffc0cb', 'field1': 'pink', 'entry_id': 972968}
#ffc0cb

esp32spi_ipconfig.py

set hostname: esp32-spi-hostname-test
ESP32 found and in idle mode
Firmware vers. bytearray(b'2.0.0alpha\x00')
MAC addr: ['0x80', '0xab', '0x58', '0x93', '0x17', '0x44']
Connecting to AP...
could not connect to AP, retrying:  ('Failed to connect to ssid', b'internet')
Connected to internet 	RSSI: -75
set ip dns
set ip config
get network_data:  192.168.0.111 192.168.0.1 255.255.255.0
ip: 192.168.0.111
My IP address is 192.168.0.111
udp in addr:  192.168.0.1 5500
connected local UDP

esp32spi_localtime.py

ESP32 local time
Fetching json from http://worldtimeapi.org/api/ip
Failed to connect, retrying
 ('Failed to connect to ssid', b'internet')
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=35, tm_wday=1, tm_yday=184, tm_isdst=True)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=35, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=36, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=37, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=38, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=39, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=40, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=41, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=42, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=43, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=44, tm_wday=0, tm_yday=184, tm_isdst=-1)
struct_time(tm_year=2023, tm_mon=7, tm_mday=3, tm_hour=21, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=184, tm_isdst=-1)

esp32spi_simpletest.py

ESP32 SPI webclient test
ESP32 found and in idle mode
Firmware vers. bytearray(b'2.0.0alpha\x00')
MAC addr: ['0x80', '0xab', '0x58', '0x93', '0x17', '0x44']
	1000guests		RSSI: -54
	internet		RSSI: -55
	BELL682		RSSI: -59
	YouBrokeMyBrush		RSSI: -69
	BELL682		RSSI: -71
	two guys, 1 basement		RSSI: -83
	1000guests		RSSI: -84
	internet		RSSI: -85
	_VelopSetupD5C		RSSI: -86
	BELL524		RSSI: -87
Connecting to AP...
could not connect to AP, retrying:  ('Failed to connect to ssid', b'internet')
Connected to internet 	RSSI: -84
My IP address is 192.168.0.20
IP lookup adafruit.com: 104.20.39.240
Ping google.com: 11 ms
Fetching text from http://wifitest.adafruit.com/testwifi/index.html
----------------------------------------
This is a test of Adafruit WiFi!
If you can read this, its working :)
----------------------------------------

Fetching json from http://api.coindesk.com/v1/bpi/currentprice/USD.json
----------------------------------------
{'time': {'updated': 'Jul 4, 2023 01:51:00 UTC', 'updatedISO': '2023-07-04T01:51:00+00:00', 'updateduk': 'Jul 4, 2023 at 02:51 BST'}, 'disclaimer': 'This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org', 'bpi': {'USD': {'code': 'USD', 'description': 'United States Dollar', 'rate_float': 31276.1, 'rate': '31,276.0832'}}}
----------------------------------------
Done!

Code done running.

esp32spi_tcp_client.py

Note wifitest.adafruit.com does not respond to pings.

Server ping: 65535 ms
Connecting
Sending
Receiving
b'HTTP/1.1 200 OK\r\nServer: nginx/1.18.0 (Ubuntu)\r\nDate: Tue, 04 Jul 2023 01:57:07 GMT\r\nContent-Type: text/html\r\nContent-Length: 69\r\nLast-Modified: Thu, 09 Dec 2021 17:26:22 GMT\r\nConnection: close\r\nETag: "61b23c3e-45"\r\nAccept-Ranges: bytes\r\n\r\nThis is a test of Adafruit WiFi!\nIf you can read this, its working :)'

esp32spi_udp_client.py

Server ping: 17 ms
Sending
Receiving
Time: struct_time(tm_year=2023, tm_mon=7, tm_mday=4, tm_hour=2, tm_min=9, tm_sec=23, tm_wday=1, tm_yday=185, tm_isdst=-1)

server/esp32spi_wsgiserver.py

Works in safari but not chrome

ESP32 SPI simple web server test!
MAC addr: ['0x80', '0xab', '0x58', '0x93', '0x17', '0x44']
MAC addr actual: ['0x44', '0x17', '0x93', '0x58', '0xab', '0x80']
open this IP in your browser:  192.168.0.20
{'b': 218, 'g': 255, 'r': 198}
{'b': 89, 'g': 255, 'r': 81}
{'b': 68, 'g': 236, 'r': 255}
{'b': 255, 'g': 108, 'r': 245}
{'b': 255, 'g': 63, 'r': 103}
{'b': 255, 'g': 195, 'r': 74}
{'b': 236, 'g': 60, 'r': 255}
{'b': 126, 'g': 85, 'r': 255}
{'b': 220, 'g': 227, 'r': 255}

Could not test GPIO

esp32spi_wpa2ent_simpletest.py - v1.7.4

code.py output:
ESP32 SPI WPA2 Enterprise test
ESP32 found and in idle mode
Firmware vers. 1.7.4
MAC addr: ['0x80', '0xab', '0x58', '0x93', '0x17', '0x44']
Connecting to AP...
......
Code stopped by auto-reload. Reloading soon.
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
ESP32 SPI WPA2 Enterprise test
ESP32 found and in idle mode
Firmware vers. 1.7.4
MAC addr: ['0x80', '0xab', '0x58', '0x93', '0x17', '0x44']
Connecting to AP...
..................................

esp32spi_wpa2ent_simpletest.py - v2.0.0alpha

ESP32 SPI WPA2 Enterprise test
ESP32 found and in idle mode
Firmware vers. 2.0.0alpha
MAC addr: ['0x80', '0xab', '0x58', '0x93', '0x17', '0x44']
Connecting to AP...
...
Connected to eduroam RSSI: -74
My IP address is [REDACTED]
IP lookup adafruit.com: 104.20.39.240
Ping google.com: 8 ms
Done!

esp32spi_wpa2ent_aio_post.py - v2.0.0alpha

ESP32 SPI WPA2 Enterprise webclient test
Posting data...{'created_at': '2023-07-04T15:07:02Z', 'id': '0FBD76TMP1YE6CC0F7H4RNTZ84', 'expiration': '2023-09-02T15:07:02Z', 'created_epoch': 1688483222, 'feed_id': 2573750, 'value': '0', 'feed_key': 'test'}
OK
Posting data...{'created_at': '2023-07-04T15:07:18Z', 'id': '0FBD76ZEZTWMSYA1Z3JCJNNPM7', 'expiration': '2023-09-02T15:07:18Z', 'created_epoch': 1688483238, 'feed_id': 2573750, 'value': '1', 'feed_key': 'test'}
OK
Posting data...{'created_at': '2023-07-04T15:07:34Z', 'id': '0FBD7749Z41VNG2HDRKDXT8JJZ', 'expiration': '2023-09-02T15:07:34Z', 'created_epoch': 1688483254, 'feed_id': 2573750, 'value': '2', 'feed_key': 'test'}
OK

[dlizotte-uwo]

There are two commits after this PR, one to configure the ESP32 in the right order when joining a network (previous arrangement would always fail to connect once before successfully connecting) and the other to update the cert file with the new AIO certificate and to remove all certs that have expired. Saves max RAM used during SSL handshake.

[rgov]

Thanks @dlizotte-uwo for the update. (I am not a maintainer or contributor to this repo, I'm just interested in using this firmware on different Espressif microcontrollers.)

I like that the changes here adopt the "components" from the ESP-IDF build system design.

I recommend splitting the arduino component off from a separate nina-fw component that contains the SPIS and WiFi libraries. I sent you a pull request here: dlizotte-uwo#1

[hathach]

current master builds with idf v5.3.2 now #70 . This can now be closed. Thank you for your effort.