Add WiFiServerSecure and ESP8266WebServerSecure, SSL enabled servers for HTTPS/SSH/etc.

[earlephilhower]

Howdy,

First of all, thanks for the amazing work in making the ESP8266 toolchain very approachable. I'd like to give back with a change that I did this weekend to support a SSL server (useful for SSH or HTTPS) mode of operations.

Example code is included with a self-signed certificate generation script to get users up to speed quickly.

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@8765da2). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master   #3001   +/-   ##
========================================
  Coverage          ?   27.6%           
========================================
  Files             ?      20           
  Lines             ?    3655           
  Branches          ?     678           
========================================
  Hits              ?    1009           
  Misses            ?    2468           
  Partials          ?     178

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8765da2...f154e97. Read the comment docs.

[earlephilhower]

Latest commit adds in a new HTTPS server class, ESP8266WebServerSecure and example HelloServerSecure. Both use the WiFiSecureServer in the previous commits. I learned more than I wanted to about C++ object slicing and hidden class variables, thanks to the existing object hierarchy, but it's run a loop of WGET's against the HelloServerSecure overnight on my system.

[earlephilhower]

Added and tested a SSL-encrypted SecureWebUpdater example. Since the SecureWebServer class is passed by pointer, there's only 2-lines of change (+key/cert addition) to enable secure, SSL encrypted, password-protected ESPP updates.

I'll squash everything to a single commit closer to the 2.5.0 release for easier merging, assuming it's accepted.

[igrr]

Thanks, this is a really awesome feature. I intend to review and merge this (along with other changes, such as @d-a-v 's LwIP PR) once 2.4.0 is released, which hopefully can be done soon — the release candidate has been out for a few weeks and no major issues (aside from memory use) have been found.

[earlephilhower]

Happy to help out. Hopefully this will make it easier for everyone to make more secure IoT things!

[jmseight]

Hi,
I would like to use the code that you created. I downloaded the Arduino master but cannot find the code for WiFiServerSecure or ESP8266WebServerSecure, and cannot find them using the search function in GitHub. I do have WiFiClientSecure. Would you please tell me how to get the code.
Thanks,
James

[earlephilhower]

@jmseight , You'll need to use GIT and get the head of the main branch then use the GitHub instructions to get this pull (#3001) merged into your local copy. See https://help.github.com/articles/checking-out-pull-requests-locally/ for more info.

[jmseight]

Sorry for not understanding how to get the items. I tried "git fetch origin pull/#3001/esp8266:master" and got "fatal: Not a git repository (or any of the parent directories): .git", for "Modifying an inactive pull request locally".

Also, I cannot find "Near the bottom of the pull request, in the merge box, click command line instructions. Follow the sequence of steps to bring down the proposed pull request." as instructed for "Modifying an active pull request locally".

[earlephilhower]

@jmseight, I think you just didn't change into the cloned Arduino directory. Here's my exact command output:

earle@server:/tmp$ git clone https://github.com/esp8266/Arduino.git
Cloning into 'Arduino'...
remote: Counting objects: 14632, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 14632 (delta 11), reused 19 (delta 7), pack-reused 14597
Receiving objects: 100% (14632/14632), 36.66 MiB | 8.23 MiB/s, done.
Resolving deltas: 100% (8477/8477), done.
Checking connectivity... done.
earle@server:/tmp$ cd Arduino/
earle@server:/tmp/Arduino$ git fetch origin pull/3001/head:pull3001
From https://github.com/esp8266/Arduino
[new ref] refs/pull/3001/head -> pull3001
earle@server:/tmp/Arduino$ git checkout pull3001
Switched to branch 'pull3001'
earle@server:/tmp/Arduino$ head libraries/ESP8266WiFi/src/WiFiServerSecure.cpp
/*
WiFiServerSecure.cpp - SSL server for esp8266, mostly compatible
with Arduino WiFi shield library
...

[jmseight]

Hi, Thank you for the instructions. I was able to download the items. Sorry for not understanding this well. However, when I tried to Include Libraries as ZIP file and chose the folder, but I get errors with enum of WEP (in user_interface.h) and errors with udp.h. I tried to replace user_interface.h and udp.h with the ones that I got from the download, but I get many other errors with the library. Currently I am using V2.3.0 of the ESP8266 board SW. I think the version that you are using have is 3.2.3, correct? I can only get up to 2.3.0 form the Board manger in Arduino IDE on mac. Is this why there is a conflict? Thanks James --- James H-J Yeh, Ph.D. Assistant Professor, Engineering and Computer Science Azusa Pacific University On Aug 16, 2017, at 9:21 PM, Earle F. Philhower, III <[email protected]> wrote: @jmseight <https://github.com/jmseight>, I think you just didn't change into the cloned Arduino directory. Here's my exact command output: earle@server:/tmp$ git clone https://github.com/esp8266/Arduino.git <https://github.com/esp8266/Arduino.git> Cloning into 'Arduino'... remote: Counting objects: 14632, done. remote: Compressing objects: 100% (27/27), done. remote: Total 14632 (delta 11), reused 19 (delta 7), pack-reused 14597 Receiving objects: 100% (14632/14632), 36.66 MiB | 8.23 MiB/s, done. Resolving deltas: 100% (8477/8477), done. Checking connectivity... done. earle@server:/tmp$ cd Arduino/ earle@server:/tmp/Arduino$ git fetch origin pull/3001/head:pull3001 From https://github.com/esp8266/Arduino <https://github.com/esp8266/Arduino> [new ref] refs#3001/head -> pull3001 earle@server:/tmp/Arduino$ git checkout pull3001 Switched to branch 'pull3001' earle@server:/tmp/Arduino$ head libraries/ESP8266WiFi/src/WiFiServerSecure.cpp /* WiFiServerSecure.cpp - SSL server for esp8266, mostly compatible with Arduino WiFi shield library ... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3001 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AdQr--vvv-SU8GinZtx5A6_EXm6YgBjcks5sY7_cgaJpZM4MN9n8>.

[chaitanyanettem]

@igrr Hi, Any update on when this might be merged?

[devyte]

This makes no sense to me. Where is there an assignment such as ESP8266Webserver = ESP8266WebServerSecure? If there is anywhere where an assignment is done, then there is likely a bug there.

[earlephilhower]

The problem is not the webserver, but the WiFiServer.available() model...

...
    WiFiClient(Secure) client = _server(Secure).available();
...

Because WiFiServer and it's children return a copy of a object, not a pointer to a new()'d one, the caller needs to know how big it will be/etc. So the WebServerSecure class can't fall back on the WebServer::handle() method and let the vtable work its magic. Object slicing, if I remember the term properly, is the problem here.

If there was an method like "virtual WiFiClient *WiFiServer.available();" it would be easier to reuse the WebServer code. The Secure HTTPS Updater example shows this use.

[devyte]

I know about object slicing. When you encounter it, it's usually a sign of a bug or of bad design (not always).
I should investigate this. Sigh, so much to do!
In any case, not a fault in your implementation, which is why I only commented and approved.

[igrr]

This can also be solved with the template-based implementation and passing by reference, to some extent. Plus a few helper methods to clear the current client and get a new one from the WebServer. Will handle in another PR.

[pieman64]

@earlephilhower thanks for this PR and @igrr for the commit.
I notice you state "that it use SHA256 and 1024 or 512 bits" but I thought 2048 bits was OK.
Is there an up to date doc which states max key size and a list of acceptable ciphers?

Any chance of seeing an example for remote OTA updates rather than local OTA updates?

[igrr]

@pieman64 here's the list of cipher suites supported by axTLS:
https://github.com/igrr/axtls-8266/blob/master/README.md
Regarding "remote OTA updates", do you mean an example of pulling an update binary from remote server?

Regarding key sizes: no theoretical limit in the library, however memory constraints of the application may limit key sizes which will be achievable.

[pieman64]

@igrr thanks for the link to the available cipher suites and the info on key sizes.

Yes I think bin updates from remote servers with SSL are probably more popular than local updates because generally you can trust your own local network.

[igrr]

Ok, essentially that's ESPhttpUpdate.update("https://server/file", version, cert_fingerprint);

There's some work in progress to make it possible to verify server certificate using root certificate rather than using a fingerprint. Once done, i will add an example about ESPhttpUpdate with root cert verification.

[pieman64]

@igrr we are already using the fingerprint system and it's working OK with a basic sketch and bin files hosted on GitHub. The problem is once you start building a meaningful sketch it runs out of memory.

[igrr]

The only real solution to that is to allocate most of the things dynamically in the sketch. Then you can go into "update" mode or "normal" mode and enable features separately.

There will be some improvement in memory usage down the road once we fix the support for fragment length negotiation extension. That will, however, require having a server which supports that extension.

[d-a-v]

Thanks for this !

[earlephilhower]

Thanks all for the input and feedback to make it a better patch.

@pieman64 while it does support 2048 bits, you may run into out of RAM issues during SSL negotiations. It has worked before (you can change the # of bits in the make-cert.sh script and try w/your own self-signed ones), but practically it puts you in a very tight RAM situation...

@igrr 's comment about dynamically allocating things is exactly what I'm doing in my own self-contained ESP8266 power plug firmware. To get enough RAM for upgrade w/SSL I have to stop and free pretty much everything when a toggle is set, then after a timeout force a system reset to get back to normal mode in case the upgrade is not done.

[stefaandesmet2003]

@earlephilhower

Hi, i'm unable to get the HelloServerSecure example working correctly, and would greatly appreciate your help.
I'm testing the https server from chrome on Win10, and always get NET::ERR_CERT_INVALID. (on the info page chrome adds : ... You cannot visit 192.168.1.22 right now because the website sent scrambled credentials that Google Chrome cannot process. ...) "Hello from esp8266 over HTTPS!" is never displayed in the browser.

I modified the example to use a AWS IOT client certificate, and then after a couple of resets and an occasional watchdog, I get the "Hello from esp8266 over HTTPS!", Chome only complaining about net::ERR_CERT_AUTHORITY_INVALID, which is normal for self-signed certificates, right?
I have no clue where to start digging. I added serial debugging and included 2 serial logs below, one from the original example, the other from the same code with my AWS client certificate.

Don't know if the "Error : ..." statement has anything to do with my problem; it seems to come from the axtls library, but i'm using the libaxtls.a from the repo.

.
Connected to TP-LINK
IP address: 192.168.1.22
MDNS responder started
HTTPS server started
=== CERTIFICATE ISSUED TO ===
Common Name (CN): 127.0.0.1
Organization (O): ESP8266
=== CERTIFICATE ISSUED BY ===
Common Name (CN): 127.0.0.1
Organization (O): ESP8266
Not Before: Sat Mar 18 14:49:18 2017
Not After: Mon Nov 25 14:49:18 2030
RSA bitsize: 512
Sig Type: SHA256
State: receiving Client Hello (1)
State: sending Server Hello (2)
State: sending Certificate (11)
State: sending Server Hello Done (14)
State: receiving Client Key Exchange (16)
State: receiving Finished (16)
State: sending Finished (16)
New secure client
Error: maximum number of certs added (1) - change of compile-time configuration required
State: receiving Client Hello (1)
State: sending Server Hello (2)
State: sending Certificate (11)
State: sending Server Hello Done (14)
State: receiving Client Key Exchange (16)
State: receiving Finished (16)
State: sending Finished (16)
New secure client
pm open,type:2 0

---

I modified the example with a AWS IOT client certificate, and then after a couple of resets and an occasional watchdog, I get the "Hello from esp8266 over HTTPS!", with the following serial output :

.
Connected to TP-LINK
IP address: 192.168.1.22
MDNS responder started
Success to open cert file
certificate size = 861
Success to open private cert file
private key size = 1193
HTTPS server started
=== CERTIFICATE ISSUED TO ===
Common Name (CN): AWS IoT Certificate
Organization (O):
Basic Constraints: critical, CA:FALSE, pathlen:10000
Key Usage: critical, Digital Signature
=== CERTIFICATE ISSUED BY ===
Common Name (CN):
Organization (O):
Organizational Unit (OU): Amazon Web Services O=Amazon.com Inc. L=Seattle ST=Washington C=US
Not Before: Thu Nov 23 21:56:42 2017
Not After: Tue Nov 25 17:31:43 1913
RSA bitsize: 2048
Sig Type: SHA256
State: receiving Client Hello (1)
State: sending Server Hello (2)
State: sending Certificate (11)
State: sending Server Hello Done (14)
State: receiving Client Key Exchange (16)
pm open,type:2 0
New secure client
State: receiving Finished (16)
State: sending Finished (16)
Error: maximum number of certs added (1) - change of compile-time configuration required
State: receiving Client Hello (1)
State: sending Server Hello (2)
State: sending Certificate (11)
State: sending Server Hello Done (14)
State: receiving Client Key Exchange (16)
New secure client
State: receiving Finished (16)
State: sending Finished (16)
method: GET url: / search:
headerName: Host
headerValue: 192.168.1.22
headerName: Connection
headerValue: keep-alive
headerName: Cache-Control
headerValue: max-age=0
headerName: Upgrade-Insecure-Requests
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8
headerName: Accept-Encoding
headerValue: gzip, deflate, br
headerName: Accept-Language
headerValue: en-GB,en;q=0.9,en-US;q=0.8,nl;q=0.7
args:
Request: /
Arguments:
Error: maximum number of certs added (1) - change of compile-time configuration required
Fatal exception 29(StoreProhibitedCause):
epc1=0x4000df0e, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000005, depc=0x00000000

Exception (29):
epc1=0x4000df0e epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000005 depc=0x00000000

ctx: cont
sp: 3fff09c0 end: 3fff0d20 offset: 01a0

stack>>>
3fff0b60: 6f727245 6d203a72 6d697861 6e206d75
3fff0b70: 40208872 666f2072 72656320 61207374
3fff0b80: 00000005 64252820 202d2029 6e616863
3fff0b90: 6f206567 6f632066 6c69706d 00ae0001
3fff0ba0: 00000005 00000000 00000005 40208cb9
3fff0bb0: 00000320 00000000 3fff533c 4022142b
3fff0bc0: 00000200 3fff7d4c 3ffefca0 3fff0c40
3fff0bd0: 3fff8334 3fff220c 00000200 401007ac
3fff0be0: 3fff8334 3fff220c 3fff533c 40220098
3fff0bf0: 3fff0c40 0000035d 3fff220c 402220f9
3fff0c00: 3fff220c 0000035d 3fff37c4 01000000
3fff0c10: 3fff0c40 000050e7 3fff533c 40221900
3fff0c20: 40100a2e 00000030 3fff0ca0 3fff0c40
3fff0c30: 00001387 000050e7 3fff8334 40208c49
3fff0c40: 3fffdab0 00000000 3fffd9d0 3ffefcec
3fff0c50: 3fff8fdc 00000000 3fff30d4 000004a9
3fff0c60: 3fff2d64 3fffdad0 3ffefcec 00000030
3fff0c70: 00000000 3fff0cd0 3fff368c 3ffefcec
3fff0c80: 3fffdad0 3fff0cd0 3ffef9f0 40209437
3fff0c90: 0000035d 00000000 3ffefd00 402080f7
3fff0ca0: 40107360 00000000 00001388 402086d4
3fff0cb0: 3ffefa24 3fff8fdc 3fff8334 40208f1a
3fff0cc0: 00000001 00000000 3ffef910 4020a65e
3fff0cd0: 40107308 00000000 00003a98 feefeffe
3fff0ce0: 00000000 00000000 00000000 feefeffe
3fff0cf0: 3fffdad0 00000000 3ffefce4 402073fc
3fff0d00: 3fffdad0 00000000 3ffefce4 4020e664
3fff0d10: feefeffe feefeffe 3ffefd00 40100a2c
<<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
v00000000
~ld
scandone

scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 1
cnt

connected with TP-LINK, channel 6
dhcp client start...
ip:192.168.1.22,mask:255.255.255.0,gw:192.168.1.1
.
Connected to TP-LINK

[earlephilhower]

That error's actually quite good compared to the normal "randomly crash when something's wrong" we seem to have everywhere!

Can you put your code up (obfuscate your client cert, of course!) and decode the exception?

You're only allowed a single cert per SSL connection, so on an server.available() it removes the old SSL context + cert and stuffs in its host + pricate key. It dos its work and then the server's wificlientsecure goes away. At that point, the orig client will have its connection dropped and reconnect after manually reloading its client certs. Or at least that's the theory.

[stefaandesmet2003]

Hi,
thanks for your fast reply. Do you mean the https server library doesn't work with you either?
To be more precise, I tried 2 versions of the HelloServerSecure example, one the unmodified example code, and a second using a AWS client certificate & private key loaded from spiffs. So i'm only using 1 certificate at a time. But only with the amazon certificate, the browser showed the "Hello from esp8266 over HTTPS!" line, after a couple resets only ..
Here is the code and a trace file for each version. My hardware is a Wemos D1 mini 4MB flash. As for library version : I cloned esp8266-arduino yesterday and copied the whole repository in the 2.4.0 directory. I added #define DEBUG_OUTPUT Serial in ESP8266WebServerSecure.cpp , to make the code work with debug enabled.
Initially i thought the resets with the AWS certificate had to do with the longer key used, but from the stack trace, it looks the cause of the resets is the same for both versions.
Hope you can shed a light on this.
HelloServerSecure.zip

[earlephilhower]

Howdy @stefaandesmet2003 .

The HelloServerSecure runs fine for me, as does a SSL MQTT client in parallel not using client SSL certs (I do that w/psychoplug). Due to axtls SSL limitations, when an incoming web SSL request comes in, the outgoign SSL MQTT link gets dropped (and reconnected next pass thru the loop).

Are you saying your attached ZIP, with the code as-commented, also gives you a failure? It looks like it's effectively the same as the base example, no?

If you're really brave, in #4273 I've replaced axtls w/BearSSL which allows for multiple SSL contexts and user-defined buffer sizes so you really can have parallel incoming and outgoing connections. I've run the mqtt_esp8266 example w/client certs on it (to my own server) for several hours w/o incident.

[stefaandesmet2003]

Hi @earlephilhower ,
yes, the original example code with the hardcoded certificate/key raises the exception 29 and NET::ERR_CERT_INVALID in Chrome. I have a working SSL MQTT client sketch using AWS client certificate (PubSubClient+WiFiClientSecure); the issue seems to be in the **ServerSecure libraries.
I'm not brave enough right now for your last suggestion :( ; i don't understand enough yet of the tls internals.

[stefaandesmet2003]

Hi @earlephilhower ,
I did some more testing. The WiFiHTTPSServer example works consistently without resets. But every first request from the browser is not correctly decrypted : the 'req' string is empty, serial prints 'invalid request', and client.print (http 404) never arrives at the browser. Every second request (probably a retry from the browser) is correctly decrypted : req : GET /gpio/1 HTTP/1.1. In the output log i added a ESP.getFreeHeap() printout every 5 seconds. Heap is ok here.

The HelloServerSecure example now works without reset for 1 browser request (i used lwip v2 lower memory variant), but
HelloServerSecure_StacktraceOnReset.txt
then esp8266 resets systematically on the 2nd request. From the same ESP.getFreeHeap() printout every 5s, you see initial heap size of 35k is reduced to 5k after the first browser request, and is not restored. Then the 2nd request (reload page) fails with a reset.
Main difference between the 2 examples is the ESP8266WebServerSecure library.
Could you please check if you have the same behavior on your hardware?
Cheers,
WiFiHTTPSServer_HelloServerSecure_CodeAndSerialOutput.zip

[earlephilhower]

Great, thanks @stefaandesmet2003!

Well, not really great but now I can reproduce what you're seeing. There is obviously a memory leak somewhere in here when used with the WebServerSecure. There's some ugly refcount stuff going on in all the SSL code (to share the single SSL_CTX memory), and I believe some logic in the Server/Client is borked when the refcnt is higher than expected that I've flubbed.

Since this pull is closed and merged, would you like to open a new issue so it's tracked properly? Memory leak in WebServerSecure on repeated connections.

[earlephilhower]

Anyone running this in their own apps, please check #4305 as I've done significant cleanup and it's now way more stable and usable, allowing parallel client and server connections, each with their own certs, and no interference between the two.

[stefaandesmet2003]

thanks for your big effort. Will give it go very soon!

[MNGregO]

Thanks to all who have worked on this! I was able to get the HTTPS example running.
Are there any plans for an SSH server example? I've tried to tweak the HelloServerSecure example to become an SSH server (instead of HTTPS) and then tried connecting with Tera Term, but I haven't been able to get it to work. I'm really new to this, so I'm sure it's really simple to switch from HTTPS to SSH, but I can't figure it out.

[d-a-v]

Like @pornin explains, it is unlikely we can have ssh on the esp8266.
Still, we have have telnet over ssl and this is great.

[MNGregO]

Thanks for the reply @d-a-v.
Starting with the example WiFiTelnetToSerial.ino from the ESP8266WiFi library for Arduino (located here) I tried to add the SSL part, but I can't get it to work. Here's what I did...
I added:
#include <WiFiClientSecure.h> #include <ESP8266WebServerSecure.h>
I changed:
WiFiServer server(23);
to
ESP8266WebServerSecure server(23);

I added the x509 certificate and rsakey arrays.

Then I had to comment out a bunch of lines that the compiler complained about:
49: server.setNoDelay(true);
59/77: if (server.hasClient()) { and }
66: serverClients[i] = server.available();
73: WiFiClient serverClient = server.available();
74: serverClient.stop();

By the time all the stuff was commented out that the compiler choked on, it won't actually do telnet - in fact it never even outputs the "Connecting to " message on the serial port.
Any further advice would be greatly appreciated.

[d-a-v]

Use WiFiClientSecure, not the webserver. *edit*: I mean WiFiServerSecure of course :-]

[MNGregO]

Thanks again, @d-a-v . I think I have it working now. Next I just have to find a telnet client that is capable of using SSL so I can test it.
If anyone is interested in the final code I came up with, let me know - it looks like we can't upload .ino files here, but I could post it here using "insert code."

[d-a-v]

You can try with

openssl s_client -connect $ip:$port