ESP8266HTTPClient: optionally skip validation when using https

[liebman]

if fingerprint is specified as '*' then skip validation.

[liebman]

I wonder if this logic would be more useful if moved to WiFiClientSecure?

[rzeldent]

Often a certificate is not available (using ip adresses / no dns). Additionally, not all types of certificates are recognized by the ESP8266 and these do expire and are changed. It would be a nice addition to add this because authentication and encrypting are two separate things and this change will make it easier to use https. I vote for this commit (after fixing)!

[devyte]

@rzeldent what do you think needs fixing?

[devyte]

@igrr I don't use this, so I don't know enough to form an opinion. What are your thoughts about this?

[rzeldent]

Hi Develo, Thanks for asking! The point is that https has two functions: Encryption and Authentication. When *Authenticating*, the DNS is verified with the certificate. This is not completely implemented on the ESP8266 so another technique is used: Certificate pinning. Certificate pinning checks if the certificate is identical to the fingerprint of the remote certificate when it was developed. It has a few downsides; if an certificate expires, a provider has multiple certificates for the same domain or the certificate cannot be decoded by the ESP8266, the verification fails.This happens quite often by mistake because of upgrading of the certificates from SHA1 to SHA2. It also shortens the lifetime of the application because it works until a certificate is renewed. When the authentication mentioned above fails, the https connection cannot be made. Untilll full certificate support is offered (never certificates can be verified and parsed using a DNS) there should be a fallback to allow a connection anyway. If such a fallback is implemented the other function *Encryption* can be used and this is a requirement for connecting to cloud services like Azure, Google and Amazon. The developers should have a choice if they want to use certificate pinning or not. This is what the patch is about: a fingerprint string with an star (*) means any fingerprint, effectively bypassing certificate checking. I hope this explanation helps, please get back if you have any more questions, Kind regards, Rene

…

On Sat, Dec 16, 2017 at 4:17 AM, Develo ***@***.***> wrote: @igrr <https://github.com/igrr> I don't use this, so I don't know enough to form an opinion. What are your thoughts about this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3933 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHRGhzrH1UtANIyXaTaF1ZJ09vAPNhcXks5tAzY5gaJpZM4Q6YFL> .

[Rotzbua]

I agree with @devyte. This patch must not be merged. The patch introduces nothing else then a big security issue. Why should a developer implement certificate pinning if it could maybe fail in future and make trouble? (devs are always lazy ;))
We should not touch one important rule: "Never enable a programmer to write insecure code".

I wonder if this logic would be more useful if moved to WiFiClientSecure?

@liebman No, WiFiClientSecure is "low" level (in the arduino environment). If you call there verify you really want to verify the certificate.

This is what the patch is about: a fingerprint string with an star (*) means any fingerprint, effectively bypassing certificate checking.

@rzeldent This enables Man in the Middle attacks. In other words the "security" is useless and nothing else than snake oil. Then you can use unencrypted connection.

Iot devices need updates anyway, so why not update your pinned certificate with your regular updates?

Also WiFiClientSecure supports now (in rc) certificate chain verification. It would be more useful to implement this for ESP8266HTTPClient.

I used WiFiClientSecure for connection to a mqtt server with letsencrypt certificate and it works :).

My conclusion:
(1) We need a implementation of chain verification for ESP8266HTTPClient.
(2) Regular updates of application are required to keep your fingerprints up to date. (anyway required to fix security issues by sdk or libs or others eg. KRACK)
(3) Or use unencrypted connection if you do not need security.

[liebman]

@Rotzbua Ok, I'll have to agree on your comment about not enabling developers to write insecure code.

(1) We need a implementation of chain verification for ESP8266HTTPClient.

Yes, I was unaware it was implemented and a quick look at it shows it should be easy to implement. Maybe even auto-detecting 'https:' in begin() so that one does not need a different api for https vs http, this always bugged me.

I'm still concerned about how to deal with root cert expiration & update for long running embedded devices.

[igrr]

Maybe even auto-detecting 'https:' in begin() so that one does not need a different api for https vs http

Originally HTTPClient would do such auto-detection. But the downside was that HTTPS code (including axTLS) was linked into the final program even if the program would never use HTTPS. That increased flash usage and also reduced the amount of available heap. Then the decision was made to split this functionality into two different APIs, so that the linker could eliminate unused code and data in case HTTPS was not used.

We need a implementation of chain verification for ESP8266HTTPClient.

That might not be necessary — ESP8266HTTPClient allows access to the underlying WiFiClientSecure object (with type cast to WiFiClient). So in theory we could call httpclient.beginHTTPS(host, port), then call httpclient.getClientSecure()->loadCACert(cert_pgm_string), and then finally call httpclient.GET(). The only issue, i think, is that the WiFiClientSecure object doesn't get created until the sendRequest (GET, POST, etc) call. So HTTPClient has to be modified a bit to create the client early, allowing certificate to be set before the connection.

I'm still concerned about how to deal with root cert expiration & update for long running embedded devices.

In any case, root cert expiration would be a lousy excuse to skip certificate verification step. One can perform OTA updates, or filesystem updates (if certificates are stored on the filesystem), or even just update one file on the filesystem (which contains root certificate). If your device connects to some web services, you will need some mechanism to update API endpoints (URLs) anyway. CA certificate can be considered another such "setting" which could be updated.

FYI, i'm going to close this PR, as I also do not want to enable easy ways of writing insecure code. For anyone who would like to use this change, please feel free to grab this PR branch.

[rzeldent]

Hi Ivan, A suggestion; maybe you can look at the source code of MongooseOS where there are no certificate problems and supports the ESP8266. There is support for all the main cloud providers and implements correctly the certificate validation: https://github.com/cesanta/mongoose-os. Kind regards, Rene

…

On Thu, Dec 21, 2017 at 3:31 AM, Ivan Grokhotkov ***@***.***> wrote: Maybe even auto-detecting 'https:' in begin() so that one does not need a different api for https vs http Originally HTTPClient would do such auto-detection. But the downside was that HTTPS code (including axTLS) was linked into the final program even if the program would never use HTTPS. That increased flash usage and also reduced the amount of available heap. Then the decision was made to split this functionality into two different APIs, so that the linker could eliminate unused code and data in case HTTPS was not used. We need a implementation of chain verification for ESP8266HTTPClient. That might not be necessary — ESP8266HTTPClient allows access to the underlying WiFiClientSecure object (with type cast to WiFiClient). So in theory we could call httpclient.beginHTTPS(host, port), then call httpclient.getClientSecure()->loadCACert(cert_pgm_string), and then finally call httpclient.GET(). The only issue, i think, is that the WiFiClientSecure object doesn't get created until the sendRequest (GET, POST, etc) call. So HTTPClient has to be modified a bit to create the client early, allowing certificate to be set before the connection. I'm still concerned about how to deal with root cert expiration & update for long running embedded devices. In any case, root cert expiration would be a lousy excuse to skip certificate verification step. One can perform OTA updates, or filesystem updates (if certificates are stored on the filesystem), or even just update one file on the filesystem (which contains root certificate). If your device connects to some web services, you will need some mechanism to update API endpoints (URLs) anyway. CA certificate can be considered another such "setting" which could be updated. FYI, i'm going to close this PR, as I also do not want to enable easy ways of writing insecure code. For anyone who would like to use this change, please feel free to grab this PR branch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3933 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHRGh4hYI3Fjkw5-8_iD0IqOfJ7HAEkoks5tCcMGgaJpZM4Q6YFL> .

[jwillmer]

I would like to make a case against this decision @igrr

1. We all like to store data in the cloud but more and more provider use SSL. We might not care about SSL for our implementations but we still need to use it since there is no HTTP access anymore.

2. In my case I point to a own (SSL) domain name that redirects to a file hosting service with the file. While I can use fingerprint for the connection to my domain I don't know the fingerprint of the redirect yet.

3. All major browsers also allow connecting to insecure websites with certificates that can't be validated

4. The burden to use fingerprint for every domain and manage them might be O.K. for corporate developed IoT solutions but for the majority of IoT (temp sensors) that is developed as a hobby projects this is too much too ask

5. In some cases I only might be interested that the connection is secure but I don't care about the authority.

6. Leaving the code in the pull request and referencing too it (in this issue) creates too problems. People move away from this library since it does not work (for them) or people modify the local files and they get a problem if they open there source code on another system where the library modifications are missing.

[Rotzbua]

1. We all like to store data in the cloud but more and more provider use SSL. We might not care about SSL for our implementations but we still need to use it since there is no HTTP access anymore.

Why do you think do they force SSL connections? If they would not care about security or api keys they would still offer unencrypted HTTP.

2. In my case I point to a own (SSL) domain name that redirects to a file hosting service with the file. While I can use fingerprint for the connection to my domain I don't know the fingerprint of the redirect yet.

Well, then use HTTP. If it is not possible: set up a proxy.
And by the way, the esp is still micro controller and not a full OS. Which means that you still have to use a pc/arm for crawling stuff.

3. All major browsers also allow connecting to insecure websites with certificates that can't be validated

Browser == micro controller?? Nothing to add to this compare....

4. The burden to use fingerprint for every domain and manage them might be O.K. for corporate developed IoT solutions but for the majority of IoT (temp sensors) that is developed as a hobby projects this is too much too ask

I believe in hobby projects. They took the burden of programming so they can take the burden to do it right.

5. In some cases I only might be interested that the connection is secure but I don't care about the authority.

That is wrong. Encryption != secure. Encryption + authentication = secure.