Provide APIs to help control TLS fingerprints

[pimterry]

Is your feature request related to a problem? Please describe.

There are servers in the wild online (at least all sites using Akamai's CDN bot management feature) which actively block all connections from Node.js clients by examining their TLS fingerprint (more details).

There are some limited options to work around this today, such as reordering cipher suites, but they have security consequences which make this hard to do safely, and which limit the set of valid configurations.

While reordering ciphers has security consequences, reordering the extensions in the client hello is a semantically meaningless & safe change that would make it possible to completely defeat TLS fingerprinting.

Unfortunately, there are no APIs exposed that would allow Node.js developers to do this today.

Describe the solution you'd like

An API to configure the order that TLS extensions are set in the client hello would be perfect. An API or command line option which simply randomized the order for each connection would also be very good (equally effective for this use case I think, but a bit less flexible for advanced tricks, like emulating another TLS client's extension order).

Randomizing once at process startup might potentially be good, perhaps via a command line option, but that creates new per-process fingerprinting opportunities that could be problematic.

Describe alternatives you've considered

Currently the only alternative is changing the list of order of ciphers, which does work to defeat fingerprinting in the short term, but provides limited scope and requires detailed TLS knowledge to do safely.

[pimterry]

(Edit: link was wrong, now fixed, correct URL is https://httptoolkit.tech/blog/tls-fingerprinting-node-js/)

[NichyX]

Yes please!

[bnoordhuis]

This is most likely blocked on openssl growing the necessary infrastructure.

Extensions are currently added to the handshake packet in fixed order, see tls_construct_extensions() in deps/openssl/openssl/ssl/statem/extensions.c.

A workaround is to pipe the tls.Socket into a duplex stream, parse the handshake packet and shuffle the extensions around. Not trivial but not impossible either.

[joaoscheuermann]

This is most likely blocked on openssl growing the necessary infrastructure.

Extensions are currently added to the handshake packet in fixed order, see tls_construct_extensions() in deps/openssl/openssl/ssl/statem/extensions.c.

A workaround is to pipe the tls.Socket into a duplex stream, parse the handshake packet and shuffle the extensions around. Not trivial but not impossible either.

Do you have any exemple code on how to do it?

[tniessen]

A workaround is to pipe the tls.Socket into a duplex stream, parse the handshake packet and shuffle the extensions around. Not trivial but not impossible either.

@bnoordhuis I tried implementing that based on your description in the past, but I don't see how to get past the Finished message. OpenSSL rightfully terminates the handshake with sslv3 alert bad record mac.

[bnoordhuis]

@tniessen The only packet that's modified is the very first one, the ClientHello, everything else is passed through verbatim. It sounds like you're also modifying subsequent packets?

[tniessen]

@bnoordhuis I am only rewriting the ClientHello, but looking at the TLS spec, the ClientHello is part of the Handshake Context, which is hashed and signed for the CertificateVerify and Finished messages. That seems to cause the TLS alert.

[joaoscheuermann]

Seems like this will not be possible with Node JS, it uses Open SSL for TLS.

I wrapped a Socket in a Duplex Stream to intercept and modify the ClientHello record. At the end, im getting this error:

Any Header extension that SSL doesn't requires throw this error... Im not aware of any config options that allow OpenSSL to ignore any headers/extensions that is not requested...

[bnoordhuis]

I've filed feature request openssl/openssl#19220.

[bnoordhuis]

The upstream consensus seems to be that this isn't going to happen anytime soon (and that it's more logical to pursue ECH) so I'm going to close this as a wontfix - cantfix, really.

[monperrus]

https://httptoolkit.tech/blog/tls-fingerprinting-node-js/

confirming that the solution described in this post still works today (dec 2023).

[pimterry]

confirming that the solution described in this post still works today (dec 2023).

I wrote that post, and filed this issue.

Note that it's not a perfect fix, and there are notable limitations there: tweaking settings like this will work well to defeat blocklists that match specific known clients, but won't work well to defeat allowlists (allowing only recognized clients - uncommon) or machine-learning based approaches (more common since that article was published - where clients with totally new & very rare signatures are blocked initially).

For those latter cases, the only possible general solution is to precisely match an existing signature, which is still impossible with Node, and likely extremely difficult to add in future. Chrome have been doing experiments around this topic (https://www.peakhour.io/blog/tls-extension-randomisation/) to help the ecosystem avoid these kinds of risks, but as far as I'm aware these techniques are still widely used by CDNs like Cloudflare, and so there are definitely servers which are effectively inaccessible when sending HTTP requests from node.

[pimterry]

The upstream consensus seems to be that this isn't going to happen anytime soon (and that it's more logical to pursue ECH) so I'm going to close this as a wontfix - cantfix, really.

@bnoordhuis See my updates upstream on OpenSSL. This actually looks tractable! (and note that ECH is not a solution). I've got the first PR open now: openssl/openssl#24161. I think overall this should be far more achievable than expected (helped significantly by Chrome's GREASE work on randomizing extension order).

If all that upstream work were done, there would still be a few things required in Node to solve this issue:

• Updating OpenSSL to a version with those changes - that'd be at least v3.4.
• An API (another TLS context option) for Custom EC Point Formats option openssl/openssl#12644 (now in progress: Add SSL_OP_LEGACY_EC_POINT_FORMATS and change default to allow only uncompressed point format openssl/openssl#26990)
• An API to register custom TLS extensions from Node, using OpenSSL's existing add_custom_ext API (this allows manually adding extension data to mimic a fingerprint, and could maybe plausibly (?) be useful for custom use cases or to add adhoc support for new TLS extensions in future).

I'd hope the first two aren't too contentious (though I'm sure updating through to the latest OpenSSL will take a while). What about the last point?

In practice I think this would be an option to provide a set of callbacks when creating a TLS context. There'd be some performance implications to using those in heavy traffic environments, but it shouldn't affect any existing code that doesn't use the new APIs at all. That said, there's an argument that this is a bit of a footgun, because if you're determined it's very possible to use these APIs to really mess up your TLS connections (though that applies to some other TLS context options too).

If that's not workable: I'm not too familiar with writing native addons myself, but is this something that would be possible to implement as a Node addon there instead? Is enough access to the raw OpenSSL context available there? If so, a native module could define a registerCustomExtension(tlsContext, extensionCallbacks) JS method as a reasonable solution - thereby making it possible to solve this issue, but less officially/obviously so to reduce any footgun risk.

Anyway, if the issues listed in that OpenSSL thread are resolved, OpenSSL in Node is updated, and some way to access those two APIs is added in Node, then I think it would be possible to completely negate the current TLS fingerprinting issues of Node TLS traffic (at least mimicking recent Firefox/Chrome fingerprints, which is generally enough) so it's worth reopening this issue.

[scrapoxy]

Hello,

This feature is very useful. From now, the only solution is to port code in Go, to use uTLS.

Any update/news in it ?

Thanks

[pimterry]

@fabienvauchelles the first OpenSSL PR above towards this has been merged, which should significantly shift the TLS fingerprints of all OpenSSL clients (Node included) to much more closely match Chrome et al in future. That said, that won't be released until later this year, it'll still take more time for Node to update, and it's still not a perfect match.

If you're interested in taking this further, there's more changes required in OpenSSL that you can contribute to, described in detail here, and changes in Node to implement this described just above here. Since the last comment I've now become a Node core contributor, and I'm very happy to help support & shepherd any changes through to release if you do want to work on this.

[scrapoxy]

Thanks for the update, @pimterry. It's great to see progress on the issue! Although contributing in C++ sounds really interesting, I'm not sure if my skills are up to par just yet. I'll take a closer look and see what I can do.

[pimterry]

Just adding an update here: there's now notable sites (like php.net) which block all requests from modern Node.js completely by TLS fingerprint. Discussion here: nodejs/help#4516.

The popularity of Node probably means there's lots of people sending bot traffic with Node, and so increasingly it's going to get flagged as a suspicious fingerprint in itself and blocked by CDNs all over the place. I haven't had much time to work on this recently, but there's some notes in my comments above if anybody has time to help work towards solutions to this.

[pimterry]

The upstream consensus seems to be that this isn't going to happen anytime soon (and that it's more logical to pursue ECH) so I'm going to close this as a wontfix - cantfix, really.

@bnoordhuis You closed this back in 2022, but I'm going to reopen it if that's OK (please LMK if you disagree) because:

• ECH doesn't actually help unfortunately (see my comments above)
• This is something we can fix! Very slowly, but it looks tractable and I'm making good progress.
• It's an increasing problem in a wide variety of places (see various comments above, and issues elsewhere e.g. Undici: Make TLS Fingerprint great again undici#1983)

---

Current state:

• I've fixed the OpenSSL SCSV signature in OpenSSL 3.4, so it matches modern approaches everywhere else: Use empty renegotiate extension instead of SCSV for TLS > 1.0 openssl/openssl#24161.
• I have an approved OpenSSL PR to change the default EC point formats used, and add configurability there, which should land in OpenSSL 3.6: Add SSL_OP_LEGACY_EC_POINT_FORMATS and change default to allow only uncompressed point format openssl/openssl#26990. This would offer a toggle between the old behaviour & the new default browser-matching configuration, accessible in Node via secureOptions.
• Chromium's work on GREASE has made some things significantly easier since this was opened - e.g. they now randomize extension order, so we don't have to. New fingerprinting techniques like JA4 ignore extension order entirely because of this.
• I think Node is currently in the process of updating to OpenSSL 3.5, which will pick up change 1 above. We'll need to eventually update to 3.6 or later though to pull in the 2nd change, so that's still a way off.
• Remaining work in Node: we'd need to expose SSL_CTX_add_custom_ext somehow, sufficiently to allow adding custom extensions to match fingerprints in full. I'm planning on working on that too later this year, when I have a proper chunk of time (but contributions from others in the meantime are very welcome), so it's ready by the time we update to OpenSSL 3.6 or later (which would be 2026 at the earliest).

On the current path, OpenSSL's TLS handshake will have no non-configurable distinguishing features by version 3.6 (which is nice for reducing fingerprintability in general), and with custom extensions support in Node in theory it'll be possible to match a wide range of common TLS fingerprints, bringing JS to more-or-less parity with the Go-based approaches. Much of this is low-level advanced TLS options that most users shouldn't be using manually, but should act as a foundation for userland pure-JS solutions analogous to https://github.com/Danny-Dasilva/CycleTLS.