Proposal: Replacing ICU with ztd.text or encoding_rs

[anonrig]

I've been mainly working on the TextDecoder performance gains for the past couple of weeks. It seems that ICU, even though is required for v8 Intl, is slow for UTF-8 encoding & decoding.

I recommend either adding ztd.text or encoding_rs with C++ bindings as a dependency and improving the performance of the TextDecoder & TextEncoder which will improve a lot of applications worldwide.

Deno uses encoding_rs and Bun uses a custom implementation.

Some good references:

• https://thephd.dev/the-c-c++-rust-string-text-encoding-api-landscape
• https://hsivonen.fi/encoding_rs/

[anonrig]

If this is not worth tsc-agenda, please remove it

[targos]

I don't know (yet) if it's worth, but it's probably too early to bring this to the tsc-agenda.

[Jarred-Sumner]

Small correction: in Bun's case, JSC.WebCore.TextDecoder is the struct name in Bun for TextDecoder. The code for encoding/decoding is mostly custom (no library) other than one case where a function from WebKit is used for copying 8-bit integers into a 16-bit integer array faster

[anonrig]

Thanks @Jarred-Sumner. I just updated the description.

[Trott]

Possible small positive side effect of doing what is proposed here: There is at least one place in the code where we use ICU for converting between UTF-8 and UTF-16 even though that part of the code doesn't have any internationalization needs. It's blocking #37954, so this could also help with that, I suppose.

[bnoordhuis]

encoding_rs is arguably way too big for just encoding UTF-8 to UTF-16. I mean, it's a great library but it does 100x more than what's needed.

I don't know ztd.text well enough to comment but a quick look at its source code suggests it's not exactly small either. It also seems to be very young and untested.

[targos]

What about https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/platform/wtf/ ?

[bnoordhuis]

I've worked quite a bit with WTF. It's great but not easy to use outside chromium's source tree. It's supposed to be standalone but it has at least a partial dependency on base/.

WebKit's fork is probably even worse, I could never gather up enough courage to even try: https://github.com/WebKit/WebKit/blob/main/Source/WTF

[anonrig]

encoding_rs is arguably way too big for just encoding UTF-8 to UTF-16. I mean, it's a great library but it does 100x more than what's needed.

I don't know ztd.text well enough to comment but a quick look at its source code suggests it's not exactly small either. It also seems to be very young and untested.

UTF8 encoding is now fast due to the recent developments. Can we consider encoding_rs for the rest of the encoding types?

[bnoordhuis]

What encodings are we talking about? Can you enumerate them?

[anonrig]

What encodings are we talking about? Can you enumerate them?

Any Buffer.from(val, ENC_1).toString(ENC_2) as well as TextDecoder (due to performance issues on initializing string_decoder).

[bnoordhuis]

I suppose? The tradeoff is performance vs. maintenance. A library like encoding_rs is probably going to be a PITA to build on IBM i or the other tier 2/tier 3 platforms.

Cross-compiling to WASM is an option but means no sharing of code or data. Each thread gets its own copy, and that's probably quite substantial for conversion tables. You'd have to measure it.