Skip to content

icu: replace the C++ libraries with the parts that Bun uses, in Rust - #49369

Open
dylan-conway wants to merge 8 commits into
mainfrom
claude/icu-safe-rust-rewrite-4f2ws7
Open

dylan-conway wants to merge 8 commits into
mainfrom
claude/icu-safe-rust-rewrite-4f2ws7

Conversation

@dylan-conway

@dylan-conway dylan-conway commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

src/icu replaces the code of ICU4C 79.1 on Linux, Windows, FreeBSD and Android. It exports the 189 functions of ICU's C API that JavaScriptCore, WTF and Bun's C++ call, for the arguments that they pass, and nothing else: 39,100 lines. The headers and the data are still ICU4C's and still come with WebKit, which does not change. macOS keeps the system's ICU.

common/ and i18n/ (34,700 lines) and the bodies of the exported functions forbid unsafe. It is in capi/ffi/, which turns C's pointers into references, and in sys/, which reads the linked data and asks the host for its locale and time zone (1,400 lines together). src/icu/README.md has the rules, and what to do when WebKit or ICU is upgraded.

It aims for the results of ICU4C 79.1, but is not the same in every case: some rare results differ on purpose, and we know of no difference that is not intended. Others that you may notice:

  • ICU_DATA and ICU_ENABLE_TENTATIVE_ERA are not read any more. ICU_TIMEZONE_FILES_DIR still replaces the four time zone files.
  • A function that is missing fails the link, but an argument that is not supported fails only at run time (U_UNSUPPORTED_ERROR, or a debug assertion). That includes a source that overlaps the buffer of u_strToLower or u_strToUpper. upgrade-webkit.md says to look for that.
  • A refused allocation is an error. It ends the process while files are opened (the time zone of the host, ICU_TIMEZONE_FILES_DIR), and if it leaves Date without a calendar.
  • $vm.dumpLineBreakData(), which builds with assertions have for WebCore, ends the process in bun. jsc still has it.

bench/snippets/intl.mjs, release builds with LTO, Zen 4, Linux, best of several rounds. Time relative to the previous build, lower is better:

cases range geometric mean
constructors 11 0.07 – 1.01 0.36
NumberFormat 6 0.47 – 0.74 0.62
DateTimeFormat 7 0.30 – 0.77 0.39
other formats, Locale 7 0.50 – 0.96 0.67
Temporal 4 0.50 – 1.02 0.73
sort with Collator 5 0.92 – 1.00 0.94
normalize 10 0.85 – 1.07 0.95
toUpperCase 5 0.47 – 1.00 0.70
Segmenter 15 0.48 – 0.90 0.75
new URL 1 0.75 0.75
all 71 0.07 – 1.07 0.64

It is more than 1% slower in four cases: normalize("NFD") and normalize("NFC") of Thai text (1.07, 1.04), normalize("NFC") of Korean text (1.02) and add in the Persian calendar (1.02). Short loops depend on the address at which they are linked: Thai NFC was at 0.79 and 0.83 in two other builds.

first call in a process before after
String(new Date(0)) 4.0 ms 0.9 ms
new Date(0).toLocaleString() 5.5 ms 1.5 ms
(1234.5).toLocaleString("de") 1.3 ms 0.6 ms

Startup without such a call is the same (3.8 ms). After all services were used in all locales, the library holds 15.6 MiB instead of 35.7 MiB, not counting decompressed data.

bun is 0.4 to 0.7 MB smaller on Linux, Android and FreeBSD, and 2.3 to 2.5 MB on Windows. With --compile --aot --trim, hello world is 451 KB smaller, and a program that uses Intl.DateTimeFormat 770 KB.

Build: bun links only ICU's data library. jsc and testaot keep ICU4C. The names of the symbols end in _79, and a static_assert fails if WebKit has another version of ICU. A local WebKit on Linux without BUN_ICU_ROOT uses the system's ICU as before, and the crate is empty (--cfg=bun_system_icu). A local WebKit leaves out three files of WTF that only WebCore uses.

Deleted: bun_icu_malloc.cpp. The crate uses the global allocator.

Left for later: the data package has items that nothing opens (brkitr/line*.brk, title.brk, ulayout.icu and others).

How did you verify your code works?

  • test/js/web/intl/intl-golden has 6,870 expressions with the results of ICU4C 79.1, in the test's process, in 40 environments and in workers. They were recorded with the previous build, so they pass there too. A debug build runs a quarter of them. They notice 3,158 of 5,793 small mistakes planted in the crate of the first commit, of which 4,042 change a value while they run. hardened-icu.test.ts runs them on a hardened build.
  • cargo test -p bun_icu calls exported functions as C++ does, and in most tests refuses each allocation in turn. It runs under Miri too.
  • Of 12,578 files of test262 and JSTests/stress, run in a jsc that is linked with this crate, 12,551 get the result of ICU4C from each call. The other 27 pass.
  • Of about 990,000 generated expressions (Intl, Temporal, Date, normalization, case mapping, regular expressions, hosts of URLs), 99.6% give the result of ICU4C. The others are the intended differences.
  • 33,217 scripts, which include those files and sweep locales, options, calendars, time zones and environments, ran with debug assertions and overflow checks, under ASan, and with every allocation tracked: no report, and none of 1.78 billion allocations lost. 74 of them stop early under ASan at an assertion of JavaScriptCore, as with ICU4C. Parts also ran under TSan, near the end of the stack, with allocations that are refused, with damaged time zone files, and without the data that --trim-locales removes.
  • The signatures of the exported functions were compared with ICU's headers, as LLVM IR, for ten targets.
  • Existing tests that changed: compile-aot-size looks for other strings of the formatters, build-local-webkit expects ICU's code only in the programs, hardened-executable expects __memmove_chk in place of __strcpy_chk, which only ICU4C called, refusable-memory covers the crate, and layers.py knows its symbols.
  • Only Linux x64 with glibc ran bun outside of CI. CI also runs the tests on Windows, musl and arm64, but only builds for Android and FreeBSD. Speed is measured only on Linux x64.
  • Windows skips the cases that set LANG, LC_* or TZ. The code that asks Windows for its time zone and locale ran under Wine instead: 293,000 combinations of zone, region and flags and 950 names of locales give the result of ICU4C.
  • ctrl-c.test.ts is the version of test: fold 57 pending test-only PRs into one PR #49196, which needs no registry. It has nothing to do with ICU.

Generated by Claude Code

claude and others added 2 commits November 21, 2026 04:17
src/icu exports the functions of ICU's C API that JavaScriptCore, WTF and
Bun call. The headers and the data still come with WebKit, and macOS keeps
the system's ICU.

Co-authored-by: dylanc <dylanc@anthropic.com>
Co-authored-by: dylanc <dylanc@anthropic.com>
@robobun

robobun commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator
Updated 1:59 AM PT - Nov 21st, 2026

✅ @claude, your commit ff971d3766b1a92d51ad0312bdd5cab7786af08a passed in Build #143440! 🎉


🧪   To try this PR locally:

bunx bun-pr 49369

That installs a local version of the PR into your bun-49369 executable, so you can run:

bun-49369 --bun

claude and others added 4 commits November 21, 2026 05:23
setlocale() takes "no_NO" for an alias where locale.alias is installed.
Nothing in a hardened build calls __strcpy_chk any more.

Co-authored-by: dylanc <dylanc@anthropic.com>
udat_applyPattern takes a rejected pattern for an empty one again.

Co-authored-by: dylanc <dylanc@anthropic.com>
Co-authored-by: dylanc <dylanc@anthropic.com>
The same change as in #49196. The test failed when the registry answered 404.

Co-authored-by: dylanc <dylanc@anthropic.com>

Copy link
Copy Markdown
Member Author

test/regression/issue/ctrl-c.test.ts failed on Debian 13 x64 in build 143423, in all four attempts. It is not caused by this change: the test installs vite from the public registry, which answered 404 for rollup-4.68.8.tgz.

I took over the version of the test from #49196, which needs no registry.


Generated by Claude Code

claude and others added 2 commits November 21, 2026 07:35
Option<&H>, Option<&mut H> and Option<Box<H>> replace the pointer types and
the magic numbers. No caller passes an object twice or casts between types.

Co-authored-by: dylanc <dylanc@anthropic.com>
No caller passes a source that overlaps its buffer. Where a source is read
while the buffer is written, an overlap is now U_UNSUPPORTED_ERROR.

Co-authored-by: dylanc <dylanc@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline findings, I also checked two candidates and ruled them out: the default-locale lookup in src/icu/common/putil.rs (an empty LC_ALL falls through to LC_MESSAGES/LANG, matching ICU4C's uprv_getPOSIXIDForDefaultLocale), and the four-property u_hasBinaryProperty in src/icu/capi/uchar.rs (every in-tree caller in URLPatternParser.cpp and stringWidth.cpp passes one of the accepted properties). This review read the diff only; the intl-golden suite, cargo test -p bun_icu and the JSC/WTF caller enumeration in WebKit were not run or re-checked here.

Extended reasoning...

The change adds a ~39k-line Rust crate replacing ICU4C's C API for Bun on Linux, Windows, FreeBSD and Android, plus build scripts, test fixtures, and CLAUDE.md edits; it touches parsing of linked and on-disk ICU data (ICU_TIMEZONE_FILES_DIR) and host locale/time zone queries, but no auth or crypto surface. Two confirmed findings are posted inline, so approval is not on the table; this body only records the two additional candidates examined and ruled out, and that no build or test was run by this review.

Comment thread src/icu/common/udata.rs
Comment on lines +205 to +211
match read_file(&path)? {
Some(file) if is_acceptable(&file) => return Ok(file),
Some(_) => error = U_INVALID_FORMAT_ERROR,
None => {}
}
}
Err(error)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Operators whose ICU_TIMEZONE_FILES_DIR holds a zoneinfo64.res (or timezoneTypes/windowsZones/metaZones) that the header check rejects lose all time zone data after merging; ICU4C skipped such a file and used the linked package. time_zone_file at src/icu/common/udata.rs:205-211 turns "found but unacceptable" into Err(U_INVALID_FORMAT_ERROR), and open_choice propagates it with ? at udata.rs:239 before ever searching the package. Fix: treat an unacceptable override file as not found for the package lookup (keep U_INVALID_FORMAT_ERROR only when the package also lacks the item), matching ICU4C's non-fatal subErrorCode, for all four names. Big-endian tz-update files, truncated or non-ResB files trigger it. [also at: src/icu/common/udata.rs:216 - An operator whose ICU_TIMEZONE_FILES_DIR holds a time zone file the reader rejects loses all time zone data, where ICU4C silently fell back to the linked package.]

Why this was flagged

ICU_TIMEZONE_FILES_DIR points at a directory whose zoneinfo64.res is big-endian, truncated below a 20-byte header, or not a ResB v1-3 bundle. read_file returns Some(file), accepts(&file) is false, so error becomes U_INVALID_FORMAT_ERROR and the loop ends with Err(error) at udata.rs:211; the match at udata.rs:213-216 only maps U_FILE_ACCESS_ERROR to Ok(None), so open_choice returns the error at udata.rs:239 and never builds the "icudt79l/zoneinfo64.res" entry or calls package::find. cached_part (src/icu/i18n/timezone.rs:185) and find_name/zones then fail, TimeZone::create_time_zone (timezone.rs:266-277) returns the Etc/Unknown fixed zone for every ID, so Date prints in UTC and every Intl.DateTimeFormat/Temporal timeZone is rejected as invalid, with no message naming the file. ICU4C's doLoadFromIndividualFiles records an unacceptable file only in the non-fatal subErrorCode and doOpenChoice goes on to the common data, so the same environment works on the base.

Verification: In src/icu/common/udata.rs, time_zone_file for a file that reads but fails is_acceptable sets error = U_INVALID_FORMAT_ERROR (line 207) and ends with Err(error) (line 211); the only error mapped back to "not found" is U_FILE_ACCESS_ERROR (line 214). open_choice propagates that with ? at line 239, before the package lookup package::find(&entry) at line 257 is ever reached.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not reproduce as a regression: the base does not fall back to the linked data either. I ran a build of the base and one of this branch with each of the four files big-endian, with another format, version or magic number, cut to 10 and to 3 bytes, and as text. Output and exit code are the same in all 28 cases (windowsZones through ucal_getHostTimeZone under Wine). Left as it is.


Generated by Claude Code

Comment thread src/icu/capi/ucal.rs Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants