Repository navigation
icu: replace the C++ libraries with the parts that Bun uses, in Rust - #49369
dylan-conway wants to merge 8 commits into
Conversation
src/icu exports the functions of ICU's C API that JavaScriptCore, WTF and Bun call. The headers and the data still come with WebKit, and macOS keeps the system's ICU. Co-authored-by: dylanc <dylanc@anthropic.com>
Co-authored-by: dylanc <dylanc@anthropic.com>
|
Updated 1:59 AM PT - Nov 21st, 2026
✅ @claude, your commit ff971d3766b1a92d51ad0312bdd5cab7786af08a passed in 🧪 To try this PR locally: bunx bun-pr 49369That installs a local version of the PR into your bun-49369 --bun |
setlocale() takes "no_NO" for an alias where locale.alias is installed. Nothing in a hardened build calls __strcpy_chk any more. Co-authored-by: dylanc <dylanc@anthropic.com>
udat_applyPattern takes a rejected pattern for an empty one again. Co-authored-by: dylanc <dylanc@anthropic.com>
Co-authored-by: dylanc <dylanc@anthropic.com>
The same change as in #49196. The test failed when the registry answered 404. Co-authored-by: dylanc <dylanc@anthropic.com>
|
I took over the version of the test from #49196, which needs no registry. Generated by Claude Code |
Option<&H>, Option<&mut H> and Option<Box<H>> replace the pointer types and the magic numbers. No caller passes an object twice or casts between types. Co-authored-by: dylanc <dylanc@anthropic.com>
No caller passes a source that overlaps its buffer. Where a source is read while the buffer is written, an overlap is now U_UNSUPPORTED_ERROR. Co-authored-by: dylanc <dylanc@anthropic.com>
There was a problem hiding this comment.
Beyond the inline findings, I also checked two candidates and ruled them out: the default-locale lookup in src/icu/common/putil.rs (an empty LC_ALL falls through to LC_MESSAGES/LANG, matching ICU4C's uprv_getPOSIXIDForDefaultLocale), and the four-property u_hasBinaryProperty in src/icu/capi/uchar.rs (every in-tree caller in URLPatternParser.cpp and stringWidth.cpp passes one of the accepted properties). This review read the diff only; the intl-golden suite, cargo test -p bun_icu and the JSC/WTF caller enumeration in WebKit were not run or re-checked here.
Extended reasoning...
The change adds a ~39k-line Rust crate replacing ICU4C's C API for Bun on Linux, Windows, FreeBSD and Android, plus build scripts, test fixtures, and CLAUDE.md edits; it touches parsing of linked and on-disk ICU data (ICU_TIMEZONE_FILES_DIR) and host locale/time zone queries, but no auth or crypto surface. Two confirmed findings are posted inline, so approval is not on the table; this body only records the two additional candidates examined and ruled out, and that no build or test was run by this review.
| match read_file(&path)? { | ||
| Some(file) if is_acceptable(&file) => return Ok(file), | ||
| Some(_) => error = U_INVALID_FORMAT_ERROR, | ||
| None => {} | ||
| } | ||
| } | ||
| Err(error) |
There was a problem hiding this comment.
🔴 Operators whose ICU_TIMEZONE_FILES_DIR holds a zoneinfo64.res (or timezoneTypes/windowsZones/metaZones) that the header check rejects lose all time zone data after merging; ICU4C skipped such a file and used the linked package. time_zone_file at src/icu/common/udata.rs:205-211 turns "found but unacceptable" into Err(U_INVALID_FORMAT_ERROR), and open_choice propagates it with ? at udata.rs:239 before ever searching the package. Fix: treat an unacceptable override file as not found for the package lookup (keep U_INVALID_FORMAT_ERROR only when the package also lacks the item), matching ICU4C's non-fatal subErrorCode, for all four names. Big-endian tz-update files, truncated or non-ResB files trigger it. [also at: src/icu/common/udata.rs:216 - An operator whose ICU_TIMEZONE_FILES_DIR holds a time zone file the reader rejects loses all time zone data, where ICU4C silently fell back to the linked package.]
Why this was flagged
ICU_TIMEZONE_FILES_DIR points at a directory whose zoneinfo64.res is big-endian, truncated below a 20-byte header, or not a ResB v1-3 bundle. read_file returns Some(file), accepts(&file) is false, so error becomes U_INVALID_FORMAT_ERROR and the loop ends with Err(error) at udata.rs:211; the match at udata.rs:213-216 only maps U_FILE_ACCESS_ERROR to Ok(None), so open_choice returns the error at udata.rs:239 and never builds the "icudt79l/zoneinfo64.res" entry or calls package::find. cached_part (src/icu/i18n/timezone.rs:185) and find_name/zones then fail, TimeZone::create_time_zone (timezone.rs:266-277) returns the Etc/Unknown fixed zone for every ID, so Date prints in UTC and every Intl.DateTimeFormat/Temporal timeZone is rejected as invalid, with no message naming the file. ICU4C's doLoadFromIndividualFiles records an unacceptable file only in the non-fatal subErrorCode and doOpenChoice goes on to the common data, so the same environment works on the base.
Verification: In src/icu/common/udata.rs, time_zone_file for a file that reads but fails is_acceptable sets error = U_INVALID_FORMAT_ERROR (line 207) and ends with Err(error) (line 211); the only error mapped back to "not found" is U_FILE_ACCESS_ERROR (line 214). open_choice propagates that with ? at line 239, before the package lookup package::find(&entry) at line 257 is ever reached.
There was a problem hiding this comment.
Does not reproduce as a regression: the base does not fall back to the linked data either. I ran a build of the base and one of this branch with each of the four files big-endian, with another format, version or magic number, cut to 10 and to 3 bytes, and as text. Output and exit code are the same in all 28 cases (windowsZones through ucal_getHostTimeZone under Wine). Left as it is.
Generated by Claude Code
What does this PR do?
src/icureplaces the code of ICU4C 79.1 on Linux, Windows, FreeBSD and Android. It exports the 189 functions of ICU's C API that JavaScriptCore, WTF and Bun's C++ call, for the arguments that they pass, and nothing else: 39,100 lines. The headers and the data are still ICU4C's and still come with WebKit, which does not change. macOS keeps the system's ICU.common/andi18n/(34,700 lines) and the bodies of the exported functions forbidunsafe. It is incapi/ffi/, which turns C's pointers into references, and insys/, which reads the linked data and asks the host for its locale and time zone (1,400 lines together).src/icu/README.mdhas the rules, and what to do when WebKit or ICU is upgraded.It aims for the results of ICU4C 79.1, but is not the same in every case: some rare results differ on purpose, and we know of no difference that is not intended. Others that you may notice:
ICU_DATAandICU_ENABLE_TENTATIVE_ERAare not read any more.ICU_TIMEZONE_FILES_DIRstill replaces the four time zone files.U_UNSUPPORTED_ERROR, or a debug assertion). That includes a source that overlaps the buffer ofu_strToLoweroru_strToUpper.upgrade-webkit.mdsays to look for that.ICU_TIMEZONE_FILES_DIR), and if it leavesDatewithout a calendar.$vm.dumpLineBreakData(), which builds with assertions have for WebCore, ends the process inbun.jscstill has it.bench/snippets/intl.mjs, release builds with LTO, Zen 4, Linux, best of several rounds. Time relative to the previous build, lower is better:NumberFormatDateTimeFormatLocaleTemporalCollatornormalizetoUpperCaseSegmenternew URLIt is more than 1% slower in four cases:
normalize("NFD")andnormalize("NFC")of Thai text (1.07, 1.04),normalize("NFC")of Korean text (1.02) andaddin the Persian calendar (1.02). Short loops depend on the address at which they are linked: Thai NFC was at 0.79 and 0.83 in two other builds.String(new Date(0))new Date(0).toLocaleString()(1234.5).toLocaleString("de")Startup without such a call is the same (3.8 ms). After all services were used in all locales, the library holds 15.6 MiB instead of 35.7 MiB, not counting decompressed data.
bunis 0.4 to 0.7 MB smaller on Linux, Android and FreeBSD, and 2.3 to 2.5 MB on Windows. With--compile --aot --trim, hello world is 451 KB smaller, and a program that usesIntl.DateTimeFormat770 KB.Build:
bunlinks only ICU's data library.jscandtestaotkeep ICU4C. The names of the symbols end in_79, and astatic_assertfails if WebKit has another version of ICU. A local WebKit on Linux withoutBUN_ICU_ROOTuses the system's ICU as before, and the crate is empty (--cfg=bun_system_icu). A local WebKit leaves out three files of WTF that only WebCore uses.Deleted:
bun_icu_malloc.cpp. The crate uses the global allocator.Left for later: the data package has items that nothing opens (
brkitr/line*.brk,title.brk,ulayout.icuand others).How did you verify your code works?
test/js/web/intl/intl-goldenhas 6,870 expressions with the results of ICU4C 79.1, in the test's process, in 40 environments and in workers. They were recorded with the previous build, so they pass there too. A debug build runs a quarter of them. They notice 3,158 of 5,793 small mistakes planted in the crate of the first commit, of which 4,042 change a value while they run.hardened-icu.test.tsruns them on a hardened build.cargo test -p bun_icucalls exported functions as C++ does, and in most tests refuses each allocation in turn. It runs under Miri too.JSTests/stress, run in ajscthat is linked with this crate, 12,551 get the result of ICU4C from each call. The other 27 pass.Intl,Temporal,Date, normalization, case mapping, regular expressions, hosts of URLs), 99.6% give the result of ICU4C. The others are the intended differences.--trim-localesremoves.compile-aot-sizelooks for other strings of the formatters,build-local-webkitexpects ICU's code only in the programs,hardened-executableexpects__memmove_chkin place of__strcpy_chk, which only ICU4C called,refusable-memorycovers the crate, andlayers.pyknows its symbols.bunoutside of CI. CI also runs the tests on Windows, musl and arm64, but only builds for Android and FreeBSD. Speed is measured only on Linux x64.LANG,LC_*orTZ. The code that asks Windows for its time zone and locale ran under Wine instead: 293,000 combinations of zone, region and flags and 950 names of locales give the result of ICU4C.ctrl-c.test.tsis the version of test: fold 57 pending test-only PRs into one PR #49196, which needs no registry. It has nothing to do with ICU.Generated by Claude Code