Repository navigation
[WTF] Unix: strip the codeset/modifier before testing for the C locale in platformLanguage() - #428
Conversation
…e in platformLanguage() setlocale(LC_CTYPE, nullptr) yields language[_territory][.codeset][@modifier]. platformLanguage() compared the whole string against "C"/"POSIX" and only stripped the codeset afterwards, so "C.UTF-8" — glibc's C.UTF-8 and the value bionic reports by default for every process — came out as the language tag "C". That is not a valid BCP 47 tag, canonicalizeUnicodeLocaleID() rejects it, and JSC's default locale fell through to ICU's uloc_getDefault(), which is en_US_POSIX there: Intl.Collator().resolvedOptions().locale was "en-US-u-va-posix", "a".localeCompare("B") was 1 and numbers lost their grouping separators. An @modifier was likewise passed through into the tag ("sr_RS@latin" -> "sr-RS@latin"). Strip everything from the first '.' or '@' first, then apply the C/POSIX check and the '_' -> '-' normalization. "C.UTF-8"/"POSIX.UTF-8" now give "en-US" like "C" does; "sr_RS@latin" gives "sr-RS"; all other inputs are unchanged.
…ip the two node getServers()-non-empty tests on Android The Intl test guards the WTF platformLanguage() fix (oven-sh/WebKit#428): bionic's default "C.UTF-8" used to yield the tag "C" and JSC fell back to en-US-u-va-posix (case-first collation, no digit grouping). It passes everywhere else today. test-dns.js/test-dns-get-server.js assert getServers().length > 0, which the platform resolver cannot satisfy by design.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughChangesLocale normalization
Merge Risk: ⚪ Minimal · up to This localized change corrects Unix locale parsing for codesets and modifiers; no actionable merge-blocking risk remains after normal checks and review. 🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
Comment |
There was a problem hiding this comment.
LGTM — small, well-reasoned fix that makes C.UTF-8 resolve to en-US instead of leaking an invalid "C" tag; the @modifier nit below is a fair follow-up but not a blocker.
Extended reasoning...
Overview
This PR touches a single function, platformLanguage() in Source/WTF/wtf/unix/LanguageUnix.cpp (~5 lines net). It reorders the POSIX-locale normalization so that the .codeset and @modifier suffixes are stripped before the string is compared against "C"/"POSIX", rather than after. The motivating bug is that bionic (Android) reports C.UTF-8 by default, which the old code failed to recognize as the C locale, returned the invalid BCP 47 tag "C", and caused Intl to fall through to ICU's en_US_POSIX — producing case-sensitive collation and un-grouped number formatting observable from JS.
Security risks
None. This is pure string normalization of a locale name read from setlocale(LC_CTYPE, nullptr). No injection surface, no auth, no memory-safety concerns — String::left() is only called when find() returned a real index, and the templated CodeUnitMatchFunction overload of String::find accepts the captureless lambda directly.
Level of scrutiny
Low. The change is mechanical and the PR description includes an explicit before/after truth table covering the empty string, bare C/POSIX, codeset-only, modifier-only, and codeset+modifier forms. I walked through each and they check out against the new code path.
Other factors
The bug-hunting pass raised one legitimate nit: for the rare codeset-less @modifier locales (ca_ES@valencia, sr_RS@latin), the old code's accidental behavior of producing an invalid tag caused JSC to fall through to uloc_getDefault(), which happened to preserve valencia as a BCP 47 variant. Stripping @modifier now yields a valid ca-ES and short-circuits that fallback, so the variant is lost. This is a real but very narrow regression (codeset-less @modifier glibc locales after an explicit setlocale), and the fix it enables — correct default collation on every Android process — clearly outweighs it. It's flagged inline as a follow-up, not a blocker, and I agree with that framing.
Preview Builds
|
…8 fix); test it via a forced C.UTF-8 locale on Linux; adapt netd's Job to the new JobContext::run signature WEBKIT_VERSION points at the preview tag until #428 merges; the Linux-run intl test fails on the previous WebKit (en-US-u-va-posix) and passes with it.
…reats C.UTF-8 as the C locale) Replaces the preview pin; #428 is the only commit between the previous pin (687eb8e1b73c) and this one.
… locale (WebKit bump) (#38246) ### What does this PR do? Running the full test suite on the `bun-linux-{x64,aarch64}-android` artifacts (API 35/29/28 emulators, `scripts/runner.node.mjs` on-device under Termux's node) surfaced Android-only problems. This lands the two that have clean fixes; the DNS work found by the same run is deliberately not in here (see below). - **Every `bun build --compile` executable segfaulted at startup.** Standalone binaries are PIE on Android (bionic requires it; Linux/FreeBSD builds are `-no-pie`), but the reader dereferenced the embedded module graph's *link-time* vaddr. It now adds the load bias — `dlpi_addr` of the object containing `BUN_COMPILED`, found by address via the existing `bun_sys::elf::find_loaded_module` (which returns 0 for the non-PIE executables, so Linux/FreeBSD are unchanged). The faulting address was exactly the payload's unrelocated vaddr, in 25+ test files. - **`Intl`'s default locale was `en-US-u-va-posix` on Android** (`"a".localeCompare("B") === 1`, sorts `A,B,a,b`, no digit grouping): bionic reports `"C.UTF-8"` from `setlocale(LC_CTYPE, nullptr)` by default and WTF's `platformLanguage()` only recognised bare `"C"`. Fixed in WTF (oven-sh/WebKit#428, merged) and picked up here by bumping `WEBKIT_VERSION` to e2f13c6aa1cd — the only commit past the previous pin. Two tests pin it: the default locale is never the posix fallback, and (Linux) forcing `C.UTF-8` via `setlocale` still yields `en-US` — that one fails on the previous WebKit. - The runner/harness support for `android` hosts (`isAndroid`, `getAbi`, `libcPathForDlopen`, …) already landed with #38242. **Not included: DNS.** `dns.resolve*`/`reverse`/`lookupService` time out on Android because c-ares has no way to learn the device's nameservers; the right mechanism is the platform resolver (`android_res_nquery` for records, bionic `getnameinfo`/`gethostbyaddr_r` for names). A working, on-device-verified transport for that existed on this branch, but it was integrated as a second transport inside the c-ares-shaped `Resolver` with special cases through its timer/cancel/setServers/teardown paths — the wrong shape to land. It will come back as its own PR structured as `Resolver` owning a `Transport`. Until then Android behaves as on main: `dns.lookup`, `fetch`, and everything `getaddrinfo`-based work; `resolve*` needs `dns.setServers()`. ### How did you verify your code works? - `--compile`: on-device with the CI artifact from this branch on x86_64 (API 28/29/35) and arm64 (API 29/35, separate Apple-Silicon-hosted emulator run) — plain and `--bytecode` outputs run where they segfaulted before; `bundler_compile.test.ts` passes on arm64; Linux `bun bd build --compile` variants unchanged. - Intl: reproduced on Linux by forcing `C.UTF-8` (`en-US-u-va-posix` on the old WebKit), green with the bump on every Linux lane (glibc/musl, x64/arm64, ASAN); the arm64 Android run confirmed the on-device symptom and root cause. - Minimum API unchanged (28): no new libc imports; verified the artifact loads and runs on an API 28 image. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
setlocale(LC_CTYPE, nullptr)yieldslanguage[_territory][.codeset][@modifier].platformLanguage()compared the whole string against"C"/"POSIX"and only stripped the codeset afterwards, so"C.UTF-8"— glibc'sC.UTF-8, and the value bionic reports by default for every process — came out as the language tag"C". That is not a valid BCP 47 tag,canonicalizeUnicodeLocaleID()rejects it, and JSC's default locale falls through to ICU'suloc_getDefault(), which isen_US_POSIXthere. Observable in Bun on Android (found by running Bun's test suite on an arm64 emulator):new Intl.Collator().resolvedOptions().locale→"en-US-u-va-posix","a".localeCompare("B")→1(−1 elsewhere),["b","A","a","B"].sort(Intl.Collator().compare)→A,B,a,b,(1234.5).toLocaleString()→"1234.5". On glibc the process starts in"C", so Linux embedders that never callsetlocalewere unaffected — but one running underC.UTF-8aftersetlocale(LC_ALL, "")hits the same path. An@modifierwas likewise passed through into the tag ("sr_RS@latin"→"sr-RS@latin").This strips everything from the first
.or@first, then applies the C/POSIX check and the_→-normalization. Truth table (old → new):C.UTF-8:C→en-US;C.utf8:C→en-US;POSIX.UTF-8:POSIX→en-US;sr_RS@latin:sr-RS@latin→sr-RS;C,POSIX, empty,en_US,en_US.UTF-8,de_DE.UTF-8@euro,ca_ES.UTF-8@valencia,zh_CN.GB18030,fr: unchanged.