Skip to content

[WTF] Unix: strip the codeset/modifier before testing for the C locale in platformLanguage() - #428

Merged
dylan-conway merged 1 commit into
mainfrom
claude/unix-language-strip-codeset
Aug 14, 2026
Merged

dylan-conway merged 1 commit into
mainfrom
claude/unix-language-strip-codeset

Conversation

@dylan-conway

Copy link
Copy Markdown
Member

setlocale(LC_CTYPE, nullptr) yields language[_territory][.codeset][@modifier]. platformLanguage() compared the whole string against "C"/"POSIX" and only stripped the codeset afterwards, so "C.UTF-8" — glibc's C.UTF-8, and the value bionic reports by default for every process — came out as the language tag "C". That is not a valid BCP 47 tag, canonicalizeUnicodeLocaleID() rejects it, and JSC's default locale falls through to ICU's uloc_getDefault(), which is en_US_POSIX there. Observable in Bun on Android (found by running Bun's test suite on an arm64 emulator): new Intl.Collator().resolvedOptions().locale → "en-US-u-va-posix", "a".localeCompare("B") → 1 (−1 elsewhere), ["b","A","a","B"].sort(Intl.Collator().compare) → A,B,a,b, (1234.5).toLocaleString() → "1234.5". On glibc the process starts in "C", so Linux embedders that never call setlocale were unaffected — but one running under C.UTF-8 after setlocale(LC_ALL, "") hits the same path. An @modifier was likewise passed through into the tag ("sr_RS@latin" → "sr-RS@latin").

This strips everything from the first . or @ first, then applies the C/POSIX check and the _→- normalization. Truth table (old → new): C.UTF-8: C → en-US; C.utf8: C → en-US; POSIX.UTF-8: POSIX → en-US; sr_RS@latin: sr-RS@latin → sr-RS; C, POSIX, empty, en_US, en_US.UTF-8, de_DE.UTF-8@euro, ca_ES.UTF-8@valencia, zh_CN.GB18030, fr: unchanged.

…e in platformLanguage()

setlocale(LC_CTYPE, nullptr) yields language[_territory][.codeset][@modifier].
platformLanguage() compared the whole string against "C"/"POSIX" and only
stripped the codeset afterwards, so "C.UTF-8" — glibc's C.UTF-8 and the value
bionic reports by default for every process — came out as the language tag
"C". That is not a valid BCP 47 tag, canonicalizeUnicodeLocaleID() rejects
it, and JSC's default locale fell through to ICU's uloc_getDefault(), which is
en_US_POSIX there: Intl.Collator().resolvedOptions().locale was
"en-US-u-va-posix", "a".localeCompare("B") was 1 and numbers lost their
grouping separators. An @modifier was likewise passed through into the tag
("sr_RS@latin" -> "sr-RS@latin").

Strip everything from the first '.' or '@' first, then apply the C/POSIX check
and the '_' -> '-' normalization. "C.UTF-8"/"POSIX.UTF-8" now give "en-US"
like "C" does; "sr_RS@latin" gives "sr-RS"; all other inputs are unchanged.
dylan-conway added a commit to oven-sh/bun that referenced this pull request Aug 14, 2026
…ip the two node getServers()-non-empty tests on Android

The Intl test guards the WTF platformLanguage() fix (oven-sh/WebKit#428):
bionic's default "C.UTF-8" used to yield the tag "C" and JSC fell back
to en-US-u-va-posix (case-first collation, no digit grouping). It passes
everywhere else today.

test-dns.js/test-dns-get-server.js assert getServers().length > 0, which the
platform resolver cannot satisfy by design.
@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: bb1a507b-5a80-4128-aca8-bddc0d2ee0c8

📥 Commits

Reviewing files that changed from the base of the PR and between 687eb8e and d7b7a6a.

📒 Files selected for processing (1)
  • Source/WTF/wtf/unix/LanguageUnix.cpp

Walkthrough

Changes

Locale normalization

Layer / File(s) Summary
Locale parsing and normalization
Source/WTF/wtf/unix/LanguageUnix.cpp
platformLanguage() strips codesets and modifiers, recognizes C.UTF-8 as the C locale, normalizes underscores to hyphens, and returns the complete language-territory value.

Merge Risk: ⚪ Minimal · up to d7b7a

This localized change corrects Unix locale parsing for codesets and modifiers; no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the bug and behavior changes, but it omits the required bug title, Bugzilla link, reviewer line, and changed-file summary. Add the required bug title and Bugzilla link, the “Reviewed by NOBODY” line, and the affected file and function entries from the repository template.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the Unix platformLanguage() change and the codeset/modifier handling involved.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small, well-reasoned fix that makes C.UTF-8 resolve to en-US instead of leaking an invalid "C" tag; the @modifier nit below is a fair follow-up but not a blocker.

Extended reasoning...

Overview

This PR touches a single function, platformLanguage() in Source/WTF/wtf/unix/LanguageUnix.cpp (~5 lines net). It reorders the POSIX-locale normalization so that the .codeset and @modifier suffixes are stripped before the string is compared against "C"/"POSIX", rather than after. The motivating bug is that bionic (Android) reports C.UTF-8 by default, which the old code failed to recognize as the C locale, returned the invalid BCP 47 tag "C", and caused Intl to fall through to ICU's en_US_POSIX — producing case-sensitive collation and un-grouped number formatting observable from JS.

Security risks

None. This is pure string normalization of a locale name read from setlocale(LC_CTYPE, nullptr). No injection surface, no auth, no memory-safety concerns — String::left() is only called when find() returned a real index, and the templated CodeUnitMatchFunction overload of String::find accepts the captureless lambda directly.

Level of scrutiny

Low. The change is mechanical and the PR description includes an explicit before/after truth table covering the empty string, bare C/POSIX, codeset-only, modifier-only, and codeset+modifier forms. I walked through each and they check out against the new code path.

Other factors

The bug-hunting pass raised one legitimate nit: for the rare codeset-less @modifier locales (ca_ES@valencia, sr_RS@latin), the old code's accidental behavior of producing an invalid tag caused JSC to fall through to uloc_getDefault(), which happened to preserve valencia as a BCP 47 variant. Stripping @modifier now yields a valid ca-ES and short-circuits that fallback, so the variant is lost. This is a real but very narrow regression (codeset-less @modifier glibc locales after an explicit setlocale), and the fix it enables — correct default collation on every Android process — clearly outweighs it. It's flagged inline as a follow-up, not a blocker, and I agree with that framing.

Comment thread Source/WTF/wtf/unix/LanguageUnix.cpp Resolved
@github-actions

Copy link
Copy Markdown

Preview Builds

Commit Release Date
d7b7a6af autobuild-preview-pr-428-d7b7a6af 2026-08-14 06:16:18 UTC

dylan-conway added a commit to oven-sh/bun that referenced this pull request Aug 14, 2026
…8 fix); test it via a forced C.UTF-8 locale on Linux; adapt netd's Job to the new JobContext::run signature

WEBKIT_VERSION points at the preview tag until #428 merges; the Linux-run
intl test fails on the previous WebKit (en-US-u-va-posix) and passes with it.
@dylan-conway
dylan-conway merged commit e2f13c6 into main Aug 14, 2026
47 checks passed
dylan-conway added a commit to oven-sh/bun that referenced this pull request Aug 14, 2026
…reats C.UTF-8 as the C locale)

Replaces the preview pin; #428 is the only commit between the previous pin
(687eb8e1b73c) and this one.
dylan-conway added a commit to oven-sh/bun that referenced this pull request Aug 14, 2026
… locale (WebKit bump) (#38246)

### What does this PR do?

Running the full test suite on the `bun-linux-{x64,aarch64}-android`
artifacts (API 35/29/28 emulators, `scripts/runner.node.mjs` on-device
under Termux's node) surfaced Android-only problems. This lands the two
that have clean fixes; the DNS work found by the same run is
deliberately not in here (see below).

- **Every `bun build --compile` executable segfaulted at startup.**
Standalone binaries are PIE on Android (bionic requires it;
Linux/FreeBSD builds are `-no-pie`), but the reader dereferenced the
embedded module graph's *link-time* vaddr. It now adds the load bias —
`dlpi_addr` of the object containing `BUN_COMPILED`, found by address
via the existing `bun_sys::elf::find_loaded_module` (which returns 0 for
the non-PIE executables, so Linux/FreeBSD are unchanged). The faulting
address was exactly the payload's unrelocated vaddr, in 25+ test files.
- **`Intl`'s default locale was `en-US-u-va-posix` on Android**
(`"a".localeCompare("B") === 1`, sorts `A,B,a,b`, no digit grouping):
bionic reports `"C.UTF-8"` from `setlocale(LC_CTYPE, nullptr)` by
default and WTF's `platformLanguage()` only recognised bare `"C"`. Fixed
in WTF (oven-sh/WebKit#428, merged) and picked up here by bumping
`WEBKIT_VERSION` to e2f13c6aa1cd — the only commit past the previous
pin. Two tests pin it: the default locale is never the posix fallback,
and (Linux) forcing `C.UTF-8` via `setlocale` still yields `en-US` —
that one fails on the previous WebKit.
- The runner/harness support for `android` hosts (`isAndroid`, `getAbi`,
`libcPathForDlopen`, …) already landed with #38242.

**Not included: DNS.** `dns.resolve*`/`reverse`/`lookupService` time out
on Android because c-ares has no way to learn the device's nameservers;
the right mechanism is the platform resolver (`android_res_nquery` for
records, bionic `getnameinfo`/`gethostbyaddr_r` for names). A working,
on-device-verified transport for that existed on this branch, but it was
integrated as a second transport inside the c-ares-shaped `Resolver`
with special cases through its timer/cancel/setServers/teardown paths —
the wrong shape to land. It will come back as its own PR structured as
`Resolver` owning a `Transport`. Until then Android behaves as on main:
`dns.lookup`, `fetch`, and everything `getaddrinfo`-based work;
`resolve*` needs `dns.setServers()`.

### How did you verify your code works?

- `--compile`: on-device with the CI artifact from this branch on x86_64
(API 28/29/35) and arm64 (API 29/35, separate Apple-Silicon-hosted
emulator run) — plain and `--bytecode` outputs run where they segfaulted
before; `bundler_compile.test.ts` passes on arm64; Linux `bun bd build
--compile` variants unchanged.
- Intl: reproduced on Linux by forcing `C.UTF-8` (`en-US-u-va-posix` on
the old WebKit), green with the bump on every Linux lane (glibc/musl,
x64/arm64, ASAN); the arm64 Android run confirmed the on-device symptom
and root cause.
- Minimum API unchanged (28): no new libc imports; verified the artifact
loads and runs on an API 28 image.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant