Repository navigation
Conversation
A TimeZoneID was an index into a table of every time zone identifier. The first name that a process
gave to a time zone built the table: two enumerations of ICU's zones and a primary identifier for
every name, about 21 million instructions. The host's zone is such a name, so the first local-time
read of a Date built the table.
Under USE(BUN_JSC_ADDITIONS) a TimeZoneID is now the order in which the process first named an
identifier, and UTC is always 0. TimeZoneRegistry (IntlObject.cpp) keeps the identifiers that the
process has named. ICU is asked about one name at a time, by the rules of the table:
- A primary identifier as ICU spells it. ICU's canonical identifier of the name has to be its own
canonical identifier, which is how ICU chooses the names of its canonical enumeration. Then the
code that the table runs for such a name has to give the name back. What that code gives a
canonical identifier is kept, because an alias has the canonical identifier of its primary.
- Every other name is looked up in the names that ucal_openTimeZones lists, which is about
1.9 million instructions, once. The table takes its aliases from that list, so a name that the
list does not have is no zone. A name of the list that is no zone (PST, SystemV/EST5) is
remembered as none.
- Intl.supportedValuesOf("timeZone") asks the same about every name of the list. Upstream's table
stays for the tests, which hold the two against each other.
A thread reads the registry without a lock, as it read the table. A zone is complete before it can
be found and never changes, and an index that is replaced stays. With a WTF::Lock around a HashMap
in its place, new Temporal.ZonedDateTime(0n, name) takes 3,210 ns a call in each of 8 threads,
against 59 ns, and zoned.timeZoneId takes 16 ns in one thread, against 7 ns.
Linux x64, ICU 79.1, release build of jsc. Instructions of the first local-time read of a Date,
over the same script with getUTCHours():
TZ before after
unset 38,651,503 17,656,954
America/Los_Angeles 38,667,174 17,673,463
america/los_angeles 38,964,180 19,499,958
Invalid/Zone 38,978,922 19,472,196
What is left opens a calendar for the zone. The next commit replaces the calendar.
First use of names, over a script that names no zone, without the JIT:
Intl.supportedValuesOf("timeZone") 21,154,760 20,898,253
one primary identifier 21,125,646 87,139
the 445 primary identifiers, one by one 22,415,614 14,644,941
an alias 21,090,074 1,926,120
all 638 names of ICU, one by one 23,465,481 22,590,582
200 primary identifiers, then supportedValuesOf 22,129,332 21,932,502
Per call: new Temporal.ZonedDateTime(0n, name) 1,608 to 1,204 instructions, with an alias 2,046 to
1,169, with "PST" (throws) 5,874 to 4,968, with a name that ICU does not list (throws) 5,907 to
5,060.
Tests:
- stress/bun-time-zone-by-name-agrees-with-table.js compares the answer by name with upstream's
table for every identifier of the table, 25 other spellings of each and names that are no zones,
and Intl.supportedValuesOf("timeZone") with the primaries of the table.
- stress/bun-time-zone-first-use-lists-no-zones.js says what each kind of name makes ICU list, and
which names are answered without a question to ICU.
- stress/bun-time-zone-named-by-many-threads.js names zones in 8 threads at once, and for the first
time in 7 threads while the eighth lists every zone. A fault there shows in some runs only.
- complex/bun-host-time-zone-*.js give the host's zone as TZ gives it: a primary identifier, another
case, an alias in another case, a name that is no zone.
DateCache, Temporal's bridge to ICU and the date patterns of the AOT compiler each opened a
UCalendar to ask a time zone for its offsets, its transitions and its names. ICU's C API has no
time zone object. The ICU that Bun bundles opens no calendar without its likely-subtags data
(Calendar::setWeekData): about 17.6 million instructions and 470 page faults, once per process.
ICUTimeZone (runtime/ICUTimeZone.h) holds one zone for the three of them.
- Where Bun links the ICU that it bundles (Linux, Windows, FreeBSD, Android) it is an
icu::TimeZone. Its queries are the calls that ucal.cpp makes on the zone of a calendar.
ICUTimeZone.cpp is the only file that sees ICU's C++ headers, and it is in no unified source.
- On Darwin Bun links the system's ICU, of which only the C API is stable, and it is a UCalendar.
Two calls on that calendar are gone, ucal_setGregorianChange and ucal_clear: they change the
fields of a calendar, and no offset, transition or name of a zone. The calendar of the AOT
compiler was UCAL_GREGORIAN and is UCAL_DEFAULT, as the other two were.
Linux x64, ICU 79.1, release build of jsc. Instructions and minor page faults of the first
local-time read of a Date, over the same script with getUTCHours():
TZ before after
unset 17,656,954 / 493 91,734 / 22
UTC 17,645,966 / 488 69,780 / 24
America/Los_Angeles 17,673,463 / 490 106,231 / 23
america/los_angeles 19,499,958 / 507 1,946,885 / 41
With the calendar arm forced on Linux the first read is 17,628,747 / 490 (TZ unset). That is the
cost of the bundled ICU. The ICU of macOS was not measured.
Per call: a Date read that misses the cache of offsets 2,928 to 2,130 instructions, a local time
to UTC 2,957 to 2,608, an offset of a named zone through Temporal 3,077 to 2,274.
JSTests/microbenchmarks/temporal-zoned-date-time-time-zone-cache.js 4,368.5 to 2,423.8 million
instructions, temporal-zoned-date-time-from-named-time-zone.js 388.5 to 280.1 million.
Tests:
- stress/bun-time-zone-holder-loads-no-locale-data.js. The shell counts the items of ICU's data
that ICU loads (icuDataItemCount(), where the shell reads the data as Bun packs it). After the
first local-time read, the name of the host's zone, and the offsets and transitions of a named
zone, to maximize a locale still loads an item. With a calendar at one of these sites it loads
none, and the test fails.
- stress/bun-temporal-time-zone-transition-is-an-offset-change.js. getTimeZoneTransition() passes
over the transitions of ICU that change no offset, forward and backward.
- aot/tests/walk/builtin-calls/formatter-of-a-named-time-zone-is-laid-down-as-a-pattern.ir. The
pattern of a formatter that names a zone is laid down only where the offsets and the stretches
that findUTCOffset gives agree with what ICU formats.
|
Status: ready for review. It waits for CI and for a maintainer's answer to the two choices under Background. How the problem was reproduced (Linux x64, main at cbf8d97):
With this PR: 0.1 million instructions and 22 faults in Not run here: macOS and Windows. The lanes of this PR compile both. |
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it introduces a lock-free process-wide TimeZoneRegistry and the fork's first use of ICU's C++ API in paths under Source/JavaScriptCore (jsc-reviewers in CODEOWNERS), a human look would still be worthwhile, as the description itself asks for.
What was reviewed:
TimeZoneRegistrypublication order (segment store,m_zoneCountrelease, index slot release) against the lock-freefind()/zone()readers, index resizing with the kept-alive replaced index, andplaceOfsegment math.- The three calendar sites moved onto
ICUTimeZone(JSDateMath.cpp,TimeZoneICUBridge.cpp,AOTIntlBuiltinOperations.cpp): the transition loop's Previous/Next search points and the droppeducal_clear/ucal_setGregorianChangeon the DarwinUCalendarpath were checked — every query sets millis before reading and zone offsets do not depend on the Gregorian cutover. - Name resolution before and after ICU's name list exists (primary, alias, other-case, non-IANA) was traced for agreement with the table rules; the RangeError-consistency concern was ruled out because every primary identifier is on the
ucal_openTimeZoneslist. - Not done in this run: no build,
testaot, or JSTests execution — the new stress/complex tests and the macOS/Windows paths were only read.
Extended reasoning...
The change replaces JSC's eager all-zones time zone table with a per-name registry under USE(BUN_JSC_ADDITIONS), adds a new ICUTimeZone wrapper (icu::TimeZone on non-Darwin, UCalendar on Darwin), rewires DateCache, Temporal's TimeZoneICUBridge and the AOT Intl helper onto it, and adds nine JSTests plus a testaot IR test and $vm/jsc hooks. It touches no auth, injection or data-exposure surface. The bug hunt ran dry with no findings and the ruled-out candidates hold up on reading, but the PR is about 1,800 lines, hand-rolls a lock-free hash index and segmented arena read without a lock, and is the fork's first use of ICU's C++ API, which the author explicitly flags as needing a maintainer decision; the paths are also covered by CODEOWNERS. Those facts decided defer over approve.
|
Preview build of 420e51a: |
|
Follow-up on the first What loads the table on this head
Why no pull request
The branch |
Problem
Datetakes 38.7 million instructions and 520 page faults, for everyTZ. Bun: 2.5 ms. Node: 0.05 ms. @sosukesuzuki asked for this fix, which finishes VM: skip eager timezone prewarm under USE(BUN_JSC_ADDITIONS) #322.DateCachethen opens aUCalendarfor the zone, which loads ICU's likely-subtags data: 18 million.Fix
TimeZoneRegistrynumbers zones as the process names them, and asks ICU about one name.ICUTimeZoneholds a zone as anicu::TimeZoneat the three calendar sites. Darwin keeps theUCalendar.TZset: 2.50 ms to 0.02 ms, 487 faults to 4. 802TZvalues give identical output.testaottest, JSTests--quick, bun's time zone tests under LeakSanitizer. Self-reviewed: 30 concerns raised, 29 addressed.Background
icu::TimeZoneis in its C++ API, which WebKit hides.Calendar::setWeekData) that keeps the calendar: 236 thousand instructions for the first read, against 106 thousand, and no per-call gain.Downsides
TZunset with a regular-file/etc/localtime, and the firstIntl.DateTimeFormat(32 million instructions).Notes
How the numbers were made
jscand bun are release builds without LTO. "main" is cbf8d97, which is Bun'sWEBKIT_VERSIONand the base of this branch. bun is 7bb427f3b in every column.perf_event_open, all threads), median of 21 runs for a first read and of 5 to 9 runs elsewhere. A first read isnew Date(1700000000000).getHours()over the same script withgetUTCHours().First local-time read
jsc, instructions / minor page faults:TZUTCAmerica/Los_AngelesUS/PacificAsia/CalcuttaEST5EDTamerica/los_angelesInvalid/Zone:/etc/localtimebun, the same read. Instructions / faults over the
getUTCHours()script, and the time of the read in the process:TZUTCAmerica/Los_AngelesAsia/Calcuttaamerica/los_angelesInvalid/ZoneTZunset: bun 5.77 ms to 3.24 ms,jsc4.79 ms to 2.94 ms.jsc2,786,766 and 2,785,293 instructions, bun 3,928,173 and 3,932,316.TZset does about 210 thousand instructions more at its start than withTZunset (4,139,658 against 3,928,173), before anyDate. Skip the walk over live Dates at a time zone change when no Date has cached a local time bun#46614 is about that.What ICU does on the first read
Entries of ICU functions while
jscruns thegetHours()script (gdb breakpoints with ignore counts):initLikelySubtagsTZEnumeration::createucal_openucal_getIanaTimeZoneIDTZAmerica/Los_Angelesamerica/los_angelesTZunsetAmerica/Los_Angelesamerica/los_angelesFirst use of other entry points
jsc,TZ=America/Los_Angeles, instructions over an empty script:new Date(0).getHours()new Date(0).toString()Temporal.Now.timeZoneId()Temporal.Now.zonedDateTimeISO().hourTemporal.Instant.fromEpochMilliseconds(0).toZonedDateTimeISO("Asia/Tokyo").hournew Temporal.ZonedDateTime(0n, "America/New_York").hournew Intl.DateTimeFormat().resolvedOptions()new Intl.DateTimeFormat("en", { timeZone: "America/New_York" }).format(0)new Date(0).toLocaleString()Intl.supportedValuesOf("timeZone")The first
Intl.DateTimeFormatformats for a locale, so it still loads the likely-subtags data. About 11 million of the rest is the set of available locales (intlDateTimeFormatAvailableLocales), which converts every locale of ICU to a language tag.First use of names, over a script with a
ZonedDateTimeof an offset. No JIT, so that no helper thread compiles the loop of the script:UTCUS/Pacific)america/new_york)supportedValuesOfsupportedValuesOf, then all 638supportedValuesOfsupportedValuesOfsupportedValuesOfA primary identifier costs about 30 thousand instructions the first time, an alias up to 60 thousand. No order of names costs more than main.
Per call, after the first use
Instructions per iteration, from runs of N and 2N iterations with
--useConcurrentJIT=0:new Temporal.ZonedDateTime(0n, "+05:00")(no name)new Temporal.ZonedDateTime(0n, name), 8 primary nameszoned.timeZoneIda.equals(b)"PST", which ICU lists and which is no zone (throws)"SystemV/PST8"(throws)new Intl.DateTimeFormat("en", { timeZone })new Intl.DateTimeFormat("en", { timeZone: "PST" })(throws)new Date(t).getHours(),t20 days on each timenew Date(y, m, d).getTime(), 20 days on each timeinstant.toZonedDateTimeISO(name).offsetNanosecondsJSTests/microbenchmarks/temporal-zoned-date-time-time-zone-cache.js: 4,389.2, 4,368.5 and 2,423.8 million instructions (252, 255 and 157 ms).temporal-zoned-date-time-from-named-time-zone.js: 408.8, 388.5 and 280.1 million (30.6, 28.4 and 22.4 ms).process.env.TZand a local read in bun, per change: 263,988 to 234,387 instructions. The change alone is 210,465 and 210,478.The registry is read without a lock
Upstream's table is built once and then read without a lock. The registry grows, and keeps that property:
TimeZoneRegistry::resolve()andzone()have nolockprefix, and on the path of a hit they call a hash and a compare.new Temporal.ZonedDateTime(0n, name)in 1, 4 and 8 threads of one process: 1,534, 1,540 and 1,541 instructions a call on main, 1,137, 1,136 and 1,138 here.Wall time per call in each thread,
jsc, median of 7 runs:WTF::Lockand aHashMapas the registrynew Temporal.ZonedDateTime(0n, name), 1 threadzoned.timeZoneId, 1 threada.equals(b), 1 threadtimeZoneIdandequals. An offset or a field of aZonedDateTimestill takes the lock of Temporal's cache of zones, as on main.Recordsinaot/ir/AOTSharedAnswers.h: records that never move, an index of atomic slots, adds under a lock.Considered: a patch to ICU in place of commit 2
Calendar::setWeekDataasks for the minimized locale before it needs it, and that is what loads the likely-subtags data. First read injsc,TZ=America/Los_Angeles, instructions / faults, with commit 1 in every row:UCalendar, stock ICU (commit 1)"en_US"UCalendar, patched ICU, calendar opened with""as nowUCalendar, patched ICU, calendar opened with"en_US"icu::TimeZone(commit 2)Dockerfile,Dockerfile.musl,Dockerfile.freebsd,Dockerfile.android,Dockerfile.windows,build-icu.ps1, andscripts/build/hardened-icu.shin bun).Intl.DateTimeFormat, which commit 2 cannot reach. This hunk alone does not remove the load there: ICU's calendar of the default century is opened without a region (gregoimp.h).TZunset and/etc/localtimea regular fileICU then finds the host's zone by comparing that file with the files under
/usr/share/zoneinfo(443 here). JavaScriptCore asks once, and ICU asks again for its own default zone. On main both happen in the first read. Here the second one moves to the firsttoString()orIntluse.jsc, times in the process:getHours()toString()getHours(), thentoString()getHours(): 3.039 ms to 0.273 ms. node: 0.299 ms./etc/localtimea symlink, which is what every other table here has: 2.583 + 0.100 ms, 1.320 + 0.092 ms and 0.104 + 0.171 ms.Same results
TZ: the 638 names of ICU, 66 invalid or unusual values, 98 names in another case. For each: the offset at 86,000 instants from 1880 to 2100, 70,000 local times that include every skipped and repeated one of four years, both names inDate.prototype.toString(), and the default zone ofIntlandTemporal.TZof a new process, and each as a change inside one process ($vm.setHostTimeZone()injsc,process.env.TZ =in bun). Each of these with the firstDateread first, withIntl.DateTimeFormat()first, and withIntl.supportedValuesOf("timeZone")first.jscat commit 1,jscat commit 2,jscat commit 2 with theUCalendararm forced, and bun.--quick: 35,334 runs, 10 failures, none about dates or zones. 7 fail on main every time in this container: BigInt tests that run out of memory,auxiliary-evacuation-with-concurrent-compiles,get-array-length-concurrently-change-mode. The other 3 arebigint-terminate-*, which wait for the watchdog. They fail under load on main too, and pass alone.testaot all: 5,036 passed, 0 failed, 45 todo.API/tests/TemporalCoreTest.cpp, which no lane builds, through a driver that callstestTemporalCore(): all tests pass, with each arm.temporal.test.ts,temporal-global.test.ts,deep-equals-temporal.test.ts,cron.test.ts,cron-local-time.test.ts,cron-parse.test.ts,in-process-cron.test.ts,intl.test.ts,bun-jsc.test.ts,sql-timezone.test.js,cookie.test.ts,fake-timers.test.ts,v8-date-parser.test.js,test/aot/time-zone-at-start.test.ts,test-process-env-tz.js,test-datetime-change-notify.js.detect_leaks=1,BUN_DESTRUCT_VM_ON_EXIT=1,test/leaksan.supp): the same files without the AOT and sqlite ones, no leak report. An earlier revision allocated the slots of the registry's index withnew[]. With that put back,temporal.test.tsaborts withDirect leak of 128 byte(s)fromTimeZoneRegistry::Index::create.What each test catches
Each row is one change to the code, and the test that fails with it.
UCalendararm on Linuxbun-time-zone-holder-loads-no-locale-data.js: ICU has loaded 9 data items after the first read, where it loads 3getTimeZoneTransitiondoes not search on past a change of rulebun-temporal-time-zone-transition-is-an-offset-change.jsfindUTCOffset: a stretch that ends an hour late, starts 1 ms early, or leaves the DST offset outformatter-of-a-named-time-zone-is-laid-down-as-a-pattern.irbun-time-zone-by-name-agrees-with-table.jssupportedValuesOflists aliasesbun-time-zone-first-use-lists-no-zones.jsbun-time-zone-first-use-lists-no-zones.js(questions to ICU)complex/bun-host-time-zone-lists-no-zones.jsadd()does not look for the identifier firstbun-time-zone-named-by-many-threads.js, in 56 runs of 60isPrimaryTimeZoneIdentifierasks whether the canonical identifier of a name is its own canonical identifier, as ICU's canonical enumeration does. ICU 79.1 has no name for which the answer is no.How a name is resolved
ucal_getCanonicalTimeZoneIDgives the canonical identifier of the name. The code that the table runs for a canonical name (primaryTimeZoneIdentifierOfCanonicalICUTimeZone) has to give the name back.ucal_openTimeZoneslists. The table takes its aliases from that list, soUS:Pacific,UTC<NUL>xand a CLDR alias that the zone data lacks cannot pass. ICU's own lookup of one name accepts the first two.bun-time-zone-by-name-agrees-with-table.jsfails when that is not so.$vm.timeZoneTable(),$vm.timeZoneInTable()and$vm.timeZoneByName()are the hooks of that test. A release build of Bun has no$vm, so upstream's table is not linked into it: bun is 10,300 bytes of text smaller.jscis 6,824 bytes larger.Not in this change
UCalendarthere, so only commit 1 changes what macOS does. What a calendar costs with Apple's ICU was not measured. The 17.6 million above is the bundled ICU on Linux.ICUTimeZone.cppmakes ICU objects only with constructors that ICU's library defines.testedkey (.github/scripts/lanes.mjs). A test that compares the two ways to find a zone in bun itself belongs in the bun PR that movesWEBKIT_VERSION: it is the only one that meets Apple's ICU.offsetChange()moves toICUTimeZone::transition()), [JSC] The host time zone is not UTC when ICU has no name for it #890 (its threeucal_openprobes becomeICUTimeZone::create(id)->offsetAt(ms), or they put the load back), [JSC] Temporal is 2.5× faster: JIT intrinsics, a sort fast path, cached options and time zone offsets #901 (its cache entry holdszone). [JSC] CET, EET, MET and WET are aliases of their city zones with every tz data #897 and Intl: accept legacy IANA primary zones (CET, CST6CDT, EET, EST5EDT, MET, MST7MDT, PST8PDT, WET) #227 change which names the table takes: the resolver runs the table's rule functions, and the by-name test fails on any difference.TimeZoneRegistryhas nothing of Bun in it but theUSE(BUN_JSC_ADDITIONS)around it. It can go to WebKit if that is wanted.Self-review
30 concerns were raised, and this is what changed for them:
fastMallocblock. LeakSanitizer reported thenew[]of its slots in bun.Intl.supportedValuesOf("timeZone")is made by the resolver. It built upstream's table, and the registry took the table over, so there were two sources of zones in production.TZchange, and what is not fixed.Not addressed: no lane runs the tests on macOS or Windows.