Skip to content

Bump WebKit: a marking helper leaves a collection that no thread conducts - #49338

Draft
robobun wants to merge 6 commits into
mainfrom
robobun/7542688f/marking-helper-leaves-idle-collection
Draft

robobun wants to merge 6 commits into
mainfrom
robobun/7542688f/marking-helper-leaves-idle-collection

Conversation

@robobun

@robobun robobun commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #49314

Problem

  • A process with Workers can stop collecting its main heap: heapUsed stays byte-identical for 17 s while heapTotal grows by 4.7 GB.
  • Stacks: the Collector Thread waits with no time limit in SlotVisitor::drainInParallelPassively. Every HeapHelper thread is parked in SlotVisitor::drainFromShared.
  • Cause: all heaps share the helper threads, and a helper stayed in a collection until it ended. A sleeping Worker with an unfinished collection kept them all.

Fix

Background

Downsides

  • One heap: churn at 3, 25 and 125 MB live within the noise (Notes). Text: +1,856 bytes.
  • A sleeping thread still leaves its own collection unfinished until it wakes or its GC timer fires.
  • The macOS OOM of the report is not reproduced.
Notes

The tests cannot fail without the pin. The change is in scripts/build/deps/webkit.ts, not in src/. With main's pin (cbf8d973991b) both tests fail: the helper marks 0 KB for the main heap (20 of 20 runs, and 20 of 20 with 12 busy loops beside it). With this pin the Atomics.wait test passes 40 of 40, 40 of 40 with 12 busy loops, 30 of 30 on 2 CPUs, and 12 of 12 on a debug+ASAN build.

Reproduction of the issue (release build, about 10 GB of RAM, 55 s): 8 idle Workers get one burst of allocation each, then the main thread reads a pipe and allocates. On main: heapUsed was byte-identical for 17.0s while heapTotal grew by 4.70GB, 6 of 6 runs. With the fix: 0.0s, 0.00GB, in the three forms of the script (pipe, 2 ms timer, less work per chunk).

A probe with the shape of the report: 25 or 32 Workers idle in their event loops, default GC settings, one empty message to each Worker while its collector thread conducts its idle collection, a main thread that works on a 2 ms timer. A first revision of the WebKit change left the main heap with no helper for the last 34 s of the run (cycles of 1.0 to 2.4 s). With the revision that is pinned here: no such cycle in 4 of 4 runs, longest cycle 195 to 782 ms.

Other ways a thread holds a collection (6 Workers, main-heap cycles with no helper, longest cycle), main's pin against the fix:

the Worker is in main fix
its event loop 4 of 6, 299 ms 0 of 16, 78 ms
Bun.spawnSync 5 of 5, 636 ms 0 of 16, 50 ms
Bun.sleepSync 5 of 5, 674 ms 1 of 13, 41 ms
a loop that does not allocate 6 of 6, 458 ms 0 of 16, 89 ms

Cost with one heap. Release builds without LTO of 7624a7ea1 with WebKit 0befb64c and with the fix on it, Linux x64, 12 cores, interleaved runs, min / median / max:

churn of records main fix
3 MB live, ns per record 269 / 278 / 290 265 / 278 / 294
25 MB live, ns per record 228 / 233 / 238 228 / 229 / 231
125 MB live, ns per record 273 / 278 / 282 270 / 275 / 280
125 MB live, peak RSS, MB 761 / 817 / 823 808 / 813 / 823
31 eden cycles, voluntary context switches of the 7 helpers 2847 to 3875 2911 to 3983

With one heap the helpers leave and come back 48 times in the 204 cycles of the 3 MB churn, and not at all in the two larger ones. BUN_JSC_gcHelpersLeaveWhenConductorIsAway=0 turns the new behaviour off.

Debug build. A debug+ASAN build with the preview (assertions on) ran the new test, a stress of 8 Workers that allocate and sleep (default, two markers, collectContinuously), and the three test directories. On this machine 7 tests of those directories pass the 5 s default limit of a debug build and pass with a longer limit (two FTL code age out tests at 6.2 and 6.6 s, five Worker destruction tests at 5.6 to 6.4 s). On release builds they take the same time with and without the fix.

The second new test (the collector thread gives the conn to a Worker that sleeps) is skipped on Windows, debug and ASAN, as its neighbours are: it needs a 4-million-object heap and Bun's idle collection on the collector thread. Its timer lands inside the idle collection in 8 of 10 runs, so the test does not require that it did.

Platforms. The new tests pass on the Linux, macOS (x64 and arm64) and Windows (x64 and arm64) lanes, and on the x64 lane with AddressSanitizer, whose JavaScriptCore has assertions on. On Windows only the Atomics.wait test runs.

What stays for later. drainInParallelPassively still waits with no time limit when the helpers are busy in another heap (oven-sh/WebKit#775 is about that wait). Workers that all mark at the same time still share one pool (#44186).


no test proof · iteration 2 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/gc/gc-controller-cadence.test.ts

@robobun

robobun commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 4:05 AM PT - Nov 20th, 2026

❌ @robobun, your commit ce47fa2 has 1 failures in Build #143262 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 49338

That installs a local version of the PR into your bun-49338 executable, so you can run:

bun-49338 --bun

@robobun

robobun commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

The one red test in CI, test/aot/order-file.test.ts on the x64 lane with AddressSanitizer, is not caused by this change. Every other lane passes, and the new tests pass on each of them.

  • In build 143262 two tests of that file passed their 300 s limits on the first attempt. In the builds of 22 other branches the file takes 220 to 296 s on this lane (294.8 s in build 143264, 296.1 s in build 143260). So those tests are close to their limits without this change.
  • In build 143250 a different test of the file failed. It kills the child as soon as the order file lists first, and the child had not yet written 8 to stdout.
  • The retries cannot pass. On this lane a retry runs on the Bun with debug information, so each bun build --compile writes a much larger executable (384 to 611 s per attempt). The shard then reaches its 45 minute limit.
  • On a local release build with AddressSanitizer of this branch, the file passes in 21 of 21 runs. It takes the same time with the new behaviour on and off (BUN_JSC_gcHelpersLeaveWhenConductorIsAway): 64.4 against 65.1 s (medians of 4 interleaved runs), and 107.0 against 106.4 s beside 12 busy loops.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSC concurrent GC stalls in SlotVisitor::drainInParallelPassively(MonotonicTime::infinity()) while HeapHelpers sleep, causing unbounded heapTotal growth

1 participant