Skip to content

test: keep the worker alive until terminate() in the work-in-flight tests of worker_threads.test.ts - #46647

Closed
robobun wants to merge 3 commits into
mainfrom
robobun/37938d08/deflake-worker-terminate-in-flight
Closed

robobun wants to merge 3 commits into
mainfrom
robobun/37938d08/deflake-worker-terminate-in-flight

Conversation

@robobun

@robobun robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Behaviour change: none

Problem

  • terminate with work in flight > a transpile queued on the thread pool that starts after terminate() in test/js/node/worker_threads/worker_threads.test.ts fails intermittently. expect(await w.terminate()).toBe(1) reports Received: 0, less often Received: undefined.
  • The worker queues one thread pool job, posts "go", and has no other work. If the job completes before the parent's terminate() lands, the worker exits on its own and terminate() resolves its code, 0.
  • The digest and zlib tests beside it fail the same way.

Fix

  • Each of the three workers now holds setInterval(() => {}, 1e6). Only terminate() can stop it, so the exit code is always 1.
  • The runtime does not change. Node v26.3.0 also resolves 0 for these workers when the parent is late.
  • The tests still terminate with a job in flight: the parent reacts to "go" as before.
  • Verified: release build under CPU load, 300 interleaved runs each: original 26 failures, fixed 0. Debug ASAN build: the three tests pass in 3 of 3 runs of the file.

Background

  • worker.terminate() resolves the worker's exit code: 1 when the parent stops a running worker. A worker that already began its own exit keeps its own code.
  • A worker exits on its own when its event loop has no pending job, timer, or listener.
  • Considered a larger big.ts (it narrows the race) and accepting 0 or 1 (it hides a terminate() that fails to stop a running worker).
Notes

Reproduction. Release build of this branch (src/ is the same as main at 6292c65), Linux x64, 12 cores:

bun test ./test/js/node/worker_threads/worker_threads.test.ts -t "a transpile queued on the thread pool"
Condition Original test With the interval
Idle machine 0 of 170 runs fail 0 of 300
12 busy-loop processes, runs interleaved 26 of 300 fail (25 Received: 0, 1 Received: undefined) 0 of 300
Older release build (367d939), idle 3 of 420 fail 0 of 300
Older release build, 12 busy loops, interleaved 20 of 200 fail 0 of 200

The sibling tests on the release build of this branch, under the same load, interleaved:

Test Original With the interval
a SubtleCrypto digest still on the work queue at terminate() 7 of 150 fail 0 of 150
an async zlib job on the thread pool at terminate() 1 of 150 fail 0 of 150

Cause. A probe ran the transpile worker with two markers: one message from the import().then() callback and one from process.on("beforeExit"). In 9000 workers across 6 parallel processes, 20 exited with code 0. Each of the 20 had posted the beforeExit marker, so its event loop drained before the termination request arrived. No worker exited 0 without that marker.

Runtime path. WebWorker::shutdown first removes the published VM handle (src/jsc/web_worker.rs:1024). request_termination sets terminated_by_parent only while the handle is published (src/jsc/web_worker.rs:559), and only that flag turns the exit code into 1. Received: undefined is the later stage of the same schedule: the parent already handled the exit, and terminate() on an exited worker resolves undefined (src/js/node/worker_threads.ts:1070), as in Node.

Node parity. The parent blocks in Atomics.wait after "go", so each job completes before terminate() is called. Five workers per case:

Worker source Bun release Node v26.3.0
transpile, digest, zlib (original) [0,0,0,0,0] each [0,0,0,0,0] each
transpile, digest, zlib (with the interval) [1,1,1,1,1] each [1,1,1,1,1] each

How late the parent must be. Time in the worker from postMessage("go") to beforeExit, 25 samples: transpile 11.4 to 22.3 ms, digest 40.1 to 102.7 ms, zlib 45.2 to 63.1 ms.

Job in flight at terminate(). The same probe with the interval: 0 of 7200 workers (6000 transpile, 600 digest, 600 zlib) exited 0. The job was already complete when terminate() landed in 20 of the 6000 transpile workers. Without the interval it was 22 of 9000.

Whole file. Release build: 161 pass. Debug ASAN build, three runs: the three changed tests pass in each run. Runs two and three: 161 pass. Run one: 160 pass and one failure in a test that this PR does not touch, worker stop ordering as seen by the worker's own handlers > terminate(): nothing of the worker's runs after the request (Received [12, 0]). #38194 describes that failure and covers it.

Related. #39067 keeps these three tests serial because of this race. With the interval they do not depend on an idle parent.


[auto-merge] gate passed · iteration 1 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/node/worker_threads/worker_threads.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/js/node/worker_threads/worker_threads.test.ts
bun test v1.4.4 (367d939d9)

test/js/node/worker_threads/worker_threads.test.ts:
(pass) support eval in worker [1169.11ms]
(pass) all worker_threads module properties are present [25.17ms]
(pass) markAsUncloneable and markAsUntransferable markers are private, unforgeable, and permanent [33.31ms]
(pass) markAsUntransferable and markAsUncloneable ignore a WebAssembly GC reference [27.42ms]
(pass) all worker_threads worker instance properties are present [185.89ms]
(pass) threadId module and worker property is consistent [244.47ms]
(pass) receiveMessageOnPort works across threads [1129.79ms]
(pass) receiveMessageOnPort works as FIFO [14.52ms]
(pass) you can override globalThis.postMessage [1011.98ms]
(pass) support require in eval [938.76ms]
cwd /workspace/bun
realpath test/js/node/worker_threads/fixture-argv.js
(pass) support require in eval for a file [847.88ms]
(pass) support require in eval for a file that doesnt exist [892.25ms]
(pass) support worker eval that throws [870.35ms]
(pass) execArgv option > inherits the parent's execArgv when falsy or unspecified [4463.68ms]
(pass) execArgv option > provides empty execArgv when passed an empty array [1900.36ms]
(pass) execArgv option > can specify an array of strings [2234.09ms]
(pass) eval does not leak source code [14551.78ms]
(pass) captured stdio backpressure > stdout write completion is withheld until the parent reads [1039.00ms]
(pass) captured stdio backpressure > large stdout survives writev batching and repeated acks [1373.46ms]
(pass) captured stdio backpressure > captured stdio that is never consumed does not prevent exit [1797.90ms]
(pass) stdio is flushed when the worker exits synchronously > captured stdout: console + raw write, then process.exit(0) [1225.21ms]
(pass) stdio is flushed when the worker exits synchronously > captured stderr: console + raw write, then p
... (truncated)
Exit: 0
diff hotspot
test/js/node/worker_threads/worker_threads.test.ts | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

gate history · 2 passed · 0 rejected · iteration 1

evidence per changed file
file                                                reads  edits  tests
test/js/node/worker_threads/worker_threads.test.ts      2      1     26

…ests of worker_threads.test.ts

Three "terminate with work in flight" tests start a worker that queues
one thread pool job, posts "go", and has nothing else to do. The parent
calls terminate() when "go" arrives and expects exit code 1.

If the job completes before terminate() lands, the worker drains its
event loop and exits on its own. terminate() then resolves that exit's
code (0), or undefined if the parent already handled the exit. Node does
the same. The assertion failed whenever the parent reacted late.

Each of the three workers now holds an interval, so only terminate()
can stop it and the exit code is always 1.
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 2ea2670 and c2f3c34.

Files selected for processing: 1
  • test/js/node/worker_threads/worker_threads.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 0 remain after this review.

Walkthrough

The worker termination tests add interval keepalives before starting pending transpilation, SubtleCrypto digest, and zlib work. This keeps each worker running until the test calls terminate().

Changes

Worker termination tests

Layer / File(s) Summary
Keep workers active during pending work
test/js/node/worker_threads/worker_threads.test.ts
The transpilation, SubtleCrypto, and zlib worker scripts start an interval before their work, keeping the workers active for the termination tests.

Suggested reviewers: dylan-conway

Priority: Low

Merge Risk: Minimal · up to c2f3c

The test-only change has no identified merge-blocking risk and is ready to merge.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main test change: keeping workers alive until terminate() in the work-in-flight worker_threads tests.
Description check Passed The description explains the intermittent failure, the interval-based fix, test verification, reproduction details, and unrelated failures. It does not use the exact template headings, but it provides…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the intermittent failure, the interval-based fix, test verification, reproduction details, and unrelated failures. It does not use the exact template headings, but it provides the required change and verification information in equivalent sections.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 1:44 AM PT - Oct 24th, 2026

❌ @robobun, your commit c2f3c34 has 2 failures in Build #131727 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 46647

That installs a local version of the PR into your bun-46647 executable, so you can run:

bun-46647 --bun

@robobun

robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status

Reproduced on a release build of main (6292c65), Linux x64, with 12 busy-loop processes in the background:

for i in $(seq 1 300); do bun test ./test/js/node/worker_threads/worker_threads.test.ts -t "a transpile queued on the thread pool" > /dev/null 2>&1 || echo fail; done | wc -l
  • The original test fails 26 of 300 runs (Expected: 1, Received: 0).
  • With the change in this PR, the same loop gives 0 failures in 300 runs.

CI

The diff is green. The three changed tests pass on every lane that ran them, in all three CI runs. Each run has one red job, and no red job fails a test that this PR changes:

  • Build 131727 (current head, c2f3c34): :darwin: 27 aarch64 - test-bun. test/js/bun/http/serve-body-leak.test.ts and test/js/web/streams/native-source-onclose-leak.test.ts fail. The two macOS x64 shards had not started when this was written.
  • Build 131561: :windows: 2019 x64 - test-bun. test/js/first_party/ws/ws-proxy.test.ts fails. Every other lane passes, macOS x64 included.
  • Build 131534: :windows: x64 - verify-baseline. A JPEG codec test in test/js/bun/jsc-stress/fixtures/simd-baseline.test.ts timed out under CPU emulation.

In build 131727 the ASAN lane ran worker_threads.test.ts twice. The first attempt failed in worker stop ordering as seen by the worker's own handlers > terminate(): nothing of the worker's runs after the request (Received [12, 0]), and the second attempt passed. This PR does not change that test, and #38194 covers that failure.

The head commit differs from the commit of build 131561 only by a shorter code comment. The PR is ready for a maintainer to merge.

PR: #46647

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

LGTM — a small test-only de-flake: a ref'd setInterval in each worker removes the race between the pool job finishing and terminate() landing, while the parent still terminates on the same "go" signal so the in-flight teardown path these tests exist for is unchanged.

Extended reasoning...

The diff adds eight lines to test/js/node/worker_threads/worker_threads.test.ts, interpolating a shared setInterval keepAlive into the three worker sources under "terminate with work in flight"; no runtime or native code changes. It touches no security-sensitive surface. The tests' stated purpose (per the describe-block header) is to exercise VM teardown with pool work pending, where a broken build crashes or trips ASAN rather than failing an assertion, and the keepAlive leaves that path intact since the parent still calls terminate() as soon as "go" arrives; the exit-code assertion now holds because only terminate() can stop a ref'd timer, matching Node. The one inline finding is a comment-style nit, and no third-party objections are outstanding.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the latest push collapses the keep-alive comment to a single line, which was the only point raised last time, and the test change itself is sound.

What was reviewed:

  • The keepAlive splice in all three workers: it sits before postMessage("go") in each, so the parent still calls terminate() only after the pool/crypto/zlib job has been queued; the in-flight teardown path these tests exist for is unchanged.
  • expect(await w.terminate()).toBe(1) can still fail: with a ref'd interval the worker cannot reach a natural exit, so a 0/undefined now means terminate() failed to stop a running worker rather than a scheduling race.
  • The fourth test (fetch streaming, subprocess) and the rest of the file are untouched; nothing is skipped or weakened.
Extended reasoning...

Test-only change confined to the "terminate with work in flight" describe block of test/js/node/worker_threads/worker_threads.test.ts: a shared setInterval(() => {}, 1e6) snippet is interpolated into three worker sources so the worker's event loop stays open until the parent's terminate() lands, removing the race where a finished job let the worker exit 0 first. It touches no runtime code and no security-sensitive surface. The only prior review feedback (a three-line narrating comment) was addressed by the follow-up commit, the changed file is not covered by CODEOWNERS, and the hunt ran dry with no findings, so the change is small and self-contained enough to approve.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #49196, which combines the open test-only PRs into one. The commit 989b2e3c5 in that PR is this change, unchanged.

Closing this one in favor of #49196.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant