Skip to content

bun test --parallel: do not hang when a test closes the worker IPC fd - #37285

Closed
robobun wants to merge 4 commits into
mainfrom
farm/2b443fa1/parallel-fd3-close-hang
Closed

robobun wants to merge 4 commits into
mainfrom
farm/2b443fa1/parallel-fd3-close-hang

Conversation

@robobun

@robobun robobun commented Aug 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Under bun test --parallel, a test file that closes fd 3 hangs the whole run forever, with the worker spinning at 100% CPU and --timeout never applying:

// fd3close.test.js (plus any sibling test files)
import { test } from "bun:test";
import { closeSync } from "node:fs";
test("close fd 3", () => { closeSync(3); }); // e.g. a close-every-fd >= 3 hygiene helper
test("after", async () => { await Bun.sleep(200); });

bun test --parallel=2 . never finishes; serial bun test passes.

Cause

fd 3 in a worker is its IPC socketpair end. closeSync(3) removes the fd from the worker's poll set without any callback, so the worker never learns the channel died: its frames go nowhere and it sits in the next-command loop forever. The coordinator does observe the resulting EOF on its end, but the clean-EOF path in Worker::on_channel_done assumed EOF means the worker process exited (the normal case) and only killed the worker on the corrupt-frame path (is_attached()). With the worker still alive, try_reap waits for an exit notification that never arrives and the run hangs.

Fix

On channel-done, kill the worker whenever its process has not exited yet, not just on a corrupt frame. A worker that outlives its channel can neither receive a file nor report one, so it is lost either way. The in-flight file is accounted as worker crashed: SIGKILL, the slot respawns if files remain, and the run continues. Channel::is_attached lost its only caller and is removed.

One wrinkle: a worker doing a deliberate exit (process.exit(7) in a test, or the normal end-of-run path) closes its channel during VM teardown, before _exit. Under BUN_DESTRUCT_VM_ON_EXIT (set on CI's ASAN lanes) that teardown is long, so the coordinator would see EOF from a still-running process and the kill would clobber the real exit status (worker crashed: SIGKILL instead of exit code 7, caught by test/regression/issue/30205.test.ts in the first CI round). Workers therefore announce deliberate exits with a new Exiting IPC frame, sent while the channel is still up: a VM cleanup hook covers process.exit, and the worker loop's tail covers normal shutdown. The coordinator skips the kill for an announced exit and waits for the real status, which also lets ASAN leak checks in the worker finish. EOF with no announcement is a lost worker and is killed.

Fixed output for the repro above:

✗ fd3close.test.js (worker crashed: SIGKILL)
(pass) ok1
(pass) ok2
 2 pass
 1 fail
Ran 3 tests across 3 files.

Verification

  • New test in test/cli/test/parallel.test.ts: fails by timeout on an unfixed build, passes in ~2-4s on the fixed debug build.
  • test/regression/issue/30205.test.ts (all 4 tests) passes with and without BUN_DESTRUCT_VM_ON_EXIT=1; the process.exit(7) fidelity failure from the first CI round reproduced locally with that env and is fixed by the exit announcement.
  • Neighboring crash-path tests (worker process.exit mid-run, raw SIGABRT abort, garbage written to fd 3) pass in both env configurations.
  • Three load-sensitive tests in parallel.test.ts (lazily scales workers, partitions by directory, work-stealing balances) fail identically with and without this change on a core-capped local machine; unrelated.

no test proof · iteration 1 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/test/parallel.test.ts

…till running

A test that closes fd 3 (e.g. fd-hygiene code closing every fd >= 3)
closes the worker's IPC socketpair end out from under it. The fd
silently leaves the worker's poll set, so the worker never learns the
channel died and spins in its next-command loop at 100% CPU, while the
coordinator got a clean EOF, assumed the worker had exited, and waited
forever for an exit notification that never came: the run hung and
--timeout never applied.

Treat channel-done with the process still running as a lost worker,
the same as the corrupt-frame path: kill it so onWorkerExit accounts
for the in-flight file (worker crashed: SIGKILL) and the slot can
respawn, and the run continues. On a normal worker exit the EOF often
lands before the exit notification; the process is an unreaped zombie
then and the kill is a no-op.
… kill preserves their status

With BUN_DESTRUCT_VM_ON_EXIT (set on CI's ASAN lanes), a worker's
process.exit(7) tears down its VM, which closes the IPC socket well
before _exit. The coordinator then saw EOF from a still-running
process, treated the worker as lost, and kill(9)ed it mid-teardown:
the file was reported as 'worker crashed: SIGKILL' instead of
'exit code 7' (caught by test/regression/issue/30205.test.ts).

Send a Kind::Exiting frame from the worker's exit paths (a VM cleanup
hook covers process.exit; run_as_worker's tail covers the normal
shutdown) before teardown closes the channel. The coordinator skips
the lost-channel kill for an announced exit and waits for the real
status, which also lets ASAN leak checks in the worker run to
completion. EOF without the announcement (test closed fd 3, corrupt
frame) is still treated as a lost worker and killed.
Comment thread src/runtime/cli/test/parallel/Frame.rs Resolved
Comment thread src/runtime/cli/test/parallel/Worker.rs Resolved
Comment thread src/runtime/cli/test/parallel/Worker.rs Resolved
Comment thread src/runtime/cli/test/parallel/runner.rs Resolved
Comment thread src/runtime/cli/test/parallel/runner.rs Resolved
Comment thread src/runtime/cli/test/parallel/runner.rs Resolved
Comment thread src/runtime/cli/test/parallel/Frame.rs Resolved
Comment thread src/runtime/cli/test/parallel/Worker.rs Resolved
Comment thread src/runtime/cli/test/parallel/Worker.rs Resolved
Comment thread src/runtime/cli/test/parallel/runner.rs Resolved
Comment thread src/runtime/cli/test/parallel/runner.rs Resolved
@coderabbitai

coderabbitai Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

The parallel test runner announces deliberate worker exits before IPC teardown. The coordinator records this state and avoids treating intentional EOF as a crash. Unexpected IPC loss still fails the affected worker while allowing other files to continue.

Parallel worker exit handling

Layer / File(s) Summary
Exit announcement protocol
src/runtime/cli/test/parallel/Frame.rs, src/runtime/cli/test/parallel/Coordinator.rs
The protocol adds Kind::Exiting, decodes discriminant 9, and records the announcement on the associated worker.
Worker deliberate-exit announcements
src/runtime/cli/test/parallel/Worker.rs, src/runtime/cli/test/parallel/runner.rs
Workers track deliberate exits. Cleanup and normal shutdown paths emit and flush the Exiting frame before teardown.
IPC loss classification and regression coverage
src/runtime/cli/test/parallel/Worker.rs, src/runtime/cli/test/parallel/Channel.rs, test/cli/test/parallel.test.ts
EOF handling kills only unexpected, still-running workers. The Windows pipe comment reflects teardown ordering. The regression test validates continued execution, aggregated results, and exit code 1.

Possibly related PRs

  • oven-sh/bun#37129: Both changes modify parallel worker coordination and crash or exit handling.

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the affected command and the primary fix for worker IPC file descriptor closure hangs.
Description check ✅ Passed The description explains the problem, cause, fix, and verification results, although it uses different headings from the repository template.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/runtime/cli/test/parallel/Worker.rs`:
- Around line 393-399: Update the worker teardown logic around the exiting-state
check and on_channel_done so Exiting is treated as pending termination rather
than proof of process exit. After fd 3 reaches EOF, allow only a bounded
teardown grace period; if the process remains alive, invoke Process::kill(9) and
continue reaping it through try_reap. Add a regression test covering a synthetic
Exiting frame followed by channel closure and verifying the worker is eventually
killed and reaped.

In `@test/cli/test/parallel.test.ts`:
- Line 163: Remove the conditional per-test timeout value in the affected
parallel test configuration so the test uses the test runner’s default timeout.
Preserve the existing ASAN/debug branching and test behavior, changing only the
local timeout setting.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9344a6f3-e49b-494b-a578-be43ef7a849b

📥 Commits

Reviewing files that changed from the base of the PR and between d2f48c6 and 52f15d5.

📒 Files selected for processing (6)
  • src/runtime/cli/test/parallel/Channel.rs
  • src/runtime/cli/test/parallel/Coordinator.rs
  • src/runtime/cli/test/parallel/Frame.rs
  • src/runtime/cli/test/parallel/Worker.rs
  • src/runtime/cli/test/parallel/runner.rs
  • test/cli/test/parallel.test.ts
💤 Files with no reviewable changes (1)
  • src/runtime/cli/test/parallel/Channel.rs

Comment thread src/runtime/cli/test/parallel/Worker.rs Resolved
Comment thread test/cli/test/parallel.test.ts Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/runtime/cli/test/parallel/runner.rs (1)

752-768: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Drain Kind::Exiting before process-exit teardown.

Channel::flush is not a completion barrier. Windows only submits uv_write, and POSIX returns when write makes no progress. The cleanup-hook path performs no event-loop drain, unlike the normal exit path. If bytes remain queued, the coordinator treats the worker as lost and may kill it. Use a teardown-safe synchronous or acknowledged send, or a bounded drain. Add POSIX and Windows backpressure tests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/runtime/cli/test/parallel/runner.rs` around lines 752 - 768, The
announce_deliberate_exit function must ensure the Kind::Exiting frame is
delivered before process-exit teardown, rather than relying on
cmds.channel.flush(). Replace this flush-only path with the channel’s
teardown-safe synchronous or acknowledged send mechanism, or add a bounded drain
that completes on both POSIX and Windows. Add backpressure tests covering both
platforms and verify the coordinator receives Kind::Exiting before treating the
worker as lost.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/runtime/cli/test/parallel/Worker.rs`:
- Around line 74-76: Update the worker termination flow around the exiting state
and on_channel_done so Kind::Exiting only marks pending termination, not
confirmed process exit. Preserve the existing skip path, but add a bounded grace
period; if has_exited() is still false afterward, force-kill the worker and reap
it via try_reap.

---

Outside diff comments:
In `@src/runtime/cli/test/parallel/runner.rs`:
- Around line 752-768: The announce_deliberate_exit function must ensure the
Kind::Exiting frame is delivered before process-exit teardown, rather than
relying on cmds.channel.flush(). Replace this flush-only path with the channel’s
teardown-safe synchronous or acknowledged send mechanism, or add a bounded drain
that completes on both POSIX and Windows. Add backpressure tests covering both
platforms and verify the coordinator receives Kind::Exiting before treating the
worker as lost.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2ac6841e-fa7b-4901-bb73-b8b51453826b

📥 Commits

Reviewing files that changed from the base of the PR and between 52f15d5 and 801e21a.

📒 Files selected for processing (4)
  • src/runtime/cli/test/parallel/Channel.rs
  • src/runtime/cli/test/parallel/Frame.rs
  • src/runtime/cli/test/parallel/Worker.rs
  • src/runtime/cli/test/parallel/runner.rs

Comment thread src/runtime/cli/test/parallel/Worker.rs Resolved
@robobun

robobun commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai On the outside-diff finding about announce_deliberate_exit relying on flush (runner.rs): a dropped Exiting frame degrades in the safe direction. The coordinator then kills the worker, marks the file as crashed, and the run continues; no hang is possible from a lost announcement, only a less precise crash label (SIGKILL instead of the worker's exit code). For the frame to be dropped, the channel must already hold a full socket buffer of undelivered output at the moment of process.exit, and those queued bytes were equally discarded by teardown before this PR. The suggested bounded event-loop drain is not safe at this call site: the cleanup hook runs in the middle of a JS stack during process.exit, where ticking the event loop could run user timers and callbacks after exit has begun, which process.exit semantics forbid. The normal end-of-run path already drains (the existing loop immediately after the announce call).

@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@robobun Thank you for the detailed explanation. I agree that the finding does not apply.

A lost Exiting frame cannot cause the hang that this PR fixes. The coordinator treats the worker as an unexpected loss, kills and reaps it, reports the in-flight file as crashed, and continues the run.

A bounded event-loop drain is unsafe in the process.exit cleanup hook. It could run user callbacks after exit begins. The existing normal shutdown path already drains after announce_deliberate_exit.


✏️ Learnings added
Learnt from: robobun
URL: https://github.com/oven-sh/bun/pull/37285

Timestamp: 2026-08-09T19:36:44.153Z
Learning: In `src/runtime/cli/test/parallel/runner.rs`, `announce_deliberate_exit` can run from a cleanup hook during `process.exit` while a JavaScript stack is active. Do not drain or tick the event loop at this call site because it can run user timers or callbacks after exit begins. If its `Exiting` frame is lost, the coordinator safely treats the worker as unexpectedly lost, kills and reaps it, marks the active test file as crashed, and continues the parallel test run. The normal end-of-run shutdown path already drains after the announcement.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed after the WindowsHandlers::on_error comment was trimmed — thanks, that resolves the prior nit. No bugs found this run. Given this changes the coordinator/worker lifecycle state machine (new Kind::Exiting frame, VM cleanup hook, kill-on-EOF gating on has_exited()/exiting) across both the usockets and uv::Pipe backends, a human look is still worthwhile.

What was reviewed

  • on_channel_done kill path: process ptr is live (reap deferred to run_pending_reaps), has_exited() guards the zombie/no-op case, SIGKILL is non-panic so no whole-run abort.
  • exiting reset on respawn (start()) and initial construction — covered.
  • Existing process.exit(7) / SIGKILL / corrupt-frame tests still take the intended paths under the new gate.
  • Windows: Exiting frame delivery when a prior uv_write is in flight — worst case degrades to a redundant TerminateProcess on an already-exiting worker, still a per-file (non-panic) failure.
Extended reasoning...

Overview

Fixes a hang in bun test --parallel when a test closes fd 3 (the worker's IPC socketpair end). The coordinator now kills any worker whose channel goes EOF while the process is still running and hasn't sent a new Kind::Exiting frame; the worker sends that frame from a VM cleanup hook (covering process.exit()) and from run_as_worker's normal tail. Channel::is_attached() (the old corrupt-frame-only kill gate) is removed along with its rationale comment. A new test in parallel.test.ts reproduces the fd-3-close hang.

Security risks

None. Internal same-binary IPC protocol between coordinator and worker; no user-facing surface, no auth/crypto, no untrusted input beyond what frame::MAX_PAYLOAD already bounds.

Level of scrutiny

Medium-high. This is a process-lifecycle / state-machine change with cross-platform branches (usockets vs uv::Pipe), unsafe raw-pointer dereferences of *mut Process, and a new race axis (Exiting-frame delivery vs channel EOF vs exit notification). It's the sort of change where an off-by-one in the ordering silently regresses an existing crash-classification test on one platform only, so it warrants a human reviewer even though the mechanism is sound and the diff is small.

Other factors

  • The prior 🟡 nit (stale on_error comment referencing the removed is_attached() distinction) is addressed in the current diff.
  • The verifier examined and ruled out the Windows in-flight-uv_write concern for announce_deliberate_exit(); even in the degraded case the coordinator's kill is a no-op or redundant TerminateProcess on a process that's already tearing down, and the resulting status is not in is_fatal_windows_exit_code, so it stays a per-file failure rather than a run-abort.
  • Neighboring tests (process.exit(7) mid-run, raw SIGABRT, corrupt-frame fd-3 write, SIGKILL junit synthetic suite) all still route correctly: process.exit sends Exiting → no coordinator kill → real exit code reported; SIGKILL/SIGABRT skip cleanup hooks → exiting=false but has_exited() (or ESRCH-ignored kill) covers it.
  • The comment-cop bot flagged several multi-line comments; those are protocol/field documentation rather than workaround justification, and the follow-up commit already consolidated the Exiting-frame rationale onto the Kind doc — leaving it to the author/reviewer whether the remaining flags need action.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs (my earlier nit about the stale on_error comment was addressed in 801e21a). Because this changes worker lifecycle/kill semantics, adds an IPC frame kind, and registers a VM cleanup hook — with a design tradeoff around trusting Kind::Exiting unconditionally that already went through one CI failure-and-fix cycle — a human sign-off on the approach would be worthwhile.

Checked: the exiting flag is reset in Worker::start() so a respawned slot doesn't inherit the prior worker's announcement; announce_deliberate_exit matches the CleanupHookFunction signature and is idempotent if fired twice (normal tail + cleanup hook); the coordinator handler for Kind::Exiting is a no-op beyond setting the flag, so a duplicate frame is harmless; the new on_channel_done kill path is strictly narrower than trusting every clean EOF (pre-PR behavior).

Extended reasoning...

Overview

The PR fixes a hang in bun test --parallel when a test closes fd 3 (the worker IPC socketpair end). It touches five Rust files in src/runtime/cli/test/parallel/ and adds one regression test:

  • Frame.rs: adds Kind::Exiting (discriminant 9) with a doc comment explaining the wire contract.
  • runner.rs: adds announce_deliberate_exit (an extern "C" fn matching CleanupHookFunction), registers it as a VM cleanup hook in WorkerLoop::begin, and calls it directly in run_as_worker's normal tail before the drain loop.
  • Worker.rs: adds an exiting: bool field (reset in start()), and rewrites on_channel_done to kill the worker when !exiting && !has_exited() instead of gating on is_attached().
  • Coordinator.rs: records w.exiting = true on the new frame.
  • Channel.rs: removes the now-dead is_attached() and trims the stale rationale in WindowsHandlers::on_error (my prior nit).
  • test/cli/test/parallel.test.ts: adds a hang-guard regression test using the file's existing isASAN || isDebug ? 60_000 : 20_000 timeout convention.

Security risks

None identified. The IPC channel is between a coordinator and workers it spawned; the fd-3 protocol is already inside the trust boundary (a test that can forge Exiting can already forge FileDone/TestDone and corrupt results, as the author noted in the coderabbit thread). No user-facing input parsing, no auth/crypto/permissions.

Level of scrutiny

Moderate-to-high. This is process-lifecycle and kill-decision logic with cross-process coordination and a documented race under BUN_DESTRUCT_VM_ON_EXIT that the first CI round caught (30205.test.ts). The fix is well-reasoned and the mechanism is clearly explained, but it is not a mechanical change: it introduces a new protocol frame, hooks into VM teardown via rare_data().push_cleanup_hook, and changes when the coordinator SIGKILLs a worker. The "trust Exiting with no bounded grace period" design was debated twice by coderabbit and defended by the author with a sound argument (any deadline reintroduces the LSAN-teardown race), but that's exactly the kind of design call a maintainer should ratify.

Other factors

  • All prior review threads (comment-cop, coderabbit ×3, my inline nit) are resolved. The comment-cop noise was addressed by consolidating the rationale onto the Kind::Exiting doc comment with one-line pointers elsewhere.
  • I verified the exiting flag is reset on respawn (start() sets self.exiting = false before spawn), so a slot reused after a deliberate-exit worker won't carry stale state into the next worker's on_channel_done.
  • announce_deliberate_exit is safe to call more than once (double Exiting frame → coordinator sets w.exiting = true twice; flush() is idempotent), so any overlap between the direct tail call and a cleanup-hook invocation via global_exit() is harmless.
  • The regression test follows the file's existing spawn-heavy conventions (drains stdout/stderr/exited concurrently, asserts specific summary strings and exit code, uses the same per-test timeout expression as the neighboring NTSTATUS/fastfail tests).

Given the subtlety of the lifecycle handling and the design tradeoff involved, I'm deferring rather than auto-approving.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Closing this PR because it has been inactive for more than 90 days.

@github-actions github-actions Bot closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants