Repository navigation
Conversation
…till running A test that closes fd 3 (e.g. fd-hygiene code closing every fd >= 3) closes the worker's IPC socketpair end out from under it. The fd silently leaves the worker's poll set, so the worker never learns the channel died and spins in its next-command loop at 100% CPU, while the coordinator got a clean EOF, assumed the worker had exited, and waited forever for an exit notification that never came: the run hung and --timeout never applied. Treat channel-done with the process still running as a lost worker, the same as the corrupt-frame path: kill it so onWorkerExit accounts for the in-flight file (worker crashed: SIGKILL) and the slot can respawn, and the run continues. On a normal worker exit the EOF often lands before the exit notification; the process is an unreaped zombie then and the kill is a no-op.
… kill preserves their status With BUN_DESTRUCT_VM_ON_EXIT (set on CI's ASAN lanes), a worker's process.exit(7) tears down its VM, which closes the IPC socket well before _exit. The coordinator then saw EOF from a still-running process, treated the worker as lost, and kill(9)ed it mid-teardown: the file was reported as 'worker crashed: SIGKILL' instead of 'exit code 7' (caught by test/regression/issue/30205.test.ts). Send a Kind::Exiting frame from the worker's exit paths (a VM cleanup hook covers process.exit; run_as_worker's tail covers the normal shutdown) before teardown closes the channel. The coordinator skips the lost-channel kill for an announced exit and waits for the real status, which also lets ASAN leak checks in the worker run to completion. EOF without the announcement (test closed fd 3, corrupt frame) is still treated as a lost worker and killed.
WalkthroughChangesThe parallel test runner announces deliberate worker exits before IPC teardown. The coordinator records this state and avoids treating intentional EOF as a crash. Unexpected IPC loss still fails the affected worker while allowing other files to continue. Parallel worker exit handling
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/runtime/cli/test/parallel/Worker.rs`:
- Around line 393-399: Update the worker teardown logic around the exiting-state
check and on_channel_done so Exiting is treated as pending termination rather
than proof of process exit. After fd 3 reaches EOF, allow only a bounded
teardown grace period; if the process remains alive, invoke Process::kill(9) and
continue reaping it through try_reap. Add a regression test covering a synthetic
Exiting frame followed by channel closure and verifying the worker is eventually
killed and reaped.
In `@test/cli/test/parallel.test.ts`:
- Line 163: Remove the conditional per-test timeout value in the affected
parallel test configuration so the test uses the test runner’s default timeout.
Preserve the existing ASAN/debug branching and test behavior, changing only the
local timeout setting.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 9344a6f3-e49b-494b-a578-be43ef7a849b
📒 Files selected for processing (6)
src/runtime/cli/test/parallel/Channel.rssrc/runtime/cli/test/parallel/Coordinator.rssrc/runtime/cli/test/parallel/Frame.rssrc/runtime/cli/test/parallel/Worker.rssrc/runtime/cli/test/parallel/runner.rstest/cli/test/parallel.test.ts
💤 Files with no reviewable changes (1)
- src/runtime/cli/test/parallel/Channel.rs
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/runtime/cli/test/parallel/runner.rs (1)
752-768: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftDrain
Kind::Exitingbefore process-exit teardown.
Channel::flushis not a completion barrier. Windows only submitsuv_write, and POSIX returns whenwritemakes no progress. The cleanup-hook path performs no event-loop drain, unlike the normal exit path. If bytes remain queued, the coordinator treats the worker as lost and may kill it. Use a teardown-safe synchronous or acknowledged send, or a bounded drain. Add POSIX and Windows backpressure tests.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/runtime/cli/test/parallel/runner.rs` around lines 752 - 768, The announce_deliberate_exit function must ensure the Kind::Exiting frame is delivered before process-exit teardown, rather than relying on cmds.channel.flush(). Replace this flush-only path with the channel’s teardown-safe synchronous or acknowledged send mechanism, or add a bounded drain that completes on both POSIX and Windows. Add backpressure tests covering both platforms and verify the coordinator receives Kind::Exiting before treating the worker as lost.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/runtime/cli/test/parallel/Worker.rs`:
- Around line 74-76: Update the worker termination flow around the exiting state
and on_channel_done so Kind::Exiting only marks pending termination, not
confirmed process exit. Preserve the existing skip path, but add a bounded grace
period; if has_exited() is still false afterward, force-kill the worker and reap
it via try_reap.
---
Outside diff comments:
In `@src/runtime/cli/test/parallel/runner.rs`:
- Around line 752-768: The announce_deliberate_exit function must ensure the
Kind::Exiting frame is delivered before process-exit teardown, rather than
relying on cmds.channel.flush(). Replace this flush-only path with the channel’s
teardown-safe synchronous or acknowledged send mechanism, or add a bounded drain
that completes on both POSIX and Windows. Add backpressure tests covering both
platforms and verify the coordinator receives Kind::Exiting before treating the
worker as lost.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 2ac6841e-fa7b-4901-bb73-b8b51453826b
📒 Files selected for processing (4)
src/runtime/cli/test/parallel/Channel.rssrc/runtime/cli/test/parallel/Frame.rssrc/runtime/cli/test/parallel/Worker.rssrc/runtime/cli/test/parallel/runner.rs
|
@coderabbitai On the outside-diff finding about |
|
Tip For best results, initiate chat on the files or code changes.
A lost A bounded event-loop drain is unsafe in the ✏️ Learnings added
You are interacting with an AI system. |
There was a problem hiding this comment.
I re-reviewed after the WindowsHandlers::on_error comment was trimmed — thanks, that resolves the prior nit. No bugs found this run. Given this changes the coordinator/worker lifecycle state machine (new Kind::Exiting frame, VM cleanup hook, kill-on-EOF gating on has_exited()/exiting) across both the usockets and uv::Pipe backends, a human look is still worthwhile.
What was reviewed
on_channel_donekill path:processptr is live (reap deferred torun_pending_reaps),has_exited()guards the zombie/no-op case, SIGKILL is non-panic so no whole-run abort.exitingreset on respawn (start()) and initial construction — covered.- Existing
process.exit(7)/ SIGKILL / corrupt-frame tests still take the intended paths under the new gate. - Windows:
Exitingframe delivery when a prior uv_write is in flight — worst case degrades to a redundant TerminateProcess on an already-exiting worker, still a per-file (non-panic) failure.
Extended reasoning...
Overview
Fixes a hang in bun test --parallel when a test closes fd 3 (the worker's IPC socketpair end). The coordinator now kills any worker whose channel goes EOF while the process is still running and hasn't sent a new Kind::Exiting frame; the worker sends that frame from a VM cleanup hook (covering process.exit()) and from run_as_worker's normal tail. Channel::is_attached() (the old corrupt-frame-only kill gate) is removed along with its rationale comment. A new test in parallel.test.ts reproduces the fd-3-close hang.
Security risks
None. Internal same-binary IPC protocol between coordinator and worker; no user-facing surface, no auth/crypto, no untrusted input beyond what frame::MAX_PAYLOAD already bounds.
Level of scrutiny
Medium-high. This is a process-lifecycle / state-machine change with cross-platform branches (usockets vs uv::Pipe), unsafe raw-pointer dereferences of *mut Process, and a new race axis (Exiting-frame delivery vs channel EOF vs exit notification). It's the sort of change where an off-by-one in the ordering silently regresses an existing crash-classification test on one platform only, so it warrants a human reviewer even though the mechanism is sound and the diff is small.
Other factors
- The prior 🟡 nit (stale
on_errorcomment referencing the removedis_attached()distinction) is addressed in the current diff. - The verifier examined and ruled out the Windows in-flight-uv_write concern for
announce_deliberate_exit(); even in the degraded case the coordinator's kill is a no-op or redundant TerminateProcess on a process that's already tearing down, and the resulting status is not inis_fatal_windows_exit_code, so it stays a per-file failure rather than a run-abort. - Neighboring tests (
process.exit(7)mid-run, raw SIGABRT, corrupt-frame fd-3 write, SIGKILL junit synthetic suite) all still route correctly:process.exitsendsExiting→ no coordinator kill → real exit code reported; SIGKILL/SIGABRT skip cleanup hooks →exiting=falsebuthas_exited()(or ESRCH-ignored kill) covers it. - The comment-cop bot flagged several multi-line comments; those are protocol/field documentation rather than workaround justification, and the follow-up commit already consolidated the
Exiting-frame rationale onto theKinddoc — leaving it to the author/reviewer whether the remaining flags need action.
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs (my earlier nit about the stale on_error comment was addressed in 801e21a). Because this changes worker lifecycle/kill semantics, adds an IPC frame kind, and registers a VM cleanup hook — with a design tradeoff around trusting Kind::Exiting unconditionally that already went through one CI failure-and-fix cycle — a human sign-off on the approach would be worthwhile.
Checked: the exiting flag is reset in Worker::start() so a respawned slot doesn't inherit the prior worker's announcement; announce_deliberate_exit matches the CleanupHookFunction signature and is idempotent if fired twice (normal tail + cleanup hook); the coordinator handler for Kind::Exiting is a no-op beyond setting the flag, so a duplicate frame is harmless; the new on_channel_done kill path is strictly narrower than trusting every clean EOF (pre-PR behavior).
Extended reasoning...
Overview
The PR fixes a hang in bun test --parallel when a test closes fd 3 (the worker IPC socketpair end). It touches five Rust files in src/runtime/cli/test/parallel/ and adds one regression test:
Frame.rs: addsKind::Exiting(discriminant 9) with a doc comment explaining the wire contract.runner.rs: addsannounce_deliberate_exit(anextern "C"fn matchingCleanupHookFunction), registers it as a VM cleanup hook inWorkerLoop::begin, and calls it directly inrun_as_worker's normal tail before the drain loop.Worker.rs: adds anexiting: boolfield (reset instart()), and rewriteson_channel_doneto kill the worker when!exiting && !has_exited()instead of gating onis_attached().Coordinator.rs: recordsw.exiting = trueon the new frame.Channel.rs: removes the now-deadis_attached()and trims the stale rationale inWindowsHandlers::on_error(my prior nit).test/cli/test/parallel.test.ts: adds a hang-guard regression test using the file's existingisASAN || isDebug ? 60_000 : 20_000timeout convention.
Security risks
None identified. The IPC channel is between a coordinator and workers it spawned; the fd-3 protocol is already inside the trust boundary (a test that can forge Exiting can already forge FileDone/TestDone and corrupt results, as the author noted in the coderabbit thread). No user-facing input parsing, no auth/crypto/permissions.
Level of scrutiny
Moderate-to-high. This is process-lifecycle and kill-decision logic with cross-process coordination and a documented race under BUN_DESTRUCT_VM_ON_EXIT that the first CI round caught (30205.test.ts). The fix is well-reasoned and the mechanism is clearly explained, but it is not a mechanical change: it introduces a new protocol frame, hooks into VM teardown via rare_data().push_cleanup_hook, and changes when the coordinator SIGKILLs a worker. The "trust Exiting with no bounded grace period" design was debated twice by coderabbit and defended by the author with a sound argument (any deadline reintroduces the LSAN-teardown race), but that's exactly the kind of design call a maintainer should ratify.
Other factors
- All prior review threads (comment-cop, coderabbit ×3, my inline nit) are resolved. The comment-cop noise was addressed by consolidating the rationale onto the
Kind::Exitingdoc comment with one-line pointers elsewhere. - I verified the
exitingflag is reset on respawn (start()setsself.exiting = falsebefore spawn), so a slot reused after a deliberate-exit worker won't carry stale state into the next worker'son_channel_done. announce_deliberate_exitis safe to call more than once (doubleExitingframe → coordinator setsw.exiting = truetwice;flush()is idempotent), so any overlap between the direct tail call and a cleanup-hook invocation viaglobal_exit()is harmless.- The regression test follows the file's existing spawn-heavy conventions (drains stdout/stderr/exited concurrently, asserts specific summary strings and exit code, uses the same per-test timeout expression as the neighboring NTSTATUS/fastfail tests).
Given the subtlety of the lifecycle handling and the design tradeoff involved, I'm deferring rather than auto-approving.
|
Closing this PR because it has been inactive for more than 90 days. |
Problem
Under
bun test --parallel, a test file that closes fd 3 hangs the whole run forever, with the worker spinning at 100% CPU and--timeoutnever applying:bun test --parallel=2 .never finishes; serialbun testpasses.Cause
fd 3 in a worker is its IPC socketpair end.
closeSync(3)removes the fd from the worker's poll set without any callback, so the worker never learns the channel died: its frames go nowhere and it sits in the next-command loop forever. The coordinator does observe the resulting EOF on its end, but the clean-EOF path inWorker::on_channel_doneassumed EOF means the worker process exited (the normal case) and only killed the worker on the corrupt-frame path (is_attached()). With the worker still alive,try_reapwaits for an exit notification that never arrives and the run hangs.Fix
On channel-done, kill the worker whenever its process has not exited yet, not just on a corrupt frame. A worker that outlives its channel can neither receive a file nor report one, so it is lost either way. The in-flight file is accounted as
worker crashed: SIGKILL, the slot respawns if files remain, and the run continues.Channel::is_attachedlost its only caller and is removed.One wrinkle: a worker doing a deliberate exit (
process.exit(7)in a test, or the normal end-of-run path) closes its channel during VM teardown, before_exit. UnderBUN_DESTRUCT_VM_ON_EXIT(set on CI's ASAN lanes) that teardown is long, so the coordinator would see EOF from a still-running process and the kill would clobber the real exit status (worker crashed: SIGKILLinstead ofexit code 7, caught bytest/regression/issue/30205.test.tsin the first CI round). Workers therefore announce deliberate exits with a newExitingIPC frame, sent while the channel is still up: a VM cleanup hook coversprocess.exit, and the worker loop's tail covers normal shutdown. The coordinator skips the kill for an announced exit and waits for the real status, which also lets ASAN leak checks in the worker finish. EOF with no announcement is a lost worker and is killed.Fixed output for the repro above:
Verification
test/cli/test/parallel.test.ts: fails by timeout on an unfixed build, passes in ~2-4s on the fixed debug build.test/regression/issue/30205.test.ts(all 4 tests) passes with and withoutBUN_DESTRUCT_VM_ON_EXIT=1; theprocess.exit(7)fidelity failure from the first CI round reproduced locally with that env and is fixed by the exit announcement.process.exitmid-run, raw SIGABRT abort, garbage written to fd 3) pass in both env configurations.parallel.test.ts(lazily scales workers,partitions by directory,work-stealing balances) fail identically with and without this change on a core-capped local machine; unrelated.no test proof · iteration 1 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/test/parallel.test.ts