Skip to content

test(bake): run the bundle dev tests concurrently - #40541

Open
robobun wants to merge 3 commits into
mainfrom
farm/465ee815/bake-bundle-test-concurrent
Open

robobun wants to merge 3 commits into
mainfrom
farm/465ee815/bake-bundle-test-concurrent

Conversation

@robobun

@robobun robobun commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/bake/dev/bundle.test.ts takes 15 to 19s per CI lane (build #106056), 10.7s locally on release, 47.8s on debug+ASAN. Its 21 cases run in sequence, each with its own dev server and usually a happy-dom client. That fixed cost is the time.
  • The cases cannot overlap today. run() in test/bake/bake-harness.ts SIGKILLs the module-global danglingProcesses set when any case ends. Under test.concurrent, the first case to finish kills every other case's server and clients.

Fix

  • The harness tracks processes per test (Dev.processes) and kills only those at teardown. A new opt-in concurrent option on devTest registers the case with test.concurrent.
  • The runner kills a timed-out test's processes only when the test is not concurrent, and bun test skips process.on("exit") listeners. So run() gives up 5s (or 10%) before the runner's timeout, and its disposers kill that test's processes. An afterAll, registered once per test file, reaps what is left at file end.
  • bundle.test.ts opts all 21 cases in. Two expectMessage calls are now awaited, the two toBeInstanceOf(Response) checks assert the status, and "commonjs forms" loses its debug logs. No case is merged, skipped, or changed in what it tests.
  • Verified: bun bd test test/bake/dev/bundle.test.ts 47.8s before, 13.3 to 13.6s after (ASAN caps concurrency at 5). Release 10.7s before, 1.2 to 1.4s after. All 155 bake dev tests pass. A hung concurrent case leaves no process behind.

Background

  • devTest gives each case its own temp directory and a dev server on port 0. dev.client() is a node process that loads the page in happy-dom and acks each build over IPC.
  • test.concurrent lets async tests in one file overlap, up to --max-concurrency (20, or 5 in an ASAN build). In CI this file runs alone in the runner's serial phase.
Notes

Per-phase timing of one case (local, release, before)

phase ms
dev server spawn to port line 58 (50 of it is the setTimeout(50) in waitForLine)
dev.client() page load and overlay check 275 to 335
one dev.write() rebuild and client ack 67 to 76 (50 of it is the coalescing Bun.sleep(50) in batchChanges)
gracefulExit 20 to 25

Debug+ASAN: 330ms to the port line, 460ms page load, 110 to 190ms per write, 800 to 1100ms gracefulExit.

The two fixed waits in the harness (50ms after a matched line, 50ms to let watcher events coalesce) are left alone. They exist for every bake dev file, the coalescing one has a stated reason (several watcher events per write on Windows), and under concurrency they overlap with other cases anyway.

Why the harness and not the file

Two earlier PRs (#40082, #40315) merged cases onto shared dev servers and were closed: #39488 carries the same restructure of this file, and #40315's own review found that the serial harness was the real cost. This PR keeps the 21 cases as they are, so it stays small against #39488, and makes the harness able to run them concurrently. The bundle.test.ts edits are one concurrent: true line per case plus the assertion fixes #39488 also has (the awaited expectMessage calls matter under concurrency: an unawaited rejection lands on whichever test is current). concurrent is opt-in so the other 18 dev files keep their sequential behavior until each one is checked.

The deadline

src/runtime/test_runner/Execution.rs (kill_dangling_processes_on_timeout) kills a timed-out test's spawned processes only when the concurrent group has one sequence. src/runtime/cli/test_command.rs (skip_exit_listeners) skips a test file's process.on("exit") listeners, so the harness's exit hook never ran under bun test, on main too. Before this PR the kill-all at the end of every case reaped a hung case's processes when the next case finished. The deadline replaces that: a case that hangs fails with "The test did not finish before the harness deadline" 5s (or 10% for short timeouts) before the runner's timeout, and its disposers SIGKILL its server and clients. Promise.race keeps observing the abandoned body, so its later rejection is not unhandled.

Probe (not committed): a concurrent: true case with timeoutMultiplier: 0.1 that awaits a promise that never settles, next to two live siblings and a final check with process.kill(pid, 0). Release: the case fails at 2.7s of its 3s timeout, both siblings pass, server and client are dead. Debug+ASAN: fails at 24.4s of 27s, same result. The afterAll reaper fires at file end (checked with a temporary log, 0 processes left).

Concurrency limits and memory

bun test runs at most --max-concurrency tests at once: 20 by default, 5 when the build has ASAN (src/options_types/context.rs). Peak RSS per case: about 40 MB for the dev server and 110 MB for the client on release, 400 MB and 120 MB on debug+ASAN. So 20 release cases need about 3 GB, 5 ASAN cases about 2.6 GB. The file is in excludeFiles of test/parallel-allowlist.json, so the CI runner runs it alone in its serial phase.

Release, pinned to 4 cores (taskset), 10 runs: 2.2 to 3.9s, all pass. Pinned to 2 cores, 5 runs: 4.0 to 4.2s, all pass. Debug+ASAN pinned to 8 cores, 4 runs: 13.4 to 13.5s, all pass.

Other harness users

bun bd test test/bake/dev/ test/bake/dev-and-prod.test.ts: 155 pass, 0 fail (22 files, 425s debug+ASAN), run after every harness change. The interactive mode (bun test/bake/dev/bundle.test.ts "<filter>") still starts: test.concurrent and afterAll throw outside the test runner the same way test does, and the harness catches them.

The harness module is evaluated once per bun test process, so an afterAll at module scope reached only the first file of a multi-file run. testImpl registers it the first time a file adds a test. stackTraceFileName used to keep the line number in the caller path on posix (it took the s of .ts before the first colon for a drive letter), so the per-file key needed that fixed too. Test names and temp directories are unchanged.

Assertions

expect() calls: 44 before, 45 after. The final fetch in "removing 'use client' from a component with a pending resolution failure" returns 500 today through a runtime error page (#40113, fix in #40115) and a "Build Failed" page once that lands. The route has an unresolved import either way, so the status is the stable assertion.

The dev server harness killed a module-global set of spawned processes
when any test ended. That made test.concurrent impossible: the first
test to finish killed every other test's server and clients.

The harness now tracks the processes per test (Dev.processes) and kills
only those at teardown. A new `concurrent` option on devTest registers
the test with test.concurrent. Dev server and client log lines carry
the test's directory name, so concurrent output stays readable.

bundle.test.ts opts every test in. Two expectMessage calls are now
awaited. The two "is a Response" checks assert the response status.
"commonjs forms" loops over the forms without debug logs and without
the timeout multiplier.

Release: 10.7s to 1.2s. Debug+ASAN: 47.8s to 13.4s.
@robobun

robobun commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review.

Reproduced the slow run with bun bd test test/bake/dev/bundle.test.ts (47.8s debug+ASAN) and USE_SYSTEM_BUN=1 bun test test/bake/dev/bundle.test.ts (10.7s release). With this branch the same commands take 13.3 to 13.6s and 1.2 to 1.4s.

Review follow-ups, both verified with probes and a full bun bd test test/bake/dev/ test/bake/dev-and-prod.test.ts run (155 pass):

@coderabbitai

coderabbitai Bot commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 81a7cb2c-e24b-4a4e-9ac0-c7a9f0bb89db

📥 Commits

Reviewing files that changed from the base of the PR and between 731aa92 and d3d9ec7.

📒 Files selected for processing (2)
  • test/bake/bake-harness.ts
  • test/bake/dev/bundle.test.ts

Included review availability: Your plan provides up to 5 included reviews per hour; 0 remain after this review.


Walkthrough

The bake harness now supports opt-in concurrent tests, per-test subprocess cleanup, deadline-bounded lifecycle operations, and exit reaping. DevServer bundle tests use concurrent execution and stronger HMR and HTTP response assertions.

Changes

Bake concurrency and cleanup

Layer / File(s) Summary
Test process ownership
test/bake/bake-harness.ts
DevServerTest supports concurrent execution. Dev and Client track their subprocesses, and client output uses per-root names.
Deadline and cleanup flow
test/bake/bake-harness.ts
Deadline bounds startup, test execution, and shutdown. Per-file and exit cleanup reap dangling processes. Windows drive-letter parsing now requires a colon at index two.
Concurrent bundle test coverage
test/bake/dev/bundle.test.ts
DevServer tests opt into concurrency. HMR assertions are awaited, route responses have stronger status and content checks, and diagnostic logging is removed from the CommonJS mutation test.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR retains all 21 bundle regression cases and adds opt-in concurrency, per-test process cleanup, timeout handling, and file-level reaping. These changes support the relevant testing, resource clea…
Out of Scope Changes check ✅ Passed The changes remain within scope. Concurrency flags, awaited assertions, response status checks, removal of debug logs, and harness cleanup all support the stated test execution and reliability objecti…
Title check ✅ Passed The title clearly identifies the primary change: enabling concurrent execution for the Bake bundle development tests.
Description check ✅ Passed The description clearly explains the problem, implementation, verification results, cleanup behavior, and scope. It does not use the exact template headings, but it provides the required purpose and v…
Full details: Linked Issues check

Explanation

The PR retains all 21 bundle regression cases and adds opt-in concurrency, per-test process cleanup, timeout handling, and file-level reaping. These changes support the relevant testing, resource cleanup, and execution objectives in [#39488].

Full details: Out of Scope Changes check

Explanation

The changes remain within scope. Concurrency flags, awaited assertions, response status checks, removal of debug logs, and harness cleanup all support the stated test execution and reliability objectives.

Full details: Description check

Explanation

The description clearly explains the problem, implementation, verification results, cleanup behavior, and scope. It does not use the exact template headings, but it provides the required purpose and verification information in equivalent sections.


Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-infrastructure only, opt-in concurrency with per-test process tracking.

Checked that the moved using _ disposer leaves no gap: await using devProcess covers startup failures before Dev is constructed, and clients only spawn after. The module-global danglingProcesses set is retained and still feeds the process.on("exit") handler, so nothing leaks on abnormal exit. The activeClient module-global is shared across concurrent tests but is only read in interactive mode. The "commonjs forms" table-driven rewrite preserves all seven original cases in order; the two added awaits and the Response→status assertions are strict tightenings.

Extended reasoning...

Overview

This PR touches only test infrastructure: test/bake/bake-harness.ts gains an opt-in concurrent?: boolean on DevServerTest that routes registration through test.concurrent, and the per-test cleanup disposer is changed from iterating the module-global danglingProcesses set to iterating a new per-Dev-instance processes set (so a finishing concurrent test kills only its own subprocesses). OutputLineStream prefixes now carry the test's temp-directory basename so interleaved output stays attributable. test/bake/dev/bundle.test.ts opts all 21 cases into concurrent: true, adds two missing awaits on c.expectMessage(...), strengthens two toBeInstanceOf(Response) liveness checks into concrete status/body assertions, and rewrites the "commonjs forms" test as a table-driven loop (dropping debug console.log scaffolding and timeoutMultiplier: 2). No production code is touched.

Security risks

None. This is test-harness and test-file code only; no user-facing surface, parsing, auth, crypto, or network-exposed paths are modified.

Level of scrutiny

Moderate for the harness change (shared by ~22 bake test files), low for the test-file edits. The harness change is deliberately opt-in: only bundle.test.ts sets concurrent: true, so the other 21 files keep their sequential behavior. I traced the disposer move: the old using _ sat before Bun.spawn and killed everything in danglingProcesses; the new one sits after new Dev(...) and kills only dev.processes. The window between spawn and Dev construction (which includes await stream.waitForLine) is already covered by await using devProcess, and no clients exist yet, so no process can leak from that gap. The module-global danglingProcesses is still populated (lines 909, 2126) and still drained in the process.on("exit") handler (line 2286), so the last-resort cleanup on abnormal exit is unchanged. The activeClient module-global is written per client but only read by maybeWaitInteractive, which runs single-test — harmless under concurrency.

Other factors

The "commonjs forms" refactor was diffed case-by-case: all seven [source, field] pairs match the original sequence exactly, and the initial { field: {} } assertion is preserved before the loop. The two now-awaited expectMessage calls fix a real hazard (unawaited rejection would misattribute under concurrency). The res.status === 500 assertion replaces a vacuous toBeInstanceOf(Response) and the PR notes the route stays failing regardless of the pending #40115 fix; the 200 + /_bun/client/ body check similarly tightens. The change follows CLAUDE.md's explicit guidance ("use test.concurrent for independent subprocess-spawning tests") and REVIEW.md's note that slow test files get scrutinized for test.concurrent. No CODEOWNERS entry covers these paths, and there are no outstanding reviewer objections in the timeline.

…'s processes die

The runner kills a timed-out test's processes only when the test is not
concurrent, and `bun test` never runs `process.on("exit")` listeners. A
concurrent test that hung would have left its dev server and clients
alive for the rest of the file, and past it on lanes without
BUN_FEATURE_FLAG_NO_ORPHANS.

`run()` now races the socket connect, the test body and the graceful
exit against a deadline 5s (or 10%) before the runner's timeout. The
rejection returns through the disposers, which kill this test's
processes. An `afterAll` reaps whatever is still in `danglingProcesses`
when the file ends.

bundle.test.ts keeps the "commonjs forms" shape from main, minus its
debug console.log lines, so the file stays close to #39488.
Comment thread test/bake/bake-harness.ts Resolved
The harness module is evaluated once for every file in a `bun test`
run, so an `afterAll` registered at module scope only reached the first
file. `testImpl` now registers it the first time a file adds a test.

`stackTraceFileName` treated the letter before the first colon of a
posix path as a drive letter and kept the line number in the result.
The caller path is now the file alone on every platform.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the per-file afterAll reaper in d3d9ec7 addresses the concern I raised; each calling file now registers its own hook via Bun.jest(caller) inside testImpl, gated by filesWithReaper.

Checked: per-test dev.processes cleanup is scoped correctly and the disposer is registered before the test body runs; Deadline pre-attaches a .catch so its rejection is never unhandled and Promise.race keeps observing the abandoned body; the stackTraceFileName colon === 1 change stops mis-parsing POSIX path:line:col while still handling Windows drive letters; the two newly-awaited expectMessage calls and the status-based assertions strengthen tests without weakening any existing coverage.

Extended reasoning...

Overview

Test-infrastructure-only change across two files. test/bake/bake-harness.ts gains per-test process tracking (Dev.processes), a concurrent opt-in that routes to test.concurrent, a disposable Deadline that races the test body/connect/gracefulExit against a timeout slightly shorter than the runner's, per-test log prefixing for interleaved output, a tightened Windows drive-letter check in stackTraceFileName, and a per-file afterAll reaper (previously my one open concern — now registered inside testImpl gated on the caller path). test/bake/dev/bundle.test.ts opts all 21 cases into concurrent: true, awaits two previously-floating expectMessage calls, tightens two toBeInstanceOf(Response) assertions to check status/body, and removes debug console.logs.

Security risks

None. No production/runtime code touched; changes are confined to the bake test harness and one test file. No network egress, credentials, or user-facing surface.

Level of scrutiny

Moderate for a harness refactor. I focused on process-lifecycle correctness under concurrency (that the per-test disposer only kills dev.processes, that it's armed before any awaited call that could hang, that Deadline doesn't leak an unhandled rejection), the reaper registration scope (fixed in d3d9ec7 — jest = Bun.jest(caller) so jest.afterAll attaches to the calling file, and filesWithReaper prevents duplicates), and that no existing assertions were weakened (the two Response checks are strictly stronger; the removed console.logs were debug noise, not assertions).

Other factors

The prior inline concern about the module-cached afterAll was directly addressed by the latest commit. No CODEOWNERS cover these paths. The PR description documents verification across release and debug+ASAN, all 155 bake dev tests passing, and a probe confirming hung concurrent cases leave no processes. Exit reason was dry_streak.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant