Skip to content

test: run the node:test fixture spawns concurrently and assert exact reports - #42149

Closed
robobun wants to merge 3 commits into
mainfrom
robobun/92a84e03/node-test-runner-concurrent
Closed

robobun wants to merge 3 commits into
mainfrom
robobun/92a84e03/node-test-runner-concurrent

Conversation

@robobun

@robobun robobun commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/js/node/test_runner/node-test.test.ts took 12 s on the debian 13 x64-asan lane. About 30 of its tests spawned a bun test <fixture> child, one at a time.
  • Most tests asserted only exitCode and a "0 fail" or "N pass" substring. That passes when a fixture registers no tests or prints an unexpected error.

Fix

  • The node:test describe is describe.concurrent, without explicit timeouts. An ASAN build caps concurrent tests at 5, so the children do not pile up.
  • runTests spawns with await using and the fixtures directory as cwd, so the children skip the repo bunfig.toml preload (a quarter of a debug child's time). Its signature and raw output stay for the other open changes to this file.
  • runTests also returns stdoutLines, report (each file header, error message and (pass|fail|skip|todo) line, in order) and summary (the printed counters). Every fixture run asserts all three with toEqual.
  • Verified: bun bd test on this file (debug+ASAN) went from 69.73 s to 14.54, 14.76 and 14.54 s in three runs, 45 pass each. Release: 1.95 s to 0.75 s. Self-reviewed: 4 concerns raised, 2 addressed, 2 are follow-ups outside this file.

Background

  • Each fixture is a node:test file that runs under bun test in a child. Hooks, assert.register() and module-scope mocks are scoped per file, and several fixtures check that isolation. So each test keeps its own child.
  • bun test reads bunfig.toml from its working directory only. At the repo root it names test/preload.ts.
Notes

Assertion changes per fixture:

  • Exact ordered (pass|fail|skip|todo) lines and exact summary counts (pass, skip, todo, fail, filtered out, plus tests and files from Ran N tests across M files) for every fixture run: 01, 02, 03, 04, 01+02, 05, 06, 07, 08, 09, 10, 11, 12 (serial and --concurrent), 13, 14a+14b, 15, 16, 16b, 17 (per --rerun-each run), 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29. Any error: / SomeError: line, unhandled-error banner or N error counter that a passing fixture did not print before fails the comparison.
  • Exact failure messages, each paired with the (fail) line it precedes: 07 (all ten, for example passed a callback but also returned a Promise, callback invoked multiple times, test timed out after 1ms), 16b (test timed out after 100ms), 24 (1 subtest failed then boom), 27 and 28 (test was expected to fail but passed), 29 (The test failed, but the error did not match the expected validation and the validator's AssertionError), 13 (the two todo bodies' errors, which proves the bodies ran under --todo).
  • Exact stdout: 02-hooks now prints the recorded hook order from its last after() and the test compares it with 02-hooks.json (alone and in the two-file run). 05-test-in-test prints the subtest order from its t.after() hook, so the test also proves that hook ran. 07 (SUB_BODY_RAN=false) and 26 (only the todo suite body prints) compare the whole stdout instead of toContain / not.toContain.
  • bun test prints concurrent tests in completion order, so the one --concurrent fixture run (12) compares its report sorted.

Timings on this machine (8 cores), bun bd test test/js/node/test_runner/node-test.test.ts (debug+ASAN): before 69.73 s, after 14.54 s / 14.76 s / 14.54 s. USE_SYSTEM_BUN=1 bun test (release): before 1.95 s, after 0.75 s. A debug child took about 2.05 s with the repo preload and 1.47 s without it. The slowest single test under debug+ASAN takes about 2 s, under the 5 s default.

Stability: five extra release runs and two debug runs under eight CPU-burning processes all passed. Fixtures with timers (07's 1 ms hook timeout against a 200 ms settle, 11's 300 ms sleeps against a 3000 ms timeout, 12's waitFor, 16b's 100 ms timeout) compare outcomes, not durations.

Review follow-ups left out on purpose, both outside this file: the ASAN-only cap of 5 concurrent tests (src/options_types/context.rs) could also apply to plain debug builds, and test/preload.ts could import a lighter module than the whole harness.

Nothing under src/ changed. No fixture writes to disk, so the concurrent children share nothing but read-only files.


[auto-merge] gate passed · iteration 2 · 3 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/node/test_runner/node-test.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/js/node/test_runner/node-test.test.ts
bun test v1.4.3 (f42e98025)

test/js/node/test_runner/node-test.test.ts:
(pass) node:test > should run async tests [1538.57ms]
(pass) node:test > should run tests with different variations [1567.93ms]
(pass) node:test > should run hooks in the right order [1723.21ms]
(pass) node:test > should run basic tests [1786.76ms]
(pass) node:test > should run all tests from multiple files [1960.10ms]
(pass) node:test > should run test() and describe() called inside another test() as subtests [1627.35ms]
(pass) node:test > should run before hooks created on a running test once and validate hook options [1618.16ms]
(pass) node:test > should support done callbacks in tests and hooks [1589.94ms]
(pass) node:test > should fail tests whose hooks, bodies, or inline suite callbacks fail [1688.84ms]
(pass) node:test > should count runtime t.todo()/t.skip() as todo/skip and keep runner timers real under mock timers [1685.01ms]
(pass) node:test > should run todo bodies under --todo instead of registering an empty function [1515.91ms]
(pass) node:test > should count runtime t.todo()/t.skip() as todo/skip under --concurrent too [1770.31ms]
(pass) node:test > should not leak file-level beforeEach hooks across files in one process [1577.63ms]
(pass) node:test > should treat only as a no-op instead of using bun:test's CI-banned only() [1456.45ms]
(pass) node:test > should forward Infinity and finite timeouts so they override the runner default [2025.80ms]
(pass) node:test > should serialize inline suites and await async describe callbacks like node [1571.67ms]
(pass) node:test > should expose the body outcome to afterEach and workerId to the context [1505.39ms]
(pass) node:test > should capture plan at first t.assert access and resolve subtests started after their parent finished [1556.73ms]
(pass) node:test > should bound plan({wait:true}) by the test's own t
... (truncated)
Exit: 0
diff hotspot
test/js/node/test_runner/fixtures/02-hooks.js      |   3 +-
 .../node/test_runner/fixtures/05-test-in-test.js   |   1 +
 test/js/node/test_runner/node-test.test.ts         | 662 +++++++++++++--------
 3 files changed, 423 insertions(+), 243 deletions(-)

gate history · 2 passed · 0 rejected · iteration 2

evidence per changed file
file                                                  reads  edits  tests
test/js/node/test_runner/fixtures/02-hooks.js             2      2     30
test/js/node/test_runner/fixtures/05-test-in-test.js      1      1     29
test/js/node/test_runner/node-test.test.ts                8     12     29

…reports

node-test.test.ts spawned ~30 `bun test` children one after another. The
describe is now concurrent, the children no longer load the repo preload,
and every fixture run asserts its exact ordered (pass)/(fail)/(skip)/(todo)
lines, error messages, summary counts and stdout instead of a substring.
…t capture

Keep runTests(filenames, env, args) and return the raw stdout/stderr too,
so the other open changes to this file still work after a rebase. The
exit-code assertion carries the child's stderr as its message. `report`
also captures `SomeError: ...` lines and the unhandled-error banner. The
mock.property spawn test is back at the end of the file, unchanged.
@github-actions github-actions Bot added the claude label Sep 9, 2026
@robobun

robobun commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Status: ready for review.

How this was verified, on an 8-core container:

  • bun bd test test/js/node/test_runner/node-test.test.ts (debug+ASAN): 69.73 s before, 14.54 s / 14.76 s / 14.54 s after, three consecutive runs, 45 pass each.
  • USE_SYSTEM_BUN=1 bun test test/js/node/test_runner/node-test.test.ts (release): 1.95 s before, 0.75 s after.
  • Five release runs and two debug runs under eight CPU-burning processes passed.
  • A deliberate change to a fixture's printed subtest order fails the parent test with a one-line diff.

@coderabbitai

coderabbitai Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 5b63b037-cd02-458f-92bb-31fe1a793f88

📥 Commits

Reviewing files that changed from the base of the PR and between 5f55496 and ff0a566.

📒 Files selected for processing (3)
  • test/js/node/test_runner/fixtures/02-hooks.js
  • test/js/node/test_runner/fixtures/05-test-in-test.js
  • test/js/node/test_runner/node-test.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 1 remains after this review.


Walkthrough

The node:test integration suite now runs fixture-specific concurrent tests with normalized output parsing and exact assertions. Hook fixtures now print recorded execution order for validation.

Changes

Node test runner validation

Layer / File(s) Summary
Fixture execution-order output
test/js/node/test_runner/fixtures/02-hooks.js, test/js/node/test_runner/fixtures/05-test-in-test.js
Fixtures now print recorded hook and subtest execution order.
Integration test harness and assertions
test/js/node/test_runner/node-test.test.ts
Tests use Bun.spawn, shared fixture execution, normalized output parsing, exact reports, summaries, statuses, diagnostics, and filtering checks.

Suggested reviewers: cirospaciari

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to ff0a5

This strengthens and accelerates node:test integration coverage without changing runtime source code, and no actionable merge-blocking risk is identified.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: concurrent node:test fixture spawns with exact report assertions.
Description check ✅ Passed The description explains the problem, implementation, verification steps, performance results, scope, and follow-up items. Although it does not use the template headings verbatim, it provides the requ…

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only hardening that strengthens assertions and speeds up the suite.

What was reviewed:

  • runTests helper: await using + Promise.all pipe drain, regex parsing of report/summary lines and duration stripping look correct; cwd: fixturesDir keeps headers to bare filenames and skips the repo preload.
  • Concurrency safety: verified no fixture writes to disk, so describe.concurrent children share nothing mutable; the one --concurrent fixture run sorts before comparing.
  • Fixture edits: the removed %AFTER% marker in 02-hooks.js has no other consumers; new console.log lines feed the exact stdoutLines assertions.
Extended reasoning...

Overview

This PR touches only test/js/node/test_runner/node-test.test.ts and two of its fixture files. It replaces ~30 loose expect.stringContaining("N pass" / "0 fail") assertions with exact toEqual comparisons against a parsed per-test report array and a summary object, switches the outer describe to describe.concurrent, drops per-test 30 s timeouts, sets cwd to the fixtures directory (avoids the repo bunfig.toml preload and yields bare-filename headers), and adds two console.log lines to fixtures so stdout can be asserted exactly. No src/ code changes.

Security risks

None. Test-only change; no network, no filesystem writes, no credential handling, no new dependencies. Children spawn bunExe() with bunEnv over read-only fixture files.

Level of scrutiny

Low-to-moderate. The change is mechanical and moves the file squarely toward the repo's stated test conventions: strongest-invariant assertions (toEqual over toContain), content asserted before exit code, await using + concurrent pipe drain, describe.concurrent for independent subprocess spawns, and no ad-hoc timeouts. I read the new runTests parser regexes (file header, error:/SomeError: lines, (pass|fail|skip|todo) with duration strip, summary counters with \r?$ for Windows) and they match Bun's test-reporter output shape. The --concurrent variant correctly sorts both sides before comparing since completion order is nondeterministic.

Other factors

No CODEOWNERS entry covers these paths. Grep confirmed no fixture performs filesystem writes, so concurrent children cannot interfere. The removed %AFTER% sentinel in 02-hooks.js has no remaining consumers anywhere in the repo. The PR timeline shows no prior reviews or outstanding objections, and the bug hunt ran to dry_streak with no findings.

@robobun

robobun commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 3:12 PM PT - Sep 9th, 2026

✅ @robobun, your commit 417400e31df6e53c1e25c0a93527257238e7bba1 passed in Build #113599! 🎉


🧪   To try this PR locally:

bunx bun-pr 42149

That installs a local version of the PR into your bun-42149 executable, so you can run:

bun-42149 --bun

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #49196, which combines the open test-only PRs into one. The commit 61a646bc3 in that PR is this change, unchanged.

Closing this one in favor of #49196.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant