Skip to content

test: run the serial macro-test spawn cases concurrently and assert exact output - #40623

Closed
robobun wants to merge 4 commits into
mainfrom
farm/d3b95a5b/macro-test-speed
Closed

robobun wants to merge 4 commits into
mainfrom
farm/d3b95a5b/macro-test-speed

Conversation

@robobun

@robobun robobun commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Five tests in test/bundler/transpiler/macro-test.test.ts spawn a bun child or run a build one after another: the sparse-key test, the destructuring test, and the three --no-macros tests. The other seven spawn tests already run concurrently.
  • Their assertions were loose: exit code only, tail and last-line matches, stringContaining, toContain.
  • The 42 s this file showed on the debian 13 x64-asan lane (build #106656) was the LeakSanitizer abort of a bun build child (the main break leaksan: follow the Bun::generateModule rename to Bun::generateInternalModule #40614 fixes) plus CI retries, not the tests.

Fix

  • The two standalone tests are test.concurrent and the --no-macros block is describe.concurrent, so all twelve spawn tests form one concurrent group.
  • The two Bun.build() cases of the --no-macros block run in this process through one test.each. The bun build --no-macros spawn stays, since it covers the CLI flag.
  • Each child test asserts stdout, stderr, exit code and signal code with one toEqual. The refusal case snapshots the exact build error. The Bun.build() cases assert the result object and the macro's marker file.
  • The "event loop routing around macros" block is unchanged. Its tests were already concurrent, and macros: run every macro on one dedicated VM thread #40059 changes its expected values.
  • Verified: bun bd test test/bundler/transpiler/macro-test.test.ts, local debug build: 5.29 s to 5.58 s before, 3.74 s to 3.85 s after. The asan lane stays red until leaksan: follow the Bun::generateModule rename to Bun::generateInternalModule #40614 lands, since the in-process Bun.build() runs the same node:fs macro. With leaksan: follow the Bun::generateModule rename to Bun::generateInternalModule #40614's suppression the file passes 10 of 10 local runs under detect_leaks=1.

Background

  • bun test runs consecutive concurrent tests as one group, up to --max-concurrency (20). A non-concurrent test or a beforeAll/afterAll hook ends the group, a describe boundary does not.
Notes
  • The LeakSanitizer report: Direct leak of 4104 byte(s) from Box::new in bun_runtime::node::node_fs_binding::create_binding. The box belongs to the NodeJSFS cell of the macro VM, which is never torn down. test/leaksan.supp covered it through leak:Bun::generateModule until compile: builtin module sources in a dedicated section so --bytecode covers non-host targets #40597 renamed that function to Bun::generateInternalModule. leaksan: follow the Bun::generateModule rename to Bun::generateInternalModule #40614 renames the entry. On main the same leak aborts the bun build child of the old file on every asan build since compile: builtin module sources in a dedicated section so --bytecode covers non-host targets #40597.
  • test/expected-durations.json records 8990 ms for this file on the asan lane and 150 ms by default. On build #106656 the runner reported 5.06 s, 3.9 s of it in the aborted child. On the asan lane the runner reports 0.9 s for the new file (build #106766).
  • Debug builds print [macro] call <name> to stdout for every macro call (src/js_parser_jsc/Macro.rs, a prettyln! that BUN_DEBUG_QUIET_LOGS does not silence). The runBun helper drops those lines so stdout can be compared exactly, as the event-loop block's run() already did. Moving that print to a scoped_log! is a separate change, after which both filters can go.
  • The CLI default (macros on without a flag) is covered by the itBundled macro cases and test/regression/issue/03830.test.ts, which spawn bun build with a macro.
  • The macro in the --no-macros block writes MACRO_RAN next to itself with import.meta.dir, so the marker lands in the temp dir whether the build runs in a child or in the test process.
  • toMatchInlineSnapshot works inside concurrent tests. Only the file snapshot matchers need the active test name.
  • The Bun.build() cases pass minify: true because the unminified bundle starts with a // <path> comment relative to the cwd, which varies between machines.
  • A one-test file that imports harness costs 2.03 s on the local debug build, so about 2 s of the remaining 3.8 s is fixed cost.
  • While measuring I ran bun build --target=bun --external '*' on this file. A wildcard --external also marks the macro import as external, so the macro host imports ./macro.ts relative to the cwd and fails unless the cwd is the source dir. Filed as bun build: a wildcard --external marks macro imports as external, so the macro fails to load #40626.

[stamp-90s] gate passed · iteration 5 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/bundler/transpiler/macro-test.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/bundler/transpiler/macro-test.test.ts
bun test v1.4.1 (731aa92da)

test/bundler/transpiler/macro-test.test.ts:
[macro] call escapeHTML
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call escape
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStrings
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call addStringsUTF16
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call identity
[macro] call default
[macro] call default
[macro] call identity
[macro] call identity
[macro] call escape
[macro] call ireturnapromise
(pass) bun builtins can be used in macros [2.29ms]
(pass) lati
... (truncated)
Exit: 0
diff hotspot
test/bundler/transpiler/macro-test.test.ts | 229 +++++++++++++++--------------
 1 file changed, 120 insertions(+), 109 deletions(-)

gate history · 1 passed · 0 rejected · iteration 5

evidence per changed file
file                                        reads  edits  tests
test/bundler/transpiler/macro-test.test.ts      8     13      0

@coderabbitai

coderabbitai Bot commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

On-demand reviews are free for the next 24 days. After that, they cost $0.25 per reviewed file.

Or wait 9 minutes for your next included review.

View limit details

Limit details: You’ve used all 5 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d0e33359-b379-4a66-8cd8-8507a7f1724c

📥 Commits

Reviewing files that changed from the base of the PR and between 47d954e and 83f9722.

📒 Files selected for processing (1)
  • test/bundler/transpiler/macro-test.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review.

Scope after self-review: the two standalone spawn tests and the --no-macros block. The "event loop routing around macros" block is byte-identical to main, since its tests were already concurrent and #40059 rewrites it.

How it was measured: bun bd test test/bundler/transpiler/macro-test.test.ts on a local debug build, three runs before (5.29 s, 5.41 s, 5.58 s) and three after the trim (3.74 s, 3.76 s, 3.85 s). On the asan lane the runner reports 0.9 s for the new file (build #106766). The release bun passes the file as well.

CI (build #107086, rebased onto the merged #40614): this file passes on every lane, including debian 13 x64-asan. The one lane still red is url.test.ts on darwin x64, which fails on main too and is reported separately. Every other red entry passed on retry or alone.

Related: #40658 moves the debug [macro] call print to a scoped logger and removes the [macro] filters from main's version of this file. Whichever lands second gets a small rebase in the two standalone tests, and runBun drops its filter then.

PR: #40623

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the hang-guard test now spreads ...rest and asserts signalCode: null, addressing the earlier nit.

What was reviewed: the runBun helper drains pipes concurrently and every rewired caller now asserts the full {stdout, stderr, exitCode, signalCode} object with toEqual; the --no-macros fixture writes MACRO_RAN via import.meta.dir so the in-process Bun.build() marker lands in the temp dir, not the repo cwd; the log.position! non-null assertion only executes on the macros: false arm where the expected value pins a concrete position, and the enabled arm expects logs: [] so the map never runs; describe.concurrent + test.each keeps every temp-dir case in the same concurrent group per the runner's grouping rules.

Extended reasoning...

Overview

Test-only refactor of test/bundler/transpiler/macro-test.test.ts: extracts a shared runBun spawn helper, converts serial subprocess tests to test.concurrent/describe.concurrent, tightens loose toMatchObject/regex assertions to exact toEqual on {stdout, stderr, exitCode, signalCode}, and collapses the Bun.build({ macros: false }) vs macros-enabled pair into one in-process test.each. The --no-macros CLI case now snapshots the exact stderr via normalizeBunSnapshot and checks the temp dir listing with readdirSync instead of two existsSync probes. No runtime code is touched.

Security risks

None. This is a test-file refactor with no changes to shipped code, no network access beyond a local Bun.serve({ port: 0 }), and all filesystem writes confined to tempDir-managed directories.

Level of scrutiny

Low-to-moderate. The change follows the harness conventions REVIEW.md enforces: Promise.all pipe draining, combined-object toEqual, bunEnv spread, using tempDir, normalizeBunSnapshot, and test.concurrent for independent subprocess cases. Assertions were strictly tightened, not loosened — every site that previously used toMatchObject, stringContaining, or regex tail-matching now pins exact values. The one prior inline nit (dropped signalCode on the hang-guard test) was addressed in the follow-up commit by spreading ...rest into the asserted object.

Other factors

The log.position! non-null assertion was considered: it only fires on the macros: false arm (the enabled arm expects logs: []), and an unexpected null would throw rather than diff — acceptable as a nit given the expected object pins concrete line/column values. The fixture's MACRO_RAN marker was moved to import.meta.dir so the in-process Bun.build() case writes into the temp dir rather than the test runner's cwd, avoiding a leftover file in the repo. The PR author documents that the in-process build triggers the same LeakSanitizer false positive #40614 suppresses; that's a merge-ordering concern the author has already called out, not a defect in this change.

@robobun

robobun commented Aug 27, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 4:24 PM PT - Aug 27th, 2026

❌ @robobun, your commit 83f9722 has 1 failures in Build #107086 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 40623

That installs a local version of the PR into your bun-40623 executable, so you can run:

bun-40623 --bun

@robobun robobun changed the title test: run the macro-test spawn cases concurrently and assert exact output test: run the serial macro-test spawn cases concurrently and assert exact output Aug 27, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — both earlier nits (signalCode on the hang-guard test, log.position! throwing on a positionless log) are addressed in 232c645 and d3c3405.

Checked: the runBun helper drains pipes concurrently and spreads bunEnv; the two test.each in-process Bun.build() cases each get their own tempDir and the fixture writes MACRO_RAN via import.meta.dir, so concurrent runs don't collide; the readdirSync().sort() assertion is strictly stronger than the old pair of existsSync(false) checks; and the --no-macros CLI path is still exercised as a subprocess so CLI-flag coverage isn't lost by moving the API cases in-process.

Extended reasoning...

Overview

This PR refactors test/bundler/transpiler/macro-test.test.ts — a single test file, no runtime code touched. It introduces a shared runBun helper (concurrent pipe drain, [macro] debug-line stripping, returns {stdout, stderr, exitCode, signalCode}), converts five previously-serial tests to test.concurrent/describe.concurrent, tightens loose stringMatching/stringContaining/last-line assertions into exact toEqual on full result objects, snapshots the --no-macros CLI stderr via normalizeBunSnapshot, and collapses the two Bun.build() API cases into an in-process test.each asserting {success, logs, outputs, macroRan}.

Security risks

None. This is a test-only refactor of transpiler macro tests; no auth, crypto, network, or user-input handling is touched. The in-process Bun.build() reads from a tempDir fixture and writes a marker file inside it.

Level of scrutiny

Low. Test refactors that tighten assertions and add concurrency are low-risk and self-verifying — if the refactor were wrong the file would fail. The main risk classes for this kind of change (concurrent tests sharing state, weakened assertions, lost coverage) were checked: each concurrent case owns its own tempDir; the macro fixture writes its marker via join(import.meta.dir, ...) so the in-process build's cwd doesn't matter; every rewritten assertion is equal-or-stronger than what it replaced (toEqual on the full object vs. tail regex, readdirSync().sort() vs. two existsSync(false), exact outputs text vs. toContain("INLINED_RESULT")); and the CLI bun build --no-macros subprocess test is retained so the CLI-flag entry point is still covered alongside the JS-API entry point.

Other factors

I raised two nits on earlier pushes (missing signalCode in the hang-guard assertion; log.position! throwing before the diff on a positionless log). Commits 232c645 and d3c3405 address both exactly as suggested — the hang-guard now asserts signalCode in its combined toEqual, and the log mapping uses log.position && {...} so an unexpected positionless log surfaces in the diff. No outstanding CHANGES_REQUESTED reviews from other reviewers. The exit reason was dry_streak. Approving.

…tput

Every test in macro-test.test.ts that spawns a bun child, or builds with
Bun.build, now runs in one concurrent group. The two Bun.build cases of
the --no-macros block run in this process through a test.each. One CLI
spawn keeps covering the --no-macros flag itself.

The child tests assert the full stdout, an empty stderr, the exit code
and the signal code in one toEqual. The --no-macros refusal snapshots
the exact build error. The Bun.build cases assert the result object
(success, log message and position, bundle text) and whether the macro
wrote its marker file.
Its tests were already concurrent, so rewriting them gained nothing, and
#40059 changes that block's expected values. The concurrency change and
the exact assertions stay on the two standalone spawn tests and on the
--no-macros block.
@robobun
robobun force-pushed the farm/d3b95a5b/macro-test-speed branch from d3c3405 to 83f9722 Compare August 27, 2026 23:04

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — thanks for the follow-ups; the log.position && {...} guard now keeps positionless logs in the diff, and leaving the event-loop routing block untouched to avoid churn with #40059 is reasonable.

What was reviewed:

  • runBun helper: await using + concurrent pipe drain + [macro] line filter, signalCode included in the returned object.
  • Converted tests assert full {stdout, stderr, exitCode, signalCode} via toEqual; CLI refusal case snapshots stderr with normalizeBunSnapshot and checks readdirSync().sort() for both no-marker and no-dist.
  • test.each Bun.build() cases: minify: true avoids the cwd-relative path comment, macroRan is read from import.meta.dir so the in-process build's marker lands in the temp dir.
Extended reasoning...

Overview

Test-only refactor of test/bundler/transpiler/macro-test.test.ts: introduces a shared runBun spawn helper, marks the five previously-serial subprocess tests as concurrent, and tightens assertions from regex/toContain/toMatchObject to exact toEqual on a composite {stdout, stderr, exitCode, signalCode} object. The two Bun.build() --no-macros cases collapse into one in-process test.each that asserts success, mapped logs (with position basename/line/column), output text, and a filesystem MACRO_RAN side-effect flag. No source changes.

Security risks

None. Test-only; no new network access, no credentials, no user-facing surface. The macro fixture writes a marker file inside a tempDir scoped with using.

Level of scrutiny

Low-to-moderate. This is a mechanical test refactor that follows the repo's stated conventions (test.concurrent for independent subprocess tests, single combined toEqual, normalizeBunSnapshot, tempDir, bunExe()/bunEnv, pipes drained concurrently). The two nits I raised on earlier revisions were both handled: 83f9722 replaced the log.position! non-null assertion with log.position && {...} so an unexpected positionless log surfaces in the diff rather than throwing; the hang-guard signalCode note was for the event-loop routing block, which 1ea2074 intentionally leaves at main's state to avoid conflicting with #40059 — a reasonable call for a pre-existing nit outside this refactor's scope.

Other factors

The refactor strengthens rather than weakens assertions (exact stdout/stderr instead of tail-match/stringContaining; readdirSync().sort() catches any unexpected file, not just the two existsSync was checking). The import.meta.dir-relative marker write is required for the in-process Bun.build() case since cwd is the test runner's, not the temp dir. No CODEOWNERS on this path; no outstanding third-party CHANGES_REQUESTED reviews in the timeline.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-08-27.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant