Skip to content

test(bundler): speed up bun-build-api.test.ts and tighten its assertions - #40196

Closed
robobun wants to merge 3 commits into
mainfrom
farm/edde83d8/speed-up-bun-build-api-test
Closed

robobun wants to merge 3 commits into
mainfrom
farm/edde83d8/speed-up-bun-build-api-test

Conversation

@robobun

@robobun robobun commented Aug 23, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/bundler/bun-build-api.test.ts takes 26 s on the debian 13 x64-asan lane (build #103933) and 157 s under bun bd test locally.
  • One case runs 400 builds of a 500-module graph: 123 s locally. The other child-process cases run one after another. The reloader fixture has two fixed Bun.sleep(1000) calls.

Fix

  • The loop becomes 40 builds that must produce identical output. It was written to overflow the FilenameStore. resolver: fix filename-store overflow panic under --bun #31504 raised that ceiling 32x to about 8.4M paths, so the unfixed resolver needs about 8,400 builds to reach it: the loop cannot detect the regression it documents. bundler: stop appending plugin-resolved paths to the FilenameStore on every bundle #39456 adds a direct gauge; once it lands, the loop can go.
  • The child-process cases at the end of the file run as test.concurrent. The skipped NumberRenamer case becomes test.concurrent.skip, because a serial entry splits the group. The thread-pool timing case stays serial on purpose.
  • The reloader fixture drops the first sleep and replaces the second with a retry loop with a deadline. The bundle thread advances the resolver generation once per queue drain, after it posts the result to JS, so a rebuild enqueued before that reuses the stale listing (bundler: advance the resolver generation once per build instead of once per queue drain #38212 fixes the thread). A listing that is never refreshed still fails after the deadline.
  • Verified: 157 s to 27 s locally over five runs. CI build #104194: 8.5 s on the x64-asan lane (was 26.3 s), 0.9 s on debian 13 x64 (was 5.6 s). No test removed, expect() calls go from 214 to 223. The release binary passes too.

Background

  • Consecutive test.concurrent entries form one group. Any serial entry, even a skipped one, starts a new group. Under ASAN a group runs 5 cases at a time.
  • FilenameStore is the process-lifetime interner for module paths. Running out is an allocation error since resolver: fix filename-store overflow panic under --bun #31504, not a panic.
  • The resolver caches directory listings per process. A listing is re-read when the build's generation is newer than the listing's. Snapshot matchers do not work in concurrent tests, so those cases stay serial.
Notes

Assertion changes:

  • "sourcemap + build error crash case" asserted nothing. It now checks success: false, no outputs, the single No matching export ... for import "C" message, and its position (line, column, lineText, file).
  • The four onEnd cases asserted toBeDefined() and length > 0. They now check the exact log list, that onEnd receives the same BuildOutput object the promise resolves with, and (for throw: true) that the rejection is an AggregateError with the same messages. The assertions moved out of the plugin callback: a throwing onEnd rejects the build promise, and the old try/catch would have swallowed that.
  • "multiple onEnd callbacks" asserted success is defined. It is true: the unused import is dropped by the TypeScript loader, so the missing file is never resolved.
  • "warnings do not fail a build" asserted the position is truthy. It now checks kind, level, message and position. Note: the runtime reports level: "warn" while packages/bun-types declares "warning". That mismatch is pre-existing and not touched here.
  • "rebuilding busts the directory entries cache" inherited stdout. It now pipes stdout and expects OK. The fixture also checks that the first build fails on the missing import and for no other reason.
  • "repeated in-process builds of a symlinked package" checked stdout + stderr. It now checks the marker on stdout and 1 pass / 0 fail on stderr.

Reloader fixture: the first push removed both sleeps. Nothing is cached before the first build, so the first sleep guarded nothing. The second sleep covered the generation window described above, and the rebuild failed once with Could not resolve: "./mutate.js" on the alpine 3.23 x64 lane of build #104111. src/bundler/BundleThread.rs reads generation per dequeued build and increments it only after queue.pop() returns null. A probe that parks build 1 in an async onLoad, writes mutate.js, and enqueues build 2 into the same drain reproduces it every time: build 2 fails, the build after the drain succeeds. The retry loop encodes exactly that.

Timing of the slow cases alone, debug ASAN build on this 12-core container: repeated builds 123 s (400 builds), ropes 13.7 s, inline sourcemap leak 8.0 s, reloader 2.75 s, thread-pool wait 1.4 s, GlobalThis 1.5 s, each deep TOML case 1.4 s. After the change the concurrent group at the end takes about 14 s (bounded by the ropes case, unchanged) and the repeated-build case takes 13.9 s inside it.

The rope race detector is unchanged: on the pre-fix release binary both the old and the new file catch the corruption in some runs on this machine (the case was designed for 16 cores).

Output determinism was probed before adding the check: 40 consecutive builds of the graph produced the same hash, size and content hash for both outputs.

Rebased on main after #40722 added a concurrent case between the ropes case and the C0 case. The only conflict was that insertion point. The rebase brings in #40417 and #40507, which add five serial bytecode cases to the Bun.build describe; two of them exceed the 5 s default timeout on this 12-core debug container (4 children doing 128 bytecode builds each), with or without this change. They pass in CI.

Related open PRs: #33704 lowers the same loop from 400 to 200 builds as part of a change across 53 files; this change supersedes that hunk. #39456 inserts a pathStoreCounts() block right after the ropes case, so one of the two needs a small rebase at that boundary.

Suites run: test/bundler/bun-build-api.test.ts with bun bd test (5 runs) and with USE_SYSTEM_BUN=1.


no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/bundler/bun-build-api.test.ts

@coderabbitai

coderabbitai Bot commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

  • Run on-demand review

On-demand reviews are free for the next 23 days. After that, they cost $0.25 per reviewed file.

Or wait 21 minutes for your next included review.

View limit details

Limit details: You’ve used all 5 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a041b5fa-6d16-4a37-8378-b7731c65cf08

📥 Commits

Reviewing files that changed from the base of the PR and between 427edf3 and 521a897.

📒 Files selected for processing (2)
  • test/bundler/bun-build-api.test.ts
  • test/bundler/fixtures/bundler-reloader-script.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 23, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. Rebased on main (one trivial conflict: #40722 inserted a case at the same spot).

How the timing was measured: bun bd test test/bundler/bun-build-api.test.ts on a debug ASAN build, 12 cores. Before: 157 s (the 400-build case alone took 123 s). After: 27 s, stable over five runs (26.9 s to 31.9 s). CI build #104194: 8.5 s on the x64-asan lane (was 26.3 s in #103933), 0.9 s on debian 13 x64 (was 5.6 s). No test removed. The release binary also passes the file.

The CI number that started this (26 s on debian 13 x64-asan, build #103933) came from three cases: the 400-build loop, the rope race detector and the inline sourcemap leak check ran one after another, plus 2 s of fixed sleeps in the reloader fixture.

The first push removed both sleeps from the reloader fixture, and the rebuild failed once on the alpine x64 lane of build #104111 (Could not resolve: "./mutate.js"). The bundle thread advances the resolver generation once per queue drain, so a rebuild enqueued before that reuses the stale listing (#38212 fixes the thread). The fixture now retries the rebuild with a deadline instead of sleeping for a fixed second. The PR body has the details.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only speedup that tightens assertions while cutting the file from ~157s to ~27s on debug ASAN.

What was reviewed:

  • The 400→40 build reduction: the comment and PR description show the original loop could no longer reach the FilenameStore ceiling after #31504, so it was dead coverage; the new determinism check is a strictly stronger assertion for what 40 builds can observe.
  • Removing the two Bun.sleep(1000) calls from the reloader fixture: the cache is generation-keyed, not mtime-keyed, and the fixture now asserts the first build fails for exactly the expected reason.
  • Making the child-process tests concurrent: the thread-pool timing case stays serial with a comment explaining why; the rope-race detector was checked against the pre-fix binary and still catches the corruption.
  • The onEnd test rewrites move assertions out of callbacks so a throwing onEnd can no longer mask a failure via the outer catch.
Extended reasoning...

Overview

Two test files touched, no production code. bun-build-api.test.ts gets: (1) five child-process tests converted to test.concurrent so they overlap; (2) the 400-build FilenameStore loop cut to 40 builds with an added byte-identical-output check and a rewritten comment explaining why the old loop could no longer reach the ceiling; (3) tightened assertions on six previously-weak cases (the crash case that asserted nothing, the four onEnd cases that used toBeDefined()/length > 0, the warnings case that only checked position was truthy). bundler-reloader-script.ts drops two fixed Bun.sleep(1000) calls, switches to throw: false with an exact error-message check on the first build, and prints OK so the parent test can assert stdout instead of inheriting it.

Security risks

None. Test-only changes; no auth, crypto, network, or untrusted-input handling touched.

Level of scrutiny

Moderate. Test-only, but the repo's review rules are explicit that weakening or de-flaking a test must preserve the property the original assertion protected. Each reduction here is justified: the 400-build loop is shown to be dead coverage (needs ~8,400 builds post-#31504), and a stronger determinism assertion replaces it; the sleeps are shown to be unnecessary because the cache is generation-keyed; the rope-race test was re-checked against the pre-fix binary. Every other assertion change is a strict tightening (exact log lists, exact positions, object identity between onEnd result and the resolved BuildOutput).

Other factors

  • The PR description is unusually thorough: per-case timing before/after, four repeat runs, release-binary verification, and an explicit note that the determinism check was probed over 40 builds before being added.
  • The onEnd throw: true rewrite fixes a real test-quality bug: assertions inside the callback would have rejected the build promise and been swallowed by the outer catch.
  • The 60s timeout on the 40-build case (down from 180s) gives ~4× headroom over the measured 13.9s under concurrent load on 12 cores; reasonable for CI.
  • The .concurrent.skip on the NumberRenamer case is cosmetic (keeps the concurrent group intact) and harmless if the grouping claim is ever wrong.
  • No prior human review comments; no CODEOWNERS on test files; no snapshot tests were made concurrent (the PR notes snapshot matchers don't work in concurrent tests and left those serial).

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only speedup that also tightens assertions.

What was reviewed:

  • Reloader fixture: retry loop replaces fixed sleeps; only the exact stale-listing error is retried, any other error throws immediately, and the 10s deadline still fails if the cache is never busted. The fd-leak check now spans all retries, which is at least as strict.
  • 400→40 build reduction: the original overflow can no longer be reached at 400 builds since #31504, so the test was already not protecting that; the new byte-identical-output check is a stronger invariant for cross-build state leaks.
  • test.concurrent conversions: snapshot tests and the timing-sensitive thread-pool case stay serial with a comment; .concurrent.skip on the NumberRenamer case keeps the concurrent group unbroken.
  • onEnd assertions moved out of the plugin callback so a throwing callback can't be swallowed by the surrounding catch.
Extended reasoning...

Overview

This PR touches only test/bundler/bun-build-api.test.ts and its fixture test/bundler/fixtures/bundler-reloader-script.ts. It reduces the file's runtime from ~157s to ~27s on a debug ASAN build by (1) cutting the 400-build loop to 40 builds with a new determinism check, (2) making the independent child-process tests test.concurrent, and (3) replacing two Bun.sleep(1000) calls in the reloader fixture with a bounded retry loop. Along the way it replaces vague assertions (toBeDefined(), length > 0, position toBeTruthy()) with exact ones on log name/level/message/position, and moves expect() calls out of plugin callbacks where a throw would have been swallowed.

Security risks

None. Test-only change; no production code, no auth/crypto/network surface.

Level of scrutiny

Moderate — while it's test-only, the review guidelines are explicit that weakening or de-flaking tests must preserve the property the original assertion protected. I checked each substantive change against that rule:

  • The 400→40 reduction: the PR description (and updated comment) explain that #31504 raised the FilenameStore overflow block size from 64 to 2048, so the ~8.4M-path ceiling now needs ~8,400 builds of this graph — the 400-build loop already couldn't detect the documented regression. The replacement (byte-identical output across 40 builds) catches a broader class of cross-build state leak. Timeout dropped from 180s to 60s, which fits the 40× smaller workload.
  • The reloader fixture retry loop only tolerates the exact expected stale-cache error (Could not resolve: "./mutate.js"); any other failure throws immediately, and a cache that never refreshes hits the 10s deadline and fails. The fd-leak baseline is captured before the retry loop and checked after, so a leak on any retry iteration still fails the test. This follows the repo rule of polling a condition with a deadline instead of a fixed sleep.
  • test.concurrent is applied only to child-process cases that don't share process-global state and don't use snapshot matchers. The thread-pool timing test stays serial with a comment explaining the 5s window would be violated by contention. .concurrent.skip on the skipped NumberRenamer case avoids splitting the concurrent group (a serial entry — even skipped — ends the group).

Other factors

  • Test count preserved (58 tests: 56 pass, 1 skip, 1 todo); expect() count went up 214 → 223.
  • Verified by the author with four bun bd test runs and the release binary.
  • Windows path separators handled in the new position assertions via replaceAll("\\\\", "/") and [\\\\/] in the regex.
  • The onEnd rewrite fixes a real vacuous-assertion pattern the review guidelines call out: expects inside a callback that, if they threw, would reject the build promise into a catch that swallowed everything.
  • No prior human or bot review comments to address; no CODEOWNERS on these test files.

@robobun

robobun commented Aug 23, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 2:47 AM PT - Aug 28th, 2026

✅ @robobun, your commit 521a89787aa89998136d9fe7806e2bc4949f5d3b passed in Build #107586! 🎉


🧪   To try this PR locally:

bunx bun-pr 40196

That installs a local version of the PR into your bun-40196 executable, so you can run:

bun-40196 --bun

Run the child-process tests at the end of the file concurrently, drop the
two fixed one second sleeps from the reloader fixture, and cut the repeated
build loop from 400 to 40 builds. The loop cannot reach the FilenameStore
ceiling it was written for since #31504 raised that ceiling to about 8.4M
entries. It now also checks that every build produces identical output.

Assert the build result in the cases that checked nothing or only checked
for a defined value: the sourcemap crash case, the four onEnd cases, the
warning position, the reloader child and the symlinked package child.
…sting refreshes

The bundle thread advances the resolver generation once per queue drain,
after it has posted the previous result to JS. A build enqueued before
that happens runs at the previous generation and reuses the stale
directory listing. The second fixed sleep used to cover that window, and
its removal made the test fail once on the alpine x64 lane.

Retry the rebuild with a deadline instead of sleeping for a fixed time.
A listing that is never refreshed still fails after the deadline.
@robobun
robobun force-pushed the farm/edde83d8/speed-up-bun-build-api-test branch from b9db5b7 to dd327e8 Compare August 28, 2026 08:13
Comment thread test/bundler/bun-build-api.test.ts Resolved
Comment thread test/bundler/bun-build-api.test.ts Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — both comment nits from the previous review are addressed in 521a897, and the rest is a well-scoped test-only tightening/de-flaking pass.

What was reviewed:

  • Reloader fixture: fixed sleeps replaced with a bounded poll loop that only tolerates the specific stale-listing error and hard-fails on anything else or past the deadline.
  • onEnd tests: assertions moved out of the plugin callback into the test body; throw: true rejection captured via .then(resolve, reject) and asserted as AggregateError with matching logs.
  • 400→40 build reduction: comment now documents why the original overflow ceiling is unreachable after #31504; the loop asserts a stronger property (byte-identical [kind, hash, contentHash] across all builds) instead of only survival.
  • test.concurrent grouping: the FIFO thread-pool test stays serial with a stated reason; the skipped NumberRenamer entry is .concurrent.skip so it doesn't split the group.
Extended reasoning...

Overview

This PR is a test-only speedup and assertion-tightening pass on test/bundler/bun-build-api.test.ts and its bundler-reloader-script.ts fixture. It replaces weak toBeDefined/toBeTruthy/length > 0 assertions with exact .toEqual/.toMatchObject checks on log shape and position, moves expect() calls out of onEnd plugin callbacks into the test body (so a failing assertion fails the test rather than being swallowed into the build's own rejection), converts independent subprocess-spawning tests to test.concurrent, cuts the 400-build FilenameStore regression loop to 40 builds while asserting deterministic output, and replaces two hardcoded Bun.sleep(1000) waits in the reloader fixture with a first-build exact-error check and a bounded retry loop for the stale-directory-listing window. No source code under src/ is touched.

Security risks

None. This change only touches test code and a test fixture. It spawns child processes of the local bun binary against temp-dir fixtures, contacts no network, and introduces no new inputs that reach production code paths.

Level of scrutiny

Moderate — enough to confirm the changes follow the repo's test conventions in REVIEW.md and CLAUDE.md, and that the one place a regression test's workload is reduced (400 → 40 builds) is justified and not a silent weakening. The PR description and the in-file comment both explain that #31504 raised the FilenameStore ceiling ~32× (to ~8.4M paths), so the unfixed port would need ~8,400 builds to hit it: neither the old 400 nor the new 40 loop can detect that specific regression any more. The new loop instead asserts a stronger invariant (identical output across every build), which resolver/graph state leaking between builds would break. This satisfies REVIEW.md's "when de-flaking, keep asserting the property the original assertion protected" — the property has shifted, but with a stated reason and a stronger replacement, and #39456 is cited as the direct-gauge follow-up.

Other factors

I posted two comment-accuracy nits on the previous push; commit 521a897 fixed both exactly (the "catch below" wording now names the rejection handler, and the concurrent-group comment now qualifies "most" and calls out the in-process sourcesContent test). No CHANGES_REQUESTED reviews are outstanding, and CODEOWNERS does not cover these paths. The reloader fixture's retry loop only tolerates the exact Could not resolve: "./mutate.js" message inside the deadline and throws on any other failure or after 10s, so a genuine cache-never-refreshes regression still fails the test. The explicit per-test timeouts (60s/180s) on the concurrent subprocess tests are the "rare outlier" case CLAUDE.md permits — each spawns dozens-to-hundreds of real bundles under debug/ASAN — and the 180s ceiling on the ropes case is unchanged from base.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #49196, which combines the open test-only PRs into one. The commit b17fad24f in that PR is this change, unchanged.

Closing this one in favor of #49196.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant