Skip to content

test/bundler: run the bundler_edgecase deep graph cases concurrently and pin more outputs - #39647

Closed
robobun wants to merge 4 commits into
mainfrom
farm/877a728d/bundler-edgecase-faster
Closed

robobun wants to merge 4 commits into
mainfrom
farm/877a728d/bundler-edgecase-faster

Conversation

@robobun

@robobun robobun commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • bundler_edgecase.test.ts takes 23s to 31s on the asan lane and up to 42s on darwin (table in Notes). Three cases take most of it: edgecase/DeepImportChain, DeepImportChainWrappedTLA and DeepImportDiamondDAG. They ran in sequence, they wrote the 7000 module chain twice, and release builds cannot fail them.
  • Seven cases only checked that the build succeeded (list in Notes). YieldKeyword even produced an empty bundle.

Fix

Background

  • The default api backend of itBundled calls process.chdir, so expectBundled.ts registers those cases as it.serial. A serial test is a barrier for the concurrent ones, so only the deep cases and five cli cases overlap.
  • Depth 7000 overflowed the old walks under debug+ASAN only: a release build's stack holds more than 10000 modules. The old O(V*E) walk took about 4.5s on the 20000 module DAG (bundler: BFS the code-splitting reachability walk for O(V+E) on shared-importer DAGs #35310), far below the cap. Bun 1.3.14, from before both fixes, passes all three cases at the full sizes.
Notes

Per-lane duration of this file as reported by bun test, in seconds. Build 101234 is the first commit of this PR (full sizes on every lane, concurrent). The other builds are main and unrelated PRs from the same day. The second commit (small sizes on release builds) passed CI as build 101410. On the release lanes the three deep cases now take about 0.1s to 0.3s in total (measured locally), so most of the time left there is the 131 serial api cases.

lane other builds (n) 101234
debian 13 x64-asan 23.2 to 31.5 (8) 19.3
darwin any aarch64 (PR builds, host dependent) 6.0 to 42.0 (8) 18.5
darwin 14 / 26 (main builds) 5.8 to 9.7 (9) not run on PRs
alpine 3.23 aarch64 16.2 to 29.8 (10) 25.1
alpine 3.23 x64 17.3 to 27.3 (10) 23.5
debian 13 aarch64 4.8 to 6.8 (10) 3.4
debian 13 x64 3.2 to 6.2 (9) 4.2
ubuntu 25.04 aarch64 4.2 to 6.6 (10) 4.5
ubuntu 25.04 x64 4.0 to 6.6 (10) 4.0
windows 11 aarch64 8.6 to 10.5 (10), 1 test 14.1, 3 tests
windows 2019 x64 3.7 to 6.6 (10), 1 test 6.6, 3 tests

Windows: every itBundled case is silently dropped on Windows today (#34552 fixes that), so the Windows numbers are the deep cases alone. Before this PR that was the DAG case only. The first commit added the two chain cases there at full size, which is the Windows slowdown in the table. The second commit runs all three at the small sizes there, because those lanes are release builds. The alpine lanes are noisy (29.8s and 27.3s on unrelated builds), so their 101234 numbers are not conclusive either way. The small sizes take that question away, because the three concurrent builds become tiny there.

Why release builds cannot fail the full sizes: with bun 1.3.14 (0d9b296, before 483fd42 and 0870c7c) the 7000 module chain bundles in 0.7s and prints 7000, the TLA variant has exactly 6999 wrappers, and the DAG bundles in 0.4s. #35310 reports the regressed DAG walk at 4.4s to 4.6s for 20000 modules on a release build, against the 60s cap. The chain variants need more than 10000 modules to overflow the 8MB main thread stack of a release build on Linux, and the darwin and Windows builds get an 18MB stack. So the full sizes only run where isDebug || isASAN is true: local bun bd runs and the asan lane. test/expected-durations.json has this file's asan median at 15.7s, above the 15s rule of test/parallel-allowlist.json, so the file stays in excludeFiles after this PR as well. The gain is wall time in the serial part of the run.

Local timings: debug build (bun bd test), two runs before (145s, 190s) and three after (78s, 80s, 103s). Release build (USE_SYSTEM_BUN=1), four runs before (6.3s to 8.2s) and three after the second commit (1.8s to 2.5s). Slowest cases before, debug build: DeepImportChain 41.1s, DeepImportChainWrappedTLA 35.7s, DeepImportDiamondDAG 35.7s, AssetEntryPoint 2.0s, every other case under 1.3s. Release build: DeepImportDiamondDAG 2.5s, DeepImportChain 1.4s, DeepImportChainWrappedTLA 1.0s, every other case under 85ms. Under the debug build most of the old chain case time was the itBundled pipeline itself: dedent, mkdirSync and writeFileSync per module took about 30s per case, against 2.4s for plain writes of the same files.

Assertion changes:

  • EmptyFile: pins the bundle. The empty file is CommonJS, so entry_default = {} is exported as default.
  • StarExternal: pins that both relative imports stay as written with external: ["*"].
  • ValidLoaderSeenAsInvalid: pins the bundle.
  • RequireUnknownExtension: the output directory holds entry.js and x-4my6q98m.aaaa, the copied file holds x, and the bundle requires the copied path.
  • ExportDefaultUndefined: pins the bundle, which exports a and no default.
  • YieldKeyword: the generator is exported so that it survives tree shaking, and the printed form of every yield is pinned (yield;, yield* 2;, yield yield;, (yield).hello;, yield 1;). Before, nothing used the generator, so the bundle was empty and only the parse was tested.
  • UsingWithSixImports: pins the bundle (using kept, bun:sqlite import kept).
  • DeepImportDiamondDAG: exact printed value of m[0], computed in the test with the same expression order, instead of typeof v.
  • DeepImportChainWrappedTLA: exactly deepChainDepth - 1 wrapper definitions (the leaf module is a constant and gets none), and the deepest one is present.
  • buildDeepGraph throws a message that names the graph, the signal or exit code, the cap, and the build's stdout and stderr. Both paths were exercised with a missing import and with a 1ms cap.

Todo cases: itBundled registers a todo case with a plain it.todo, not it.serial.todo, so under describe.concurrent the adjacent api todo cases (RuntimeExternalImport and RuntimeExternalImport2, and the three OverwriteInput* cases) overlap in a bun test --todo run and can race on process.chdir. CI never passes --todo, so CI is not affected. The fix is it.serial.todo in the todo branch of itBundled, which is inside a hunk of #38655, so this PR only documents it in the comment above the describe.

Diff scope: the three deep cases use test.concurrent explicitly, so they also overlap if the describe ever goes back to plain describe, and the block keeps the same closing lines as before. The second describe at the end of the file (one api case, one cli case) is left as it is. edgecase/AwsCdkLib is currently not registered at all because it sets timeoutScale, which #34542 fixes. A dry-run merge of ten open PRs that add cases to this file (#38984, #35617, #36122, #38505, #38471, #31667, #33807, #34734, #38757, #38533) gives the same result against this branch as against main. #38629 adds an import line next to the import lines this PR changes, so it gets a conflict on those lines. No deep-import-* directories remain in the temp dir after a run.


no test proof · iteration 1 · Platform-specific test-only change; deferring to CI.

…tputs

Switch the main describe to describe.concurrent so the backend=cli cases
and the three deep import graph cases overlap. Write the 7000 module chain
once for both chain cases, with plain fs calls instead of the itBundled
fixture pipeline, and register the DAG case first so its build runs while
the chain fixture is written.

Add exact output checks to EmptyFile, StarExternal, ValidLoaderSeenAsInvalid,
RequireUnknownExtension, ExportDefaultUndefined, YieldKeyword and
UsingWithSixImports, which only checked that the build succeeded. The DAG
case now checks the value computed through all 20000 modules, and the
wrapped TLA case counts the wrappers.
@coderabbitai

coderabbitai Bot commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c6c286c7-b667-4044-9766-45102f7a7490

📥 Commits

Reviewing files that changed from the base of the PR and between a356964 and 0a83176.

📒 Files selected for processing (1)
  • test/bundler/bundler_edgecase.test.ts

Included review availability: Your plan provides up to 5 included reviews per hour; 0 remain after this review.


Walkthrough

The bundler edge-case suite now runs concurrently, captures inline output snapshots, validates copied assets, and tests generated syntax. Deep dependency-graph tests use spawned builds, generated fixtures, timeout handling, runtime checks, wrapper-count assertions, and cleanup.

Changes

Bundler edge-case validation

Layer / File(s) Summary
Bundle output snapshots and asset checks
test/bundler/bundler_edgecase.test.ts
The suite runs concurrently and validates emitted output for external imports, loaders, unknown-extension assets, default exports, generators, and using declarations.
Deep dependency-graph build tests
test/bundler/bundler_edgecase.test.ts
Generated diamond and chain fixtures are built in spawned processes. Tests enforce timeouts and process errors, then check runtime values, initialization wrappers, and temporary-directory cleanup.

Suggested reviewers: jarred-sumner

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the primary changes: concurrent deep-graph tests and additional output assertions.
Description check ✅ Passed The description explains the problem, implementation, verification results, scope, and test evidence, although it uses different section headings than the template.

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: test only change, ready for a maintainer. The diff is green. The red CI status is unrelated, see below.

Commits:

CI: test/bundler/bundler_edgecase.test.ts passed on every lane in all four builds (101234, 101410, 101420, 101431). Builds 101234 and 101410 were fully green. Builds 101420 and 101431 each failed in one job, darwin any aarch64, on test/js/node/tls/node-tls-server.test.ts, which this PR does not touch and which also fails on main. It is reported to main-break triage. I will not push again only to rerun CI.

How the numbers were taken, against a356964:

  • bun bd test test/bundler/bundler_edgecase.test.ts (debug+ASAN, full sizes): 145s and 190s before, 78s to 103s after.
  • USE_SYSTEM_BUN=1 bun test test/bundler/bundler_edgecase.test.ts (release, small sizes): 6.3s to 8.2s before, 1.8s to 2.5s after.
  • The per-lane CI table for the first commit is in the description.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only change that strengthens assertions and cuts file runtime roughly in half.

What was reviewed:

  • Verified expectBundled.ts registers api-backend cases as it.serial, so describe.concurrent only overlaps the cli-backend and plain test.concurrent cases; each of those builds in its own directory.
  • Checked the shared deepChainRoot() for races: it's fully synchronous, so the first concurrent caller populates deepChainDir before yielding, and both chain cases write distinct outfiles into the shared read-only fixture. afterAll disposes it.
  • Confirmed bunRun trims stdout and toSpawn(expected) does an exact string compare, so toSpawn(String(deepChainDepth)) and toSpawn(String(values[0])) match console.log(v) output; the DAG test computes values[i] with the same left-to-right addition order the bundled code evaluates.
  • The seven new inline snapshots are on api-backend (serial) cases, so no snapshot-write concurrency concern.
Extended reasoning...

Overview

This PR touches a single test file, test/bundler/bundler_edgecase.test.ts, with two independent goals: (1) run the three deep-import-graph regression guards concurrently and share the 7000-module chain fixture between the two chain cases, and (2) add onAfterBundle inline-snapshot assertions to seven itBundled cases that previously only checked that the build succeeded. No production code changes.

Security risks

None. Test-only; no new external inputs, no network, no auth/crypto paths.

Level of scrutiny

Low-to-medium. It's a test refactor, so the risk is (a) breaking existing regression guards or (b) introducing flakiness. I checked both. The deep-graph cases keep the same graph sizes (7000 chain depth, 20000 DAG modules) and the same 60 s build timeout, so the stack-overflow and O(V*E) regressions from #34554/#35310 are still guarded. The chain cases were already backend: "cli" before, and now spawn bun build directly with the same flags — same code path exercised. The concurrency model was checked against expectBundled.ts:1911, which registers api-backend cases as it.serial, so the ~130 default-backend cases stay sequential under describe.concurrent and their process.chdir/configRef state is safe.

Other factors

The shared deepChainDir between two concurrent tests is race-free because deepChainRoot() is entirely synchronous (tempDir + a loop of writeFileSync), so the first test body to reach it completes all 7003 writes before yielding at await buildDeepGraph; the second call sees the cache. Both builds only read the m*.js files and write to distinct outfiles (chain.out.js vs tla.out.js), and afterAll cleans the directory via Symbol.dispose. bunRun (harness.ts:526) trims stdout and toSpawn(expectedStdout) (harness.ts:737) does an exact compare, so the new stdout assertions line up with console.log(v). The DAG expected value is computed with deps.reduce((sum, d) => sum + values[d], i), which evaluates (i + v[i+1]) + v[2i+3] — identical addition order to the generated export const v = i + v_{i+1} + v_{2i+3}, so double-precision results match even if the sum grows large. The seven new snapshots are strict improvements over the previous no-assertion cases (YieldKeyword in particular was tree-shaken to an empty bundle before adding export). The PR description records verified before/after timings on both debug and release builds.

@robobun

robobun commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

The review above raises no change requests, so the diff stays as pushed (b9530d2). The status comment above stays the place to look for the current state. I am also running a second self-review of the diff and will push or comment if it finds something.

… ASAN builds

A release build from before #34554 and #35310 bundles all three graphs at
the full sizes, so release lanes cannot fail these cases and only pay for
writing 27000 files. Keep the full sizes where the cases can fail (debug
and ASAN builds) and run the same shapes at a small size elsewhere.

buildDeepGraph now names the graph and the build cap in its error, and
prints the build's stdout and stderr, instead of failing through a
toEqual diff.
@robobun

robobun commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed a3631ad after a second self-review of the first commit. It found that a release build from before #34554 and #35310 passes all three deep cases at the full sizes (I confirmed this with bun 1.3.14), so the release lanes paid for 27000 files without a chance to fail. The deep cases now keep the full sizes on debug and ASAN builds only and use 500 and 1000 modules elsewhere. The build helper also reports the graph, the cap and the build's stderr when it fails, instead of a toEqual diff. The description now has the per-lane CI numbers for the first commit. The rate limited bot review above needs no action.

@robobun

robobun commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 3:18 PM PT - Aug 19th, 2026

❌ @robobun, your commit 8d93ee4 has 1 failures in Build #101431 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 39647

That installs a local version of the PR into your bun-39647 executable, so you can run:

bun-39647 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the inline nit, I also checked the shared deepChainRoot() state between the two concurrent chain tests — tempDir is synchronous so the first caller fully initializes deepChainDir before yielding, and each build writes a distinct outfile in that directory. I also confirmed the adjacent cli-backend itBundled cases (the NodeEnv* group, InvalidLoaderSegfault, OutWithTwoFiles) each build under a unique per-id root, so describe.concurrent does not let them interfere.

Extended reasoning...

This is a test-only refactor that introduces describe.concurrent, reworks the three deep-graph guards to share a fixture directory and overlap via subprocess builds, gates the full graph sizes on isDebug || isASAN, and adds inline-snapshot assertions to seven previously build-only cases. I verified bunRun trims stdout so toSpawn(String(N)) matches, that the DAG value recurrence in the test uses the same left-to-right addition order as the emitted export const v = i + v[i+1] + v[2i+3], and that expectBundled.ts does register api-backend cases as it.serial (line 1911). The one finding — it.todo cases inheriting concurrency under --todo — is real but CI-invisible; the release-build size reduction is a deliberate coverage tradeoff a maintainer should sign off on.

Comment thread test/bundler/bundler_edgecase.test.ts Resolved
@robobun

robobun commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Confirmed the todo finding. With bun test --todo, two plain it.todo cases inside describe.concurrent start at the same time, and it.serial.todo cases run one after the other (checked with a small file of four todo cases). So in a --todo run of this file, RuntimeExternalImport and RuntimeExternalImport2, and the three OverwriteInput* cases, can race on process.chdir. The CI runner never passes --todo, so CI does not see it. The other two files that already put itBundled cases under describe.concurrent have no todo cases, so this file is the first one where it can happen.

The fix is it.serial.todo in the todo branch of itBundled. That line sits inside a hunk of #38655, and this PR stays out of expectBundled.ts on purpose, so I did not change it here. 0a83176 notes the exception in the comment above the describe, and the description's Notes record it.

The shared chain fixture and the per-id roots of the cli cases are as described in the review.

@robobun

robobun commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Both bot reviews of 0a83176 report nothing to act on, and the one review thread is resolved. Builds 101234 and 101410 passed on every lane. Build 101420 for 0a83176, which only changes a comment, is running. Nothing else is pending on my side. The status comment at the top stays current.

@robobun

robobun commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Build 101420 (for 0a83176) failed in one job, darwin any aarch64, on test/js/node/tls/node-tls-server.test.ts. That test is not touched by this PR and fails on main as well, so it went to main-break triage. bundler_edgecase.test.ts passed on every lane, as in builds 101234 and 101410. 8d93ee4 is an empty commit that runs CI again (build 101431). The coderabbit summary of 0a83176 has no actionable comments.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and the bug-hunting pass found no new issues (the earlier --todo note is acknowledged and documented in 0a83176). Given that this reduces the deep-graph regression guards to 500/1000 modules on every release lane and restructures the file's concurrency, a maintainer sign-off on that coverage trade-off would still be worthwhile.

What was reviewed:

  • describe.concurrent vs the it.serial guard in expectBundled.ts — api-backend cases stay serial; only cli and the three plain test.concurrent cases overlap.
  • Shared deepChainRoot() fixture — both consumers only read the m*.js files and write distinct outfiles; afterAll disposes it.
  • buildDeepGraph — pipes drained concurrently, signalCode checked before exitCode, error message includes stdout/stderr.
  • New bunRun/toSpawn and isDebug/isASAN imports resolve in test/harness.ts.
Extended reasoning...

Overview

Test-only change to test/bundler/bundler_edgecase.test.ts. Three parts: (1) the top-level describe becomes describe.concurrent; (2) the three deep-import-graph regression tests are rewritten to spawn bun build via a shared buildDeepGraph helper, share one chain fixture, and scale their module counts by isDebug || isASAN (7000/20000 on debug+ASAN, 500/1000 on release); (3) seven previously assertion-free itBundled cases now pin exact bundle output with toMatchInlineSnapshot or explicit checks, and YieldKeyword gains an export so the generator survives tree-shaking.

Security risks

None. Pure test changes; no production code, no auth/crypto/permissions surface.

Level of scrutiny

Moderate. It's test-only and CI-green on all lanes (builds 101234, 101410), but two aspects warrant a maintainer look rather than auto-approval:

  • Coverage reduction on release lanes. The deep-graph tests were added as regression guards for #34554 (stack overflow on 7000-module chains) and #35310 (O(V*E) DAG walk). This PR runs them at 500/1000 modules on every non-debug, non-ASAN lane. The author's argument — that a release build from before both fixes passes the full sizes anyway, so release lanes can't fail them — is well-documented and empirically checked against bun 1.3.14, but it's the kind of "weaken a regression guard for speed" decision REVIEW.md flags for explicit sign-off.
  • Known gap left open. My earlier finding (adjacent api-backend it.todo cases race on process.chdir under --todo) is confirmed by the author but intentionally not fixed here to avoid conflicting with #38655; only a comment documents it. That's a reasonable scoping call, but a human should confirm they're OK merging with the gap noted rather than closed.

Other factors

The concurrency change itself looks safe: expectBundled.ts:1911 registers api-backend cases as it.serial, which acts as a barrier inside describe.concurrent, so only the five backend: "cli" cases and the three explicit test.concurrent cases overlap, each in its own per-id root. The shared chain fixture is read-only from both consumers with distinct outfile names, and afterAll disposes it. buildDeepGraph follows the harness conventions (await using, concurrent pipe drain, bunEnv, checks signalCode before exitCode, includes stderr in the thrown message). The added inline snapshots strengthen previously vacuous tests without weakening any existing assertion. Nothing in this run surfaced beyond the already-resolved --todo thread.

@robobun

robobun commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

The review asks for a maintainer decision on the sizes, so here is the choice in one place.

What a3631ad changes: the three deep cases use 7000 and 20000 modules when isDebug || isASAN is true (local bun bd runs and the asan lane, which runs on PR builds) and 500 and 1000 modules on release builds (every other lane, and every lane on main builds).

What the release lanes lose: nothing that was measured. bun 1.3.14, which predates both fixes, passes all three cases at the full sizes, and #35310 reports the regressed walk at about 4.5s against the 60s cap on a release build. The small sizes still build and run the same shapes there.

What they gain: the per-lane table in the description. The largest items are about 9s on windows aarch64, where these three cases are the whole file today, and the slow darwin hosts, where the file took 24s to 42s.

If you would rather keep the full sizes everywhere, the two size lines in a3631ad (deepChainDepth and N) are the only thing to change back. The first commit alone still passed CI on every lane (build 101234).

The --todo note stays a comment here on purpose. The line to change is in expectBundled.ts, inside a hunk of #38655.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-08-19.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant