Skip to content

test(aot): run the layered runs of callbacks-named-by-another-program-of-a-test-run.test.ts concurrently and state every line - #47663

Closed
robobun wants to merge 2 commits into
mainfrom
robobun/ed2258f6/callbacks-named-layered-runs-concurrent
Closed

robobun wants to merge 2 commits into
mainfrom
robobun/ed2258f6/callbacks-named-layered-runs-concurrent

Conversation

@robobun

@robobun robobun commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts takes 467 s of the 496 s of its parallel batch on debian 13 x64-asan.
  • Its six bun test --aot runs in layers go one after another. Each compiles Bun's own JavaScript library again.
  • Two of a run's 18 lines are checked, by substring. Nothing checks that both test files ran.

Fix

  • Each run in layers is its own test, and the six are concurrent. bun test --max-concurrency says how many build at once.
  • Every run must print the 18 stated lines and the summary of two passing test files, and exit with 0.
  • A debug build skips the file and says why. Plain Bun lists flatIntoArrayWithCallback there, so main's file fails at line 91.
  • Verified: CI below, local runs in Notes. bun bd test: 3 fail in 5.5 s before, 6 skip in 2.4 s after.
CI: x64-asan, x64, aarch64 Before After
the file, in its parallel batch 467, 153, 102 s 372, 99, 65 s
the file, alone 158, 67, 37 s
the shard that holds it 1487, 668, 475 s 1478, 647, 486 s

Background

  • In layers, the package and Bun's own code are one compiled program, the application a second, each test file one more.
  • The six builds cost the same processor time as before, and a batch keeps every processor busy: the file ends sooner, its shard does not.
  • Considered one project for all cases, and a Bun with a compiled library. Both save builds and change which program knows of the naming.
Notes

What the file asks, and what stays as it was. The project, its three variants (the package, the application or the test file gives the names), the commands and the words TestRunLayers:1,TestRunPartFiles:1 and TestRunLayers:2,TestRunPartFiles:1 are unchanged. Each variant still runs under plain Bun and in both forms of layers: nine commands before, twelve now, because each of the six tests starts its own plain run (0.05 s with the release build, 0.2 to 0.4 s with release-asan). run() of harness/run.ts still fails a run that was not built in the layers it asked for. A test has 300 s. It holds one run in layers, where a test of main holds two and has 240 s, but up to six run at once: a test then took 158 s in CI and at most 275 s locally. With two at once in a batch of CI a test took about 124 s (372 s for three rounds of two).

Assertions, before and after.

Before After
Lines of a run 18 counted. Lines 5 and 6 checked by substring. The runs in layers compared with the plain run All 18 stated in the file and compared exactly, for the plain run and for each run in layers
Test files of the project not checked 2 pass, 0 fail, 2 expect() calls, Ran 2 tests across 2 files. for every run
Exit code first, alone, for the plain run last, in the same comparison as the output, for every run

Two faults show the difference. Each was put into the project text of both versions of the file and run with the release build.

Fault Main's file This PR
a.test.js prints the line "one that the package calls" before it names each, so every run prints each there 3 pass 6 fail in 0.24 s. The diff shows that line three times
b.test.js skips its test 3 pass 6 fail in 0.18 s: 1 pass, 1 skip, 1 expect() calls

Where the time goes. One run in two layers with the release build and BUN_JSC_logAOTPhaseTimes=1: the layer that holds the package and Bun's own code takes 18.4 s of 19.4 s (250 units, 11,028 functions). The layer of the application, both test files and both workers take under 1.3 s together. So the file is six builds of Bun's library, and the processor time stays the same: 152 and 158 s before, 152 and 154 s after (release, 4 CPUs). Only the order changes. The layer holds that much of the library because the project makes two functions from text at run time (see the last section).

All local runs. Linux x64, 12 vCPUs, 15.6 GB, taskset for fewer CPUs. Each row is one round, taken in turn. The host takes a changing part of the machine: the rounds differ more than the columns of a round.

release-asan (bun run build:asan), 8 CPUs, the environment that scripts/runner.node.ts gives a file on the lane with AddressSanitizer:

Round Main, limit raised to 900 s This PR, --max-concurrency=8 This PR, --max-concurrency=2
1 1048 s 276 s 405 s
2 385 s 154 s 212 s
3 653 s 148 s 265 s
4 442 s 169 s 337 s
  • Main's file as it is, in a fifth round: 722 s. Two tests timed out at 240 s, and the third passed after 238 s. This PR right after it: 181 s. On this machine a test of main is at its limit, so the column above has the limit raised and nothing else changed. A test of main holds two runs in layers and took 95 to 363 s. A test of this PR holds one, and took 59 to 140 s with two at once and 141 to 275 s with six at once.
  • With the limit of 300 s, the second commit: 161 s with six at once, a test took 148 to 158 s.
  • Through scripts/runner.node.ts itself (--exec-path build/release-asan/bun-asan, 8 CPUs), this PR took 190 and 131 s. Main's file failed there after 722 s, and with the raised limit the runner ended it at the 1080 s that it gives a file.
  • --max-concurrency=8 is what the runner passes when the file runs in a process of its own. --max-concurrency=2 is what a parallel batch gets there.
  • Most memory in use at once, beyond what was in use before the run: 1.8 GB for main, 8.8 to 9.5 GB for this PR with six runs at once, 3.3 GB with two. The lane has 64 GB (r7i.2xlarge).

release (bun run build:release, bun-profile), 4 CPUs, the environment of a lane without a sanitizer:

Round Main This PR, --max-concurrency=4 This PR, --max-concurrency=2
1 74.5 s 45.5 s 47.2 s
2 68.1 s 42.8 s 49.6 s
  • Most memory at once: 0.3 GB for main, 1.0 GB for this PR with four at once, 0.5 GB with two.
  • With 12 CPUs and no flag: main 83 and 100 s, this PR 29, 38 and 39 s.

debug (bun bd test):

  • Main: 3 fail in 5.5 s (15.6 s with the check of the build). The first comparison with a text fails: Received: "SEEN one of the test file | probed < visited < step < for each < in a package < flatIntoArrayWithCallback < forEachOf < work < <anonymous>". A debug build sets showPrivateScriptsInStackTraces (src/jsc/bindings/ZigGlobalObject.cpp, under BUN_DEBUG), so plain Bun lists the private helper that flatMap calls. Compiled code does not list it. No build with AddressSanitizer alone does this, so the lane with it is not affected.
  • This PR: 6 skip in 2.4 s (3.6 s with the check of the build), and one SKIPPED: line on stderr with the reason.
  • One run in layers alone took 280 and 416 s with the debug build. A variant of this PR without the skip, with the frame removed from the lines and a limit of 900 s, ran for 26 minutes: five tests at once timed out, and the sixth took 666 s alone. A debug build cannot run this file in a useful time, and it would need the lines of another build. The lane with AddressSanitizer has assertions too.

CI, before. Build 136323: 467 s on debian 13 x64-asan, 153 s on x64 and 102 s on aarch64, the only lanes that run the file. It ran inside a batch of 35 files: bun test --parallel=7 --max-concurrency=2 on x64-asan (the batch took 496 s), --parallel=3 --max-concurrency=2 on x64 (153 of 265 s). The file is in such a batch because test/parallel-allowlist.json lists the folder and was made before the file existed. This PR leaves that list alone.

CI, this PR. The same shard of each lane held the file in all three builds.

Main, build 136323 This PR in its batch, build 136558 This PR alone, build 136542
The file, debian 13 x64-asan 467 s 372 s 158 s
The file, debian 13 x64 153 s 99 s 67 s
The file, debian 13 aarch64 102 s 65 s 37 s
The batch, x64-asan 496 s 481 s
The batch, x64 265 s 255 s
The shard, x64-asan 1487 s 1478 s canceled by the second push
The shard, x64 668 s 647 s 663 s
The shard, aarch64 475 s 486 s 457 s
  • Build 136558 (the second commit) ran the file inside its batch, as main does: --max-concurrency=2, beside six other files on x64-asan and two on the others. That column is like for like with main.
  • Build 136542 (the first commit, a limit of 600 s and nothing else different) ran the file in a process of its own, in front, as the runner does with a file that a PR changes: --max-concurrency=8 beside two other files on x64-asan, --max-concurrency=4 beside one on the others.
  • The shard takes as long as before in either way. The file is six builds of Bun's library, the builds cost the same processor time as before, and the batches of a shard keep every processor busy. What changes is when the file ends and how long one test takes: about 156 s of a limit of 240 s on main (x64-asan, two runs in layers to a test), about 124 s of 300 s now.
  • The other long files of the x64-asan batch moved both ways between the two builds: 320 to 345 s, 223 to 266 s, 223 to 189 s.
  • Alone on x64-asan, the two files that ran beside it took 159 and 172 s. In build 136323 they took 58 and 65 s. Six runs in layers at once keep most of the eight processors of that lane busy, and what runs beside them is slower for that time.

Designs that were not taken.

  • One project that holds the three variants, run once in each form: two builds of the library and not six. The lowest layer would then hold the package that gives the names, for every variant. The variant in which only the test file gives the names asks what happens when no program below knows of it.
  • A copy of Bun that holds its library compiled (scripts/build/embed-compiled-library.ts, as the-library-that-bun-holds-compiled-is-the-lowest-layer.test.ts makes one): one build of the library. The package is then a program of its own above Bun's own code. In the Bun under test the two are one program.
  • Two runs in layers at once inside each of three concurrent tests: four builds at once where the runner asked for two tests.
  • The project without its two new Function("f", "return f"). Because of them the lowest layer holds about twice as much of Bun's library: 249 units and 11,025 functions, against 128 and 4,007 when both are written f => f, and one run takes 10.6 s and not 6.5 s (release, 4 CPUs). They hand each callback through code that the compiler cannot read, and the tests ask about callbacks that it has lost sight of. No commit on main has both layers and the fault that these tests were written for, so a cheaper project cannot be shown to find it.

[auto-merge] gate passed · iteration 0 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts
bun test v1.4.4 (367d939d9)

test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts:
SKIPPED: test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts: a debug build of Bun lists a helper of flatMap in the stacks of its plain run, and takes five minutes or more for one run in layers
(skip) a stack taken before the package names the callbacks of the other programs shows the names, over one layer
(skip) a stack taken before the package names the callbacks of the other programs shows the names, over two layers
(skip) a stack taken before the application names the callbacks of the other programs shows the names, over one layer
(skip) a stack taken before the application names the callbacks of the other programs shows the names, over two layers
(skip) a stack taken before the test file names the callbacks of the other programs shows the names, over one layer
(skip) a stack taken before the test file names the callbacks of the other programs shows the names, over two layers

 0 pass
 6 skip
 0 fail
Ran 6 tests across 1 file. [8.96s]
Exit: 0
diff hotspot
...-named-by-another-program-of-a-test-run.test.ts | 68 ++++++++++++++++------
 1 file changed, 51 insertions(+), 17 deletions(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                                      reads  edits  tests
…allbacks-named-by-another-program-of-a-test-run.test.ts      1      0     30

… concurrently and state every line

Each `bun test --aot` run in layers is a test of its own, and the six
tests are concurrent, so `bun test --max-concurrency` says how many of
them are built at once. A run compiles all of Bun's own JavaScript, and
the six runs went one after the other.

Every run, plain Bun's too, must print the 18 lines that the file
states and the summary of two passing test files, and exit with 0.
Before, two of the 18 lines were checked, by substring.

A debug build skips the file and says why. Plain Bun lists a private
helper of flatMap in a stack there, so the file failed at its first
comparison, and one run in layers takes five minutes or more.
@github-actions github-actions Bot added the claude label Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 3f3a9ff and 90e55a7.

Files selected for processing: 1
  • test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 0 remain after this review.

Walkthrough

The test now checks exact stack output, test-run summaries, and exit codes for plain and AOT runs at two test-run layer settings. Debug builds skip these tests.

Changes

AOT callback test coverage

Layer / File(s) Summary
Run output assertions
test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts
The test parses the final summary from stderr and checks expected stack lines, successful two-test summaries, and exit code 0 for each run configuration. Debug builds skip the tests. The timeout increases from 240 to 300 seconds.

Suggested reviewers: jarred-sumner

Merge Risk: Minimal · up to 90e55

The change strengthens AOT output checks while isolating concurrent test fixtures. No actionable merge-blocking issue was identified; normal checks should pass before merging.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly identifies the main changes: concurrent layered AOT test runs and exact output-line assertions. It is somewhat long but remains specific and relevant.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description clearly explains what the PR changes and how the change was verified. It includes implementation details, performance measurements, debug-build behavior, CI results, and test evidence.…
Full details: Description check

Explanation

The description clearly explains what the PR changes and how the change was verified. It includes implementation details, performance measurements, debug-build behavior, CI results, and test evidence. The required template headings are not used verbatim, but the required information is present and complete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@robobun

robobun commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: closed without a merge. #47942 rewrites the front end of the ahead-of-time compiler and its tests, and it deletes test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts, so this change no longer applies.

What this pull request measured, for the record:

  • CI, the file inside its parallel batch (build 136558, as on main): 372 s on debian 13 x64-asan, 99 s on x64, 65 s on aarch64. On main: 467, 153 and 102 s (build 136323).
  • CI, the file in a process of its own (build 136542): 158, 67 and 37 s.
  • The shard that holds the file took as long as before (1487 and 1478 s on x64-asan). The six builds cost the same processor time, so the file ended sooner and the shard did not.
  • On a debug build main's file fails at its first comparison with a text: plain Bun lists flatIntoArrayWithCallback in the stack there.

The description has every run and how it was taken.

@robobun

robobun commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 2:05 PM PT - Nov 1st, 2026

❌ @robobun, your commit 90e55a7 has 40 failures in Build #136558 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 47663

That installs a local version of the PR into your bun-47663 executable, so you can run:

bun-47663 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline note on the per-test timeout, I also checked the hardcoded summary lines against the runner's summary printer in src/cli/test_command.rs (N pass, N fail, N expect() calls, Ran N tests across N files. plus a [N.NNms]/[N.NNs] suffix from src/bun_core/output.rs) — the trim-and-strip regex in seenIn matches that output exactly, and the AOT test run | ... lines that could land after the summary are gated on BUN_JSC_logAOTPhaseTimes, so they only matter when a developer has that set. The test.skipIf(isDebug).concurrent.each chain is also handled by the alwaysSkipped proxy in test/aot/supported.ts when --aot is unsupported.

Extended reasoning...

The change rewrites one test file, test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts, splitting six layered bun test --aot runs into concurrent per-layer tests and replacing substring checks with the full expected 18-line stdout plus a stderr summary and exit code; it touches no production code and no security-sensitive surface. The restructure is correct as far as I can trace it, but the choice to run six Bun-library builds concurrently (the author reports up to ~9.5 GB peak) and the exact-output coupling to stack-frame rendering are design trade-offs a test/aot owner should weigh, so a human look is still worthwhile rather than an automated approve.

…ot 600 s

A test holds one run in layers. With six of them at once a test took
158 s on the lane with AddressSanitizer (build 136542) and at most
275 s in the local runs. On main a test holds two runs in layers and
has 240 s.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the second commit brings the per-test ceiling down to 300 s, close to the measured worst case for one layered run, and the rest is a test-only tightening that replaces two substring checks with the full 18-line output, the summary and the exit code.

What was reviewed:

  • Summary parsing against print_summary/print_elapsed in src: durations are only ms or s, bunEnv sets NO_COLOR, and the AOT test run | largest worker line prints only under BUN_JSC_logAOTPhaseTimes, so nothing lands after Ran 2 tests across 2 files. in the slice.
  • The isDebug skip: showPrivateScriptsInStackTraces is set under BUN_DEBUG in ZigGlobalObject.cpp, so the flatMap helper frame is debug-only and the SKIPPED: stderr line follows the test/aot/README.md convention.
  • test.skipIf(...).concurrent.each(...) is a chain used across the suite, and supported.ts's alwaysSkipped proxy handles it when --aot is unsupported.
Extended reasoning...

The PR touches one test file, test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts, splitting three sequential tests into six concurrent ones and asserting every output line, the bun test summary and the exit code with a single toEqual. It touches no security-sensitive surface. The only prior review comment was this bot's nit on the 600 s timeout, which the second commit lowered to 300 s; no CODEOWNERS entry covers test/aot, the bug hunt ran dry, and the stderr parsing and skip condition were checked against the runtime source.

@Jarred-Sumner

Copy link
Copy Markdown
Collaborator

Closing: #47942 rewrites the front end of the ahead-of-time compiler and its tests for the engine's new compiler (oven-sh/WebKit#997, merged), so this no longer applies and would conflict with it. If the problem still shows once that is in, please open a new pull request against main.

@robobun

robobun commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

Understood. #47942 deletes test/aot/callbacks-named-by-another-program-of-a-test-run.test.ts, and no file in its tree has the same project, so nothing is left to speed up once it lands. I do not plan a new pull request for this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants