Skip to content

test/aot: fail a module-identity case whose bun test run is not whole - #48718

Open
robobun wants to merge 1 commit into
mainfrom
robobun/45750027/module-identity-whole-run
Open

robobun wants to merge 1 commit into
mainfrom
robobun/45750027/module-identity-whole-run

Conversation

@robobun

@robobun robobun commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/aot/module-identity-15.test.ts takes 189 s on debian 13 x64-asan (build 140086). Its tests are in test/aot/harness/module-identity.ts. This PR does not make it faster (Notes).
  • A case passes on too little. Its bun test run, the specification, only has to print one SAID line and not crash.
  • test/aot/module-identity.known.txt can name a case twice. The later line counts in silence.

Fix

  • The bun test run must be whole. Its summary counts every test file, each counted test has its line, and the exit code matches the failures.
  • "every listed case exists" also fails on a case that is listed twice.
  • A failed list comparison has the differing columns in its message.
  • Verified: bun bd test test/aot/module-identity-15.test.ts takes 733 s and 782 s before, 967 s and 1,025 s after. At the same time on 27 cases: 369 s and 371 s (Notes). The release canary passes all 20 files, and three planted faults fail only here.

Background

  • A case is a project whose test file mocks a module. It runs under bun test and bun test --aot. The list names the known differences.
  • The time is the compile in bun test --aot: 0.7 to 0.9 s of CPU in a release build. bun test needs 15 ms.
  • The rule reads the summary line of bun test. A change of its words fails every case.
  • Considered several cases in one compile: the module's header forbids it.

Behaviour change: none

Notes

The files. The shared module drives these 20 wrapper files. None of them changes.

  • test/aot/module-identity-1.test.ts
  • test/aot/module-identity-2.test.ts
  • test/aot/module-identity-3.test.ts
  • test/aot/module-identity-4.test.ts
  • test/aot/module-identity-5.test.ts
  • test/aot/module-identity-6.test.ts
  • test/aot/module-identity-7.test.ts
  • test/aot/module-identity-8.test.ts
  • test/aot/module-identity-9.test.ts
  • test/aot/module-identity-10.test.ts
  • test/aot/module-identity-11.test.ts
  • test/aot/module-identity-12.test.ts
  • test/aot/module-identity-13.test.ts
  • test/aot/module-identity-14.test.ts
  • test/aot/module-identity-15.test.ts
  • test/aot/module-identity-16.test.ts
  • test/aot/module-identity-held-1.test.ts
  • test/aot/module-identity-held-2.test.ts
  • test/aot/module-identity-held-3.test.ts
  • test/aot/module-identity-held-4.test.ts

Time of test/aot/module-identity-15.test.ts, before and after. The diff adds a pattern match over lines that are already read. So the two sides do the same work.

Bun Command Before After
Debug build with ASAN of 7f964e6, first pair bun bd test test/aot/module-identity-15.test.ts 732.9 s, 3,350 s of CPU, 109 pass, 1 fail 966.6 s, 3,810 s of CPU, 110 pass
The same, second pair the same 782.1 s, 3,597 s of CPU, 110 pass 1,024.8 s, 3,964 s of CPU, 108 pass, 2 fail
Release canary of 7f964e6 bun test test/aot/module-identity-15.test.ts 11.7 s, 14.0 s and 14.3 s alone, 92 s to 100 s of CPU. 77.0 s and 83.7 s when both sides run at the same time. 110 pass 11.7 s alone, 92 s of CPU. 76.8 s and 84.2 s when both sides run at the same time. 110 pass
  • The after runs are slower, and the diff is not the cause. The machine is a VM with 12 processors. During a debug run it gets about a third of the processor time that it asks for (steal in /proc/stat). That share swings by the minute.
  • The same child commands show it. The 13 looped cases took a median of 16.6 s in the second run before and 71.2 s in the second run after. The 85 straight cases took 27.8 s and 40.6 s before, 40.6 s and 37.1 s after.
  • Both sides at the same time share the machine alike. For that the module of main lay beside the changed one under another name, with a wrapper file of its own. On 27 cases (-t "straight \| (named|namespace) ") the module of main takes 369.3 s and 903 s of CPU. The changed one takes 370.9 s and 884 s of CPU. Each side passes its 27 cases.
  • An ASAN build runs 5 concurrent tests by default (src/options_types/context.rs:648). CI passes --max-concurrency=8 to this file on its 8 processors.
  • A debug run can lose a case to AOT: debug/ASAN compiler crash in LinkFlow::findWhatSomeLaneWasGiven #48169. The compiler then ends with an AddressSanitizer report in the stage "scan and Link". That happened to one case of the first run before and to two cases of the second run after, each time another case. CI has no debug build.

Where the time is. Release canary of 7f964e6, one case at a time.

Run of one case CPU
bun test 12 ms to 42 ms
bun test --aot 0.7 s to 0.9 s
Program that bun test --aot compiles Instructions of the stages Load, Lower and the engine (BUN_JSC_logAOTPhaseTimes=1) Files of Bun's library that it loads
A test file with an empty test 0.64 billion 34
A test file that only calls vi.doMock() 1.84 billion 243
60 cases, every 33rd of the 1,967 1.86 billion to 2.21 billion 243
  • So 83% to 99% of such a compile is work that a test file with one mock already costs. The files of the case add the rest.
  • All 1,967 bun test runs together take 44 s of CPU. The 20 wrapper files together take about 2,100 s of CPU.
  • In a debug run the process of the test file takes about 2% of the CPU (62 s against 2,950 s for its children, at 89 of the 110 cases).
  • On 8 processors a case costs 0.71 s to 0.95 s of CPU. That holds with 1, 2, 4, 8, 16 and 24 cases under way (48 cases). So another number of cases at once does not change the work.
  • In build 140086 the 20 files take 2,259 s on debian 13 x64-asan. They take 514 s on debian 13 x64. The ASAN machine has 8 processors and runs 3 such files at a time, each with 8 cases under way. So a file is bound by processor time.
  • test/aot/module-identity-15.test.ts is the slowest of the 16 in builds 140086 and 140295. That holds on debian 13 x64-asan and on debian 13 aarch64. Its cases are not heavier. In a run of every case its part costs 129 s of CPU. The 16 parts cost 124 s to 137 s. The runner puts the same files beside it in each build. One of them is test/aot/node-http-calls-what-another-file-laid-over-a-method-of-a-request.test.ts (178 s and 164 s on the ASAN lane).

What does not make it faster.

  • Several cases in one compile or one process. The header of the module says why each run has a folder and a process of its own. The compiler also decides for the whole program which files may load again.
  • One compile for the four runs of a held project. They differ in the environment alone, but bun test --aot has no way to run a program that is already compiled.
  • Duplicates. 12 of the 1,967 projects have the files, the arguments and the environment of another case. The two cases of each pair are in different wrapper files.
  • Laying out only the projects that a file runs. Each wrapper file lays out all 1,967 projects to run its part. That takes 0.14 s in a release build and 7 s in a debug build.
  • The environment of the children. On an ASAN build test/harness.ts gives the children of every file under test/aot Malloc=1 and its ASAN options. The runner sets the exception checks (BUN_JSC_validateExceptionChecks) for the file, and the children inherit them. Other files use both, so this PR leaves them. I have no release build with ASAN to measure what they cost.

The new rule holds today. I ran all 1,967 projects under bun test before I wrote it. Every run prints the summary line. It counts as many files as its command names, and as many tests as it has lines for. 1,962 runs exit with 0 and "0 fail". 5 exit with 1 and "1 fail": a require() of an async module throws inside their test. That was the canary of 7f964e6. The canary of ccb3611 gives the same counts.

Every case with the change. All 20 wrapper files pass with the changed module on the release canary of 7f964e6: 1,968 tests. The branch is now on main at 9ea3c69. The canary of that commit passes all 20 files again: 1,968 tests.

CI of this PR, first push. Build 140670, on main at ccb3611. All 20 wrapper files pass at the first try on the three lanes that run them. Build 140651 is the build of main at the same commit. A build of main does not run the ASAN lane.

Lane module-identity-15, main module-identity-15, this PR The 20 files, main The 20 files, this PR
debian 13 x64 44.5 s 49.9 s 562.1 s 571.7 s
debian 13 aarch64 32.3 s 35.1 s 392.3 s 393.3 s
debian 13 x64-asan not run 170.3 s not run 2,495.8 s
  • The base moved since build 140086, which the Problem quotes. On debian 13 x64 the 20 files took 513.9 s there, and 562.1 s in the build of main at ccb3611.
  • Two jobs of build 140670 are red. Each is a stored program of test/aot/parallel that this PR does not touch. one-data-file-under-two-types/untyped-import-answers-before-text/main.mjs fails on ubuntu 25.04 aarch64. lists/elements-beside-numeric-arrays.js fails on ubuntu 25.04 x64, and also in build 140651 of main. Two builds of main behind AOT: update the engine; stores in defaults of parameters, paths that stores rule out #48714 pass both programs (140786, 140802). So the branch is rebased onto main.

CI of this PR, second push. Build 140933, on main at 9ea3c69. All 20 wrapper files pass again on the three lanes.

Lane module-identity-15 The 20 files
debian 13 x64 26.2 s 546.1 s
debian 13 aarch64 17.8 s 373.1 s
debian 13 x64-asan 227.0 s 2,717.4 s
  • On the ASAN lane module-identity-15 took 189.3 s, 170.3 s and 227.0 s in builds 140086, 140670 and 140933. The files beside it decide that, as above.
  • One job of build 140933 is red, and it runs no code of this PR. In one shard of debian 13 x64-asan the runner script stopped with TypeError: Invalid URL in getFileUrl (scripts/buildkite.ts:367). It was writing the report of vendor/elysia/test/response/stream.test.ts, which failed its first try and passed its second.

Planted faults. Release canary, each run filtered to the named case.

Fault main This PR
process.exit(0) after the SAID line of reduced | unmocked below a barrel passes fails: the run has no summary line
process.exit(0) in the first of the two test files of reduced | loaded earlier through a barrel passes, and the second test file runs in neither run fails: the run has no summary line
A second, stale line for straight | namespace dependency mock.module reset in the list every test passes "every listed case exists, and is listed once" fails and names the case
  • With BUN_AOT_REWRITE_MODULE_IDENTITY=1 a run of two cases leaves the list byte for byte as it is.
  • test/aot/module-identity.known.txt does not change.

no test proof · iteration 5 · no src or test change; test-proof not applicable

@robobun

robobun commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 11:05 PM PT - Nov 12th, 2026

❌ @robobun, your commit 9a291fe has 1 failures in Build #140933 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 48718

That installs a local version of the PR into your bun-48718 executable, so you can run:

bun-48718 --bun

@robobun

robobun commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. This PR is #48718.

CI:

  • The change is green in both builds. All 20 module-identity wrapper files pass on the three lanes that run them: debian 13 x64, debian 13 aarch64 and debian 13 x64-asan.
  • Second push, build 140933, on main at 9ea3c69: one job is red, and it runs no code of this PR. In one shard of debian 13 x64-asan the runner script stopped with TypeError: Invalid URL in getFileUrl (scripts/buildkite.ts:367). It was writing the report of vendor/elysia/test/response/stream.test.ts, which failed its first try and passed its second. test/aot/module-identity-9.test.ts passed in that shard before the stop.
  • The four darwin any x64 jobs of that build expired in their queue, because no agent took them. Every other job passed.
  • First push, build 140670, on main at ccb3611: two jobs were red on stored programs of test/aot/parallel that this PR does not touch. main passes both programs since AOT: update the engine; stores in defaults of parameters, paths that stores rule out #48714, so the branch was rebased.

How the numbers in the description were measured:

  • Before and after: bun bd test test/aot/module-identity-15.test.ts on a debug build with ASAN of 7f964e6, two runs on each side in turns. Then both sides at the same time on 27 cases, because the speed of the machine swings.
  • Every case: the release canary runs the 20 wrapper files with the changed module. It did so on 7f964e6, and again on 9ea3c69 after the rebase: 1,968 tests pass each time.
  • Where the time is: the CPU time of each child process, and BUN_JSC_logAOTPhaseTimes=1 on the bun test --aot run of a case.
  • The planted faults: the release canary, with the run filtered to the named case, with the module of main and with the changed module.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The module-identity harness validates test and file totals and exit status for each plain test run. It also rejects duplicate known-list entries and includes mismatch details in the expected comparison.

Changes

Module identity harness

Layer / File(s) Summary
Test output validation
test/aot/harness/module-identity.ts
The harness extracts named test files and reported test and file totals. It checks that the totals match the run and that the exit status matches the failure count.
Known-list checks and diagnostics
test/aot/harness/module-identity.ts
The known-list check rejects duplicate entries as well as unknown cases. The expected comparison receives classification and differing-column details.

Suggested reviewers: jarred-sumner, alii

Merge Risk: Minimal · up to 9a291

This change tightens validation in the module-identity AOT tests and does not alter product behavior. No actionable merge-blocking risk was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: the module-identity case now fails when its bun test run is not whole. The raw summary confirms the plain-run check now validates totals, rejects crashes, requires S…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template: it has a problem statement, a fix section, and verification evidence for the timing and planted-fault checks. It also documents the behavior change (no…
Full details: Title check

Explanation

The title names the main change: the module-identity case now fails when its bun test run is not whole. The raw summary confirms the plain-run check now validates totals, rejects crashes, requires SAID output, and checks exit codes. The title is specific and concise, and it is consistent with the PR objectives.

Full details: Description check

Explanation

The description follows the repository template: it has a problem statement, a fix section, and verification evidence for the timing and planted-fault checks. It also documents the behavior change (none) and the CI runs. The template's "What does this PR do?" and "How did you verify your code works?" content is present in substance, though the headings do not match the template exactly. The description is long and contains a large amount of measurement detail, but the required information is complete.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 395e175 and 9a291fe.

Files selected for processing: 1

  • test/aot/harness/module-identity.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — a small, self-contained tightening of the module-identity test harness with no production code touched.

What was reviewed:

  • The new ran / exitCode expectations against the runner source: print_summary always prints "Ran N test(s) across M file(s). ", files is counted once per file (not per --rerun-each repeat) while each repeat's (pass) line and summary count increment, the " N fail" line is always printed, and exit code 1 is set iff failures, unhandled errors between tests, or load failures occur — consistent with what testFilesOf and plain.answer.tests.length compare against for straight, looped, shared, reduced and held cases (including --preload ./setup.mjs and --rerun-each=3 args, which do not match .test.<ext>).
  • expect(actual, message): Bun's custom label replaces only the expect(...).toBe(...) signature and keeps the string diff, so the replaced console.error loses nothing.
  • Not run here: bun bd test test/aot/module-identity-*.test.ts and the planted-fault checks the description reports; this reading relies on the runner source and the author's reported runs.
Extended reasoning...

The diff changes only test/aot/harness/module-identity.ts (+33/-20): the plain bun test run of each case must now report a summary line whose test and file counts match the printed test lines and the test files named on the command, with an exit code that matches the fail count; the known-list check also flags keys listed twice; and a console.error dump becomes the custom message of the final expect. It touches no production code and no security-sensitive surface, and the file is not covered by CODEOWNERS. Approval rests on cross-checking the hard-coded summary and exit-code expectations against src/runtime/cli/test_command.rs and on every case definition in module-identity-cases.ts and module-identity-held-cases.ts naming its test files explicitly as .test.mjs/.test.ts args; the four candidate issues from the ruled-out list (exit 1 with "0 fail", rerun load failures, non-.test. file names, summary wording changes) are all intended tightenings or hypothetical future drift rather than defects in this change.

The run of a case under `bun test` is its specification. It passed with one
SAID line and no crash. So a project whose two runs both ended before the last
test file passed.

- The `bun test` run must count every test file that its command names, have a
  line for each test that it counts, and exit with the code of the failures
  that it counts.
- "every listed case exists" also fails on a case that
  module-identity.known.txt names twice. The later line counted in silence.
- A case whose line differs from the listed one fails with the differing
  columns in its message. They went to stderr before.
@robobun
robobun force-pushed the robobun/45750027/module-identity-whole-run branch from 395e175 to 9a291fe Compare November 13, 2026 01:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant