Skip to content

test(lint): share runs in js-parsers.test.ts and check how every run ends - #45880

Closed
robobun wants to merge 1 commit into
mainfrom
robobun/1cdbba4e/faster-js-parsers-test
Closed

robobun wants to merge 1 commit into
mainfrom
robobun/1cdbba4e/faster-js-parsers-test

Conversation

@robobun

@robobun robobun commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/cli/lint/js-parsers.test.ts takes 68 s to 77 s on debian 13 x64-asan, 1 s to 6 s elsewhere. 56 of its 242 bun lint runs start a JavaScript engine, and with LeakSanitizer each ends about 5 s later.
  • lint() normalized every output: 41 s of the runner's thread in a debug build, 29 s for outputs no test reads.
  • About 100 runs had no exit code checked.

Fix

  • lint() normalizes an output when a test first reads it.
  • Rows with a file each share a run: 40 line-break tests take 10 runs, 13 directive tests 1, 15 --fix tests 1. Six tests start both runs together. 186 runs.
  • resultsOf() checks that a -f json run ends as its results say, and each test states its exit code. Still 196 tests. expect() calls: 490, now 629.
  • Verified: bun bd test test/cli/lint/js-parsers.test.ts --timeout 270000, debug build, in turns: 264, 259, 308 s, then 202, 230, 218 s. ASAN lane, four runs: 63.6, 73.3, 63.5, 62.6 s.

Background

  • An ASAN build runs 5 tests of a file at a time (src/options_types/context.rs:534). A waiting test holds one place.
  • A run with an engine leaves one leak that test/leaksan.supp suppresses by name, so LeakSanitizer starts llvm-symbolizer (test(lint): make rule-throws.test.ts faster and strengthen its assertions #45755): 4 s of CPU for each of the 56 runs. That bounds the lane, and only a src/ change removes it.
  • Considered one test per table. Rejected: rows stay tests, and more runs at once defeat the limit of 5.

Behaviour change: none

Notes

The lane, before and after. The time of the file on debian 13 x64-asan - test-bun:

Build File Time
127750, 127767, 127781, 127785, 128925, 128929 main, 180 tests 72.3, 77.4, 75.6, 71.1, 68.5, 68.6 s
128927 main, 196 tests (after #45800) 70.8 s
128086 the first commit of this PR, 180 tests 63.6 s
129068 the second commit of this PR, 196 tests 73.3 s
129111 the third commit of this PR, 196 tests 63.5 s
130145 this PR after the rebase, 196 tests 62.6 s
  • Main's seven runs have a median of 71.1 s. The four runs of this PR have a median of 63.6 s, and one of them is inside the spread of main.
  • In build 129068 the first 35 s were slow: the tests with an engine ended 12 s apart, not 5 s. After that they ended 5 s apart, as in every other build.
  • The pair that is like for like has 196 tests and the file at place 16 of its job: 70.8 s on main (build 128927), 63.5 s here (build 129111). At the end of the tests with an engine of the first two groups the two are 1.5 s apart, after the JSON tests 4.4 s, at the end 7.3 s.
  • The other lanes in build 129111: 1.2 s to 4.9 s. In the builds of main's file: 1.2 s to 6.3 s.
  • Why the lane cannot gain much: the 56 runs with an engine need about 250 s of CPU for llvm-symbolizer, and 5 tests at a time use 5 of the 8 cores for it. This PR starts the same 56 runs. It removes 56 runs without an engine, which take about 0.5 s each there.

Where the time goes on the lane. The log of the ASAN lane has a time for each batch of tests that end. In build 127781 the file takes 75.6 s:

Time Tests done What fits
0 to 34 s 27 processors and languages: 22 tests, 34 of their 35 runs start an engine
34 to 41 s 58 more JSON tests without an engine
41 to 53 s 10 more the JSON tests with an engine, 5 at a time
53 to 65 s 76 more the rest of JSON and Markdown, 5 of them with an engine
65 to 75 s the last 9 the 6 Markdown tests with an engine: 5, then 1 alone

The log does not name the tests, so this is what fits the log, not what the log says.

Which runs end with the symbolizer. I put a script at ASAN_SYMBOLIZER_PATH that writes down its parent and then starts llvm-symbolizer. With the variables that scripts/runner.node.ts sets on the lane, 56 runs start it, on main and here: the runs with a processor, with a rule of a plugin in JavaScript, and the runs that send a file to the eslint package. One such run on the debug build: 2.0 s to 2.2 s, and 6.3 s to 6.4 s with the variables. A run with an eslint.config.mjs and built-in rules only: 1.4 s to 1.5 s, and 2.0 s with the variables.

A model of the lane. 5 tests at a time in the order of the file, 5.5 s for a run with an engine, 0.6 s for a run without. For the file with 180 tests it gave 74.6 s for main and 62.6 s for the first commit. The model lets runs with an engine overlap at no cost. On the lane they share 8 cores, so it says too much.

The test runner's thread. normalizeBunSnapshot makes about 15 passes over a text. On the debug build it needs 14.3 s for 6.4 MB. files whose parser options request types all go to one engine prints 6.4 MB twice and reads only raw. In one run of the file on main the normalization of all outputs took 41.5 s. With the default limit of 5 s the tests that waited for the thread then failed after 15 s to 17 s. The getters keep what they normalized, so the four rows that read one SARIF output normalize it once.

The runs that are shared.

  • U+2028 and U+2029 aren't line breaks in JSON: 4 files (line feeds, byte order mark, carriage returns, both) and 10 formats were 40 runs. Now one run for each format lints the four files. Each file is ten lines longer than the one before it, so the second "a" is on line 54, 64, 74 or 84, and a place in the output belongs to one file. A test still fails alone: with the expected line of one file changed, its 10 rows fail and the other 30 pass.
  • disables every rule it lists: 13 rows, each with its own file, in one run. A row reads the result of its file.
  • --fix turns .. into emphasis: 15 rows that bun check: 1.4x to 7.8x faster on large projects. Fixes for bun check, bun lint and bun fmt #45800 added, each with its own file, in one run with --fix -f json. A row expects the fixed text of its file and no problem left in it. With the expected text of one row changed, that row fails and the other 14 pass.
  • The first test that asks starts a shared run (once). A row can still run alone with -t.
  • If a shared run fails, all rows that read it fail: 4 for a format, 13 for the directives, 15 for the fixes.

The runs that start together. handles a processor that throws, .., a file with a language and a processor .., extglobs and brace ranges .., files whose parser options request types .., handle many keys out of order, with comments and --fix removes what is unused, and nothing else each started two runs one after the other.

What keeps its own run, and why.

  • The tables options are validated, languageOptions: .. is rejected and an invalid regex in the options of ..: an error in the configuration ends a run.
  • the whole file goes to the eslint package and eslint-plugin-json is treated like any other package ..: each row has its own configuration or flag.
  • runs in linear time: a row that is slow fails alone, and names the input.

Order. runs in linear time is now the last block of the Markdown tests, after the whole file goes to the eslint package. Six tests with an engine ended the file: 5 at a time, then one alone. Now the 14 tests without an engine run beside them. The model gives 2.4 s for this.

Assertions.

  • resultsOf(run) parses the output of a -f json run. It expects stderr to be Linted N files or ✓ No problems in N files for the N results and nothing else, and the exit code to be 1 only if a message is an error. A run that ended with another exit code fails with its stderr, not with a JSON syntax error. problems(run) is the summary of these results. 73 of the 186 runs go through them.
  • That check compares a run with itself, so each test that uses it also states the exit code it expects (review). The two shared runs with rows have no line of their own: each row expects no problem in its file.
  • Absent texts became exact output: not.toContain("error") and not.toContain("not linted") (3 tests) are the exact stderr. not.toContain("the eslint package linted") (2 tests) is JavaScript: 0 engines loaded 0 modules of --timing. --doctor (2 tests) compares the whole files object. several such files among others compares the path and the number of problems of all 22 results. if a named block is the whole file, .. compares the paths of the results. with an .eslintrc, .. compares all of stdout and stderr. One not.toContain is left: the six formats of if eslint isn't installed do not name the file that was skipped.
  • The 22 runs that end with an error (15 with a refused configuration, 7 with a rule that throws) compare all of stdout, stderr and the exit code. They used toContain on stderr.
  • runs in linear time (14 rows) expected no fatal message and an exit code below 2. Each row now expects how often each rule reports. ESLint 10.13 with @eslint/markdown 8.0.3 reports the same numbers for 200 and for 2,000 repeats.
  • handle many keys out of order, with comments expected more than half of the keys. It now expects the number, which is what ESLint 10.13 with @eslint/json 2.1.1 reports for 300 and for 3,000 keys.
  • The two --stdin tests went around lint() and did not read stderr. They use lint() now, compare stderr, and have an inline snapshot of the pretty output where they looked for one line.
  • The two tests about deprecated layout rules expect the note on stderr: 2 deprecated formatting rules did not run: indent, semi.
  • The --fix tests expect the line about what was fixed, for example Fixed 25 problems in 4 files. The excerpt tests expect the line of the problem with the mark under it.

Timeouts. bun bd test <file> gives a test 5 s. A run with a configuration file in JavaScript starts two processes, and on a debug build that takes seconds. On this machine 90 of the 196 tests ran out of the 5 s (144 and 95 of 180 with main's file). On a debug build the file now sets its default limit to 120 s, which the tests of the first two groups already have. Release builds and the lanes of CI keep their limits: the condition is isDebug.

Local numbers. Debug ASAN build on Linux x64, a VM with 12 vCPUs that gets about 3 cores. Before and after in turns. The time of one run differs by up to 50 s from the next.

Before After
bun bd test <file> --timeout 270000, 3 runs, 196 tests 264, 259, 308 s 202, 230, 218 s
CPU time of all processes in these runs 545 to 595 s 438 to 473 s
the same before #45800, 7 runs, 180 tests 233, 224, 217, 164, 190, 224, 168 s 211, 152, 153, 149, 141, 152, 151 s
bun bd test <file>, 2 runs, 180 tests 249 s (144 tests fail), 187 s (95 fail) 109 s (6 fail), 104 s (24 fail)
with the variables of the ASAN lane, 4 runs, 180 tests 251, 297, 230, 357 s 238, 261, 228, 287 s
  • Without --timeout a test has 5 s. That is too short for a debug build on this machine, on main and here. Every test that fails ran out of it.
  • With the variables of the lane the symbolizer needs 4 s of CPU and 1.4 GB for each of the 56 runs. On about 3 cores that hides the rest. The lane has 8 vCPUs and 64 GB.
  • Peak with the variables of the lane, 180 tests: 12 llvm-symbolizer at once on main, 15 here. The least available memory of the 15 GB: 4.0 GB on main, 2.7 GB here.

[auto-merge] gate passed · iteration 1 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/cli/lint/js-parsers.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/cli/lint/js-parsers.test.ts
bun test v1.4.4 (367d939d9)

test/cli/lint/js-parsers.test.ts:
bun lint c.none: exit code 2
error: Key "processor": Could not find "none" in plugin "fenced".

(pass) bun lint with processors > a rule in JavaScript sees the name of the block and the name of the file [27599.61ms]
(pass) bun lint with processors > a processor that throws, blocks without a name, and a processor that does not exist [28033.05ms]
(pass) bun lint with processors > a processor that an object of the configuration has itself, without supportsAutofix: no fixes [28083.35ms]
(pass) bun lint with processors > --fix fixes the blocks in the file [28149.20ms]
(pass) bun lint with processors > the blocks are linted with the configuration for their own names, next to other files [29693.75ms]
bun lint b.txt: exit code 2
✓ No problems in 0 files [6.90s]
error: 1 file was not linted because it needs eslint, which is not installed (1 *.txt): b.txt
  Run bun add -d eslint
  Everything else was linted. Use --allow-unsupported to make this a warning.

(pass) bun lint with processors > a file with a language and a processor goes to the processor [8104.76ms]
(pass) bun lint with languages > so is a file that a parser of its own reads [8035.66ms]
(pass) bun lint with languages > the deprecated rules about layout do not run there either: ESLint 8 [8122.92ms]
(pass) bun lint with languages > the deprecated rules about layout do not run there either: ESLint 9 [8781.63ms]
(pass) bun lint with languages > a file in the language of a plugin is linted by the Linter of the eslint that is installed [11422.44ms]
(pass) bun lint with languages > the whole configuration file, which exports a promise, and a processor, which is called once [6087.25ms]
(pass) bun lint with languages > so is a file for which eslint-plugin-html is configured, which changes the Linter when it is loaded [7999.17ms]
(pass) b
... (truncated)
Exit: 0
diff hotspot
test/cli/lint/js-parsers.test.ts | 665 +++++++++++++++++++++++++--------------
 1 file changed, 432 insertions(+), 233 deletions(-)

gate history · 1 passed · 1 rejected · iteration 1

evidence per changed file
file                              reads  edits  tests
test/cli/lint/js-parsers.test.ts     15      3     73

@robobun

robobun commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: open, ready for review.

How this was measured:

  • The lane: the log of debian 13 x64-asan - test-bun. Main's test/cli/lint/js-parsers.test.ts takes 68.5 s to 77.4 s in seven builds (127750, 127767, 127781, 127785, 128925, 128927, 128929). This PR took 63.6 s, 73.3 s, 63.5 s and 62.6 s in its four builds (128086, 129068, 129111, 130145).
  • Locally: a debug ASAN build on Linux x64. bun bd test test/cli/lint/js-parsers.test.ts --timeout 270000 on main and on this branch in turns: 264, 259, 308 s before, 202, 230, 218 s after.
  • The same file with the variables that scripts/runner.node.ts sets on the ASAN lane, to count the runs that end with llvm-symbolizer: 56 before and after.

The description has the numbers.

@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 53 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: oven-sh/bun/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: efc6c668-2bdf-41da-b288-132bb7897b60
📥 Commits

Reviewing files that changed from the base of the PR and between 9634520 and c6881cb.

📒 Files selected for processing: 1
  • test/cli/lint/js-parsers.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:53 PM PT - Oct 21st, 2026

✅ @robobun, your commit c6881cbbc1cc5ee4250f89796d898de43a64cefa passed in Build #130145! 🎉


🧪   To try this PR locally:

bunx bun-pr 45880

That installs a local version of the PR into your bun-45880 executable, so you can run:

bun-45880 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline findings, I also checked the two shared-run matrices (formatted and contained via once()) for vacuous rows: a missing per-file result fails through expect(undefined).toEqual([]) rather than passing, and each row's expected line number or path is unique to its file. The temp-dir names used by the lazy stdout/stderr getters are captured before using dir is disposed, so late normalization still strips the path.

Extended reasoning...

The change rewrites assertions and run-sharing in a single integration test file for bun lint, with no production code and no security-sensitive surface. Two inline findings were posted; this note records what else was examined and ruled out.

Comment thread test/cli/lint/js-parsers.test.ts Resolved
Comment thread test/cli/lint/js-parsers.test.ts Resolved
@robobun
robobun force-pushed the robobun/1cdbba4e/faster-js-parsers-test branch from 2d22f28 to bbc7858 Compare October 19, 2026 20:40

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread test/cli/lint/js-parsers.test.ts Resolved
@robobun robobun changed the title test(lint): make js-parsers.test.ts faster and check how every run ends test(lint): share runs in js-parsers.test.ts and check how every run ends Oct 19, 2026
…ends

lint() normalizes stdout and stderr when a test first reads them. It
normalized both for every run before. In a debug build that took 41 s of
the test runner's thread, 29 s of it for two 6 MB outputs that no test
reads, and no other test made progress in that time.

Rows of three tables share their runs, each row with its own file: one
run for each format lints the four files of the line-break tests, one
run lints the 13 files of the directive tests, and one run fixes the 15
files of the emphasis tests. Six tests start their two runs together.
The file starts 186 `bun lint` processes. It started 242.

resultsOf() reads the results of a `-f json` run and expects the run to
end the way a run with these results ends: standard error has the
summary line for that number of files and nothing else, and the exit
code is 1 only if a problem is an error. That only compares a run with
itself, so each test also states the exit code it expects. Checks that
a text is absent are now checks of the exact output, and the tests of
linear time expect the number of problems that ESLint reports.

On a debug build the file sets its default limit for a test to 120 s.
`bun bd test` gives a test 5 s, and a run with a configuration file in
JavaScript starts two debug processes. Release builds and CI keep their
limits.
@robobun
robobun force-pushed the robobun/1cdbba4e/faster-js-parsers-test branch from 1bb64be to c6881cb Compare October 21, 2026 12:52

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the squashed commit and found no bugs; the points from my earlier passes are addressed here (the --fix count at test/cli/lint/js-parsers.test.ts:1517 is now exact, and reported/realms keep an explicit exitCode 1 beside resultsOf). Because this is a 675-line restructuring of a test file that I could not execute, a human look at the new exact expectations would still be worthwhile.

What was reviewed:

  • lint()'s lazy <dir> replacement: it captures realpath-then-given before the temp dir is disposed, in the same order harness's normalizeBunSnapshot(…, dir) uses, so output is unchanged.
  • pathOf's bun-lint-js-parsers_\w+ matches harness's mkdtemp suffix (six alphanumerics); the ?.messages / ?.suppressedMessages rows fail on an undefined result rather than passing vacuously.
  • Shared once() runs: each row asserts on its own file so a single-row regression still fails alone; a shared-run failure rejects every dependent row, which is the stated tradeoff.
  • The one thing I could not verify by reading: the per-rule counts in "runs in linear time" (e.g. count - 2 for emphasis markers) were derived at count = 2000 on debug/ASAN; the release lane runs them at 20000.
Extended reasoning...

The diff rewrites only test/cli/lint/js-parsers.test.ts (+437/-238): memoized output normalization, a resultsOf helper that checks stderr and exit code against the parsed JSON, three parameterized suites collapsed onto shared runs, exact stderr/exitCode assertions added across most tests, and the "runs in linear time" suite moved and tightened to exact per-rule counts. No production code and no security-sensitive surface is touched; no CODEOWNERS entry covers the path. The latest push addresses my prior inline comments, and the setDefaultTimeout call is now gated to debug builds with a stated reason. Deferring rather than approving because the change is large, replaces many loose assertions with exact values I could not execute, and the new count expectations are only stated as verified on the debug and ASAN lanes.

@robobun

robobun commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator Author

About the counts in runs in linear time at 20,000 repeats, which the last review could not verify by reading:

  • The release lanes run these rows with count = 20_000. They pass in builds 128086, 129111 and 130145, which are green on every lane.
  • I also ran the 14 inputs at 20,000 repeats through a debug build. The numbers follow the same formulas as at 200 and at 2,000.
  • At 200 and at 2,000 repeats, ESLint 10.13 with @eslint/markdown 8.0.3 reports the same numbers.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #49196, which combines the open test-only PRs into one. The commit 8a0eb63c9 in that PR is this change, unchanged.

Closing this one in favor of #49196.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant