Skip to content

test(lint): start fewer runs in prettier-rule.test.ts and check how every run ends - #47064

Closed
robobun wants to merge 2 commits into
mainfrom
robobun/3ccf2dc1/faster-prettier-rule-test
Closed

robobun wants to merge 2 commits into
mainfrom
robobun/3ccf2dc1/faster-prettier-rule-test

Conversation

@robobun

@robobun robobun commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • test/cli/lint/prettier-rule.test.ts starts 36 bun lint processes. Six repeat a run in a second format to read one message.
  • Seven runs had their exit code checked. None had its whole stderr compared.
  • On a debug build, bun bd test fails the file: one test needs over 5 s.

Fix

  • Six tests (runs the package's rule: ...) read the report and the note from one -f stylish run. Now 30 runs and 18 tests.
  • Two bun/format tests start their runs together.
  • Each run has its output, stderr and exit code compared. Exact values replace toContain, not.toContain and toMatchObject.
  • Verified: bun bd test test/cli/lint/prettier-rule.test.ts --timeout 270000, debug build, 24 runs each in turns: median 17.7 s, then 14.6 s. Without the flag: 9 of 9 runs fail, then 6 of 6 pass.

Background

  • When bun lint's own prettier/prettier cannot match Prettier's output, the rule of eslint-plugin-prettier runs in a JavaScript engine. A note on stderr says why. -f json prints no note.
  • Five runs of four other tests keep a -f json report of such a run.
  • debian 13 x64-asan shows no clear gain: 20.4 s and 17.4 s in two builds, 18.4 to 27.2 s before. Five tests run at a time there, and ten take 6.5 to 11 s each (test(lint): make rule-throws.test.ts faster and strengthen its assertions #45755).
  • On a debug build, the tests with a config in JavaScript get 120 s, as in doctor.test.ts.
  • Concurrent tests cannot use snapshot matchers, so toEqual has the expected text.
Notes

The ASAN lane. The time of the file on debian 13 x64-asan - test-bun, from the log of each build.

Build File Time
133512 main 18.40 s
133525 main 19.39 s
133504 main 20.22 s
133521 main 20.39 s
133524 main 21.08 s
133443 main 25.81 s
133498 main 27.15 s
133645 this PR, first commit 20.40 s
133655 this PR, second commit 17.38 s

The second commit changes limits on a debug build only, not what runs. In both builds of this PR the file is the first test file of its shard, because the runner starts the files that a pull request changes first. Main's file is at place 13, after config-files.test.ts.

The log has a time for each batch of tests that end. It does not name the tests. This is what fits the order of the file:

main, build 133525 this PR, build 133645
The two tests at the top, with no rule in JavaScript end at 0.5 s end at 0.5 s
The first five tests with a rule in JavaScript end at 8.3 s (10 such runs) end at 8.8 to 10.0 s (5 such runs)
The next tests with one end at 15.9 s (7 such runs) end at 18.7 s (8 such runs)
an option that isn't Prettier's starts at 12.9 s, ends at 19.3 s is a row of the table
The short tests after them 9.7 to 12.9 s, most of them one at a time 10.6 s, then 19.8 s

What the lane shows:

  • An ASAN build runs 5 tests of a file at a time (src/options_types/context.rs:534).
  • A test with a rule in JavaScript takes 6.5 to 11 s there. It takes little longer with 10 such runs in flight than with 2: in build 133525, 7.8 s with 10, 7.5 s with 7 and 6.5 s with 2.
  • So the number of such runs does not set the time of the file. The number of such tests does. There are ten, before and after, and five run at a time: two rounds.
  • Main's file has one of the ten at the end, behind five short tests, so it ends a part of a round later. In this file that test is a row of the table. That is the only gain on the lane: about 3 s by the times above.
  • Two builds do not prove it, because the builds differ by more. A round takes 6.4 to 8.1 s in build 133655, 7.8 s in build 133525, 8.3 to 9.9 s in build 133645, and 11 to 12 s in builds 133443 and 133498. Build 133655 has two tests end at 0.5 s, six between 6.8 and 8.6 s, eight at 15.3 s and the last two at 16.9 s.

The other lanes. No change that the numbers can show. On the release lanes all 18 tests run at once.

Lane main, build 133525 this PR, build 133645
debian 13 x64 0.33 s 0.42 s
debian 13 aarch64 0.30 s 0.29 s
ubuntu 25.04 x64 0.37 s 0.36 s
ubuntu 25.04 aarch64 0.32 s 0.33 s
alpine 3.23 x64 0.46 s 0.48 s
alpine 3.23 aarch64 0.33 s 0.34 s
windows 2019 x64 0.81 s 1.03 s
windows 11 aarch64 0.73 s 1.02 s
macOS aarch64 0.22 s not done when I read the build
macOS x64 0.45 s not done when I read the build

The two columns are not like for like: this file is the first test file of its shard in build 133645 (place 3 of its job, after two installs), and main's file is at place 14 or later. #45768 shows what the first place costs on Windows with other files. No Windows machine was available for this work, so the 0.2 to 0.3 s there are not explained.

Which runs are slow on the ASAN lane. The lane sets detect_leaks=1. A run that executes a rule in JavaScript leaves one 80-byte allocation that test/leaksan.supp suppresses by name (Zig__GlobalObject__create), so LeakSanitizer starts llvm-symbolizer when the process ends. A run with an eslint.config.mjs and built-in rules only leaves none. One run on a debug build, alone:

Run bunEnv only with the variables of the lane
.oxlintrc.json, -f unix 0.2 s 0.2 to 0.4 s
eslint.config.mjs, built-in rule 1.4 s 2.0 s (3.8 s the first time)
eslint.config.mjs, the package's rule 1.8 s 7.4 to 7.9 s

I put a script at ASAN_SYMBOLIZER_PATH that counts its calls and then starts llvm-symbolizer. With the variables that scripts/runner.node.ts sets on the lane, main's file starts it 19 times and this file 13 times. Both pass that way (debug build, --max-concurrency=2: the symbolizer needs 1.8 GB for the debug binary, and this machine has 16 GB).

What the 30 runs are.

Before After
Runs that execute the package's rule 19 13
Other runs with an eslint.config.mjs 9 9
Runs with an .oxlintrc.json 8 8

The six runs that are gone are the -f json runs of the five table rows and of an option that isn't Prettier's. Each compared one thing: the report of a.js is the stub's message. The -f stylish run of the same project prints that message with its position, its severity and the rule's name, and stderr has the note. The format does not change which rule runs: in src/lint/driver/run.rs it decides only whether the text of a file is kept and whether the notes are printed. an option that isn't Prettier's had the same body as the table rows, so it is a row now, under the same name.

A -f json report of a file that the package's rule linted stays in: the syntax error test, the Tailwind test (two runs), the disable comment test, and the test with two plugins. In those runs stderr is now compared with the summary line alone, which also shows that -f json prints no note.

The three tests that need the fix of the built-in rule and the note keep both runs.

Assertions. expect() calls: 45 before, 43 after. One call now compares what several did, and more:

  • lint() returns the reports, stderr without the time, and the exit code. Each test compares all three with toEqual. Before a report is parsed, lint() checks that the exit code is 0 or 1, and a failure prints stderr.
  • --fix-dry-run produces Prettier's output also checks that no report is left after the fixes, the summary line, and exit code 0.
  • The -f stylish runs compare stdout. Two tests had not.toContain("ran in JavaScript"). One of them read the stderr of a -f json run, which never has a note. Both compare the whole stderr now.
  • bun/format reports the files it can't check compared single keys and used toMatchObject. It now compares every file of each run. That adds the config file, which is linted there, and the unused disable comment in off.js once the file can be checked.
  • The -f unix runs compare the whole lines about bun/format, an empty stderr and the exit code. The lines of oxlint's default rules stay out, as before.

Local numbers. Debug build (ASAN) of 92e819d on Linux x64. The machine has 12 logical CPUs, but 12 busy processes take 3.4 times as long as one, so the runs are bound by CPU time and vary.

Measure Before After
bun bd test <file> --timeout 270000, 24 runs each in turns 13.0 to 28.8 s, median 17.7 s 8.3 to 22.1 s, median 14.6 s
CPU time of that command and all its processes, 6 runs each in turns 57.0 to 61.5 s, median 60.7 s 44.3 to 48.8 s, median 47.2 s
bun bd test <file>, no flag 9 of 9 runs fail: 1 to 12 tests pass 5 s 6 of 6 runs pass: 8.5 to 11.2 s

The 24 runs of this file are from both commits and from the edits before them. None of those changes which processes run.

The limit on a debug build. A run with an eslint.config.mjs takes 1.4 s or more there, and up to 5 s while four other tests run. With no limit, this file passed 3 of 6 runs of bun bd test with no flag. So the 15 tests with such a run get { timeout }, which is 120 s on a debug build and nothing on any other build: there the command line or the default sets the limit. The three tests with an oxlint config have the default everywhere. 120 s is what the other lint test files give their slow tests (doctor.test.ts:60, js-plugins.test.ts:63, react-compiler.test.ts:68). A limit on a test wins over --timeout, here as there.

Not done.

  • Start the runs of later tests before their turn, so that all 13 runs with a rule in JavaScript overlap. That would take a round off the ASAN lane. It would also put more than 5 tests' processes in memory at once, which the limit of an ASAN build is there to prevent.
  • The cost of each run with a rule in JavaScript on that lane is in src/, not in this file (test(lint): make rule-throws.test.ts faster and strengthen its assertions #45755 has the stack of the allocation).
  • Projects that share one bun lint process: each case keeps its own working directory and its own node_modules.

[auto-merge] gate passed · iteration 0 · 1 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/cli/lint/prettier-rule.test.ts'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/cli/lint/prettier-rule.test.ts
bun test v1.4.4 (367d939d9)

test/cli/lint/prettier-rule.test.ts:
(pass) prettier/prettier > reports the same as the package's rule [3946.92ms]
(pass) prettier/prettier > --fix-dry-run produces Prettier's output [4147.41ms]
(pass) prettier/prettier > runs the package's rule: a different minor version of prettier [5054.70ms]
(pass) prettier/prettier > runs the package's rule: prettier isn't installed [5564.04ms]
(pass) prettier/prettier > runs the package's rule: a different major version of prettier [7155.74ms]
(pass) prettier/prettier > runs the package's rule: a different major version of the plugin [3705.60ms]
(pass) prettier/prettier > runs the package's rule: a Prettier plugin that isn't built in [3879.75ms]
(pass) prettier/prettier > runs the package's rule: an option that isn't Prettier's [4616.18ms]
(pass) prettier/prettier > runs the package's rule: a syntax error that only the formatter finds [4615.76ms]
(pass) prettier/prettier > reads the versions of the plugin and the prettier that the config loaded [2520.71ms]
(pass) prettier/prettier > prettier-plugin-tailwindcss: runs the package's rule, unless Tailwind CSS isn't installed [3773.12ms]
(pass) prettier/prettier > a disable comment also turns off the package's rule, and isn't reported as unused [3806.29ms]
(pass) prettier/prettier > bun/format: one package's formatter config doesn't affect another package's files [543.47ms]
(pass) prettier/prettier > bun/format skips the scripts in a component [801.32ms]
(pass) prettier/prettier > bun/format respects allowUnsupported in [fmt], like bun fmt [1065.56ms]
(pass) prettier/prettier > bun/format reports the same without anything installed [2575.75ms]
(pass) prettier/prettier > with two plugins under the same name, each file uses the one its config block loads [3481.11ms]
(pass) prettier/prettier > bun/format reports the files it can't 
... (truncated)
Exit: 0
diff hotspot
test/cli/lint/prettier-rule.test.ts | 333 +++++++++++++++++++++++-------------
 1 file changed, 211 insertions(+), 122 deletions(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                 reads  edits  tests
test/cli/lint/prettier-rule.test.ts     10     11     29

… ends

Six tests read the report of the rule and the note from one `-f stylish`
run. Each made a second run with `-f json` before. The case "an option
that isn't Prettier's" is now a row of their table. The file starts 30
`bun lint` processes, not 36. 13 of them run a rule in JavaScript, not 19.

Two `bun/format` tests start their runs together.

Each run has its output, its stderr and its exit code compared.

On a debug build a test of this file has 60 s, not 5 s.
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 92e819d and 77db2cc.

Files selected for processing: 1
  • test/cli/lint/prettier-rule.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 7:50 AM PT - Oct 27th, 2026

✅ @robobun, your commit 77db2ccb76a83788c1ffa24ee07a7a27e965ccef passed in Build #133655! 🎉


🧪   To try this PR locally:

bunx bun-pr 47064

That installs a local version of the PR into your bun-47064 executable, so you can run:

bun-47064 --bun

@robobun

robobun commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. The change is in test/cli/lint/prettier-rule.test.ts only. It needs the review of a maintainer to merge.

CI on the current commit (build 133655):

  • Every job that has run passed, 179 of 181. Two macOS x64 jobs still wait for a machine.
  • This file passed on every lane that ran, at the first attempt.
  • Four test files that this pull request does not touch failed once and passed on a retry: test/js/bun/css/nth-anplusb-ident.test.ts and test/regression/issue/10170.test.ts on macOS aarch64, test/js/node/test/parallel/test-watch-mode-kill-signal-default.mjs on debian 13 x64-asan, and test/js/node/process/process-stdin.test.ts on windows 11 aarch64.

The review has no open thread. Its one request, the limit of a test on a debug build, is done in 77db2cc.

What the numbers are, and how they were measured:

  • The ASAN lane shows no clear gain. The log of debian 13 x64-asan - test-bun has 20.40 s and 17.38 s for this file (builds 133645 and 133655). Main's file takes 18.40 s to 27.15 s in seven builds (133443, 133498, 133504, 133512, 133521, 133524, 133525). The description says why: five tests run at a time there, and ten tests still wait for a run that takes 6.5 to 11 s.
  • A debug build (ASAN) of 92e819d on Linux x64: bun bd test test/cli/lint/prettier-rule.test.ts --timeout 270000 with main's file and with this file in turns, 24 runs each. Median 17.7 s before, 14.6 s after. The CPU time of the command and all its processes, 6 runs each: median 60.7 s before, 47.2 s after.
  • The same command with no flag: main's file fails 9 of 9 runs on the 5 s limit of a test. This file passes 6 of 6.
  • The runs that end with llvm-symbolizer: the command with the variables that scripts/runner.node.ts sets on the ASAN lane, and a script at ASAN_SYMBOLIZER_PATH that counts its calls. 19 before, 13 after.
  • The other lanes: 0.3 to 0.5 s on Linux before and after, and about 1 s on Windows. The description has the table.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Comment thread test/cli/lint/prettier-rule.test.ts Resolved
…vaScript

Replaces the default limit that the file set for debug builds. The tests
whose runs have an `eslint.config.mjs` get 120 s on a debug build, the
number that the other lint test files give their slow tests. On every
other build they keep the limit of the command line or the default. The
three tests with an oxlint config keep the default on every build.
@robobun robobun changed the title test(lint): make prettier-rule.test.ts faster and check how every run ends test(lint): start fewer runs in prettier-rule.test.ts and check how every run ends Oct 27, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only tightening of assertions, and the follow-up commit moves the debug-build limit to a per-test { timeout } on the eslint.config.mjs tests, the same shape as the sibling lint test files.

What was reviewed:

  • Confirmed { timeout: undefined } falls through to the CLI --timeout and then the default (b.f64.optional maps to None in src/runtime/test_runner/ScopeFunctions.rs:624), so the oxlint tests and non-debug builds keep whatever limit the caller set; CI's ASAN lane is covered by the runner's tripled --timeout, so the isDebug-only guard (siblings also include isASAN) is fine there.
  • Checked that normalizeBunSnapshot(text, dir) strips the [Nms] suffix and trims, so the exact "Linted N files" / stylish stdout expectations are stable, and that expect(value, message) plus toBeOneOf are supported matchers.
  • Walked each deleted assertion (not.toContain("ran in JavaScript"), toMatchObject on bun/format reports, the standalone "an option that isn't Prettier's" test) to its replacement — every one is now covered by a stricter whole-object toEqual or the test.each row with the same name; no check was dropped.
Extended reasoning...

The change touches only test/cli/lint/prettier-rule.test.ts, rewriting its assertions from partial checks to whole-object toEqual on normalized stdout/stderr/exitCode, parallelizing a few sequential runs, and folding one standalone test into the test.each matrix. It touches no security-sensitive surface and no CODEOWNER-covered path. The prior review's one nit (a file-wide setDefaultTimeout under isDebug) was addressed by commit 77db2cc, which replaced it with a per-test timeout on only the tests that evaluate a JavaScript config, matching the pattern in the neighboring lint test files. The bug hunt ran dry, the harness semantics the test relies on were verified from source, and no existing assertion was weakened, so a human look is not needed.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Folded into #49196, which combines the open test-only PRs into one. The commit 0b851419c in that PR is this change, unchanged.

Closing this one in favor of #49196.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant