Skip to content

logger: let a Source own its position index, and delete the Log's LineColumnTracker - #35550

Draft
robobun wants to merge 10 commits into
mainfrom
farm/2b658520/fix-quadratic-error-line-scan
Draft

robobun wants to merge 10 commits into
mainfrom
farm/2b658520/fix-quadratic-error-line-scan

Conversation

@robobun

@robobun robobun commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • LineColumnTracker (src/ast/lib.rs:2234) keeps four scans in the Log. A lookup that none can resume walks from byte 0 and keeps nothing.
  • Many errors are then quadratic: 9.2 s for 1.2 MB of "x" has already been declared. A note from bun_ast::range_data never reaches the tracker: 17.1 s for 520 KB.
  • The tracker knows a source by its address. bun install prints b/package.json:4:99 for 1:138, and most of a bunfig.toml token.

Fix

  • Source::init_error_position keeps the scans in the Source. A lookup that none can resume starts at a checkpoint, one for each 1024 bytes.
  • The tracker is deleted. No call site changes, and positions stay the same.
  • An arena never drops a Source, so three places release the index.
  • Verified: test/bundler/transpiler/transpiler.test.js (six tests that main fails), test/cli/install/bun-install.test.ts, Miri. Self-reviewed: 12 concerns raised, 10 addressed.

Background

Downsides

  • Source grows from 112 to 120 bytes and owns an allocation: a maintainer must accept that (Notes). .text: +10,240 bytes.
  • bun build of the one-line file from the report stays slow (37 s for 64 KB) until logger: indent the caret relative to the windowed line excerpt #41658.
  • A flood that is linear on main costs 2% to 3% more cycles. An arena Source with no release leaks 272 bytes.
Notes

Question for a maintainer (@Jarred-Sumner, @dylan-conway). May bun_ast::Source get one 8-byte atomic slot that owns a boxed index? It is the first field of Source that owns something other than its text, and &Source is no longer Freeze.

Why at the root. No user reported this class. The one-line input is a maintainer's synthetic repro, and the other inputs came from review. #31273 added one scan and #31341 made it four. Each left the next order of lookups quadratic. The comment on #31273: "Do not solve quadratic behavior by limiting the count. Solve it by fixing the underlying cause."

Which part fixes which input. The report was one long line with many errors. There the cost on main is the walk to the end of the line for each note. #44811 bounds that walk for every lookup and does not touch Source: it alone fixes that input for Bun.Transpiler and bun run. This PR bounds it only for a lookup from a checkpoint. The walk from byte 0 is the cost of the other inputs, and only this PR removes it.

Inputs, release builds of main b1305dd and of this branch, Linux x64.

Input main This branch
One line of redeclared consts, notes at 40 early places, 128 / 256 / 512 KB 750 / 2,784 / 11,870 ms 17 / 31 / 60 ms
The same at 1 / 8 / 50 MB 133 s at 1 MB (from the report) 0.11 / 1.3 / 9.5 s
let redeclared, notes at 6 places 60 KB apart, 0.6 / 1.2 MB 2,399 / 9,242 ms 67 / 128 ms
let x; and {var x;} blocks, note from range_data, 260 / 520 KB 4,312 / 17,148 ms 28 / 56 ms
bun run of the 512 KB line 11,945 ms 125 ms
bun build of a 64 KB one-line file 36.5 s, 157 MB of output 37.1 s, 157 MB of output
  • For twice the input, main takes 4.0 times the instructions in the first four rows. This branch takes 1.86 to 1.99 times.
  • In well-formed UTF-8, a lookup from a checkpoint reads at most 1027 bytes for the position and 81 for the line end. The unit test a_lookup_from_a_checkpoint_reads_a_bounded_part_of_the_source counts the bytes. A run of continuation bytes where the excerpt is cut is read to its end, as on main.
The three inputs
// One long line: bun line.mjs <KB>
const chunk = i => `const a_${i%10}=1,b_${(i+1)%10}=[1,2,3],c_${(i+2)%10}={x:'${String(i).padStart(8,"0")}'},d_${(i+1)%10}=(e)=>e*2;`;
let s = "", i = 0; while (s.length < Number(process.argv[2]) << 10) s += chunk(i++);
const t0 = performance.now(); try { new Bun.Transpiler({ loader: "ts" }).transformSync(s); } catch {} console.log(Math.round(performance.now() - t0), "ms");
// Notes at <names> far places: bun flood-let.js 6 20000
const k = Number(process.argv[2]), repeats = Number(process.argv[3]);
const names = ["x", "y", "z", "w", "v", "u"].slice(0, k);
const filler = Buffer.alloc(3 * repeats, "{}\n").toString();
const dups = names.map(n => "let " + n + ";\n").join("");
const input = names.map(n => filler + "let " + n + ";\n").join("") + Buffer.alloc(dups.length * repeats, dups).toString();
const t0 = performance.now(); try { new Bun.Transpiler({ loader: "js" }).transformSync(input); } catch {} console.log(Math.round(performance.now() - t0), "ms");
// A note from range_data: bun flood-var.js 40000
const repeats = Number(process.argv[2]);
const input = Buffer.alloc(3 * repeats, "{}\n").toString() + "let x;\n" + Buffer.alloc(10 * repeats, "{var x;}\n ").toString();
const t0 = performance.now(); try { new Bun.Transpiler({ loader: "js" }).transformSync(input); } catch {} console.log(Math.round(performance.now() - t0), "ms");

Cost. User-space instructions and cycles from perf_event_open, minimum of 5 runs in turn (7 for the build), with a second copy of the main binary as control. perf and valgrind are not in the build container.

Loop, for each call or diagnostic main This branch Control
transformSync of 2 KB of TypeScript 496,798 instructions, 120.7 K cycles 494,343 (-0.49%), 121.3 K (+0.6%) 496,794, 121.6 K (+0.8%)
One syntax error after 4 KB 767,730 instructions, 186.5 K cycles 765,176 (-0.33%), 185.3 K (-0.6%) 767,733, 190.1 K (+2.0%)
300,000 redeclarations of one name on short lines (linear on main) 4,645 instructions, 1,305 cycles 4,613 (-0.7%), 1,332 (+2.1%) 4,641, 1,298 (-0.5%)
Eight names redeclared in turn, 40,000 diagnostics, notes in the first 56 bytes (linear on main) 4,975 instructions, 1,372 cycles 5,018 (+0.9%), 1,413 (+3.0%) 4,975, 1,360 (-0.9%)
bun build of 2,000 modules, two batches 191.07 M, 190.56 M instructions 190.78 M, 190.26 M 190.56 M, 190.76 M
A macro returns the same invalid 256 KB JSON 800 times (Bun.build) 56.1 G instructions, 3,300 ms 61.2 G (+9.0%), 3,540 ms
  • A lookup takes the index out of the slot and puts it back: two atomic swaps. That is the cycle cost in rows 3 and 4. An instruction count does not show it.
  • The macro row: Macro.rs:1139 builds a new Source for each result, so each error starts a new index and walks the 256 KB again. On main the address key resumed the scan of the call before. Every other producer has one Source for a file.
  • size_of (rustc, x86_64): Source 112 to 120, Log 48 to 40, Graph::InputFile 192 to 200, transpiler::ParseResult 1376 to 1384, parse_task::Success 1152 to 1152. The boxed index has 272 bytes. The boxed tracker had 256.
  • The first located diagnostic of a Source boxes the index. On main the first one of a Log boxes the tracker. range_data on a source with no other diagnostic now allocates once, where main allocates nothing (AsyncModule.rs:753, :946, Package.rs:3171).
  • Checkpoints start with the second lookup that no scan can resume. They take 40 bytes for each 1024 bytes, up to the highest such offset (4% of that part of the text). One Unexpected end of file inside a character allocates none.

Where a Source is never dropped.

  • Released: Transpiler::parse_internal (src/bundler/transpiler.rs, the arena copy that Bun.Transpiler, bun run, import and require parse), parse_worker::run_with_source_code (src/bundler/ParseTask.rs, every bundler parse), and the rows of graph.input_files in BundleV2::deinit_without_freeing_arena (linker diagnostics).
  • No release: JSTranspiler.rs:706 and :1124 and RuntimeTranspilerStore.rs:784 are only a virtual_source, which parse_internal clones. bundle_v2.rs:7363 has no text. bun build --watch leaks its BundleV2 by design, and the error return of scan_module_graph_from_cli ends the process.
  • Checked with LeakSanitizer as CI sets it (detect_leaks=1, BUN_DESTRUCT_VM_ON_EXIT=1): bun run, require and import, bun build with a parse error and with a linker error, Bun.build with both, bun test, transformSync, scan and transform report nothing. With the three releases removed, all seven abort with Direct leak of 272 byte(s) from Box<bun_ast::PositionIndex>.

Positions.

  • The scan that finds a position is the loop of main, not changed. A checkpoint is taken where a walk from the last checkpoint is between two characters, so it is a place that the walk from byte 0 also passes.
  • Unit tests compare positions inside a character with the values that main reports, and a lookup from a checkpoint with a scan to the end of the line (each kind of line break 79 to 82 bytes after the offset).
  • a note that is found from a checkpoint has the line, column and text of a walk from the start runs at the interval that ships: notes at and across multiples of 1024 bytes, CR, CR LF, LF, U+2028 and U+2029 at 79 to 81 bytes after a name, and an end of file inside a character. It compares each position with one counted in the test.
  • A copy of the position code of this branch, run outside the tree against the scan of main: 13.3 million positions in generated sources (every kind of line break, characters of 1 to 4 bytes, ill-formed sequences, intervals of 8, 64 and 1024 bytes) have the same line, column and excerpt.
  • An offset inside a character went to the walk from byte 0 on main. It now starts at a checkpoint too.

The address key. On main a reused buffer gives the next file the key of the last one. Two file: folders with a package.json of 260 bytes each: b/package.json:4:99 for an error at 1:138. A ~/.bunfig.toml and a bunfig.toml of 5000 bytes each: main prints 4e5f6g7h8i9j0k1l2m3n4o5p6q7r8", url = 5 } at bunfig.toml:3:38, because the stale line start is past the token key that the masking looks for. With the index in the Source there is no key. The four tests of #46277 are here and fail on main. Not tested here: the manifests of lifecycle scripts (Package/Scripts.rs) and of a chain of local tarballs (processDependencyList.rs), which #46277 measured by hand.

Not changed here.

Other PRs on these lines.

Self-review. Addressed: known answers from main in the unit tests, a test at the interval that ships, the line-end search on every kind of line break, test sizes that the Miri lane runs as written, a Send assertion for the index, #[must_use] on the release guard, cycles beside instructions, the rows for one error and for the macro, the four tests of #46277, and this body. Not done: the restack on #44811 (it is not merged), and one helper that places a Source in an arena for all five arena sites with a lint against a bare alloc (it waits for the answer on the field).

Tests.

  • The flood tests put 16 MB of comment in the source, so that a walk for each of 10,000 diagnostics takes minutes and one walk takes about 2 s on a debug build. Entry points: transformSync (three inputs), bun run, Bun.build with parse errors, and Bun.build with a linker note in another file.
  • parse error notes also pins line, column and lineText of the five notes that the parser builds with range_data or Location::init_or_null. Those five pass on main.
  • The Miri run of bun_ast takes about 70 s longer (15 tests in the position module, 4 before).
  • Suites run on the debug build: test/bundler/transpiler/transpiler.test.js, test/js/bun/transpiler/, test/js/bun/jsonc/jsonc.test.ts, test/js/bun/resolve/resolve-error.test.ts, test/js/bun/util/inspect-error.test.js, test/cli/install/bun-install.test.ts, test/cli/install/redacted-config-logs.test.ts, test/bundler/bundler_loader.test.ts, test/bundler/esbuild/importstar.test.ts, test/bundler/esbuild/packagejson.test.ts, test/regression/issue/{03830,11793,12782,19850,23649}, cargo miri test -p bun_ast, cargo clippy --workspace, rust:check-all for Windows x64 and macOS arm64.

no test proof · iteration 5 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/cli/install/bun-install.test.ts

…ng single lines are linear

When a parse produces many diagnostics on one long line whose note locations
cycle (e.g. 'X has already been declared' notes pointing back to a handful of
early declarations), all four scan cursors eventually overshoot the note
offset and the tracker fell back to Source::init_error_position, which runs
scan_line_end (O(line length)) on every call. 256KB took ~8s, 512KB ~33s.

Evict the least-advanced cursor and restart it instead of falling back, and
reuse another cursor's cached line_end when the line_start matches so the
restarted cursor doesn't rescan to end-of-line either.
@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

The line-column tracker now initializes cursors for previously unseen offsets and reuses compatible cached line ends before scanning. Tests cover cycling offsets, repeated newline layouts, and long single-line TypeScript redeclaration diagnostics with bounded runtime.

Line-column tracking correction

Layer / File(s) Summary
Cursor and line-end resolution
src/ast/lib.rs
LineColumnTracker::error_position updates cursor selection and reuses matching cached line-end positions before falling back to scanning.
Offset and transpiler regression coverage
src/ast/lib.rs, test/js/bun/transpiler/transpiler-error-longline-quadratic.test.ts
Tests validate tracker results against full scans and check long-line transpiler diagnostics, error counts, positions, runtime, and process output.

Suggested reviewers: jarred-sumner

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Title check Warning The title names a logger change (letting a Source own its position index and deleting the Log's LineColumnTracker). The changeset summary describes changes to LineColumnTracker::error_position in src/… Retitle the PR so it matches the change in the summarized diff. For example: "ast: fix quadratic error-position scanning for long single-line inputs". Alternatively, update the PR summary and file-level summaries so they describe the Source…
Passed Passed checks: 3 passed

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is long and substantive, and it does not follow the repository template. It has no "What does this PR do?" heading and no "How did you verify your code works?" heading. Its content cov…
Full details: Description check

Explanation

The description is long and substantive, and it does not follow the repository template. It has no "What does this PR do?" heading and no "How did you verify your code works?" heading. Its content covers the problem, the fix, the downsides, the measured results, and the tests. The Fix section and the Problem section together explain what the change does, and the Tests and Cost sections explain verification. Two gaps remain. First, the verification lists test suites, but the robobun evidence block notes that the bun-install test runs only on CI and was not run on this machine. Second, the text is dense, and a reviewer must read a long Notes section to find the open maintainer question about the Source field.

Full details: Title check

Explanation

The title names a logger change (letting a Source own its position index and deleting the Log's LineColumnTracker). The changeset summary describes changes to LineColumnTracker::error_position in src/ast/lib.rs with cursor selection and line-end caching, plus a new regression test. The title does not describe the cursor and line-end caching change shown in the summary, and it refers to a Source-owned index and a deleted Log tracker that the provided summary does not mention. The title and the summary therefore describe different changes, so the title does not match the changeset as summarized.

Resolution

Retitle the PR so it matches the change in the summarized diff. For example: "ast: fix quadratic error-position scanning for long single-line inputs". Alternatively, update the PR summary and file-level summaries so they describe the Source-owned index and the removal of LineColumnTracker from the Log.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Comment thread src/ast/lib.rs Resolved
Comment thread src/ast/lib.rs Resolved
@robobun

robobun commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 7:39 PM PT - Nov 7th, 2026

❌ @robobun, your commit 7af3d7a has 2 failures in Build #138452 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 35550

That installs a local version of the PR into your bun-35550 executable, so you can run:

bun-35550 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the timeout nit from the previous pass was addressed in e4e7361.

What was reviewed:

  • Eviction path: reset cursor is ScanCursor::default() → advance(contents, 0, offset) → scan_line_end, byte-identical to the old init_error_position fallback it replaces, just cached.
  • line_end reuse: line_start uniquely identifies a line so borrowing another cursor's cached end is sound; the self-cursor is excluded by the line_end.is_some() filter, and reset_for clears all cursors on source change so no cross-source staleness.
  • Unit test diffs every probe against init_error_position on both single-line and newline-per-10-bytes inputs; integration test drains pipes concurrently, asserts exact error/line/column, branches the perf bound on isDebug || isASAN, and no longer sets an explicit timeout.
Extended reasoning...

Overview

Two files: ~20 net lines in LineColumnTracker::error_position (src/ast/lib.rs) plus a Rust unit test and a new integration test. The change replaces the fall-through-to-init_error_position path (when all four scan cursors are ahead of the requested offset) with (1) evicting the least-advanced cursor and restarting it from 0, and (2) reusing a sibling cursor's cached line_end when both are on the same line_start. Output is unchanged; only redundant end-of-line scans are eliminated.

Security risks

None. Pure safe Rust over an in-memory byte slice; no FFI, no unsafe, no allocation, no user-facing API surface. The path only affects diagnostic line/column computation.

Level of scrutiny

Medium — hot path in the parser's diagnostic pipeline, but the correctness argument reduces to "the evicted cursor does exactly what init_error_position did" (verified by reading init_error_position at src/ast/lib.rs:2843, which is ErrorPositionState::default() + advance(0, offset) + scan_line_end — the identical sequence). The line_end reuse is safe because line_start is a byte offset that uniquely names a line, and reset_for zeroes all cursors when the tracked Source changes. The new unit test asserts every probe against the ground-truth full scan on both single-line and multi-line inputs, which directly guards the identity claim.

Other factors

  • The prior review's only finding (explicit 120_000 test timeout) was addressed in e4e7361: timeout dropped, workload shrunk to 128 KB, inner limit tightened to 500 ms release / 3 s debug+ASAN.
  • Both comment-cop threads are resolved (comment blocks removed in 72b14a7).
  • Integration test follows harness conventions: bunEnv/bunExe, concurrent pipe drain, stderr asserted empty, exact error object asserted before exitCode, perf bound branched on isDebug || isASAN.
  • Verified column 567 arithmetic: 10 chunks × 56 bytes + "const ".len + 1-based = 567.
  • No CODEOWNERS on src/ast/.

@robobun

robobun commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

This comment is about the earlier version of this PR (a cursor eviction in LineColumnTracker). That version is replaced. The PR body says how the current version and #44811 fit together.


#44811 changes the same function. It removes the cached line end from LineColumnTracker: the search for the end of the line becomes a search of the 83 bytes after the position, done in Location::init_or_null_impl.

What that means for this PR:

The second of the two PRs to land needs a rebase.

@robobun

robobun commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: draft. The diff is complete and it waits for a maintainer's answer on the question in the next comment. This PR is #35550.

How I reproduced it (release build of main b1305dd, Linux x64):

  • let x; and 40,000 {var x;} blocks through Bun.Transpiler: 17.1 s. This branch: 56 ms.
  • let redeclarations with notes at 6 places, 1.2 MB: 9.2 s. This branch: 128 ms.
  • One 512 KB line of redeclared consts: 11.9 s. This branch: 60 ms.
  • bun install with two file: folders whose package.json files have 260 bytes each: b/package.json:4:99 for an error at 1:138. This branch: 1:138.
  • bun install with a ~/.bunfig.toml and a bunfig.toml of 5000 bytes each: 29 characters of the token are printed at bunfig.toml:3:38. This branch: the line is masked at 2:72.

The scripts and the tables are in the Notes of the PR body. The earlier version of this PR (a cursor eviction in the tracker) is replaced.

CI: build 138452 is finished and red on two test/aot tests. No test of this diff fails on any lane, and the GitHub checks pass. test/aot/loader-access-in-unreachable-code.test.ts fails with the same numbers (131008 for 32768 on debian 13 aarch64) in builds 138451, 138417 and 138408 of three other branches. test/aot/parallel/gc/local-set-to-null-lets-go-in-a-slow-version.ts is also red in build 138398 of main.

…eColumnTracker

Source::init_error_position resumes one of four scans, as the tracker did.
An offset that no scan can resume to starts at a checkpoint, one for each
1024 bytes, so it costs at most 1027 bytes of scan and 81 for the line end.
The index is in the Source, so a note from range_data or
Location::init_or_null uses it too, and two sources on one Log do not reset
each other.

This replaces the cursor eviction that this branch had.
…d take the install tests of #46277

- A checkpoint is taken where a walk to the next character boundary stops. The loop that scans for a position is the one main has.
- Unit tests compare positions inside a character with the values main reports, and a lookup from a checkpoint with a scan to the line end.
- A test at the interval that ships: notes at and across multiples of 1024 bytes, with each kind of line break 79 to 81 bytes after them.
- The install tests are from #46277: three file: folder cases and a bunfig.toml whose token stays masked.
@robobun robobun changed the title transpiler: fix quadratic error-position scan on long single lines with many diagnostics logger: let a Source own its position index, and delete the Log's LineColumnTracker Nov 7, 2026
@robobun

robobun commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

@Jarred-Sumner @dylan-conway one question decides the shape of this PR.

May bun_ast::Source get one 8-byte atomic slot that owns a boxed position index? Source grows from 112 to 120 bytes, &Source is no longer Freeze, and three places release the index for a Source in an arena. On error-free paths I measure no cost: transformSync takes 0.5% fewer instructions and a 2,000-module build is within noise (0.2%).

Order: #44811 alone fixes the one-line input of the report (the cost there is the walk to the end of the line) and does not touch Source. It should land first. After it, this PR drops its own line-end search.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants