Repository navigation
Conversation
…ng single lines are linear When a parse produces many diagnostics on one long line whose note locations cycle (e.g. 'X has already been declared' notes pointing back to a handful of early declarations), all four scan cursors eventually overshoot the note offset and the tracker fell back to Source::init_error_position, which runs scan_line_end (O(line length)) on every call. 256KB took ~8s, 512KB ~33s. Evict the least-advanced cursor and restart it instead of falling back, and reuse another cursor's cached line_end when the line_start matches so the restarted cursor doesn't rescan to end-of-line either.
WalkthroughChangesThe line-column tracker now initializes cursors for previously unseen offsets and reuses compatible cached line ends before scanning. Tests cover cycling offsets, repeated newline layouts, and long single-line TypeScript redeclaration diagnostics with bounded runtime. Line-column tracking correction
Suggested reviewers: Pre-merge checks:
|
|
Updated 7:39 PM PT - Nov 7th, 2026
❌ @robobun, your commit 7af3d7a has 2 failures in
🧪 To try this PR locally: bunx bun-pr 35550That installs a local version of the PR into your bun-35550 --bun |
There was a problem hiding this comment.
LGTM — the timeout nit from the previous pass was addressed in e4e7361.
What was reviewed:
- Eviction path: reset cursor is
ScanCursor::default()→advance(contents, 0, offset)→scan_line_end, byte-identical to the oldinit_error_positionfallback it replaces, just cached. line_endreuse:line_startuniquely identifies a line so borrowing another cursor's cached end is sound; the self-cursor is excluded by theline_end.is_some()filter, andreset_forclears all cursors on source change so no cross-source staleness.- Unit test diffs every probe against
init_error_positionon both single-line and newline-per-10-bytes inputs; integration test drains pipes concurrently, asserts exact error/line/column, branches the perf bound onisDebug || isASAN, and no longer sets an explicit timeout.
Extended reasoning...
Overview
Two files: ~20 net lines in LineColumnTracker::error_position (src/ast/lib.rs) plus a Rust unit test and a new integration test. The change replaces the fall-through-to-init_error_position path (when all four scan cursors are ahead of the requested offset) with (1) evicting the least-advanced cursor and restarting it from 0, and (2) reusing a sibling cursor's cached line_end when both are on the same line_start. Output is unchanged; only redundant end-of-line scans are eliminated.
Security risks
None. Pure safe Rust over an in-memory byte slice; no FFI, no unsafe, no allocation, no user-facing API surface. The path only affects diagnostic line/column computation.
Level of scrutiny
Medium — hot path in the parser's diagnostic pipeline, but the correctness argument reduces to "the evicted cursor does exactly what init_error_position did" (verified by reading init_error_position at src/ast/lib.rs:2843, which is ErrorPositionState::default() + advance(0, offset) + scan_line_end — the identical sequence). The line_end reuse is safe because line_start is a byte offset that uniquely names a line, and reset_for zeroes all cursors when the tracked Source changes. The new unit test asserts every probe against the ground-truth full scan on both single-line and multi-line inputs, which directly guards the identity claim.
Other factors
- The prior review's only finding (explicit
120_000test timeout) was addressed in e4e7361: timeout dropped, workload shrunk to 128 KB, inner limit tightened to 500 ms release / 3 s debug+ASAN. - Both comment-cop threads are resolved (comment blocks removed in 72b14a7).
- Integration test follows harness conventions:
bunEnv/bunExe, concurrent pipe drain, stderr asserted empty, exact error object asserted beforeexitCode, perf bound branched onisDebug || isASAN. - Verified column 567 arithmetic: 10 chunks × 56 bytes + "const ".len + 1-based = 567.
- No CODEOWNERS on
src/ast/.
|
This comment is about the earlier version of this PR (a cursor eviction in #44811 changes the same function. It removes the cached line end from What that means for this PR:
The second of the two PRs to land needs a rebase. |
|
Status: draft. The diff is complete and it waits for a maintainer's answer on the question in the next comment. This PR is #35550. How I reproduced it (release build of main b1305dd, Linux x64):
The scripts and the tables are in the Notes of the PR body. The earlier version of this PR (a cursor eviction in the tracker) is replaced. CI: build 138452 is finished and red on two |
…dratic-error-line-scan
…eColumnTracker Source::init_error_position resumes one of four scans, as the tracker did. An offset that no scan can resume to starts at a checkpoint, one for each 1024 bytes, so it costs at most 1027 bytes of scan and 81 for the line end. The index is in the Source, so a note from range_data or Location::init_or_null uses it too, and two sources on one Log do not reset each other. This replaces the cursor eviction that this branch had.
…d take the install tests of #46277 - A checkpoint is taken where a walk to the next character boundary stops. The loop that scans for a position is the one main has. - Unit tests compare positions inside a character with the values main reports, and a lookup from a checkpoint with a scan to the line end. - A test at the interval that ships: notes at and across multiples of 1024 bytes, with each kind of line break 79 to 81 bytes after them. - The install tests are from #46277: three file: folder cases and a bunfig.toml whose token stays masked.
|
@Jarred-Sumner @dylan-conway one question decides the shape of this PR. May
Order: #44811 alone fixes the one-line input of the report (the cost there is the walk to the end of the line) and does not touch |
…dratic-error-line-scan
Problem
LineColumnTracker(src/ast/lib.rs:2234) keeps four scans in theLog. A lookup that none can resume walks from byte 0 and keeps nothing."x" has already been declared. A note frombun_ast::range_datanever reaches the tracker: 17.1 s for 520 KB.bun installprintsb/package.json:4:99for1:138, and most of abunfig.tomltoken.Fix
Source::init_error_positionkeeps the scans in theSource. A lookup that none can resume starts at a checkpoint, one for each 1024 bytes.Source, so three places release the index.test/bundler/transpiler/transpiler.test.js(six tests that main fails),test/cli/install/bun-install.test.ts, Miri. Self-reviewed: 12 concerns raised, 10 addressed.Background
Log: it does not reach a note in another file, and the address key stays.Downsides
Sourcegrows from 112 to 120 bytes and owns an allocation: a maintainer must accept that (Notes)..text: +10,240 bytes.bun buildof the one-line file from the report stays slow (37 s for 64 KB) until logger: indent the caret relative to the windowed line excerpt #41658.Sourcewith no release leaks 272 bytes.Notes
Question for a maintainer (@Jarred-Sumner, @dylan-conway). May
bun_ast::Sourceget one 8-byte atomic slot that owns a boxed index? It is the first field ofSourcethat owns something other than its text, and&Sourceis no longerFreeze.Loggets: the three floods in the table below, when the six notes that the parser and the lockfile build withrange_datago throughLog::add_range_error_fmt_with_noteandLog::add_range_warning_fmt_with_note.range_data(js_parser: reject a second class constructor and a label that repeats an enclosing label #40793 adds one), and no address key (thebun installpositions and the token).Logand reroute the six notes. logger: key the line and column scan on a Source id, not on the address of its text #46277 then stays the fix for the address key.Why at the root. No user reported this class. The one-line input is a maintainer's synthetic repro, and the other inputs came from review. #31273 added one scan and #31341 made it four. Each left the next order of lookups quadratic. The comment on #31273: "Do not solve quadratic behavior by limiting the count. Solve it by fixing the underlying cause."
Which part fixes which input. The report was one long line with many errors. There the cost on main is the walk to the end of the line for each note. #44811 bounds that walk for every lookup and does not touch
Source: it alone fixes that input forBun.Transpilerandbun run. This PR bounds it only for a lookup from a checkpoint. The walk from byte 0 is the cost of the other inputs, and only this PR removes it.Inputs, release builds of main b1305dd and of this branch, Linux x64.
consts, notes at 40 early places, 128 / 256 / 512 KBletredeclared, notes at 6 places 60 KB apart, 0.6 / 1.2 MBlet x;and{var x;}blocks, note fromrange_data, 260 / 520 KBbun runof the 512 KB linebun buildof a 64 KB one-line filea_lookup_from_a_checkpoint_reads_a_bounded_part_of_the_sourcecounts the bytes. A run of continuation bytes where the excerpt is cut is read to its end, as on main.The three inputs
Cost. User-space instructions and cycles from
perf_event_open, minimum of 5 runs in turn (7 for the build), with a second copy of the main binary as control.perfandvalgrindare not in the build container.transformSyncof 2 KB of TypeScriptbun buildof 2,000 modules, two batchesBun.build)Macro.rs:1139builds a newSourcefor each result, so each error starts a new index and walks the 256 KB again. On main the address key resumed the scan of the call before. Every other producer has oneSourcefor a file.size_of(rustc, x86_64):Source112 to 120,Log48 to 40,Graph::InputFile192 to 200,transpiler::ParseResult1376 to 1384,parse_task::Success1152 to 1152. The boxed index has 272 bytes. The boxed tracker had 256.Sourceboxes the index. On main the first one of aLogboxes the tracker.range_dataon a source with no other diagnostic now allocates once, where main allocates nothing (AsyncModule.rs:753,:946,Package.rs:3171).Unexpected end of fileinside a character allocates none.Where a
Sourceis never dropped.Transpiler::parse_internal(src/bundler/transpiler.rs, the arena copy thatBun.Transpiler,bun run,importandrequireparse),parse_worker::run_with_source_code(src/bundler/ParseTask.rs, every bundler parse), and the rows ofgraph.input_filesinBundleV2::deinit_without_freeing_arena(linker diagnostics).JSTranspiler.rs:706and:1124andRuntimeTranspilerStore.rs:784are only avirtual_source, whichparse_internalclones.bundle_v2.rs:7363has no text.bun build --watchleaks itsBundleV2by design, and the error return ofscan_module_graph_from_cliends the process.detect_leaks=1,BUN_DESTRUCT_VM_ON_EXIT=1):bun run,requireandimport,bun buildwith a parse error and with a linker error,Bun.buildwith both,bun test,transformSync,scanandtransformreport nothing. With the three releases removed, all seven abort withDirect leak of 272 byte(s)fromBox<bun_ast::PositionIndex>.Positions.
a note that is found from a checkpoint has the line, column and text of a walk from the startruns at the interval that ships: notes at and across multiples of 1024 bytes, CR, CR LF, LF, U+2028 and U+2029 at 79 to 81 bytes after a name, and an end of file inside a character. It compares each position with one counted in the test.The address key. On main a reused buffer gives the next file the key of the last one. Two
file:folders with apackage.jsonof 260 bytes each:b/package.json:4:99for an error at1:138. A~/.bunfig.tomland abunfig.tomlof 5000 bytes each: main prints4e5f6g7h8i9j0k1l2m3n4o5p6q7r8", url = 5 }atbunfig.toml:3:38, because the stale line start is past thetokenkey that the masking looks for. With the index in theSourcethere is no key. The four tests of #46277 are here and fail on main. Not tested here: the manifests of lifecycle scripts (Package/Scripts.rs) and of a chain of local tarballs (processDependencyList.rs), which #46277 measured by hand.Not changed here.
lineText(logger: indent the caret relative to the windowed line excerpt #41658, logger: keep an excerpt of a line over 1,000 bytes wherever the error is #43313, logger: mask secrets in the source line before the excerpt window #43316).bun buildprints every error, and each caret line hascolumn - 1spaces. logger: indent the caret relative to the windowed line excerpt #41658 indents the caret by the excerpt.Sourceresume the old scans. The result stays in bounds. No producer does that.bun check(line_and_characterinsrc/sema/driver/lib.rs) andbun lint --format json(Utf16Offsetsinsrc/lint/linter/message.rs) have their own memos with the same kind of hole. Each has a separate task.Other PRs on these lines.
excerpt_line_endand the line ends that the scans keep go away here.Sourcean id to key the tracker. If the slot is accepted, this PR replaces it.range_data. It gets the index with no change.unsafeblocks frombun_sys, uWS, the HTTP client, the shell, the package manager,bun_jsc, the Web APIs,node:fsand the bundler #47948 also editsrc/ast/lib.rs.Self-review. Addressed: known answers from main in the unit tests, a test at the interval that ships, the line-end search on every kind of line break, test sizes that the Miri lane runs as written, a
Sendassertion for the index,#[must_use]on the release guard, cycles beside instructions, the rows for one error and for the macro, the four tests of #46277, and this body. Not done: the restack on #44811 (it is not merged), and one helper that places aSourcein an arena for all five arena sites with a lint against a barealloc(it waits for the answer on the field).Tests.
transformSync(three inputs),bun run,Bun.buildwith parse errors, andBun.buildwith a linker note in another file.parse error notesalso pins line, column andlineTextof the five notes that the parser builds withrange_dataorLocation::init_or_null. Those five pass on main.bun_asttakes about 70 s longer (15 tests in the position module, 4 before).test/bundler/transpiler/transpiler.test.js,test/js/bun/transpiler/,test/js/bun/jsonc/jsonc.test.ts,test/js/bun/resolve/resolve-error.test.ts,test/js/bun/util/inspect-error.test.js,test/cli/install/bun-install.test.ts,test/cli/install/redacted-config-logs.test.ts,test/bundler/bundler_loader.test.ts,test/bundler/esbuild/importstar.test.ts,test/bundler/esbuild/packagejson.test.ts,test/regression/issue/{03830,11793,12782,19850,23649},cargo miri test -p bun_ast,cargo clippy --workspace,rust:check-allfor Windows x64 and macOS arm64.no test proof · iteration 5 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/cli/install/bun-install.test.ts