Skip to content

replMode: decide between an object literal and a block in the parser - #43770

Closed
robobun wants to merge 5 commits into
mainfrom
robobun/62c33db8/transpiler-async-repl-object-literal
Closed

robobun wants to merge 5 commits into
mainfrom
robobun/62c33db8/transpiler-async-repl-object-literal

Conversation

@robobun

@robobun robobun commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • With replMode: true, transformSync() and scan() parse ( + input + ) if the input starts with { and does not end with ;. A block fails: transformSync("{ let x = 1; x }") throws Expected "}" but found "x".
  • transform() and scanImports() parse the input as written (src/runtime/api/JSTranspiler.rs). An object literal fails: transform("{a: 1, b: 2}") throws Expected ";" but found ":".

Fix

  • Parser::parse_repl_expression_or_statements (src/js_parser/parse/parse_entry.rs) reads such an input as one expression. If that fails, a second parser reads statements. The copies are deleted.
  • Correct because each reading has its own parser, options and log. Error positions are in the input: { a: } fails at 1:6, not 1:7.
  • bun repl asks for the expression only and still evaluates a block as written.
  • Verified: test/js/bun/transpiler/repl-transform.test.ts (bun 1.4.3 fails 55 of 106), test/js/bun/repl/repl.test.ts. Self-reviewed: 13 concerns raised, 13 addressed.

Background

  • replMode is a Bun.Transpiler option for REPL input. Node's REPL evaluates (input) first, so that { a: 1 } is an object, then the input as written.
  • Considered a retry in JSTranspiler.rs: it keeps the copy and the shifted positions. No user reported this.

Downsides

  • Each parsed file pays one never-taken repl_mode test in Parser::_parse (+2 instructions). The binary grows by 10.5 KB.
  • With replMode, a block costs two more parser setups: +22 allocations per transform() call, +3 µs per scanImports() call.
  • {a: 1} // note is an object literal (Node prints 1). transformSync("{a:1}) + (2") now fails.
Notes

Repro (columns: transformSync, transform, scan, scanImports, with new Bun.Transpiler({ loader: "ts", replMode: true })):

bun 1.4.3
"{ let x = 1; x }"         REFUSES takes REFUSES takes
"{ foo() }"                REFUSES takes REFUSES takes
"{ if (a) b }"             REFUSES takes REFUSES takes
"{ setup(); }\nrender()"   REFUSES takes REFUSES takes
"{a: 1}; 3"                REFUSES takes REFUSES takes
"{a: 1} // note"           REFUSES takes REFUSES takes
"{a: 1, b: 2}"             takes REFUSES takes REFUSES
"{a: 1, b: require('y')}"  takes REFUSES takes REFUSES
"{a, b}"                   takes takes takes takes

this PR: every cell is "takes". "{ a: }" is refused by all four on both builds.

1,500 generated inputs that start with a brace (a seeded generator: blocks, object literals, and 16 suffixes such as ;, // note, .a, \nrender()). 1,019 of them are valid as statements or as one expression.

transformSync transform scan scanImports
takes, bun 1.4.3 444 643 444 644
takes, this PR 1019 1019 1019 1062
lost against 1.4.3 0 0 0 0
valid but refused, 1.4.3 575 376 575 376
valid but refused, this PR 0 0 0 0
  • transform() returns the same code as transformSync(), or both refuse: 564 of 1,500 on 1.4.3, 1,500 with this PR.
  • The four methods agree on takes or refuses: 632 on 1.4.3, 1,457 with this PR. The other 43 are invalid inputs that only scanImports() takes, because it has no visit pass (1 on 1.4.3).
  • The output is the one the reading gives on 1.4.3 for all 1,500: the code of transform() where the input is statements, and the code of transformSync("(" + input + "\n)") where it is one expression.

Measured cost (release builds for linux-x64 of main 177da66 and of this PR at fc508ca)

  • replMode off: Parser::_parse gets 2 instructions in front of the top-level statement list (cmpb $0x0, <repl_mode>, jne), and none after it. This is from the disassembly (llvm-objdump) of both builds, for _parse::<false> and _parse::<true>. The build machine has no perf counters and no valgrind, so there is no instruction count of a whole run. The frame of _parse::<false> is 10,008 B (was 9,992 B). The frame of _parse::<true> stays at 9,584 B.
  • Size: .text 80,719,717 B to 80,730,213 B (+10,496 B, size -A). The stripped binary grows by 8,192 B. The new code is cold: read_repl_leading_brace (2 copies of 1,525 B), read_repl_expression (4 copies of 825 B), and +658 B and +1,431 B in the two copies of _parse.
  • Allocations per call (calls of the mimalloc allocation functions counted with gdb, the difference between 1,500 and 500 calls):
input transformSync transform scan scanImports
replMode off, const a: number = 1 26 to 26 32 to 32 24 to 24 10 to 10
replMode on, const a: number = 1 43 to 43 49 to 49 41 to 41 10 to 10
replMode on, {a: 1, b: 2} 44 to 45 threw, now 51 42 to 43 threw, now 11
replMode on, { foo() } threw, now 62 46 to 68 threw, now 67 11 to 28
  • Parser entries per call with replMode on: 1 for an input that does not start with {, 2 for an object literal, 3 for a block. The first entry only starts the others. The expression reading of a block stops at the first token that is not the expected one (Lexer::expected_is_fatal), so a block does not pay a full parse pass.
  • Time per call (7 interleaved rounds of main, PR, main, median, with the difference between the two runs of main as the noise):
    • { let x = 1; x }: scanImports() 8.4 µs to 11.6 µs (noise 1.1%). transform() 131 µs to 144 µs (noise 4.4%).
    • {a: 1, b: 2}: transformSync() 13.7 µs to 13.9 µs (noise 0.9%).
    • A block of 181 KB: transform() +3.5% (noise 1.1%), scanImports() +0.8% (noise 2.9%). An object literal of 181 KB: transformSync() -2.4% (noise 3.0%).

Which errors, if the input is not statements either

  • The errors of the statements, at positions in the input. {a: 1, b: } is Expected ";" but found ":" at 1:9.
  • The errors of the expression, if only the visit pass rejects it: {a: () => { const c = 1; c = 2 }, b: 2} is Cannot assign to "c" because it is a constant.
  • The errors of the expression, if the input ends inside it and the first error of the statements is earlier: { a: 1, b: 2, is Expected identifier but found end of file. A REPL that reads input line by line can look for end of file to ask for the next line. dannote/just-bun (lib/console.ts) does that with transformSync(). A run of 28 object literals typed line by line reaches the last line in 25 on 1.4.3 and in 28 with this PR. For 16 blocks the numbers are 2 and 14.
  • Where both builds refuse an input of the 1,500, the error text differs for 437 inputs in transformSync() and 42 in transform().

Differences from the REPL of Node.js v26.3.0 (lib/repl.js L458-L462 and L550-L561)

  • {a: 1} // note: Node prints 1, because the comment takes the ). Here it is an object literal in the four methods and in bun repl. bun repl 1.4.3 prints 1.
  • { await x; y }: Node rejects it. Here it is a block, as in transform() and scanImports() on 1.4.3.
  • If both readings fail and _=input is valid, Node reports the error of (input). Here the rule above decides.

bun repl

  • Interactive output is the same as 1.4.3 for blocks and object literals ({ let x = 1; x } prints 1, { function f() {} } then typeof f prints "function"), with two changes: { a: 1 } // note prints the object (was 1), and {a:1}) + (2 is a SyntaxError (was "[object Object]2").
  • Of 50 bun repl -e scripts, 1 changes: {a: 1} // note exits with code 0 (was 1).

What the four methods still do not share

  • scanImports() has no visit pass. It keeps the expression reading of {}\n/require('a')/g.x = 1 and returns the require-call.
  • scanImports() builds its own parser options. For example it rejects class A { accessor x = 1 }, with and without replMode.

Open question for a maintainer: Parser::_parse gets one test of repl_mode in front of the top-level statement list. There are three places for the decision.

  1. This PR: the test in _parse (+2 instructions per parsed file, never taken without replMode). The parser already tests repl_mode once per file in P::to_ast (src/js_parser/p.rs).
  2. No change in the parser: keep ( + input + ) and add a second parse at the three sites in JSTranspiler.rs and in repl.rs. The copy and the shifted positions stay.
  3. A const parameter on _parse and _scan_imports: no test at run time, and a second copy of both functions (about 59 KB for _parse, about 10 KB for _scan_imports).

Out of scope, reported separately

  • bun repl -e and -p exit with code 1 for a block that ran.
  • Bun.Transpiler.scan() lists no import() when dead code elimination is off, which replMode turns off.
  • The debug console of the VS Code extension puts a { input in parentheses with no second attempt.
  • bun repl gives a statement no completion value, so a block cannot go through the REPL transform yet. That is why bun repl asks for the expression reading only.

Open PRs that touch the same parse sites in JSTranspiler.rs: #40405, #43771, #30540, #37601, #42433, #42430. This PR returns TransformTask::run to the text of main and removes lines from get_parse_result.


[human-review] gate passed · iteration 0 · 3 files touched

fails on main (without fix)
ASAN without fix: 17 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/bun/transpiler/repl-transform.test.ts
bun test v1.4.3 (367d939d9)

test/js/bun/transpiler/repl-transform.test.ts:
(pass) Bun.Transpiler replMode > basic transform output > simple expression wrapped in value object [3.67ms]
(pass) Bun.Transpiler replMode > basic transform output > variable declaration with await [2.92ms]
(pass) Bun.Transpiler replMode > basic transform output > const becomes var with await [3.19ms]
(pass) Bun.Transpiler replMode > basic transform output > let becomes var with await [4.18ms]
(pass) Bun.Transpiler replMode > basic transform output > no async wrapper when no await [5.11ms]
(pass) Bun.Transpiler replMode > basic transform output > function declaration with await [2.94ms]
(pass) Bun.Transpiler replMode > basic transform output > class declaration with await [3.10ms]
(pass) Bun.Transpiler replMode > basic transform output > top-level `this` is preserved (not rewritten to `exports`) [2.98ms]
(pass) Bun.Transpiler replMode > basic transform output > `this` inside a nested call is preserved [3.24ms]
(pass
... (truncated)

release without fix: 17 FAILED
bun test v1.4.3-canary.1 (367d939d9)

test/js/bun/transpiler/repl-transform.test.ts:
(pass) Bun.Transpiler replMode > basic transform output > simple expression wrapped in value object [0.12ms]
(pass) Bun.Transpiler replMode > basic transform output > variable declaration with await [0.06ms]
(pass) Bun.Transpiler replMode > basic transform output > const becomes var with await [0.03ms]
(pass) Bun.Transpiler replMode > basic transform output > let becomes var with await [0.03ms]
(pass) Bun.Transpiler replMode > basic transform output > no async wrapper when no await [0.14ms]
(pass) Bun.Transpiler replMode > basic transform output > function declaration with await [0.06ms]
(pass) Bun.Transpiler replMode > basic transform output > class declaration with await [0.04ms]
(pass) Bun.Transpiler replMode > basic transform output > top-level `this` is preserved (not rewritten to `exports`) [0.03ms]
(pass) Bun.Transpiler replMode > basic transform output > `this` inside a nested call is preserved [0.03ms]
(pass) Bun.Transpiler replMode > REPL session with node:vm > simple expression returns value object [0.52ms]
(pass) Bun.Transpiler replMode > REPL session with node:vm > arit
... (truncated)
passes on PR (with fix)
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/js/bun/transpiler/repl-transform.test.ts
bun test v1.4.3 (367d939d9)

test/js/bun/transpiler/repl-transform.test.ts:
(pass) Bun.Transpiler replMode > basic transform output > simple expression wrapped in value object [3.51ms]
(pass) Bun.Transpiler replMode > basic transform output > variable declaration with await [3.56ms]
(pass) Bun.Transpiler replMode > basic transform output > const becomes var with await [3.73ms]
(pass) Bun.Transpiler replMode > basic transform output > let becomes var with await [3.28ms]
(pass) Bun.Transpiler replMode > basic transform output > no async wrapper when no await [4.92ms]
(pass) Bun.Transpiler replMode > basic transform output > function declaration with await [3.26ms]
(pass) Bun.Transpiler replMode > basic transform output > class declaration with await [3.43ms]
(pass) Bun.Transpiler replMode > basic transform output > top-level `this` is preserved (not rewritten to `exports`) [2.86ms]
(pass) Bun.Transpiler replMode > basic transform output > `this` inside a nested call is preserved [3.30ms]
(pass
... (truncated)

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped)
  target       linux-x64-gnu
  build type   Release
  build dir    ./build/release
  revision     edd2745202
  features     lto, baseline

23 deps, 136 codegen, 1176 objects in 747ms

ninja: Entering directory `/workspace/bun/build/release'
[1/4] fetch lolhtml
[lolhtml] up to date
[2/4] fetch rust-argon2
[rust-argon2] up to date
[2/4] cargo plan → /workspace/bun/build/release/rust-target/plan.json
244 units: 172 lib, 16 proc-macro (host), 19 custom-build (host), 15 run custom-build, 17 lib (host), 4 run custom-build (host), 1 rlib
[3/4] reconfigure
[1/1499] mkdir stamps
[2/1499] mkdir codegen
[3/1499] install /workspace/bun
bun install v1.4.3-canary.1 (367d939d9)

Checked 26 installs across 65 packages (no changes) [24.00ms]
[4/1499] install /workspace/bun/packages/bun-error
bun install v1.4.3-canary.1 (367d939d9)

Checked 1 install across 2 packages (no changes) [1.00ms]
[5/1499] install /workspace/bun/src/node-fallbacks
bun install v1.4.3-canary.1 (367d939d9)

Checked 111 installs across 104 packages (no changes) [5.00ms]
[6/1499] rustc unicode_ident 
[7/1499] gen node-fallbacks
... (truncated)
diff hotspot
src/runtime/api/JSTranspiler.rs               | 64 +++++++++++++--------
 src/runtime/cli/repl.rs                       | 21 +------
 test/js/bun/transpiler/repl-transform.test.ts | 82 +++++++++++++++++++++++++--
 3 files changed, 120 insertions(+), 47 deletions(-)

gate history · 1 passed · 0 rejected · iteration 0

evidence per changed file
file                                           reads  edits  tests
src/runtime/api/JSTranspiler.rs                    8     14     24
src/runtime/cli/repl.rs                            1      1     20
test/js/bun/transpiler/repl-transform.test.ts      4      6     20

In replMode, an input that looks like an object literal is parsed inside
parentheses, so `{ a: 1 }` is an expression. Only get_parse_result() did
this, so only transformSync() and scan() had it. transform() and
scanImports() build their own source. transform() parsed `{ a: 1 }` as a
block with a label, and both methods threw a syntax error for
`{a: 1, b: 2}`.

get_parse_result() also wrapped the input of a data loader. In replMode,
transformSync('{"a":1}', "json") threw `Unexpected (`, and the text
loader returned "({hi})".

repl_mode_source() now decides for all four methods. It wraps only for a
JavaScript-like loader. The wrap itself is wrap_likely_object_literal(),
which the CLI REPL also calls in place of its own copy.
@robobun

robobun commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for review. The design changed since the first version: the parser now decides between an object literal and a block, and no method parses a copy of the input in parentheses.

How to reproduce on bun 1.4.3:

const t = new Bun.Transpiler({ loader: "ts", replMode: true });
t.transformSync("{ let x = 1; x }"); // throws: Expected "}" but found "x"
await t.transform("{ let x = 1; x }"); // a block
t.transformSync("{a: 1, b: 2}"); // the value wrapper around { a: 1, b: 2 }
await t.transform("{a: 1, b: 2}"); // throws: Expected ";" but found ":"
t.scanImports("{a: 1, b: require('y')}"); // throws the same error

With this PR the four methods read each input the same way: a block is a block and an object literal is an object literal.

USE_SYSTEM_BUN=1 bun test test/js/bun/transpiler/repl-transform.test.ts fails 55 of 106 tests. bun bd test on this branch passes all 106.

One question is open for a maintainer (in the Notes of the description): is one never-taken test of repl_mode per parsed file in Parser::_parse acceptable (+2 instructions), or must the parser stay as it is.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

We couldn't safely recover the incremental review. No full review was started, and the last reviewed checkpoint was preserved. Retry later, or explicitly request a full review by commenting @coderabbitai full review.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: oven-sh/bun/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Essentials

Run ID: 3349bbb8-62a2-4fd2-8a94-e65df30403a5

📥 Commits

Reviewing files that changed from the base of the PR and between 1016a7a and edd2745.

📒 Files selected for processing (3)
  • src/runtime/api/JSTranspiler.rs
  • src/runtime/cli/repl.rs
  • test/js/bun/transpiler/repl-transform.test.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 5 remain after this review.

Walkthrough

Changes

The transpiler now centralizes REPL source preprocessing. JavaScript-like loaders wrap likely object literals in REPL mode, while non-JavaScript loaders retain raw source. Transform, parsing, scanning, CLI behavior, and tests use the shared logic.

REPL preprocessing

Layer / File(s) Summary
Shared preprocessing helpers
src/runtime/api/JSTranspiler.rs
Adds loader-aware preprocessing and exposes wrap_likely_object_literal. is_likely_object_literal is now private.
Runtime and CLI integration
src/runtime/api/JSTranspiler.rs, src/runtime/cli/repl.rs
Applies shared preprocessing to transforms, parsing, import scanning, and CLI REPL transformation.
REPL API validation
test/js/bun/transpiler/repl-transform.test.ts
Adds coverage for synchronous and asynchronous transforms, scanning, byte input, syntax errors, non-JavaScript loaders, and non-REPL parsing.

Suggested reviewers: dylan-conway

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to edd27

The implementation is mergeable with the parameterized-test structure updated to follow the repository test convention.

Pre-merge checks: Passed 4
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the main change: deciding whether REPL input is an object literal or a block during parsing.
Description check Passed The description explains the problem, fix, behavior changes, trade-offs, scope, and verification results. It does not use the exact template headings, but it provides the required information in equiv…

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline finding, I also checked the two arena-lifetime paths the wrap newly creates: in TransformTask::run the wrapped bytes die with the per-call arena before the rejection is built on the JS thread, but Location::init_or_null_impl (src/ast/lib.rs) stores line_text as Cow::Owned and file borrows the static stdin name, so the deferred log.to_js reads no arena memory; in scan_imports, named_imports_to_js consumes the import records before arena drops. Neither is affected.

Extended reasoning...

The change factors the replMode object-literal wrap in src/runtime/api/JSTranspiler.rs into repl_mode_source/wrap_likely_object_literal and applies it to transform(), transformSync(), scan(), and scanImports(), with the CLI REPL in src/runtime/cli/repl.rs sharing the helper and allocating in the parse arena; no security-sensitive surface is touched. One inline finding (error positions shifted by the injected paren for transform/scanImports) is already posted, so this note only records the lifetime checks that were ruled out.

Comment thread src/runtime/api/JSTranspiler.rs Resolved
With repl_mode, an input that starts with `{` and does not end with `;`
is read as one expression by a parser of its own. If the input is not one
expression, or the visit pass rejects it, another parser reads it as
statements. Bun.Transpiler and bun repl no longer parse a copy of the
input inside parentheses. A block is not a syntax error in transformSync()
and scan(), an object literal is not a block in transform() and
scanImports(), and an error position is a position in the input.

bun repl asks for the expression reading only. It still evaluates a block
as written.
@robobun
robobun marked this pull request as draft October 14, 2026 05:24
Comment thread src/js_parser/parse/parse_entry.rs Resolved
Comment thread src/js_parser/parse/parse_entry.rs Resolved
Comment thread src/js_parser/parse/parse_entry.rs Resolved
Comment thread src/js_parser/parse/parse_entry.rs Resolved
@robobun

robobun commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 2:51 AM PT - Oct 14th, 2026

✅ @robobun, your commit 002e3d7ec3137a1fca59a0098abadb1bb81d9b6d passed in Build #125404! 🎉


🧪   To try this PR locally:

bunx bun-pr 43770

That installs a local version of the PR into your bun-43770 executable, so you can run:

bun-43770 --bun

Comment thread src/js_parser/parse/parse_entry.rs Resolved
Comment thread src/js_parser/repl_transforms.rs Resolved
Comment thread src/js_parser/repl_transforms.rs Resolved
Comment thread src/runtime/cli/repl.rs Resolved
@robobun robobun changed the title Bun.Transpiler: apply the replMode object literal wrap in every method replMode: decide between an object literal and a block in the parser Oct 14, 2026
The reading of a block as one expression is dropped, so the parser does
not continue after the first token that is not the expected one.
@robobun
robobun marked this pull request as ready for review October 14, 2026 08:42
@robobun
robobun requested a review from alii as a code owner October 14, 2026 08:42

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline finding, I also checked error attribution when both readings fail in src/js_parser/parse/parse_entry.rs (parse_repl_expression_or_statements / scan_imports_of_repl_expression_or_statements): the reported log now carries positions in the caller's own input with no (-wrapper shift, and the statements-first rule is the deliberate, documented choice, so it is not a regression. expected_is_fatal is reset on every exit of read_repl_expression, and core::mem::take(&mut p.options) is backed by the existing Options: Default impl.

Extended reasoning...

The change moves the REPL object-literal-vs-block decision from two textual (+code+) wrappers (CLI REPL and Bun.Transpiler) into the parser via a new ReplLeadingBrace option, a scratch-log two-reading strategy in _parse and _scan_imports, and a lexer expected_is_fatal flag; it touches no security-sensitive surface. One confirmed finding is posted inline; this note records only the additional concerns examined and ruled out this run.

Comment thread src/runtime/cli/repl.rs Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-10-14.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant