Skip to content

js_parser: do not fold .length of a folded string with a non-ascii segment - #38989

Open
robobun wants to merge 1 commit into
mainfrom
farm/9dbc435e/rope-length-non-ascii
Open

robobun wants to merge 1 commit into
mainfrom
farm/9dbc435e/rope-length-non-ascii

Conversation

@robobun

@robobun robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Problem

  • With --minify-syntax, .length on a constant-folded string concatenation folds to the UTF-8 byte count when one piece is a non-ascii string stored as UTF-8: bun build --minify-syntax --define 'X="é"' turns (X + "x").length into 3, and `${X}x`.length into 3. The JS value is 2.
  • Non-ascii source literals are stored as UTF-16 and never join a rope, so this only affects strings bun injects as UTF-8 E::Strings: --define values, .env / process.env inlining, inlined import.meta.dir/file/path/url, stringified regexps and bigints.
  • Cause: EString::javascript_length (src/ast/e.rs) returned rope_len for any rope, but rope_len is the byte total of the segments (EString::push, fold_string_addition.rs) and nothing makes those bytes ascii. Template::fold flattens its rope with resolve_rope_if_needed, which leaves rope_len set, so a folded template took the same branch with non-ascii data.
  • src/js_parser/fold.rs carried a private copy of the function with the same bug; that copy is what the .length rewrite called.

Fix

  • javascript_length walks the segments of a UTF-8 string and returns None as soon as one contains a non-ascii byte; otherwise the byte length is the JS length. UTF-16 strings are unchanged (they are never ropes and already count code units).
  • Correct because a UTF-8 byte count equals the UTF-16 code unit count exactly when every byte is ascii. This is the check the non-rope UTF-8 branch already made, applied to every segment. None leaves .length in the output, which is what a non-rope non-ascii string already did (X.length printed as "é".length before this change).
  • All-ascii ropes still fold (("a" + "b").length -> 2); for them the walk is the is_all_ascii scan the non-rope case already did.
  • fold.rs calls the method instead of its copy, so there is one implementation.
  • Tests: test/bundler/transpiler/transpiler.test.js ("rewrite string to length with non-ascii utf-8 strings from define") and test/bundler/bundler_minify.test.ts (minify/StringLengthFoldingNonAsciiDefine). They cover a non-ascii define segment at the head of the rope, in the tail, the template literal path, a 4-byte (2 code unit) character, and ascii ropes still folding. Both fail on bun 1.4.0 (3 / 5 printed) and pass with this change.
  • Also ran transpiler.test.js, bundler_minify.test.ts and bundler_string.test.ts in full (291 pass), and cargo clippy -p bun_ast -p bun_js_parser (clean).

Background

  • Rope: fold_string_addition folds "a" + "b" by linking the right E::String onto the left one through its next / end pointers instead of copying bytes. rope_len caches the byte total of the chain; the printer (or resolve_rope_if_needed) flattens it later.
  • E::String stores its text either as UTF-8 bytes or as UTF-16 code units (is_utf16). len() is bytes in the first case and code units in the second. Only UTF-8 strings are joined into ropes; the lexer stores any source literal containing non-ascii as UTF-16, while defines and the other synthesized strings go through EString::init and stay UTF-8.
  • JS .length counts UTF-16 code units: "é" is 2 bytes of UTF-8 and 1 code unit, "😋" is 4 bytes and 2 code units.
Repro on bun 1.4.0
$ cat len.ts
console.log((process.env.NONASCII + "x").length);
console.log(process.env.NONASCII + "x");
console.log(`${process.env.NONASCII}x`.length);
console.log(("a" + "b").length);

$ bun build --no-bundle --minify-syntax --target=bun --define 'process.env.NONASCII="é"' len.ts
console.log(3);
console.log("\xE9x");
console.log(3);
console.log(2);

With this change:

console.log("\xE9x".length);
console.log("\xE9x");
console.log("\xE9x".length);
console.log(2);

EString::javascript_length returned rope_len for any rope, but rope_len
counts UTF-8 bytes, so `(DEFINE + "x").length` folded to the byte count
when the define value (or any other UTF-8 segment) was non-ascii. The
same happened to template literals folded through Template::fold, which
flattens the rope but keeps rope_len set.

Check every segment for ascii before using the byte length, the same
check the non-rope UTF-8 case already did, and make fold.rs call the
method instead of carrying its own copy of the logic.
@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 4 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e6ab13b6-994e-4e6d-8885-51dfbbcf01df

📥 Commits

Reviewing files that changed from the base of the PR and between 732491c and afe356b.

📒 Files selected for processing (4)
  • src/ast/e.rs
  • src/js_parser/fold.rs
  • test/bundler/bundler_minify.test.ts
  • test/bundler/transpiler/transpiler.test.js

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Reproduced on bun 1.4.0 with bun build --no-bundle --minify-syntax --define 'X="é"' on console.log((X + "x").length): prints console.log(3), JS value is 2. Same with `${X}x`.length and through new Bun.Transpiler({ define, minify: { syntax: true } }).

Fix is in this PR: EString::javascript_length checks every rope segment for ascii before using the byte length, and fold.rs uses it instead of its own copy. Tests: test/bundler/transpiler/transpiler.test.js and test/bundler/bundler_minify.test.ts (minify/StringLengthFoldingNonAsciiDefine), both fail on 1.4.0 and pass with the change.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small, correct fix that makes .length folding bail on non-ASCII UTF-8 rope segments instead of returning the byte count.

What was reviewed:

  • javascript_length rewrite: verified all previous cases (UTF-16, non-rope UTF-8 ascii/non-ascii, all-ascii rope) return the same result; only non-ascii ropes and flattened-but-still-rope_len-set templates now return None.
  • Confirmed the deleted fold.rs shim was the only duplicate and the direct str_.javascript_length() call is unambiguous (single inherent method).
  • Third caller in react_compiler/diagnostics/js_string.rs already handles None with a fallback.
  • Tests cover head/tail non-ascii segments, template path, 4-byte codepoint, and ascii ropes still folding.
Extended reasoning...

Overview

Rewrites EString::javascript_length in src/ast/e.rs to walk every rope segment and return None when any UTF-8 segment contains a non-ASCII byte, instead of blindly returning rope_len (a byte count). Deletes the private duplicate e_string_javascript_length in src/js_parser/fold.rs and calls the shared method. Adds tests in bundler_minify.test.ts and transpiler.test.js.

Security risks

None. Pure constant-folding logic in the minifier; no untrusted input handling, allocation, or FFI.

Level of scrutiny

Low-medium. The function is ~10 lines and the change is a strict tightening: every case that previously returned a correct Some(n) still does (UTF-16 → slice16().len(); all-ASCII UTF-8, roped or not → len() which equals byte count which equals code-unit count for ASCII). The only behavioral change is that non-ASCII UTF-8 ropes (and templates flattened via resolve_rope_if_needed, which leaves rope_len set) now return None instead of the wrong byte count. None is always safe — it just leaves .length in the output.

Other factors

  • Traced len() and rope_len invariants: push sets rope_len to the byte total and always sets next on the head; resolve_rope_if_needed flattens data to match rope_len before clearing next. So Some(self.len()) is correct for the all-ASCII case whether the rope is resolved or not.
  • Checked the third caller (react_compiler/diagnostics/js_string.rs::len_utf16) — it already falls back on None, and only sees more Nones where it previously got wrong answers.
  • Tests exercise the variant matrix (non-ascii at head, in tail, as sole segment, template path, emoji surrogate pair) and confirm all-ascii ropes still fold to a number. PR description states both fail on 1.4.0 and pass with the change.
  • The fold.rs shim deletion is exactly the "one implementation" cleanup REVIEW.md asks for; verified only one fn javascript_length remains so the direct method call compiles.

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 7:05 AM PT - Aug 15th, 2026

❌ @robobun, your commit afe356b has some failures in Build #97902 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 38989

That installs a local version of the PR into your bun-38989 executable, so you can run:

bun-38989 --bun

@robobun

robobun commented Aug 15, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 10:01 AM PT - Aug 15th, 2026

✅ @robobun, your commit afe356bcbe00287b106816324eac9e44fc6145ac passed in Build #97902! 🎉


🧪   To try this PR locally:

bunx bun-pr 38989

That installs a local version of the PR into your bun-38989 executable, so you can run:

bun-38989 --bun

@robobun

robobun commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator Author

Two open PRs cover the .length fold of this PR.

This PR merges cleanly with main 3f3a9ff. It conflicts with each of the two in src/ast/e.rs and src/js_parser/fold.rs. If #46915 or #47655 merges, the parser fold of this PR is not needed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant