Skip to content

parser: key the memo of the a ? (b) : c => d attempt by what the attempt reads - #45939

Closed
robobun wants to merge 6 commits into
mainfrom
robobun/d526420d/conditional-arrow-memo-key
Closed

robobun wants to merge 6 commits into
mainfrom
robobun/d526420d/conditional-arrow-memo-key

Conversation

@robobun

@robobun robobun commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Fix

  • The key is the : offset and arrow_data.allow_await. The attempt allows private names everywhere. The real parse still reports one outside a class.
  • The attempt reads further now, so the snapshot also restores has_import_meta, has_with_scope and needs_jsx_import.
  • Verified: test/bundler/transpiler/transpiler.test.js (10 new inputs and one new test, each fails on main). Self-reviewed: 6 concerns raised, 6 addressed.

Background

Downsides

  • Input that compiled only through a stale outcome is an error now (Notes).
  • The frame of parse_paren_expr grows by 32 B in TypeScript: nesting is 2% less deep (((((1)))): 4,713 to 4,629).
  • A debug build runs the attempt again on each memo hit and asserts the outcome. Nested attempts are about 2x slower there.
Notes

Inputs

Save each as a .ts file and run bun build --no-bundle <file>. bun run cannot show the difference: it fails before and after with Cannot use 'await' as an identifier in a module.

// t1.ts
x = y ? async (h) : R => await /x? class { p = /.y/g.test(s) ? (a) : T => #q in o : e; #q = 1 } : no;

// t2.ts
x = y ? async (h) : R => await /x? class { #p = 1; static async m(){ q = (p = /.y/g.test(s) ? (a) : T => #p in o : e) => body; } } : no;

// t3.ts (two lines)
x = y ? async (h) : R => q ? (p = await /x? async (a, c = /.y/g) : T => await
z : no) : w;
file main (017b243) this PR tsc 6.0.2
t1.ts error: Expected ";" but found ":" p = /.y/g.test(s) ? (a) => (#q in o) : e; the same tree
t2.ts q = (p = /.y/g.test(s) ? a : (T) => (#p in o)) => body;, no error q = (p = /.y/g.test(s) ? (a) => (#p in o) : e) => body; the same tree
t3.ts error: "await" can only be used inside an "async" function async (a, c = /.y/g) => await z the same tree

The inputs are built for the memo: they need await before a / in the body of a discarded async (…) : attempt. The test has seven more of this kind. One of them is an error with this PR and was printed without its : g on main:

x = y ? async (h) : R => await /x? class { #q = 1; m() { q = c1 ? /.y/g.test(s) ? (a) : T => #q in o : g; } } : no;

tsc rejects it too, and main rejects it without the await /x? … / prefix. Inputs of this kind compiled on main only through the stale outcome. Some of them are accepted by tsc and depend on a second bug ((a as T) : U => … is taken for an arrow function). Those come back with the change that fixes that bug, not here.

How one : is reached twice

In t1.ts the parser first tries async (h) : R => … as an async arrow function. In that attempt await is a keyword, so /x? class { p = / is a regular expression, and the text after it is .y / g.test(s) ? (a) : T => #q in o : e. The attempt for (a) : runs outside a class. No : follows the body of async (h), so that attempt is discarded. The real parse reads async(h) as a call, and await / x ? class { … } : no as a division and a conditional. It reaches the same (a) : inside the class and takes the stored outcome.

On main the attempt outside the class fails on #q, and the real parse took "not an arrow function" from the memo. t3.ts is the same with the other state: the attempt reaches : T after (p = await /x? async (a, c = /.y/g), which is not async, and the real parse reaches it after async (a, c = /.y/g).

Why this key

The attempt parses a return type and an arrow function body. Besides the source after the :, the body reads:

  • arrow_data.allow_await: parse_arrow_body makes it the state of the body. It is in the key.
  • allow_private_identifiers, for #q in o. tsc's parser has no such state, and esbuild removed it (v0.25.8). The attempt now sets it, so its outcome does not depend on the class around it. A private name outside a class is an error of the real parse, as before.
  • allow_in. It is always true at this point: the ? sets it for the true branch, and the mark that starts the attempt is set at the same place. A debug assertion states it, and the key leaves it out.
  • The await and yield state of the enclosing function, and its this and super flags. parse_arrow_body replaces the first two. The other flags only log errors, and a logged error does not change the outcome.

Check with a build of the parser that is not committed: at each attempt, compute the outcome again with the memo off, under each of 864 combinations of the states that are not in the key (await and yield of the enclosing function, is_this_disallowed, allow_super_call, allow_super_property, is_top_level, is_return_disallowed, strict mode of the scope). 51,347 attempts, 44,363,808 runs: no outcome changes, and allow_in is true at all of them. (Run on the version of this PR that still had the private name state in the key.)

Other parsers: tsc's notParenthesizedArrow is a set of token offsets, with no async state. tsc 6.0.2 reports TS1005 for x = y ? async (h) : R => await /x + /.y ? z/g ? 0 : (a) : T => d;. Main and this PR print x = y ? async(h) : (R) => await / x + /.y ? z/g ? 0 : (a) => d;.

Not in this PR

Checks

  • A debug build of the parser alone (assertions on, so each memo hit of the real parse is computed again): 410,492 generated inputs print the same as main. The 3,781 TypeScript files of this repo parse with no assertion.
  • Mutations of the new code on a copy of the parser, each run with the debug check and without it: the key without the async state, the attempt without the private name line, two wrong shifts of the offset, a lookup that never hits, an outcome that is not stored, nothing stored, an entry appended out of order, a hit that reads entry 0, a hit that reads the neighbour entry, a hit that returns the other outcome, and the ? without allow_in. The inputs of the changed test block catch 11 of 12 in both modes. The twelfth (the offset shifted by one bit, not two) is not a defect: one bit holds the two values that allow_await takes here.
  • Release builds of main (99ee03e) and this PR (8b3724c):
    • Output of the tracked files of this repo (3,893 TypeScript and 7,999 JavaScript files): identical.
    • size -A: .text 82,170,617 to 82,171,129 B. nm -S: parser symbols +389 B (parse_paren_expr +171 B and +122 B in the two TypeScript copies, restore_parser_snapshot +48 B each). The JavaScript copies do not change.
    • Stack frame of parse_paren_expr (sub rsp plus pushes): 1,088 B to 1,120 B, and 1,104 B to 1,120 B in the scan-only copy (TypeScript). 448 B and 464 B in JavaScript, unchanged. The attempt is inlined into it, as on main.
    • Deepest input that parses, default stack: parentheses 4,713 to 4,629, chains of attempts 3,644 to 3,593. At a 1 MB stack (ulimit -s 1024): parentheses 504 to 495, chains of attempts 389 to 384.
    • An earlier version of this PR had the attempt in a #[cold] function of its own. That took the snapshot out of the frame of parse_paren_expr (1,088 B to 736 B, parentheses 505 to 633 at 1 MB), and made chains of attempts less deep (390 to 350). A review asked for no such trade, so the attempt is inlined again.
    • test/bundler/transpiler/transpiler.test.js, debug build: 259 pass with this PR. With main, the changed block and the new test fail.
  • The debug parser on a chain of 320 nested attempts: 280 ms and 81 MB on main, 670 ms and 148 MB with this PR (the second run of each attempt).
  • Tests with the debug build of 8b3724c: test/bundler/transpiler/transpiler.test.js (259 pass), test/bundler/esbuild/ts.test.ts (59), test/cli/run/transpiler-cache.test.ts (24), test/bundler/transpiler/runtime-transpiler.test.ts (15), decorators.test.ts (25), decorator-metadata.test.ts (5), es-decorators.test.ts (427), scope-mismatch-panic.test.ts (21), test/bundler/bundler_regressions.test.ts (9), jsx-production.test.ts (32).

Between the "?" and ":" of a conditional, the TypeScript parser tries
"(b) : T => body" as an arrow function and keeps it only if a ":" follows
the body (#40818). The outcome is stored in `ts_conditional_arrow_attempts`
by the offset of the ":" alone. The attempt also reads two parser states:
whether the arrow function is async, and whether private names are allowed.

One ":" can be reached in two states. A discarded `async (h) :` attempt
reads `await /x.../` in its body as a regular expression, and the real
parse reads a division. The text after it can then belong to a class in
one reading and not in the other, or follow an `async (` head in one
reading and a plain `(` head in the other. The real parse took the outcome
of the other state: an error for valid input, or an arrow function that
silently lost its ": e".

The key now holds the offset of the ":" and `arrow_data.allow_await`. The
attempt allows private names wherever it runs, so that its outcome does
not depend on the class around it. The real parse still reports a private
name outside a class. `allow_in` is always true at this point, and a debug
assertion says so. A debug build also computes the attempt again for each
memo hit of the real parse and compares.

The attempt without the memo is its own cold function, so the parser
snapshot is no longer in the stack frame of `parse_paren_expr`.
@robobun

robobun commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator Author

Status

Reproduced on main (017b243) with a release build. Save each input as a .ts file and run bun build --no-bundle <file>:

  • x = y ? async (h) : R => await /x? class { p = /.y/g.test(s) ? (a) : T => #q in o : e; #q = 1 } : no; fails with Expected ";" but found ":"
  • x = y ? async (h) : R => await /x? class { #p = 1; static async m(){ q = (p = /.y/g.test(s) ? (a) : T => #p in o : e) => body; } } : no; prints q = (p = /.y/g.test(s) ? a : (T) => (#p in o)) => body;, with no error and without the : e

With this PR (#45939) both print as tsc 6.0.2 reads them. The third input (two lines) is in the PR description.

@robobun

robobun commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:26 AM PT - Oct 27th, 2026

✅ @robobun, your commit 8b3724c68efc6ca2403ba6fbb6977f33e86dc4c0 passed in Build #133462! 🎉


🧪   To try this PR locally:

bunx bun-pr 45939

That installs a local version of the PR into your bun-45939 executable, so you can run:

bun-45939 --bun

…itional-arrow-memo-key

# Conflicts:
#	src/jsc/RuntimeTranspilerCache.rs
@robobun
robobun marked this pull request as ready for review October 22, 2026 02:21
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 67a9a61 and 8b3724c.

Files selected for processing: 4
  • src/js_parser/p.rs
  • src/js_parser/parse/parse_skip_typescript.rs
  • src/jsc/RuntimeTranspilerCache.rs
  • test/bundler/transpiler/transpiler.test.js

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

Walkthrough

The TypeScript parser now keys speculative conditional-arrow attempts by colon offset and parser state. Parser snapshots restore additional flags. Tests cover nested parsing contexts and discarded JSX attempts. The runtime transpiler cache format version changes to 44.

Changes

TypeScript conditional-arrow parsing

Layer / File(s) Summary
Memo key and speculative parse
src/js_parser/p.rs, src/js_parser/parse/parse_skip_typescript.rs
The memo key includes allow_await, and the parser checks allow_in. Parser snapshots capture and restore additional flags. A helper performs the speculative parse. Stack overflow and allocation errors propagate; other parse errors produce false.
Regression cases and cache version
test/bundler/transpiler/transpiler.test.js, src/jsc/RuntimeTranspilerCache.rs
Tests cover nested parses across differing contexts, parenthesized and async arrow heads at depth 40, and discarded JSX attempts. The runtime transpiler cache format version changes from 43 to 44.

Suggested reviewers: jarred-sumner

Priority: Normal

Merge Risk: Minimal · up to 8b372

The parser now distinguishes await state and restores flags from discarded attempts. No merge-blocking risk was identified.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title names the change accurately: the memo for the a ? (b) : c => d speculative attempt is re-keyed by the state the attempt reads. The raw summary confirms the memo key now includes the colon …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is complete and relevant. It explains the problem, fix, background, downsides, and verification results. It does not use the template headings exactly, but it provides the required inf…
Full details: Title check

Explanation

The title names the change accurately: the memo for the a ? (b) : c =&gt; d speculative attempt is re-keyed by the state the attempt reads. The raw summary confirms the memo key now includes the colon offset and arrow_data.allow_await, and the parser snapshot is extended. The title is specific and single-sentence, though at 80 characters it is slightly long and omits the snapshot change.

Full details: Description check

Explanation

The description is complete and relevant. It explains the problem, fix, background, downsides, and verification results. It does not use the template headings exactly, but it provides the required information under equivalent sections.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Beyond the inline findings, I also checked the key packing in src/js_parser/parse/parse_skip_typescript.rs:1610 — AwaitOrYield has discriminants 0..2, which fit in the two bits freed by << 2 before the outcome bit, and ts_conditional_arrow_attempts is intentionally outside parser_snapshot, so entries inserted by nested attempts during the debug re-run are keyed and harmless. The existing 2^40 nesting inputs are preserved as the first iteration of the new loop rather than mutated.

Extended reasoning...

The change re-keys the TypeScript conditional-arrow attempt memo by : offset plus allow_await, widens the memo to u64, extracts the attempt into a cold function that forces allow_private_identifiers, adds a debug re-run assertion on memo hits, bumps the transpiler cache version to 42, and adds transpiler tests. It touches no security-sensitive surface. Two confirmed inline findings (unrestored module flags from a discarded attempt now reachable past private names, and reduced nesting depth from the extra frame) already signal that a human should look; this note only records what else was examined.

… and stays in the frame of its caller

The attempt allows private names, so it can read further than before.
`has_import_meta`, `has_with_scope` and `needs_jsx_import` that it set
stayed set after the restore. The snapshot holds them now.

The attempt is inlined into its one release call site again. As a
function of its own it made chains of attempts nest less deep.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest push addresses both points from my earlier review (the three module flags are now in ParserSnapshot, with a scanImports test, and the attempt body is #[inline(always)] so no extra frame per nesting level); this run found no bugs. Because the fix rests on a global argument that the memo key now covers every parser state the attempt reads, a human look at that argument is still worthwhile.

What was reviewed:

  • Memo key coverage: arrow_data is built at src/js_parser/parse/mod.rs:561-568 from opts.is_async alone, so allow_await is its only varying field; allow_in is forced true by the ? at src/js_parser/parse/parse_suffix.rs:452 and the AfterQuestionAndBeforeColon flag only propagates into nested conditional branches where it stays true; strict-mode/super/this checks in the body only log (mark_strict_mode_feature returns Ok).
  • Debug-build re-run on a memo hit: gated on lexer.is_log_disabled so attempts nested inside another attempt are not re-run, errors are swallowed via .ok(), and it uses the same snapshot/restore path as a miss, so it adds no new side effects; the 40-deep chain test now runs for both heads.
  • << 2 in the key is correct for the 3-valued AwaitOrYield; the stored entry shifts by one more for the outcome bit, and the lookup masks consistently.
  • Nit: the PR description is stale relative to the code (it says #[cold]/#[inline(never)] and EXPECTED_VERSION 41 to 42; the code has #[inline(always)] and 43 to 44).
Extended reasoning...

The change touches the TypeScript parser's speculative parse of a ? (b) : c => d : e (src/js_parser/parse/parse_skip_typescript.rs), widens and re-keys its memo by : offset plus allow_await, forces allow_private_identifiers during the attempt, extends ParserSnapshot in src/js_parser/p.rs with three more restored flags, bumps the transpiler cache version, and adds transpiler tests. It touches no security-sensitive surface (no auth, injection, or data exposure), only parser correctness and a cache-version bump. Both inline findings from the prior run were addressed by the two newest commits and the bug hunt ran dry with no new findings, but the correctness of the memo key depends on a whole-parser invariant argument (allow_in only debug-asserted, other state assumed log-only) that merits a maintainer's judgment rather than an automated approval. No CODEOWNERS entry covers the changed files.

@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-10-27.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants