Skip to content

allow importing of files with top level awaits in bundler - #14218

Merged
Jarred-Sumner merged 14 commits into
mainfrom
snoglobe/tla
Sep 30, 2024
Merged

Jarred-Sumner merged 14 commits into
mainfrom
snoglobe/tla

Conversation

@snoglobe

Copy link
Copy Markdown
Contributor

What does this PR do?

fixes #12508 - we now implement validateTLA from esbuild and properly flag is_async_or_has_async_dependency.

  • Documentation or TypeScript types (it's okay to leave the rest blank in this case)
  • Code changes

How did you verify your code works?

@robobun

robobun commented Sep 27, 2024 •

Copy link
Copy Markdown
Collaborator

@small-angel small-angel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i have tiny nitpicks. overall this looks correct porting from esbuild. thank you for the regression test.

Comment thread src/bundler/bundle_v2.zig Resolved
Comment thread src/js_ast.zig Resolved
Comment thread src/bundler/bundle_v2.zig
// Stop now if there were errors
if (this.log.hasErrors()) {
return &[_]Chunk{};
return error.BuildFailed;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im still so surprised this hasnt caused any problems. or maybe it has.

Comment thread src/bundler/bundle_v2.zig Outdated
// async. This happens when you call "import()" on this module and code
// splitting is off.
if (Index.isValid(Index.init(result_tla_check.parent))) {
c.graph.meta.items(.flags)[source_index].is_async_or_has_async_dependency = true;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need to call .items here

we can hoist it out to the parent

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done 6928acb

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.items continues to be called

Comment thread src/bundler/bundle_v2.zig Outdated
const parent_source_index = other_source_index;

if (parent_result.top_level_await_keyword.len > 0) {
tla_pretty_path = c.parse_graph.input_files.items(.source)[other_source_index].path.pretty;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't need to call .items here either

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done 6928acb

Comment thread src/bundler/bundle_v2.zig Outdated
else
std.fmt.allocPrint(c.allocator, "This require call is not allowed because the transitive dependency \"{s}\" contains a top-level await", .{tla_pretty_path}) catch bun.outOfMemory();

const source: Logger.Source = c.parse_graph.input_files.items(.source)[source_index];

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does a stack copy and then only references by pointer

let's avoid the stack copy entirely via &

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Outdated

// Build up a chain of notes for all of the imports
while (true) {
const parent_result = c.parse_graph.ast.get(other_source_index);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to call .get() here? This loads data from many different arrays.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only top_level_await_keywords and tla_check are referenced?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Outdated
}

pub fn validateTLA(c: *LinkerContext, source_index: Index.Int) js_ast.TlaCheck {
var result_tla_check: *js_ast.TlaCheck = &c.parse_graph.ast.items(.tla_check).ptr[source_index];

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This .ptr access avoids safety checks. Can we avoid doing that?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was using it while debugging and missed it when reviewing. fixed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Outdated

pub fn validateTLA(c: *LinkerContext, source_index: Index.Int) js_ast.TlaCheck {
var result_tla_check: *js_ast.TlaCheck = &c.parse_graph.ast.items(.tla_check).ptr[source_index];
const result = c.parse_graph.ast.get(source_index);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't get the entire AST for only a handful of necessary fields

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Outdated
// does it have a JS AST?
if (!(id < import_records_list.len)) continue;

_ = this.validateTLA(id);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since most projects do not use any top-level await, maybe we can add a boolean in ParseGraph or LinkerGraph that tracks whether any modules at all used top-level await, and then only run this nested for loop over all the import records of all the files when any file has top-level await? We can set that boolean as we receive results from the parser threads to avoid any extra passes

@Jarred-Sumner Jarred-Sumner left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several performance-related comments

Needs more tests.

stdout: "0\n1",
},
});
itBundled("default/TopLevelAwaitImport", {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs more tests.

This only tests one level deep.

Please add tests for:

  • require
  • dynamic import
  • import statement

These tests need to involve multiple files and permutations of those.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about require in-between for each of those?

Comment thread src/bundler/bundle_v2.zig Outdated
return hasher.digest();
}

pub fn validateTLA(c: *LinkerContext, source_index: Index.Int) js_ast.TlaCheck {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest passing the slices in to the function, so that we avoid doing any extra work

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Outdated
else
std.fmt.allocPrint(c.allocator, "This require call is not allowed because the transitive dependency \"{s}\" contains a top-level await", .{tla_pretty_path}) catch bun.outOfMemory();

const source: Logger.Source = input_files[source_index];

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still making a stack copy here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig

// Require of a top-level await chain is forbidden
if (record.kind == .require) {
var notes = std.ArrayList(Logger.Data).init(c.allocator);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This leaks memory. notes is cloned in addRangeErrorWithNotes (not the contents within the notes, but the slice is)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is fine; c.allocator is an arena so it all gets handled

- optimize usages of ast.items
- don't make stack copy of source
- pass slices to validateTLA
- don't leak memory on notes
Comment thread src/bundler/bundle_v2.zig Outdated
// does it have a JS AST?
if (!(id < import_records_list.len)) continue;

const tla_keywords = this.parse_graph.ast.items(.top_level_await_keyword);

@Jarred-Sumner Jarred-Sumner Sep 28, 2024 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are now calling .items for every reachable file. This is an improvement over calling .items for every reachable file for every import record for every import record's record, but we can still move this one level up. It might even already be there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops missed that

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Resolved
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
Comment thread src/bundler/bundle_v2.zig Outdated

if (result_tla_check.depth == 0) {
result_tla_check.depth = 1;
if (result_top_level_await_keyword.len > 0) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (result_top_level_await_keyword.len > 0) {
if (tla_keywords[source_index].len > 0) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/bundler/bundle_v2.zig Resolved
snoglobe and others added 3 commits September 27, 2024 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Top level await in Bun.build: error: "await" can only be used inside an "async" function

4 participants