Skip to content

compile: give a text module its own embedded path, and keep the sqlite record when a file import shares it - #46760

Open
robobun wants to merge 2 commits into
mainfrom
robobun/9640ff3e/compile-text-file-same-path
Open

robobun wants to merge 2 commits into
mainfrom
robobun/9640ff3e/compile-text-file-same-path

Conversation

@robobun

@robobun robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Since 1.4.1, a bun build --compile executable gives a file and a text import of same-named, equal-bytes files one embedded path. file first: the text import runs the file as JavaScript (SyntaxError: Unexpected identifier 'of' at /$bunfs/root/data-srdqwvvw.txt:1:1). text first: the file import reads the string body.
  • to_bytes keeps the first file at a path (src/standalone_graph/StandaloneModuleGraph.rs:1600), so file before a sqlite embed dies the same way.

Fix

  • A text module's [hash] covers its stored form (ParseTask.rs): it never prints a raw copy's path.
  • to_bytes keeps a sqlite embed's record over another copy of the same bytes. The file import reads either.
  • Verified: test/bundler/bundler_compile.test.ts (6 new tests, all fail on main). Also compile-asset-bunfs and 11 more suites (Notes).
  • Self-reviewed: 9 concerns raised, 8 addressed. Not taken: the .node text import fix, a separate mechanism (Notes).

Background

Downsides

  • One-time renames: the [hash] of each text module and of each hashed chunk that reaches one, and the executable's BuildArtifact.hash.
  • Sizes: 12 more hashed bytes per text module, 3,145,728 bytes for a file + text pair of a 3 MB file, 4,096 bytes of bun binary.
Notes

History. No user has reported this. There is no issue.

  • file + text: 1.3.13 and 1.4.0 run both import orders. 1.4.1, 1.4.2, 1.4.3 and main (6292c65) do not. bun build --compile: embed text imports as pre-encoded strings #40177 made a text import an embedded string body and made to_bytes skip a repeated path.
  • file + sqlite embed: one order fails in every release. Through 1.4.0 the last record at a path won, so sqlite first died at startup. Since 1.4.1 the first wins, so file first dies. This branch is the first build that runs both orders.
  • When the file's bytes parse as JavaScript, main runs them at startup and the text import is the module namespace object, with exit 0.

Outcomes, release builds of main and of this branch. Each row is two files with one basename and equal bytes, imported in that order. A row is right when every import gives what it gives in a control build where the two files have different basenames.

imports, in order main this branch
file, text startup SyntaxError right
wasm, text startup SyntaxError right
file, sqlite startup SyntaxError right
wasm, sqlite startup SyntaxError right
text, file file reads the string body, not in Bun.embeddedFiles right
text, wasm same right
text, sqlite TypeError: the database import is undefined right
sqlite, text the text import is the database module right
file, wasm and wasm, file right right
sqlite, file and sqlite, wasm right right
napi, text and text, napi wrong wrong
  • Ordered pairs with a wrong result: main 10 of 14, this branch 2 of 14.
  • The 2 left are a text import of a .node file. See "Not changed here".
  • One file as the image of a page and as a text import of its server: main serves 62616420efbfbdefbfbd for the bytes 62616420fffe and does not list the file in Bun.embeddedFiles, in both entry orders. This branch serves the bytes and lists the file.
  • The same file + sqlite pair under --asset-naming='[name].[ext]' also runs in both orders (compile/SameNameSameBytes/FileFirst/NoHash). There the content hash, not the path, tells a copy from another file.

Two different files at one embedded path. This PR does not change that case. Main keeps the first file and exits 0 (through 1.4.0 it kept the last), and so does this branch. Examples: two same-named files with different bytes under --asset-naming='[name].[ext]', or an asset printed at the name of a chunk. #46655 adds copied assets to the linker's duplicate output path check, for every build mode, and fails those builds. It lets copies with equal bytes share a path, so it does not fix the pairs in the table. An earlier version of this branch failed such a build in to_bytes. The self-review moved that out: it changes builds that exit 0 today, and #46655 is where that decision is open.

Costs for builds that never have such a pair (release builds of main 6292c65 and of this branch, gdb breakpoint hit counters on bun-profile).

  • Build, per embedded text module: +2 xxhash update calls and +12 bytes hashed. 100 text imports: 1,126 to 1,326 calls, 15,399 to 16,599 bytes, the same on a second run. 100 file imports: 1,121 calls on both builds.
  • Allocator calls for that whole build (12 mimalloc entry points): 27,949 and 27,968 on main, 27,938 and 28,013 on this branch. The difference between two runs of one build is larger than the difference between the builds. The added code allocates nothing.
  • to_bytes (from the entry of to_executable to its call of target_executable): allocator calls 8, 1,033, 10 and 12 for hello world, 1,000 file assets, two equal text files, and a page that shares an image with its server. The counts are the same on both builds.
  • The new branch runs once for each repeated path and not at all without one. It reads one loader tag and two hashes, and no file body: the only memcmp in that window is the 17-byte path, on both builds, also when the shared image is 64 MB.
  • Executables of builds with no text module: 0 differing bytes in 3 fixtures (hello world, 1,000 file assets, a page and its server that share one image). cmp, main against this branch, onto one runtime. Main against main is also 0.
  • Names, compile/TextImport fixture: 6 of 6 text modules change, 0 of 1 file assets. Bun.embeddedFiles names do not change.
  • Names, a --splitting fixture with 4 hashed chunks and 2 text modules: 2 of 2 text modules and 3 of 4 chunks change. The 3 chunks embed or reach the path of a text module. BuildArtifact.hash of the executable changes with a text import (t502y850 to x6zyrah2) and does not change without one.
  • A file + text pair of a 3 MB file: the executable grows by 3,145,728 bytes. On main the two imports share one record. 1.4.0 also stored the text and the file.
  • Release binary: .text +768 bytes (size -A), 4,096 bytes on disk. get_ast +661, to_executable +323, register_embedded_asset -201 (nm -S). __bun_fetch_builtin_module +0. No run-time source changes.
  • perf, valgrind, strace and bloaty are not installed on this machine.

Not changed here.

  • A text import of a .node file exits 1 with ERR_DLOPEN_FAILED, with or without a second import. 1.4.0 prints the string. require() sends every id that ends in .node to process.dlopen before it reads the record (src/js/builtins/CommonJS.ts). It is the only extension of 23 probed that fails. The self-review proposed to keep such a module inline in this PR. It is a separate mechanism with no report, so it is left for its own change.
  • A text import built onto a runtime older than 1.4.1 (--compile-executable-path) fails with SyntaxError at startup. The older runtime does not know the string body record.
  • --asset ./c/logo.file plus a file import of the same file under --asset-naming='[name].[ext]' fails the build with collides with another embedded file, on main and on this branch.
  • Two sqlite embeds of equal bytes under one basename share one Database object.
  • process.dlopen and bun:ffi read the stored body of a record. For a text module that is the string body.

Related open PRs.

Suites. Release builds of main and of this branch: bundler_compile, bundler_compile_splitting, bundler_compile_prelinked, bun-build-compile, bun-build-api, cli, compile-asset-bunfs, bundler_html_server, bundler_naming, bundler_loader, standalone, html-import-manifest, text-loader. This branch: 0 failures. Main: the 6 new tests fail. The debug build of this branch passes bundler_compile (91 tests). In the other suites it fails only tests that exceed their 5 s or 60 s timeout in a debug build (a debug --compile takes 4 to 30 s here). Those with a default timeout pass when the timeout is longer.

Self-review. Addressed: the build error for two different files at one path is removed and left to #46655. The rename in Downsides names chunks and BuildArtifact.hash. A test at a path without [hash] is added. The file + sqlite history, the severity of the file first order and the .node note are corrected above. The hand-written table of record forms is gone with the error. Comments are one line. Not taken: the .node fix, for the reason above.


no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/bundler/bundler_compile.test.ts

…record of a shared file

A text import in a standalone executable is embedded as a string body. A
file import is embedded as the file's bytes. Both took the path
[name]-[hash].[ext] from a hash of the source bytes, so two same-named
files with equal bytes printed one path, and to_bytes kept the first
output file at it. The other import then read a record of the wrong
form: the text import ran the file as JavaScript at startup, or the file
import read the string body.

- The [hash] of a text module covers its stored form. A text module and
  a raw copy of the same bytes print two paths.
- When a file import and a sqlite embed of the same bytes share a path,
  to_bytes keeps the record of the database import, in either import
  order. require() opens a database by the loader in its record, and
  the file import reads the same bytes.
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 179664e and 4c60042.

Files selected for processing: 3
  • src/bundler/ParseTask.rs
  • src/standalone_graph/StandaloneModuleGraph.rs
  • test/bundler/bundler_compile.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

Walkthrough

Changes

The change passes explicit content hashes for embedded assets, distinguishes text-loader assets from raw files, and handles matching duplicate SQLite assets during standalone output generation. Compile tests cover loader combinations, import order, backends, decoding, file identity, and page asset bytes.

Embedded asset identity

Layer / File(s) Summary
Asset hash propagation
src/bundler/ParseTask.rs
register_embedded_asset uses caller-provided hashes. Text-loader assets include the "text" tag in the hash input. SQLite and NAPI assets hash source contents.
Duplicate destination resolution
src/standalone_graph/StandaloneModuleGraph.rs
to_bytes replaces a retained duplicate only for matching Loader::SqliteEmbedded hashes.
Embedded asset behavior tests
test/bundler/bundler_compile.test.ts
Tests cover loader combinations, import order, backend modes, decoding, file identity, SQLite contents, asset names, and page bytes.

Suggested reviewers: jarred-sumner, alii

Priority: Normal

Merge Risk: Minimal · up to 4c600

No merge-blocking regression was identified in the changed embedded-asset behavior; this is ready for normal checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly summarizes the two main changes: separate embedded paths for text modules and SQLite record retention for shared paths.
Description check Passed The description explains the problem, fix, verification, background, limitations, and trade-offs. It does not use the template headings exactly, but it provides the required information in equivalent …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the problem, fix, verification, background, limitations, and trade-offs. It does not use the template headings exactly, but it provides the required information in equivalent sections.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 11:43 AM PT - Oct 24th, 2026

✅ @robobun, your commit 4c600420143a34993cca8b4ae0e8b2bb8cf86820 passed in Build #132092! 🎉


🧪   To try this PR locally:

bunx bun-pr 46760

That installs a local version of the PR into your bun-46760 executable, so you can run:

bun-46760 --bun

@robobun

robobun commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status

Reproduced on 1.4.3 and on main (6292c65) with two same-named files with equal bytes, one imported as file and one as text:

mkdir a b && printf 'bytes of data' > a/data.txt && printf 'bytes of data' > b/data.txt
cat > e1.ts <<'EOF2'
import f from "./a/data.txt" with { type: "file" };
import t from "./b/data.txt" with { type: "text" };
console.log(f, t);
EOF2
bun build e1.ts --compile --outfile=e1 && ./e1

The build exits 0. ./e1 prints SyntaxError: Unexpected identifier 'of' at /$bunfs/root/data-srdqwvvw.txt:1:1 and exits 1. With this branch it prints /$bunfs/root/data-srdqwvvw.txt bytes of data.

CI on 4c60042 (build 132092): no job failed (179 of 181 passed, 2 were still queued when I wrote this). 9 test files failed once and passed on retry. This diff touches none of them. All review threads are resolved.

PR: #46760

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

No output file carries Loader::Sqlite: a sqlite import that is not
embedded stays external, and only Loader::SqliteEmbedded registers an
embedded asset.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

The follow-up commit narrows the repeated-path check in to_bytes to Loader::SqliteEmbedded, which addresses the dead Loader::Sqlite arm raised last time. Beyond the inline nit, I also checked whether a file import whose bytes literally begin with text could collide with a text module's new [hash] — ContentHasher::write length-prefixes each chunk (src/bundler/bundle_v2.rs:8306), so "text" + contents and a single raw buffer never hash the same way.

Extended reasoning...

The change touches the compile-time embedded-asset hash in src/bundler/ParseTask.rs (text modules now hash a "text" tag before the contents, giving them a distinct [hash] from a file import of the same bytes) and the path-dedup loop in src/standalone_graph/StandaloneModuleGraph.rs::to_bytes (a later SqliteEmbedded record with an equal content hash replaces the kept record). No security-sensitive surface is involved; the hash equality guard also keeps the entry-point pointer in module_files intact since a chunk and a database cannot share a content hash. A human look is still worthwhile because the fix changes the [hash] of every text module in compile builds (renames, plus a second embedded copy for file+text pairs), which is a user-visible output decision rather than a mechanical fix.

Comment thread test/bundler/bundler_compile.test.ts Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant