Repository navigation
codegen: use node:crypto instead of Bun.CryptoHasher - #41224
Conversation
bundle-modules.ts computes internalModulesStamp with createHash from node:crypto in place of Bun.CryptoHasher. The stamp is the first four bytes of the SHA-256 digest, big-endian. readUInt32BE(0) reads them from the digest Buffer, also when the Buffer is a view into a larger ArrayBuffer.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Essentials Run ID: 📒 Files selected for processing (5)
Included review availability: Your plan provides up to 10 included reviews per hour; 0 remain after this review. WalkthroughThe change adds a shared SHA-256 ChangesSource stamping
Merge Risk: ⚪ Minimal · up to Code generation now derives its source stamp through node:crypto so it can run without Bun-specific hashing APIs. The generated outputs and stamp remain unchanged, and the compatibility behavior is covered by isolated tests. 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
|
Status: ready for review. The test is removed, as requested.
|
There was a problem hiding this comment.
LGTM — straightforward build-tooling swap from Bun.CryptoHasher to node:crypto.
What was reviewed:
h.update()inputs are strings fromoutputs(line 286); both hashers default to UTF-8, so digest bytes match.readUInt32BE(0)is big-endian like the oldDataView.getUint32(0), and is correct regardless of Buffer poolbyteOffset— stamp value is unchanged.- Import style matches the file's existing unprefixed
fs/fs/promisesimports.
Extended reasoning...
Overview
This PR touches a single build-time codegen script, src/codegen/bundle-modules.ts, swapping the SHA-256 hasher used to compute internalModulesStamp from new Bun.CryptoHasher("sha256") to Node's createHash("sha256"), and reading the first 4 digest bytes via Buffer.readUInt32BE(0) instead of wrapping the digest's underlying ArrayBuffer in a DataView. The stated motivation is enabling the codegen step to run under Node (part of the build-without-Bun effort).
Security risks
None. This is offline build tooling that fingerprints already-bundled internal module source strings. No untrusted input, no runtime code path, no auth/crypto/permissions surface — the hash is used purely as a 32-bit content stamp, not for integrity or security.
Level of scrutiny
Low. The change is four lines in a codegen script. The key correctness question is whether the stamp value is preserved: (1) the outputs map holds strings (set at line 286), and both Bun.CryptoHasher.update(string) and Node's Hash.update(string) encode as UTF-8, so digest bytes are identical; (2) DataView.prototype.getUint32(0) is big-endian by default, and Buffer.readUInt32BE(0) is explicitly big-endian, so the extracted uint32 is identical. The switch to readUInt32BE is also strictly safer under Node, where a Buffer's .buffer may point into a shared pool at a nonzero byteOffset — readUInt32BE respects byteOffset while the old new DataView(buf.buffer) pattern would not. The PR author reports byte-identical InternalModuleRegistryConstants.bin before/after, which matches the analysis.
Other factors
Import placement follows the file's alphabetical block and matches the existing unprefixed "fs" / "fs/promises" style. No CODEOWNERS entry covers src/codegen/. The bug hunt exited on dry_streak with no candidate findings. As a pure no-op refactor with byte-identical output, no new test is required per REVIEW.md. No outstanding reviewer objections in the timeline.
sourceStamp() in helpers.ts hashes the sources with node:crypto and returns the first four bytes of the digest as a big-endian u32. bundle-modules.ts calls it for internalModulesStamp. The test runs sourceStamp() with the Bun APIs disabled and checks two known digests. bundle-modules.ts itself bundles every builtin module first, which is too slow for a test in a debug build.
A debug or ASAN build can print to stderr. The test checks the exit code, and stderr stays in the checked object so that a failure shows it.
There was a problem hiding this comment.
LGTM — build-script-only API swap with byte-identical output and known-answer test coverage.
What was reviewed:
sourceStamp()semantics:createHash("sha256").update(str)is UTF-8 in both runtimes, anddigest().readUInt32BE(0)matches the oldDataView(...).getUint32(0)(both big-endian, and Bun's digest buffer is not pooled so byteOffset was already 0).- Test vectors:
0xfb8e20fcand0xe3b0c442are the correct leading 4 bytes of SHA-256("ab") and SHA-256(""). runWithoutBunApis/ preload fixture:DISABLED_BUN_APIS=""is falsy so the default path stubs everything;process.execPathis a documented, intentional deviation frombunExe()for source-lints.
Extended reasoning...
Overview
This PR replaces Bun.CryptoHasher with node:crypto's createHash in the codegen script that computes internalModulesStamp, extracting the logic into a testable sourceStamp() helper in src/codegen/helpers.ts. It adds a source-lint test that runs the helper with all Bun.* functions stubbed to throw, plus two small helper files (without-bun-apis.ts and its preload fixture) that other PRs in the #6887 series will reuse. The call site in bundle-modules.ts now passes an array of bundled outputs instead of iterating inline; the iteration order and ?? "" fallback are preserved.
Security risks
None. This is build-time tooling that hashes generated module source text to produce a 32-bit cache-invalidation stamp. No user input, no runtime code path, no auth/crypto in the security sense — SHA-256 here is used purely as a content fingerprint, and the choice of algorithm and endianness only needs to be stable across builds.
Level of scrutiny
Low-to-moderate. The only correctness question that matters is whether the stamp value is unchanged, since a silent change would invalidate --compile --bytecode caches without a version bump. I checked: (1) Bun.CryptoHasher.update(string) and createHash().update(string) both encode as UTF-8; (2) the old new DataView(digest.buffer).getUint32(0) read big-endian at absolute offset 0 of the underlying ArrayBuffer, which in Bun is not a pooled slab, so it is equivalent to readUInt32BE(0); (3) the test's known-answer vectors are the actual first four bytes of SHA-256("ab") and SHA-256(""). The PR description also states debug and release codegen output was diffed byte-for-byte before and after.
Other factors
No CODEOWNERS entries cover these paths. The bug-hunt exited on dry_streak with no findings. Two commits landed after the earlier review round — extracting the helper into helpers.ts and relaxing the test's stderr assertion to toMatchObject({ exitCode: 0 }) so incidental stderr noise doesn't flake it while still surfacing the error text on failure. The process.execPath-instead-of-bunExe() choice is deliberate and commented (source-lints run under the released bun), and the fixture naming follows the *-fixture.ts convention. The change is small, self-contained, and does not touch shipped runtime code.
|
remove the tests |
A maintainer asked for the PR without the test.
|
Removed in 65b707a. The PR now changes only the source files. |
### Problem - The codegen scripts read and write files with `Bun.file()` and `Bun.write()`. Node has no `Bun` global, so these scripts cannot run under Node. This is one of the blockers for a build without bun (#6887). ### Fix - Replace each call with `node:fs`: `readFileSync` or `readFile`, and `writeFileSync` or `writeFile`. - `Bun.write()` creates missing parent directories. `generate-jssink.ts` and `bundle-functions.ts` now call `mkdirSync` before they write. `bundle-functions.ts` does it once per run. - Verified: the codegen output of a debug and a release configure is byte-identical before and after this change, with bun 1.3.13, the CI version. ### Background - The build runs each codegen script as a ninja edge. The scripts write generated C++, Rust, and JS into `build/<profile>/codegen`. - Today bun runs these scripts. #34876 makes them run under Node too. This PR is one part of that change, split out for review, with #41222, #41224, and #41225. <details><summary>Notes</summary> - `compress-embed.ts` and the `.zst` write in `build-fallbacks.ts` are in #41222, because that PR edits the same lines. - `Bun.file().text()` removes a UTF-8 byte order mark. `readFileSync(path, "utf8")` keeps it. No tracked file under `src/` starts with a byte order mark, so the output does not change. - `cppbind.ts` keeps its async shape and uses `fs/promises`. - oxlint reports the same warnings as on main. - Comparison method: `node scripts/build.ts --profile=<debug|release> --build-dir=<dir> --configure-only`, then `ninja -C <dir> codegen`, in a new build directory each time. Then `diff -r` of the two codegen directories. The ninja log shows that every edited script ran. - One `Bun.write` stays in `bundle-functions.ts`, inside a commented-out line. </details> <!-- robobun:evidence:begin --> --- **[human-review]** gate passed · iteration 3 · 10 files touched <details><summary>fails on main (without fix)</summary> ```console ASAN without fix: 1 FAILED $ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/internal/source-lints/codegen-generate-node-errors.test.ts bun test v1.4.1 (a6c4cc2) test/internal/source-lints/codegen-generate-node-errors.test.ts: 15 | using dir = tempDir("generate-node-errors", {}); 16 | 17 | const { stderr, exitCode } = await runWithoutBunApis([script, String(dir)]); 18 | 19 | // stderr is in the object so that a failure shows the script's error. 20 | expect({ exitCode, stderr }).toMatchObject({ exitCode: 0 }); ^ error: expect(received).toMatchObject(expected) { - "exitCode": 0, + "exitCode": 1, + "stderr": + " 5 | for (const key of Object.getOwnPropertyNames(Bun)) { + 6 | if (only && !only.has(key)) continue; + 7 | const desc = Object.getOwnPropertyDescriptor(Bun, key); + 8 | if (!desc?.writable || typeof desc.value !== "function") continue; + 9 | (Bun as unknown as Record<string, unknown>)[key] = () => { + 10 | throw new Error(`Bun.${key} is disabled: build scripts must use node: APIs`); + ^ + error: Bun.file is di ... (truncated) release without fix: 1 FAILED bun test v1.4.1-canary.1 (a6c4cc2) test/internal/source-lints/codegen-generate-node-errors.test.ts: 15 | using dir = tempDir("generate-node-errors", {}); 16 | 17 | const { stderr, exitCode } = await runWithoutBunApis([script, String(dir)]); 18 | 19 | // stderr is in the object so that a failure shows the script's error. 20 | expect({ exitCode, stderr }).toMatchObject({ exitCode: 0 }); ^ error: expect(received).toMatchObject(expected) { - "exitCode": 0, + "exitCode": 1, + "stderr": + " 5 | for (const key of Object.getOwnPropertyNames(Bun)) { + 6 | if (only && !only.has(key)) continue; + 7 | const desc = Object.getOwnPropertyDescriptor(Bun, key); + 8 | if (!desc?.writable || typeof desc.value !== "function") continue; + 9 | (Bun as unknown as Record<string, unknown>)[key] = () => { + 10 | throw new Error(`Bun.${key} is disabled: build scripts must use node: APIs`); + ^ + error: Bun.file is disabled: build scripts must use node: APIs + at <anonymous> (/workspace/bun/test/internal/source-lints/without-bun-apis-fixture.ts:10:15) + at /workspace/bun/src/codegen/genera ... (truncated) ``` </details> <details><summary>passes on PR (with fix)</summary> ```console ASAN with fix: all passed $ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" test/internal/source-lints/codegen-generate-node-errors.test.ts bun test v1.4.1 (a6c4cc2) test/internal/source-lints/codegen-generate-node-errors.test.ts: (pass) generate-node-errors.ts writes its outputs with the Bun APIs disabled [1397.07ms] 1 pass 0 fail 3 expect() calls Ran 1 test across 1 file. [3.44s] __F:0:S:0 release with fix: all passed $ bun scripts/build.ts --profile=release [configured] bun-profile → bun (stripped) target linux-x64-gnu build type Release build dir ./build/release revision 8affceb features baseline 23 deps, 131 codegen, 1172 objects in 707ms ninja: Entering directory `/workspace/bun/build/release' [1/1244] install /workspace/bun bun install v1.4.1-canary.1 (a6c4cc2) Checked 25 installs across 62 packages (no changes) [9.00ms] [2/1244] gen bindgenv2 [3/1244] install /workspace/bun/packages/bun-error bun install v1.4.1-canary.1 (a6c4cc2) Checked 1 install across 2 packages (no changes) [1.00ms] [4/1244] fetch zlib [zlib] up to date [5/1244] gen ErrorCode+*.h [6/1244] install /workspace/bun/src/node-fallbacks bun install v1.4.1-canary.1 (a6c4cc2) Checked 111 installs across 104 packages (no changes) [8.00ms] [7/1244] fetch tinycc [tinycc] up to date [8/1243] fetch libjpeg-turbo [libjpeg-turbo] up to date [9/1216] gen .bind.ts → GeneratedBindings.cpp [10/1216] gen node-fallbacks/react-refresh.js Bundled 1 module in 8ms react-refresh.js 4.81 KB (entry point) [11/1216] gen bake.{client,server,error}.js -> bake.client.js, bake.server. ... (truncated) ``` </details> <details><summary>diff hotspot</summary> ``` src/codegen/bundle-functions.ts | 7 +++--- src/codegen/bundle-modules.ts | 3 +-- src/codegen/cppbind.ts | 15 ++++++------ src/codegen/create-hash-table.ts | 3 ++- src/codegen/generate-jssink.ts | 10 ++++---- src/codegen/generate-node-errors.ts | 3 ++- src/node-fallbacks/build-fallbacks.ts | 5 ++-- .../codegen-generate-node-errors.test.ts | 28 ++++++++++++++++++++++ .../source-lints/without-bun-apis-fixture.ts | 12 ++++++++++ test/internal/source-lints/without-bun-apis.ts | 25 +++++++++++++++++++ 10 files changed, 91 insertions(+), 20 deletions(-) ``` </details> **gate history** · 1 passed · 0 rejected · iteration 3 <details><summary>evidence per changed file</summary> ``` file reads edits tests src/codegen/bundle-functions.ts 3 5 8 src/codegen/bundle-modules.ts 4 7 10 src/codegen/cppbind.ts 1 7 8 src/codegen/create-hash-table.ts 1 2 8 src/codegen/generate-jssink.ts 2 2 8 src/codegen/generate-node-errors.ts 1 2 8 src/node-fallbacks/build-fallbacks.ts 2 4 8 …ernal/source-lints/codegen-generate-node-errors.test.ts 1 3 8 test/internal/source-lints/without-bun-apis-fixture.ts 1 0 13 test/internal/source-lints/without-bun-apis.ts 0 1 12 ``` </details> <!-- robobun:evidence:end -->
### Problem - `src/codegen/compress-embed.ts` and `src/node-fallbacks/build-fallbacks.ts` call `Bun.zstdCompressSync()`. Node has no `Bun` global, so these scripts cannot run under Node. This is one of the blockers for a build without bun (#6887). ### Fix - Call `zstdCompressSync()` from `node:zlib` with `ZSTD_c_compressionLevel` 19, the same level as before. The `Bun.file()` and `Bun.write()` calls on these lines use `node:fs`. - One output changes. `node-fallbacks/crypto.js.zst` is 71109 bytes (it was 71233). It inflates to the same 477363 bytes. All other `.zst` files are byte-identical. - Verified: debug and release codegen with bun 1.3.13, the CI version. ### Background - Release builds embed zstd-compressed copies of assets that bun does not execute, for example the node fallback polyfills. The binary inflates them on first use with `bun_zstd::decompress_alloc`. - `Bun.zstdCompressSync()` uses the one-shot zstd API (`ZSTD_compress`). `node:zlib` uses the streaming API (`ZSTD_compressStream2`), in bun and in Node. The two APIs can encode the same input differently. - This PR is one part of #34876, split out for review, with #41223, #41224, and #41225. <details><summary>Notes</summary> - `Bun.zstdDecompressSync()` inflates the new `crypto.js.zst` to the same bytes as `crypto.js`. That function calls `bun_zstd::decompress_alloc`, the same function that the release binary uses (`src/resolver/node_fallbacks.rs`, and `embed_compressed!` in `src/zstd/lib.rs`). - bun 1.3.13, bun 1.4.0, and Node 26.3.0 write the same bytes through `node:zlib`. So codegen writes the same `.zst` files under bun and under Node. - Both the old and the new frames store the content size in the frame header. - Node's zstd options have no `level` field. The level goes in `params`. - Comparison method: `node scripts/build.ts --profile=<debug|release> --build-dir=<dir> --configure-only`, then `ninja -C <dir> codegen`, in a new build directory each time. Then `diff -r` of the two codegen directories. - #41223 (the `node:fs` part) leaves these two lines alone, so the two PRs do not conflict. </details>
…package-manager=npm (#41225) ### Problem - `npm install` fails in the repo root with `npm error code EUNSUPPORTEDPROTOCOL` and `Unsupported URL Type "workspace:"`. The `workspace:` protocol came from `packages/@types/bun`, a one-line alias of the DefinitelyTyped package whose only job was to let tsc resolve the type reference `bun` to the checkout's `packages/bun-types`. - The build installs three package.json files with bun (the repo root, `packages/bun-error`, `src/node-fallbacks`). A machine without bun cannot run those steps (#6887). ### Fix - Remove `packages/@types/bun`, its workspace entry, its devDependency, and its `resolutions` entry. The four tsconfigs that reached it (`src`, `src/js`, `src/codegen/bindgenv2`, `scripts/build`) now list `packages/bun-types` in `types`, as `test/tsconfig.json` already does. - Pin the root devDependencies to the versions in `bun.lock`. - Add `--package-manager=npm`. Each install then runs `scripts/build/npm-ci.ts`: it writes `package-lock.json` from that directory's `bun.lock`, runs `npm ci` and `npm rebuild`, and deletes the file again. The default stays bun. - Verified: for the root, the translated lockfile is byte-identical to what `npm install --package-lock-only` writes. The codegen output is identical with both package managers, on Linux and on Windows, and no lockfile is left behind. tsc error counts are unchanged, and the bun-types integration test passes. ### Background - The root install provides esbuild and lezer for codegen. - npm reads the lockfile only from the directory of `package.json`, so the translated file lives there for the duration of `npm ci`. `.gitignore` already ignores `package-lock.json`. - This PR is one part of #34876, split out for review, with #41222, #41223, and #41224. <details><summary>Notes</summary> - `npm-ci.ts` reads only built-ins, so it runs under bun or node (the ninja rule uses `cfg.jsRuntime`, the runtime that ran configure). It emits a lockfileVersion 3 file: the root entry, the workspace folder and its link, and one `node_modules/...` entry per resolved package with `version`, `resolved`, `integrity`, `dev`, `optional`, `os`, `cpu`, `bin`, and the dependency maps. `dev` means no prod edge reaches the package, and `optional` means no non-optional edge reaches it, which is how npm sets them. bun.lock has no tarball URL for the default registry, so the URL is built from the name and version. - The lockfile is deleted in a `finally`, so a failed `npm ci` also leaves no file. The exit code of `npm ci` is the exit code of the rule. - `npm ci` runs install scripts only for packages that the lockfile flags with `hasInstallScript`, which `bun.lock` does not record. So the script runs `npm rebuild` after the install, with both `package-lock.json` and npm's hidden `node_modules/.package-lock.json` gone. npm then reads the real package.json files, and esbuild's postinstall puts the native binary in place. A blanket flag in the lockfile would make npm 11 warn that every package has install scripts. - `npm ci` replaces `node_modules` itself, so a tree that `bun install` wrote does not get in the way. `bun install` keeps `bun.lock` and ignores a `package-lock.json` next to it. - Codegen comparison, in a worktree on Linux: npm on a clean tree, bun over npm's tree, and npm over bun's tree. All give the same 141 files, and no lockfile is left in any of the three directories. On Windows, the `codegen` target passes with npm, with a clean `git status`. The `node-fallbacks` lockfile has 16 nested entries, and `npm ci` accepts the translated file. - Windows: npm is `npm.cmd`, so `npm-ci.ts` spawns it through a shell there, with the path quoted (the default install is under `C:\Program Files`). npm writes `node_modules/.bin/esbuild.cmd`, and `findTool` takes a `windowsExt` option for `npm.cmd`. - `bun.lock` loses the `@types/bun` entries, and its root specs become the exact versions. `bun install --frozen-lockfile` passes with bun 1.3.13 and bun 1.4.0. - Earlier versions of this PR kept the alias with `"*"` specs, ran `npm install --no-package-lock` after an `rm -rf node_modules`, and then committed a `package-lock.json` from a standalone script. Review replaced each of those with the current shape. - The release workflow edits `types/bun/package.json` in a DefinitelyTyped checkout, not this folder. `.github/workflows/bun-types.yml` loses its two path entries for the folder, and `.aikido` its exclude entry. </details> <!-- robobun:evidence:begin --> --- **no test proof** · iteration 4 · no src or test change; test-proof not applicable <!-- robobun:evidence:end --> --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Problem
src/codegen/bundle-modules.tscomputesinternalModulesStampwithnew Bun.CryptoHasher("sha256"). Node has noBunglobal, so this script cannot run under Node. This is one of the blockers for a build without bun (allow building without a bootstrap #6887).Fix
sourceStamp()insrc/codegen/helpers.tshashes the sources withcreateHash("sha256")fromnode:crypto.bundle-modules.tscalls it.digest().readUInt32BE(0). The old code usednew DataView(digest.buffer).getUint32(0). In Node, a smallBuffercan be a view into a shared pool, so.bufferdoes not start at the digest.readUInt32BE(0)reads the same big-endian value in both runtimes.InternalModuleRegistryConstants.bin(debug0xedff4287, release0x0704e3bd).Background
internalModulesStampidentifies the bundled sources of the builtin modules.bun build --compile --bytecodeembeds bytecode for these modules, and the stamp ties that bytecode to the sources.Notes
sourceStamp()lives inhelpers.ts, next to the other shared codegen helpers.bundle-modules.tsbundles every builtin module before it computes the stamp. That takes about 10 seconds with a release bun, and more than five minutes with a debug build.scripts/build/xmac.mjsalso usesBun.CryptoHasher. It is a vendored, generated file (from jarred-sumner/xmac), and it runs only for a darwin cross-compile from Linux. This PR does not change it.hash.update()encodes a string as UTF-8 in both runtimes.node scripts/build.ts --profile=<debug|release> --build-dir=<dir> --configure-only, thenninja -C <dir> codegen, in a new build directory each time. Thendiff -rof the two codegen directories.