Repository navigation
Conversation
|
This PR may be a duplicate of:
🤖 Generated with Claude Code |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR implements Node.js-compatible fs.readdir({ withFileTypes: true, encoding: 'buffer' }) by returning Dirent objects whose name and parentPath are Buffers when appropriate; changes touch Zig types, argument parsing, directory enumeration (non-recursive and recursive), C++ bindings, and tests. ChangesDirent Buffer Name and Path Support
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/runtime/node/node_fs.zig`:
- Around line 2234-2238: The code currently chooses how to store
Dirent.parentPath based on args.encoding instead of the flag indicating the
original argument type; update all branches that decide Dirent.parentPath (look
for logic near path_is_buffer, args.encoding, Dirent.parentPath, and calls to
toBunString) to first check path_is_buffer and only treat parentPath as binary
when path_is_buffer == true; when path_is_buffer is false, continue to honor
args.encoding for returning a JS string (use toBunString for string conversions)
and when true, preserve raw bytes (return a Buffer/TypedArray) regardless of
args.encoding so binary paths keep byte transparency.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 33f79881-bfe9-48f0-bd97-f7c152b4fe71
📒 Files selected for processing (4)
src/jsc/bindings/NodeDirent.cppsrc/runtime/node/node_fs.zigsrc/runtime/node/types.zigtest/js/node/fs/fs.test.ts
…oding Reviewers on #30484 caught that the storage representation for dirent_path was chosen on args.encoding == .buffer while emission is driven by args.path_is_buffer. Those flags are independent, so the mismatch produces corrupt parentPath whenever one is set and the other isn't: - string path + encoding='buffer' (name=Buffer, parentPath=string): path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café' came back as '/tmp/café'. - Buffer path + default encoding (name=string, parentPath=Buffer): path went through toBunString(.utf8) which replaces invalid UTF-8 bytes with U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF got lost. Storage now tracks path_is_buffer at all three Dirent sites (readdirWithEntries, readdirWithEntriesRecursiveAsync, readdirWithEntriesRecursiveSync). name storage stays gated on args.encoding == .buffer, which matches name_as_buffer. Two new regression tests in fs.test.ts cover both sides: - non-ASCII string path + encoding='buffer' - Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/runtime/node/node_fs.zig`:
- Around line 4840-4846: The recursive async branch currently sets
dirent_path_prev to bun.String.cloneUTF8(path_u8) for non-buffer paths, which
diverges from the non-recursive and sync recursive branches that use
jsc.WebCore.encoding.toBunString(path_u8, args.encoding); update the recursive
async path decoding to call jsc.WebCore.encoding.toBunString(path_u8,
args.encoding) when args.path_is_buffer is false (keep
bun.String.cloneLatin1(path_u8) when args.path_is_buffer is true) so
Dirent.parentPath matches the sync behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 49f5e651-b35c-4316-a683-cfe37f6462f8
📒 Files selected for processing (2)
src/runtime/node/node_fs.zigtest/js/node/fs/fs.test.ts
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
src/runtime/node/node_fs.zig (1)
4837-4858:⚠️ Potential issue | 🟠 Major | ⚡ Quick winHonor
args.encodingin the recursive async Dirent path too.This branch still hardcodes UTF-8 for both
Dirent.parentPathand non-bufferDirent.name, sofs.promises.readdir(..., { recursive: true, withFileTypes: true, encoding: "latin1" | "hex" | ... })diverges from the non-recursive and recursive sync implementations that already usetoBunString(..., args.encoding).Suggested fix
- dirent_path_prev = if (args.path_is_buffer) - bun.String.cloneLatin1(path_u8) - else - bun.String.cloneUTF8(path_u8); + dirent_path_prev = if (args.path_is_buffer) + bun.String.cloneLatin1(path_u8) + else + jsc.WebCore.encoding.toBunString(path_u8, args.encoding); @@ - .name = if (args.encoding == .buffer) - bun.String.cloneLatin1(utf8_name) - else - bun.String.cloneUTF8(utf8_name), + .name = if (args.encoding == .buffer) + bun.String.cloneLatin1(utf8_name) + else + jsc.WebCore.encoding.toBunString(utf8_name, args.encoding),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/runtime/node/node_fs.zig` around lines 4837 - 4858, The recursive async Dirent branch is hardcoding UTF-8 for parentPath and non-buffer name; update the conversion to honor args.encoding by replacing the direct bun.String.cloneLatin1/cloneUTF8 uses with the encoding-aware helper (e.g., call toBunString or a similar helper) for both utf8_name and path_u8 so entries.append uses toBunString(utf8_name, args.encoding) for .name when not buffer and toBunString(path_u8, args.encoding) for .path when path_as_buffer is false, ensuring dirent_path_prev, entries.append, args.encoding, bun.String.cloneLatin1/cloneUTF8 and utf8_name are updated accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/js/node/fs/fs.test.ts`:
- Around line 2713-2779: Modify the two async tests ("readdir (async callback):
Dirent with Buffer name" and "promises.readdir: Dirent with Buffer name") to
exercise a Buffer input path and assert parentPath semantics: pass
Buffer.from(String(dir)) instead of String(dir) to fs.readdir /
promises.readdir, and add assertions that each Dirent.entry.parentPath is an
instance of Buffer and that its string value matches the directory (e.g.,
parentPath.toString() === String(dir)); reference the Dirent.parentPath property
when adding these checks.
---
Duplicate comments:
In `@src/runtime/node/node_fs.zig`:
- Around line 4837-4858: The recursive async Dirent branch is hardcoding UTF-8
for parentPath and non-buffer name; update the conversion to honor args.encoding
by replacing the direct bun.String.cloneLatin1/cloneUTF8 uses with the
encoding-aware helper (e.g., call toBunString or a similar helper) for both
utf8_name and path_u8 so entries.append uses toBunString(utf8_name,
args.encoding) for .name when not buffer and toBunString(path_u8, args.encoding)
for .path when path_as_buffer is false, ensuring dirent_path_prev,
entries.append, args.encoding, bun.String.cloneLatin1/cloneUTF8 and utf8_name
are updated accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 88e65d0b-5e9a-45b3-9060-d7d188cec6fa
📒 Files selected for processing (2)
src/runtime/node/node_fs.zigtest/js/node/fs/fs.test.ts
There was a problem hiding this comment.
Thanks — all my earlier concerns are addressed and the bug-hunting pass found nothing new on this revision; deferring to a human reviewer given the FFI signature change and ref-counting-sensitive C++ paths, plus CI is currently red on 00ad4b1.
Extended reasoning...
Overview
This PR fixes fs.readdir({ withFileTypes: true, encoding: 'buffer' }) to return Dirent objects with Buffer names (and Buffer parentPath when the input path was a Buffer), matching Node.js. It touches four files: Arguments.Readdir and three readdir implementations in src/runtime/node/node_fs.zig, the Dirent struct and extern signature in src/runtime/node/types.zig, the Bun__Dirent__toJS C++ binding in src/jsc/bindings/NodeDirent.cpp (new bunStringToBuffer helper, two new bool parameters), and ~230 lines of new tests in fs.test.ts.
Security risks
None identified. The change is confined to how directory entry names/paths are surfaced to JS — no auth, no user-controlled allocation sizes beyond what readdir already handled, no new syscalls. The new bunStringToBuffer copies bytes into a fresh JS Buffer via WebCore::createBuffer, which is the standard pattern.
Level of scrutiny
Medium-high. The change itself is a well-scoped Node-compat fix, but it:
- Changes an
extern "C"FFI signature shared between Zig and C++ (Bun__Dirent__toJSgains twoboolparams) — ABI mismatches here fail loudly, but the build is currently red on both build-zig and build-cpp across all platforms, so this hasn't been verified green yet. - Adds a new code path in C++ that calls
transferToWTFString()on aBunString*whose underlying storage is ref-counted and shared across multipleDirententries (viadirent_path.ref()on the Zig side). The string-path branch already did this, so the pattern is established, but the Buffer branch bypasses thepreviousPathJSString cache and produces a fresh Buffer per entry — worth a human sanity-check on ref balance. - Threads two independent flags (
name_as_buffer←encoding == .buffer,path_as_buffer←path_is_buffer) through three separate readdir implementations (non-recursive, recursive-async, recursive-sync) plus a Windows-only UTF-16 branch. My earlier review caught a storage/emission mismatch between these flags; that's now fixed, but the duplication across four sites is the kind of thing that benefits from a maintainer's eye.
Other factors
- Prior feedback resolved: My three earlier inline comments (storage gated on wrong flag → fixed in fd8929b; redundant per-entry
path_buffer_poolchurn → fixed in 131c96a; macOSEILSEQtest gating → fixed in 131c96a) and CodeRabbit's two findings (same storage-flag issue; recursive-asynccloneUTF8divergence → fixed in 00ad4b1; asyncparentPathtest coverage → added in c512595) are all addressed and marked resolved. - CI is red: robobun reports build failures on 00ad4b1 across every platform (build-zig and build-cpp). The latest commit c512595 only touches tests, so the build state is unlikely to have changed. This needs to go green before merge regardless.
- Test coverage is thorough: 14 new tests covering sync/callback/promises × string/Buffer path × recursive/non-recursive, plus non-ASCII and raw-byte regression cases for the storage/emission bug I flagged earlier.
- No CODEOWNERS hit that I'm aware of, but
node_fs.zigand JSC bindings are core runtime code.
There was a problem hiding this comment.
No issues found on this revision — all earlier feedback has been addressed — but the change spans a Zig↔C++ extern signature, ref-counted string handling in Bun__Dirent__toJS, and three separate readdir code paths (plus a Windows UTF-16 branch), so it's worth a human pass before merging.
Extended reasoning...
Overview
This PR fixes fs.readdir({ withFileTypes: true, encoding: 'buffer' }) to return Dirent[] with Buffer name/parentPath instead of a flat Uint8Array[]. It touches:
src/runtime/node/node_fs.zig—Arguments.Readdir.tag()priority fix, newpath_is_bufferflag, and Latin-1 vs.toBunStringstorage selection in three readdir paths (non-recursive, recursive async, recursive sync) plus the Windowsis_u16Dirent branch.src/runtime/node/types.zig—Direntgainsname_as_buffer/path_as_bufferand theBun__Dirent__toJSextern signature grows twoboolparams.src/jsc/bindings/NodeDirent.cpp— newbunStringToBufferhelper;Bun__Dirent__toJSrestructured so thepreviousPathJSString cache only applies to the string path and Buffer paths get a freshWebCore::createBufferper Dirent.test/js/node/fs/fs.test.ts— ~15 new tests covering sync/callback/promises × string/Buffer path × recursive, plus non-ASCII and raw-0xFF-byte regression cases.
Security risks
None identified. The change is read-only fs metadata emission; no new syscalls, no user-controlled allocation sizes beyond what readdir already does, and createBuffer copies from a bounded WTF::String span.
Level of scrutiny
Medium-high. fs.readdir is a hot Node-compat API, and the change crosses an FFI boundary (extern signature change must stay in lockstep on both sides), reshuffles ref-counted BunString consumption inside Bun__Dirent__toJS, and encodes a subtle invariant (storage representation must match the independent name_as_buffer/path_as_buffer emission flags) across three code sites. Earlier revisions had exactly that storage/emission mismatch and a macOS test-gating bug; both were fixed in follow-up commits, but the pattern suggests human eyes on the final state are warranted.
Other factors
- All prior inline comments (mine and coderabbit's) are marked resolved and the diff reflects the fixes (
path_is_buffergating,re_encoding_bufferreuse,isMacOSskip,toBunStringin recursive async). - Test coverage is thorough, including the non-ASCII-path and raw-byte regressions that would catch the previously-flagged storage/emission mismatch.
- The C++
previousPathcache restructuring preserves the original one-transferToWTFString-per-call ref-count contract on the string path and bypasses the cache on the Buffer path, which looks correct but is the kind of thing a maintainer should confirm.
|
CI reds on this PR are unrelated flake, not caused by this diff:
Every build phase passed on every platform (Linux/macOS/FreeBSD/Android/Windows, all arches). The other Windows test lanes passed too. I already spent my one re-roll so I am handing off — needs a maintainer to kick CI again or merge past the flake. |
…oding Reviewers on #30484 caught that the storage representation for dirent_path was chosen on args.encoding == .buffer while emission is driven by args.path_is_buffer. Those flags are independent, so the mismatch produces corrupt parentPath whenever one is set and the other isn't: - string path + encoding='buffer' (name=Buffer, parentPath=string): path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café' came back as '/tmp/café'. - Buffer path + default encoding (name=string, parentPath=Buffer): path went through toBunString(.utf8) which replaces invalid UTF-8 bytes with U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF got lost. Storage now tracks path_is_buffer at all three Dirent sites (readdirWithEntries, readdirWithEntriesRecursiveAsync, readdirWithEntriesRecursiveSync). name storage stays gated on args.encoding == .buffer, which matches name_as_buffer. Two new regression tests in fs.test.ts cover both sides: - non-ASCII string path + encoding='buffer' - Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
f961b56 to
87690f3
Compare
There was a problem hiding this comment.
Thanks — b7758f4 ports the fix to types.rs/node_fs.rs, which resolves the Rust↔C++ ABI mismatch I flagged on Bun__Dirent__toJS (and should clear the readdir/opendir/glob CI failures from 87690f3). I didn't find further issues, but given the FFI signature change plus byte-preservation logic threaded through four readdir paths across three platforms, this warrants a human pass once CI reports on the new commit.
Extended reasoning...
Overview
This PR fixes #30482: fs.readdir({ withFileTypes: true, encoding: 'buffer' }) was returning a flat Uint8Array[] instead of Dirent[] with Buffer-typed name/parentPath. The fix spans six files: the C++ binding Bun__Dirent__toJS gains two bool params and a bunStringToBuffer helper; the Rust Dirent struct and extern declaration are updated to match; args::Readdir gains path_is_buffer and tag() is reordered so withFileTypes wins; and all four readdir code paths (non-recursive, recursive sync, recursive async, Windows UTF-16) thread name_as_buffer/path_as_buffer through with Latin-1 storage for raw-byte preservation. The .zig siblings are updated in lockstep as the porting reference. ~230 lines of test coverage are added.
Security risks
None identified. No auth, crypto, permissions, or untrusted-input parsing. The change is confined to how directory entry names/paths are encoded into JS values; the only external-data surface is filesystem entry names, which were already being read.
Level of scrutiny
High. This is not a mechanical change:
- It alters an
extern "C"signature on a hot path (Bun__Dirent__toJSis called once per directory entry for everyreaddir({withFileTypes:true})). Any caller/callee arity or ABI mismatch is UB — the 87690f3 CI run demonstrated exactly that before the Rust side was updated. - The byte-preservation logic (Latin-1 storage gated on
path_is_bufferforparentPathvs.encoding == .bufferforname) is subtle and was already iterated on twice in this PR's review history. - Platform-specific branches (Windows UTF-16→UTF-8→Latin-1 transcode reusing
re_encoding_buffer) need a reviewer familiar with the Windows path-encoding conventions in this codebase.
Other factors
- All five of my and CodeRabbit's earlier inline comments have been addressed in subsequent commits (path_is_buffer gating, recursive-async alignment, re_encoding_buffer reuse, macOS test gating, async parentPath assertions).
- The most recent commit (b7758f4) directly addresses my last 🔴 — the Rust port was missing entirely and the C++ signature change would have caused UB on every
withFileTypescall. The diff now shows the Rust extern, struct fields,tag(), and allappend_entry*impls updated consistently. - robobun's CI status is stale at 87690f3 (pre-Rust-port); the widespread fs/readdir/opendir/glob failures there are explained by the now-fixed ABI mismatch, but there is no green CI on b7758f4 yet.
- Test coverage is thorough (sync/callback/promises × string/Buffer path × recursive, plus non-ASCII and raw-0xFF regression tests with correct platform gating).
|
CI on b7758f4 currently shows two darwin aarch64 shards marked |
|
Update on CI: every build lane passes, every test lane passes except |
…oding Reviewers on #30484 caught that the storage representation for dirent_path was chosen on args.encoding == .buffer while emission is driven by args.path_is_buffer. Those flags are independent, so the mismatch produces corrupt parentPath whenever one is set and the other isn't: - string path + encoding='buffer' (name=Buffer, parentPath=string): path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café' came back as '/tmp/café'. - Buffer path + default encoding (name=string, parentPath=Buffer): path went through toBunString(.utf8) which replaces invalid UTF-8 bytes with U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF got lost. Storage now tracks path_is_buffer at all three Dirent sites (readdirWithEntries, readdirWithEntriesRecursiveAsync, readdirWithEntriesRecursiveSync). name storage stays gated on args.encoding == .buffer, which matches name_as_buffer. Two new regression tests in fs.test.ts cover both sides: - non-ASCII string path + encoding='buffer' - Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
b7758f4 to
3012e8e
Compare
…oding Reviewers on #30484 caught that the storage representation for dirent_path was chosen on args.encoding == .buffer while emission is driven by args.path_is_buffer. Those flags are independent, so the mismatch produces corrupt parentPath whenever one is set and the other isn't: - string path + encoding='buffer' (name=Buffer, parentPath=string): path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café' came back as '/tmp/café'. - Buffer path + default encoding (name=string, parentPath=Buffer): path went through toBunString(.utf8) which replaces invalid UTF-8 bytes with U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF got lost. Storage now tracks path_is_buffer at all three Dirent sites (readdirWithEntries, readdirWithEntriesRecursiveAsync, readdirWithEntriesRecursiveSync). name storage stays gated on args.encoding == .buffer, which matches name_as_buffer. Two new regression tests in fs.test.ts cover both sides: - non-ASCII string path + encoding='buffer' - Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
578bf6c to
1056ad9
Compare
…oding Reviewers on #30484 caught that the storage representation for dirent_path was chosen on args.encoding == .buffer while emission is driven by args.path_is_buffer. Those flags are independent, so the mismatch produces corrupt parentPath whenever one is set and the other isn't: - string path + encoding='buffer' (name=Buffer, parentPath=string): path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café' came back as '/tmp/café'. - Buffer path + default encoding (name=string, parentPath=Buffer): path went through toBunString(.utf8) which replaces invalid UTF-8 bytes with U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF got lost. Storage now tracks path_is_buffer at all three Dirent sites (readdirWithEntries, readdirWithEntriesRecursiveAsync, readdirWithEntriesRecursiveSync). name storage stays gated on args.encoding == .buffer, which matches name_as_buffer. Two new regression tests in fs.test.ts cover both sides: - non-ASCII string path + encoding='buffer' - Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
1056ad9 to
5c0a672
Compare
|
Rebased on main. The conflict was the Zig→Rust port completing on main: |
There was a problem hiding this comment.
Thanks for addressing all the earlier feedback — I have nothing further on the latest revision, but since this threads new flags across the Rust↔C++ FFI boundary and reworks encoding/byte-preservation across several readdir paths, it's worth a human pass before merge.
Extended reasoning...
Overview
This PR fixes fs.readdir({ withFileTypes: true, encoding: 'buffer' }) to return Dirent objects with Buffer names (matching Node.js) instead of a flat Uint8Array[]. It touches four files: src/jsc/bindings/NodeDirent.cpp (C++ binding — adds a bunStringToBuffer helper and extends Bun__Dirent__toJS from 5→7 params), src/runtime/node/types.rs (adds name_as_buffer/path_as_buffer to Dirent and updates the extern signature), src/runtime/node/node_fs.rs (~250 lines: tag() priority fix, path_is_buffer plumbing, Latin-1 storage for raw bytes, ReaddirEntry trait signature changes including replacing apply_encoding with path_is_buffer), and test/js/node/fs/fs.test.ts (+14 tests).
Security risks
None identified. The change is read-only filesystem enumeration; no auth, crypto, permissions, or untrusted-input parsing is involved. The new bunStringToBuffer helper copies bytes into a fresh GC-managed Buffer and handles null/empty strings explicitly.
Level of scrutiny
Moderate-to-high. This is production-critical core node:fs code on a hot path (readdir with withFileTypes), and it changes an extern "C" signature across the Rust↔C++ boundary — exactly the class of change where a missed call site is an ABI bug rather than a compile error. The encoding semantics (Latin-1 storage vs. UTF-8 decode, gated on path_is_buffer vs. encoding) are subtle and went through three rounds of correction during review. There are platform-specific branches (Windows UTF-16→UTF-8→Latin-1) that CI exercises but a reviewer should sanity-check.
Other factors
I left six inline comments over this PR's lifetime — one critical ABI mismatch (Rust extern not updated), one storage/emission flag mismatch, one macOS test gating issue, and three cleanup nits — all now resolved, with the most recent two addressed in commits 2890bd7 and 5c0a672. The bug-hunting pass on the current revision found nothing. Test coverage is thorough (sync/async/promises/recursive × string/Buffer path × non-ASCII + raw-byte regression cases). No CODEOWNERS match the touched paths. CI has been green modulo unrelated infra flakes per the robobun comments.
That said, the change is neither small nor mechanical: it modifies a trait signature used by three impls, removes a parameter (apply_encoding) whose semantics were folded into the new flow, and adds two struct fields that every Dirent construction site must now populate. This exceeds my bar for bot-only approval, so deferring to a human reviewer.
…compat fixes (#32813) Four `node:fs` divergences from Node, found by a differential run against Node v26.3.0. The headline is silent data corruption: the `encoding` argument of `fs.writeSync(fd, string, position, encoding)` was parsed and then ignored. ### 1. `fs.write` / `fs.writeSync` / `filehandle.write` ignore the string encoding ```js const fd = fs.openSync(p, "w"); fs.writeSync(fd, "abc", 0, "utf16le"); // Node: writes 6 bytes 61 00 62 00 63 00 // Bun: writes 3 bytes 61 62 63 (raw UTF-8, encoding ignored) ``` Same for `hex`, `base64`, `latin1`, `ascii`, and for the async `fs.write(fd, string, position, encoding, cb)` and `filehandle.write(string, position, encoding)`, which share the argument parser. **Cause:** `args::Write::from_js` in `src/runtime/node/node_fs.rs` parses the encoding into `args.encoding`, but the string was already converted to UTF-8 by `StringOrBuffer::from_js` a few lines earlier, and `write_inner` / `pwrite_inner` only ever look at `args.buffer.slice()`. A second bug hid the first for some spellings: the parser only advanced past the position slot when it held a *number*, so `fs.writeSync(fd, "abc", null, "utf16le")` never even reached the encoding argument. Node's `WriteString` binding takes the position from `args[2]` regardless of its type and the encoding strictly from `args[3]`. **Fix:** consume the position slot whatever its type (using it only when it is a number, and only when non-negative, matching the sibling Buffer arm), then, once the encoding is known, re-encode the string with `StringOrBuffer::from_js_with_encoding`. That helper is the same path `writeFileSync(p, s, "utf16le")` (which already worked) and `Buffer.from(s, enc)` use, and it already matches Node byte-for-byte for every encoding. The one encoding name the re-encode must not apply to is `"buffer"`: Node's `StringBytes::Write` treats `BUFFER` the same as `UTF8`, while the in-tree string encoder's `Buffer` arm is a verbatim latin1 / low-byte copy, so `writeSync(fd, "\u00e9", 0, "buffer")` would have written 1 byte instead of Node's 2. The re-encode therefore skips both `Encoding::Utf8` and `Encoding::Buffer`, keeping the original UTF-8 conversion for both, and the test table covers the `"buffer"` name for both 8-bit and 16-bit source strings. The one exception is the `"buffer"` encoding name: Node writes UTF-8 for it (`writeSync(fd, "\u00e9", 0, "buffer")` writes `c3 a9`), while the encoder's `Buffer` arm is a verbatim latin1 / low-byte copy, so it is excluded from the re-encode and keeps the original UTF-8 conversion. ### 2. `fs.readdir` with `{ encoding: "buffer" }` returns `Uint8Array[]`, not `Buffer[]` ```js const r = fs.readdirSync(dir, { encoding: "buffer" }); Buffer.isBuffer(r[0]); // Node: true Bun: false (plain Uint8Array) ``` **Cause:** `ret::Readdir::to_js` used `MarkedArrayBuffer::to_js`, which builds a plain `Uint8Array`. The sibling `StringOrBuffer::to_js` (used by `readFile`, `readlink`, `realpath`) already uses `to_node_buffer`, which goes through `JSBufferSubclassStructure`. **Fix:** use `to_node_buffer` there too. Both transfer ownership of the mimalloc-backed bytes to JSC via the same deallocator, so it is a drop-in. This covers the sync, async, callback, and recursive variants, which all funnel through `Readdir::to_js`. ### 3. `fs.Dir` is not disposable ```js using d = fs.opendirSync(dir); // Bun: TypeError, d[Symbol.dispose] is undefined ``` Node's `Dir` prototype has `Symbol.dispose` and `Symbol.asyncDispose` (and both are no-ops when the directory is already closed, so they compose with an explicit `close()`). **Fix:** add both to the `Dir` class in `src/js/node/fs.ts`, delegating to `closeSync()` / `close()` behind an already-closed guard, using the same `this.#handle >= 0` check the class's `Symbol.asyncIterator` already uses. ### 4. An unrecognized flag string throws `ERR_INVALID_ARG_TYPE` ```js fs.openSync(p, "bogus"); // Node: TypeError [ERR_INVALID_ARG_VALUE] // Bun: TypeError [ERR_INVALID_ARG_TYPE] ``` Node's `stringToFlags` throws `ERR_INVALID_ARG_VALUE` for every unrecognized flag string (including the empty string). `FileSystemFlags::from_js` used the `ERR_INVALID_ARG_TYPE` helper. Code that switches on `err.code` gets the wrong branch. **Fix:** switch the three string-flag error sites to `ErrorCode::INVALID_ARG_VALUE`. The messages are unchanged. ### 5. Three more `fs.write` gaps, surfaced by vendoring Node's `test-fs-write.js` Node's `test/parallel/test-fs-write.js` was missing from the vendored Node test suite. It is Node's own regression test for this exact bug class (its [nodejs/node#38168](nodejs/node#38168) block asserts `fs.writeSync(fd, 'abcd', 0, 'hex')` returns 2). Vendoring it surfaced three more gaps on `fs.write`'s string path, all fixed here: - **`validateEncoding` was never called.** Node rejects an odd-length string written with the `hex` encoding (`ERR_INVALID_ARG_VALUE`, `is invalid for data of length N`) from `write`, `writeSync`, and `filehandle.write`. The C++ validator already existed in `NodeValidator.cpp` and was exported from `internal/validators` with no callers; this wires it into the three string write paths where Node calls it. Messages are byte-identical to Node v26.3.0. - **`fs.write` was missing `writeSync`'s `typeof buffer !== "string"` check**, so `new String('x')` (and every other non-string, non-view value) reached the native parser through the string signature instead of throwing `ERR_INVALID_ARG_TYPE`. - **`fs.write` branched on `$isTypedArrayView`, which excludes `DataView`**, so `fs.write(fd, dataView, offset, length, cb)` was silently parsed as the `(position, encoding)` string signature. It now uses Node's `isArrayBufferView` predicate, matching the `writeSync` sibling one function below. ### Tests - `test/js/node/fs/fs.test.ts`: a table of `writeSync(fd, string, ...)` argument shapes whose expected bytes were each produced by Node v26.3.0; the async `fs.write` + `filehandle.write` variants; `readdir` `{ encoding: "buffer" }` across the sync / recursive / promise / callback entry points; the flag error code for `openSync` and `open`. - `test/js/node/fs/dir.test.ts`: `using`, `await using`, and disposing an already-closed `Dir`. - `test/js/node/test/parallel/test-fs-write.js`: Node's upstream test, vendored via `bun node:test:cp`. On the unfixed build it fails on its first assertion (`'ümlaut sechzig'` written with `latin1` reads back as `'ümlaut sechzig'`); with the fix it exits 0. The file is byte-for-byte identical to `nodejs/node@main` with no local patch. Upstream runs it under V8's `--expose_externalize_string` flag; the `createExternalizableString` / `externalizeString` / `isOneByteString` helpers it needs are V8 test globals with no JavaScriptCore equivalent (present upstream since at least Node v22), so they are polyfilled in `test/js/node/test/common/index.js` next to the existing `--expose-gc` polyfill. The test file stays unmodified, and any other vendored Node test that uses that flag works without a local patch. All fail on the current build and pass with the fix. The vendored Node suites `test-fs-write-sync`, `test-fs-write-sync-optional-params`, `test-fs-write-optional-params`, `test-fs-write-buffer`, `test-fs-open`, `test-fs-open-numeric-flags`, `test-fs-readdir*`, `test-fs-opendir`, and `test-fs-promises-file-handle-{write,append-file,dispose}` still pass. ### Out of scope, noted for completeness - `readdirSync(dir, { withFileTypes: true, encoding: "buffer" })`: Node returns `Dirent[]` with `Buffer` names; Bun ignores `withFileTypes` when the encoding is `buffer` and returns a flat array (now of `Buffer`s instead of `Uint8Array`s). Matching Node needs a `Dirent` variant whose `name` is a `Buffer`, a larger change than the one-line swap here. That half is #30482, and #30484 already has a fix open for it (the `Dirent` name representation, the `args::Readdir::tag` precedence, and `NodeDirent.cpp`). The two PRs touch disjoint code: #30484 does not change `ret::Readdir::Buffers` conversion, and this PR does not touch `Dirent`, so neither replaces the other. - Flag strings are case-insensitive in Bun (`"A+"` opens; Node throws `ERR_INVALID_ARG_VALUE`). The uppercase aliases are hand-enumerated in `FILE_SYSTEM_FLAGS_MAP` and date back to the original implementation, so I left that deliberate leniency for a maintainer to decide on rather than silently tightening it. #32966 now does exactly that: it rewrites `FileSystemFlags::from_js` to match Node's `stringToFlags` (case-sensitive, exhaustive, always `ERR_INVALID_ARG_VALUE`) and subsumes the small `types.rs` error-code change in this PR, so whichever of the two lands second rebases that hunk away. The rest of this PR is independent of #32966.
`fs.readdir`/`readdirSync` with `{ withFileTypes: true, encoding: "buffer" }`
previously returned a flat `Uint8Array[]` instead of `Dirent[]`.
`Arguments.Readdir.tag()` matched the `.buffer` encoding before checking
`with_file_types`, so every withFileTypes+buffer call fell through to the
raw-buffer code path and never constructed Dirents.
Fix:
- withFileTypes takes priority in tag() regardless of encoding.
- When encoding == .buffer, store the name/path bytes as Latin-1 so raw
filesystem bytes survive round-trip without the UTF-8-via-UTF-16
replacement that toBunString(.buffer) does.
- Thread two flags (name_as_buffer, path_as_buffer) through Dirent to the
C++ Bun__Dirent__toJS, which emits the corresponding fields as Buffer
instead of JSString.
- Track path_is_buffer on Arguments.Readdir so Dirent.parentPath mirrors
Node: Buffer iff the path arg was a Buffer, independent of encoding.
Covers sync, async callback, promises, and recursive paths. Tests in
test/js/node/fs/fs.test.ts.
Closes #30482. Same root cause as #27914.
…oding Reviewers on #30484 caught that the storage representation for dirent_path was chosen on args.encoding == .buffer while emission is driven by args.path_is_buffer. Those flags are independent, so the mismatch produces corrupt parentPath whenever one is set and the other isn't: - string path + encoding='buffer' (name=Buffer, parentPath=string): path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café' came back as '/tmp/café'. - Buffer path + default encoding (name=string, parentPath=Buffer): path went through toBunString(.utf8) which replaces invalid UTF-8 bytes with U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF got lost. Storage now tracks path_is_buffer at all three Dirent sites (readdirWithEntries, readdirWithEntriesRecursiveAsync, readdirWithEntriesRecursiveSync). name storage stays gated on args.encoding == .buffer, which matches name_as_buffer. Two new regression tests in fs.test.ts cover both sides: - non-ASCII string path + encoding='buffer' - Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
1. test/fs.test.ts: gate the raw 0xFF-byte test with `isWindows || isMacOS`. macOS APFS/HFS+ reject non-UTF-8 filenames with EILSEQ, so `mkdirSync` of a 0xFF-named dir fails before any assertion. Same gate pattern as `test/js/bun/util/filesystem_router.test.ts:475`. 2. src/runtime/node/node_fs.zig: reuse the `re_encoding_buffer` that's already allocated once outside the loop (for `is_u16 and encoding != .utf8`) in the Windows Dirent + encoding=buffer branch, matching the pattern the neighboring `bun.String` branch already uses. Saves one pool acquisition per directory entry on Windows.
Cover the async codepaths with Buffer-path parentPath assertions so regressions in the path_is_buffer threading can be caught independently from the sync path.
windows-11-aarch64-test-bun shard failed on an unrelated flaky test (test/cli/install/bun-install-registry.test.ts peer-dep resolution expecting '1.0.10'). Diff is green on every other shard across Linux, macOS, FreeBSD, Android, and Windows-2019.
The Rust rewrite (#30412) replaced the Zig node_fs implementation with src/runtime/node/node_fs.rs, bypassing this PR's original Zig changes and reintroducing the original bug: tag() matched .buffer encoding before with_file_types. Port the whole fix: - args::Readdir gains path_is_buffer (set in from_js when the path arg was a PathLike::Buffer). - tag() now prioritizes with_file_types over encoding. - Dirent struct gains name_as_buffer / path_as_buffer flags and the Bun__Dirent__toJS FFI signature grows two bool args to match NodeDirent.cpp — without this, the bools were reading garbage from the stack and could emit Buffers for plain string Dirents (test-fs-readdir-recursive.js, pnpm-migration saw parentPath as an object). - ReaddirEntry trait's three append_entry* methods gain path_is_buffer; Dirent impl stores name/path as Latin-1 for the buffer emission paths (preserves raw filesystem bytes 1:1 instead of UTF-8-via-UTF-16 round-tripping). - All three readdir paths (non-recursive, recursive async, recursive sync, plus the Windows UTF-16 arm) gate dirent_path storage on path_is_buffer and thread it to append_entry*. - Recursive-async was using apply_encoding=false (clone_utf8); align with sync/non-recursive by passing args.encoding through.
After aligning the recursive-async path with sync (both pass apply_encoding=true), the parameter was invariantly true: the clone_utf8 else-arms in the BunString and Dirent impls were dead and the trait doc described a sync/async distinction that no longer exists. Drop the parameter and the dead branches.
Both call sites are identical since apply_encoding was removed; the labels no longer contrast anything and the sync-side comment described an argument that is no longer passed.
5c0a672 to
56da557
Compare
|
Rebased onto current main ( Re-verified that the bug still reproduces on today's main and that this branch fixes it: const fs = require("fs");
const dir = fs.mkdtempSync("/tmp/readdir-");
fs.mkdirSync(dir + "/subdir");
console.log(fs.readdirSync(dir, { withFileTypes: true, encoding: "buffer" }));
// main: [ <Buffer 73 75 62 64 69 72> ]
// this branch / node: [ Dirent { name: <Buffer 73 75 62 64 69 72>, ... } ]Checked against node v26 on three shapes, all now match:
Non-utf8 filenames round-trip byte-exact ( Locally on the debug (ASAN) build: For the record, the failures on the previous build (65063) were all flake on lanes this diff doesn't touch:
One divergence from node this PR deliberately leaves alone: on Windows, |
|
CI status (build #68767): neither red lane is caused by this diff, which touches only
The readdir/Dirent tests pass on every lane that has run them and under the local fail-before/pass-after gate. The re-trigger re-roll is spent, and re-running will not turn CI green since the cookie failure is deterministic. Flagging for a maintainer: the |
|
Superseded by #36064, which combines this with the cp walker fix so |
|
Re-checked on a debug build of main that includes #32813 (which touched Current state of the candidates, for whoever picks this up: this PR (readdir only) and #36064 (the same readdir change plus the |
|
Correction to the note above: #36064 only returns Dirents with a Buffer |
|
Closing in favor of #42812, which fixes the same bug (#27914) on current main with a smaller change. This branch is about 2,000 commits behind main and conflicts with it. I built #42812 and compared it with Node v26.3.0 on Linux and Windows. All The 6 tests that fail there cover the one thing this PR has and #42812 does not: a #36064 carries another copy of the readdir change next to its cp change. Its cp walker and tests pass unchanged on top of #42812. |
Closes #30482. Same root cause as #27914.
Repro
Node.js:
Bun before:
Cause
Arguments.Readdir.tag()insrc/runtime/node/node_fs.zigmatched the.bufferencoding before checkingwith_file_types:So every
withFileTypes: true, encoding: 'buffer'call fell through tothe flat
Buffer[]branch and never constructed Dirents.Fixing just the branch ordering would produce Dirents whose
namewas aString, not a Buffer — the
Direntstruct and the C++Bun__Dirent__toJSonly knew how to emit strings.
Fix
tag(): withFileTypes wins, regardless of encoding.Arguments.Readdirrecordspath_is_bufferatfromJStime. Node'srule for
parentPathis: Buffer iff the user-suppliedpathwas aBuffer, independent of the
encodingoption.Direntstruct gainsname_as_bufferandpath_as_bufferflags andthreads them to
Bun__Dirent__toJS.encoding == .buffer, the Zig side stores name/path bytes withbun.String.cloneLatin1so raw filesystem bytes survive 1:1.toBunString(.buffer)goes through UTF-16 and replaces invalid UTF-8sequences with U+FFFD — wrong for a Buffer that's supposed to be
byte-transparent.
Bun__Dirent__toJSreads.span8()directly and hands the bytes toWebCore::createBufferwhen a field should be a Buffer. Fresh Bufferper Dirent for
path(Node behavior; the JSString cache is bypassed).All four paths covered: sync, async callback, promises, recursive (async
and sync).
Tests
test/js/node/fs/fs.test.tsgains adescribe.concurrentblock with8 tests covering every combination — sync, async callback, promises,
recursive + string/Buffer path — plus two controls asserting the other
two branches (withFileTypes-only, encoding=buffer-only) still behave
unchanged.
Gate:
USE_SYSTEM_BUN=1 bun test …— 6 fail, 2 pass (controls).bun bd test …— 8 pass.Notes
Supersedes the now-stale #27914 → #27917 attempt. That PR predates the
src/bun.js → src/runtimemove and has merge conflicts; this one isrebased on current main, preserves raw filesystem bytes for non-UTF-8
names, and handles Node's
parentPathBuffer/String asymmetry.