Skip to content

fs.readdir: return Dirent with Buffer name when encoding is "buffer" - #30484

Closed
robobun wants to merge 10 commits into
mainfrom
farm/19bbeef8/fix-readdir-buffer-dirent
Closed

robobun wants to merge 10 commits into
mainfrom
farm/19bbeef8/fix-readdir-buffer-dirent

Conversation

@robobun

@robobun robobun commented May 11, 2026

Copy link
Copy Markdown
Collaborator

Closes #30482. Same root cause as #27914.

Repro

const fs = require('fs');
fs.readdir(Buffer.from('.'), { withFileTypes: true, encoding: 'buffer' }, (err, entries) => {
  console.log(entries[0]);
});

Node.js:

Dirent {
  name: <Buffer 72 65 61 64 64 69 72 2d 62 75 66 66 65 72 2e 63 6a 73>,
  parentPath: <Buffer 2e>,
  Symbol(type): 1
}

Bun before:

Uint8Array(18) [ 114, 101, 97, 100, 100, 105, 114, ... ]

Cause

Arguments.Readdir.tag() in src/runtime/node/node_fs.zig matched the
.buffer encoding before checking with_file_types:

return switch (this.encoding) {
    .buffer => .buffers,             // <-- claims the whole buffer branch
    else => if (this.with_file_types) .with_file_types else .files,
};

So every withFileTypes: true, encoding: 'buffer' call fell through to
the flat Buffer[] branch and never constructed Dirents.

Fixing just the branch ordering would produce Dirents whose name was a
String, not a Buffer — the Dirent struct and the C++ Bun__Dirent__toJS
only knew how to emit strings.

Fix

  • tag(): withFileTypes wins, regardless of encoding.
  • Arguments.Readdir records path_is_buffer at fromJS time. Node's
    rule for parentPath is: Buffer iff the user-supplied path was a
    Buffer, independent of the encoding option.
  • Dirent struct gains name_as_buffer and path_as_buffer flags and
    threads them to Bun__Dirent__toJS.
  • When encoding == .buffer, the Zig side stores name/path bytes with
    bun.String.cloneLatin1 so raw filesystem bytes survive 1:1.
    toBunString(.buffer) goes through UTF-16 and replaces invalid UTF-8
    sequences with U+FFFD — wrong for a Buffer that's supposed to be
    byte-transparent.
  • Bun__Dirent__toJS reads .span8() directly and hands the bytes to
    WebCore::createBuffer when a field should be a Buffer. Fresh Buffer
    per Dirent for path (Node behavior; the JSString cache is bypassed).

All four paths covered: sync, async callback, promises, recursive (async
and sync).

Tests

test/js/node/fs/fs.test.ts gains a describe.concurrent block with
8 tests covering every combination — sync, async callback, promises,
recursive + string/Buffer path — plus two controls asserting the other
two branches (withFileTypes-only, encoding=buffer-only) still behave
unchanged.

Gate:

  • USE_SYSTEM_BUN=1 bun test … — 6 fail, 2 pass (controls).
  • bun bd test … — 8 pass.

Notes

Supersedes the now-stale #27914 → #27917 attempt. That PR predates the
src/bun.js → src/runtime move and has merge conflicts; this one is
rebased on current main, preserves raw filesystem bytes for non-UTF-8
names, and handles Node's parentPath Buffer/String asymmetry.

@robobun

robobun commented May 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

@github-actions

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. fix(fs): readdir with withFileTypes and encoding "buffer" returns Dirent with Buffer name #27917 - Same fix for fs.readdir returning Dirent with Buffer name when encoding: "buffer" and withFileTypes: true (this PR explicitly supersedes fix(fs): readdir with withFileTypes and encoding "buffer" returns Dirent with Buffer name #27917)

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented May 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR implements Node.js-compatible fs.readdir({ withFileTypes: true, encoding: 'buffer' }) by returning Dirent objects whose name and parentPath are Buffers when appropriate; changes touch Zig types, argument parsing, directory enumeration (non-recursive and recursive), C++ bindings, and tests.

Changes

Dirent Buffer Name and Path Support

Layer / File(s) Summary
Dirent Data Model
src/runtime/node/types.zig
Dirent struct gains name_as_buffer and path_as_buffer booleans; extern Bun__Dirent__toJS updated and Dirent.toJS() forwards flags.
Argument Parsing
src/runtime/node/node_fs.zig
Arguments.Readdir adds path_is_buffer; fromJS() sets it; tag() prioritizes with_file_types and maps encoding == .buffer to buffers variant.
Non-Recursive Directory Reading
src/runtime/node/node_fs.zig
readdirWithEntries chooses dirent_path creation based on path_is_buffer, uses Latin‑1 cloning for encoding == .buffer, handles Windows UTF‑16→UTF‑8→Latin‑1, and sets name_as_buffer/path_as_buffer.
Recursive Async Directory Reading
src/runtime/node/node_fs.zig
readdirWithEntriesRecursiveAsync updates dirent_path_prev caching keyed on path_is_buffer, emits Latin‑1 names for buffer encoding, and sets buffer flags.
Recursive Sync Directory Reading
src/runtime/node/node_fs.zig
readdirWithEntriesRecursiveSync mirrors async recursive changes for consistent behavior.
C++ Host Binding
src/jsc/bindings/NodeDirent.cpp
Adds bunStringToBuffer helper preserving 8‑bit bytes and transcoding 16‑bit strings to UTF‑8; extends Bun__Dirent__toJS to emit Buffer or JSString for name/path based on flags with previousPath caching.
Test Coverage
test/js/node/fs/fs.test.ts
Adds concurrent suite validating sync, callback, promise, and recursive readdir/readdirSync behaviors with withFileTypes:true + encoding:'buffer', plus control and non‑ASCII/posix cases.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: returning Dirent objects with Buffer names when encoding is set to 'buffer'.
Description check ✅ Passed The description comprehensively covers what the PR does, the root cause, the fix approach, testing, and references the linked issue.
Linked Issues check ✅ Passed All coding requirements from #30482 are met: Dirent objects returned when withFileTypes is true, Buffer name/parentPath emitted correctly, raw bytes preserved, and Node.js parity across all code paths.
Out of Scope Changes check ✅ Passed All changes directly address the requirements in #30482; no out-of-scope modifications detected.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/runtime/node/node_fs.zig`:
- Around line 2234-2238: The code currently chooses how to store
Dirent.parentPath based on args.encoding instead of the flag indicating the
original argument type; update all branches that decide Dirent.parentPath (look
for logic near path_is_buffer, args.encoding, Dirent.parentPath, and calls to
toBunString) to first check path_is_buffer and only treat parentPath as binary
when path_is_buffer == true; when path_is_buffer is false, continue to honor
args.encoding for returning a JS string (use toBunString for string conversions)
and when true, preserve raw bytes (return a Buffer/TypedArray) regardless of
args.encoding so binary paths keep byte transparency.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 33f79881-bfe9-48f0-bd97-f7c152b4fe71

📥 Commits

Reviewing files that changed from the base of the PR and between 450072b and 18dad3e.

📒 Files selected for processing (4)
  • src/jsc/bindings/NodeDirent.cpp
  • src/runtime/node/node_fs.zig
  • src/runtime/node/types.zig
  • test/js/node/fs/fs.test.ts

Comment thread src/runtime/node/node_fs.zig Resolved
Comment thread src/runtime/node/node_fs.zig Resolved
robobun added a commit that referenced this pull request May 11, 2026
…oding

Reviewers on #30484 caught that the storage representation for dirent_path
was chosen on args.encoding == .buffer while emission is driven by
args.path_is_buffer. Those flags are independent, so the mismatch produces
corrupt parentPath whenever one is set and the other isn't:

- string path + encoding='buffer' (name=Buffer, parentPath=string):
  path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café'
  came back as '/tmp/café'.
- Buffer path + default encoding (name=string, parentPath=Buffer): path
  went through toBunString(.utf8) which replaces invalid UTF-8 bytes with
  U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF
  got lost.

Storage now tracks path_is_buffer at all three Dirent sites
(readdirWithEntries, readdirWithEntriesRecursiveAsync,
readdirWithEntriesRecursiveSync). name storage stays gated on
args.encoding == .buffer, which matches name_as_buffer.

Two new regression tests in fs.test.ts cover both sides:
- non-ASCII string path + encoding='buffer'
- Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/runtime/node/node_fs.zig`:
- Around line 4840-4846: The recursive async branch currently sets
dirent_path_prev to bun.String.cloneUTF8(path_u8) for non-buffer paths, which
diverges from the non-recursive and sync recursive branches that use
jsc.WebCore.encoding.toBunString(path_u8, args.encoding); update the recursive
async path decoding to call jsc.WebCore.encoding.toBunString(path_u8,
args.encoding) when args.path_is_buffer is false (keep
bun.String.cloneLatin1(path_u8) when args.path_is_buffer is true) so
Dirent.parentPath matches the sync behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 49f5e651-b35c-4316-a683-cfe37f6462f8

📥 Commits

Reviewing files that changed from the base of the PR and between 18dad3e and fd8929b.

📒 Files selected for processing (2)
  • src/runtime/node/node_fs.zig
  • test/js/node/fs/fs.test.ts

Comment thread src/runtime/node/node_fs.zig Resolved
Comment thread src/runtime/node/node_fs.zig Resolved
Comment thread test/js/node/fs/fs.test.ts Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
src/runtime/node/node_fs.zig (1)

4837-4858: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Honor args.encoding in the recursive async Dirent path too.

This branch still hardcodes UTF-8 for both Dirent.parentPath and non-buffer Dirent.name, so fs.promises.readdir(..., { recursive: true, withFileTypes: true, encoding: "latin1" | "hex" | ... }) diverges from the non-recursive and recursive sync implementations that already use toBunString(..., args.encoding).

Suggested fix
-                        dirent_path_prev = if (args.path_is_buffer)
-                            bun.String.cloneLatin1(path_u8)
-                        else
-                            bun.String.cloneUTF8(path_u8);
+                        dirent_path_prev = if (args.path_is_buffer)
+                            bun.String.cloneLatin1(path_u8)
+                        else
+                            jsc.WebCore.encoding.toBunString(path_u8, args.encoding);
@@
-                        .name = if (args.encoding == .buffer)
-                            bun.String.cloneLatin1(utf8_name)
-                        else
-                            bun.String.cloneUTF8(utf8_name),
+                        .name = if (args.encoding == .buffer)
+                            bun.String.cloneLatin1(utf8_name)
+                        else
+                            jsc.WebCore.encoding.toBunString(utf8_name, args.encoding),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/runtime/node/node_fs.zig` around lines 4837 - 4858, The recursive async
Dirent branch is hardcoding UTF-8 for parentPath and non-buffer name; update the
conversion to honor args.encoding by replacing the direct
bun.String.cloneLatin1/cloneUTF8 uses with the encoding-aware helper (e.g., call
toBunString or a similar helper) for both utf8_name and path_u8 so
entries.append uses toBunString(utf8_name, args.encoding) for .name when not
buffer and toBunString(path_u8, args.encoding) for .path when path_as_buffer is
false, ensuring dirent_path_prev, entries.append, args.encoding,
bun.String.cloneLatin1/cloneUTF8 and utf8_name are updated accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/js/node/fs/fs.test.ts`:
- Around line 2713-2779: Modify the two async tests ("readdir (async callback):
Dirent with Buffer name" and "promises.readdir: Dirent with Buffer name") to
exercise a Buffer input path and assert parentPath semantics: pass
Buffer.from(String(dir)) instead of String(dir) to fs.readdir /
promises.readdir, and add assertions that each Dirent.entry.parentPath is an
instance of Buffer and that its string value matches the directory (e.g.,
parentPath.toString() === String(dir)); reference the Dirent.parentPath property
when adding these checks.

---

Duplicate comments:
In `@src/runtime/node/node_fs.zig`:
- Around line 4837-4858: The recursive async Dirent branch is hardcoding UTF-8
for parentPath and non-buffer name; update the conversion to honor args.encoding
by replacing the direct bun.String.cloneLatin1/cloneUTF8 uses with the
encoding-aware helper (e.g., call toBunString or a similar helper) for both
utf8_name and path_u8 so entries.append uses toBunString(utf8_name,
args.encoding) for .name when not buffer and toBunString(path_u8, args.encoding)
for .path when path_as_buffer is false, ensuring dirent_path_prev,
entries.append, args.encoding, bun.String.cloneLatin1/cloneUTF8 and utf8_name
are updated accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 88e65d0b-5e9a-45b3-9060-d7d188cec6fa

📥 Commits

Reviewing files that changed from the base of the PR and between 0a999d6 and 131c96a.

📒 Files selected for processing (2)
  • src/runtime/node/node_fs.zig
  • test/js/node/fs/fs.test.ts

Comment thread test/js/node/fs/fs.test.ts Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — all my earlier concerns are addressed and the bug-hunting pass found nothing new on this revision; deferring to a human reviewer given the FFI signature change and ref-counting-sensitive C++ paths, plus CI is currently red on 00ad4b1.

Extended reasoning...

Overview

This PR fixes fs.readdir({ withFileTypes: true, encoding: 'buffer' }) to return Dirent objects with Buffer names (and Buffer parentPath when the input path was a Buffer), matching Node.js. It touches four files: Arguments.Readdir and three readdir implementations in src/runtime/node/node_fs.zig, the Dirent struct and extern signature in src/runtime/node/types.zig, the Bun__Dirent__toJS C++ binding in src/jsc/bindings/NodeDirent.cpp (new bunStringToBuffer helper, two new bool parameters), and ~230 lines of new tests in fs.test.ts.

Security risks

None identified. The change is confined to how directory entry names/paths are surfaced to JS — no auth, no user-controlled allocation sizes beyond what readdir already handled, no new syscalls. The new bunStringToBuffer copies bytes into a fresh JS Buffer via WebCore::createBuffer, which is the standard pattern.

Level of scrutiny

Medium-high. The change itself is a well-scoped Node-compat fix, but it:

  • Changes an extern "C" FFI signature shared between Zig and C++ (Bun__Dirent__toJS gains two bool params) — ABI mismatches here fail loudly, but the build is currently red on both build-zig and build-cpp across all platforms, so this hasn't been verified green yet.
  • Adds a new code path in C++ that calls transferToWTFString() on a BunString* whose underlying storage is ref-counted and shared across multiple Dirent entries (via dirent_path.ref() on the Zig side). The string-path branch already did this, so the pattern is established, but the Buffer branch bypasses the previousPath JSString cache and produces a fresh Buffer per entry — worth a human sanity-check on ref balance.
  • Threads two independent flags (name_as_buffer ← encoding == .buffer, path_as_buffer ← path_is_buffer) through three separate readdir implementations (non-recursive, recursive-async, recursive-sync) plus a Windows-only UTF-16 branch. My earlier review caught a storage/emission mismatch between these flags; that's now fixed, but the duplication across four sites is the kind of thing that benefits from a maintainer's eye.

Other factors

  • Prior feedback resolved: My three earlier inline comments (storage gated on wrong flag → fixed in fd8929b; redundant per-entry path_buffer_pool churn → fixed in 131c96a; macOS EILSEQ test gating → fixed in 131c96a) and CodeRabbit's two findings (same storage-flag issue; recursive-async cloneUTF8 divergence → fixed in 00ad4b1; async parentPath test coverage → added in c512595) are all addressed and marked resolved.
  • CI is red: robobun reports build failures on 00ad4b1 across every platform (build-zig and build-cpp). The latest commit c512595 only touches tests, so the build state is unlikely to have changed. This needs to go green before merge regardless.
  • Test coverage is thorough: 14 new tests covering sync/callback/promises × string/Buffer path × recursive/non-recursive, plus non-ASCII and raw-byte regression cases for the storage/emission bug I flagged earlier.
  • No CODEOWNERS hit that I'm aware of, but node_fs.zig and JSC bindings are core runtime code.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found on this revision — all earlier feedback has been addressed — but the change spans a Zig↔C++ extern signature, ref-counted string handling in Bun__Dirent__toJS, and three separate readdir code paths (plus a Windows UTF-16 branch), so it's worth a human pass before merging.

Extended reasoning...

Overview

This PR fixes fs.readdir({ withFileTypes: true, encoding: 'buffer' }) to return Dirent[] with Buffer name/parentPath instead of a flat Uint8Array[]. It touches:

  • src/runtime/node/node_fs.zig — Arguments.Readdir.tag() priority fix, new path_is_buffer flag, and Latin-1 vs. toBunString storage selection in three readdir paths (non-recursive, recursive async, recursive sync) plus the Windows is_u16 Dirent branch.
  • src/runtime/node/types.zig — Dirent gains name_as_buffer/path_as_buffer and the Bun__Dirent__toJS extern signature grows two bool params.
  • src/jsc/bindings/NodeDirent.cpp — new bunStringToBuffer helper; Bun__Dirent__toJS restructured so the previousPath JSString cache only applies to the string path and Buffer paths get a fresh WebCore::createBuffer per Dirent.
  • test/js/node/fs/fs.test.ts — ~15 new tests covering sync/callback/promises × string/Buffer path × recursive, plus non-ASCII and raw-0xFF-byte regression cases.

Security risks

None identified. The change is read-only fs metadata emission; no new syscalls, no user-controlled allocation sizes beyond what readdir already does, and createBuffer copies from a bounded WTF::String span.

Level of scrutiny

Medium-high. fs.readdir is a hot Node-compat API, and the change crosses an FFI boundary (extern signature change must stay in lockstep on both sides), reshuffles ref-counted BunString consumption inside Bun__Dirent__toJS, and encodes a subtle invariant (storage representation must match the independent name_as_buffer/path_as_buffer emission flags) across three code sites. Earlier revisions had exactly that storage/emission mismatch and a macOS test-gating bug; both were fixed in follow-up commits, but the pattern suggests human eyes on the final state are warranted.

Other factors

  • All prior inline comments (mine and coderabbit's) are marked resolved and the diff reflects the fixes (path_is_buffer gating, re_encoding_buffer reuse, isMacOS skip, toBunString in recursive async).
  • Test coverage is thorough, including the non-ASCII-path and raw-byte regressions that would catch the previously-flagged storage/emission mismatch.
  • The C++ previousPath cache restructuring preserves the original one-transferToWTFString-per-call ref-count contract on the string path and bypasses the cache on the Buffer path, which looks correct but is the kind of thing a maintainer should confirm.

@robobun

robobun commented May 11, 2026

Copy link
Copy Markdown
Collaborator Author

CI reds on this PR are unrelated flake, not caused by this diff:

  • debian-13-x64-asan-test-bun — test/js/web/fetch/fetch-http2-client.test.ts fails after 4 retries (HTTP/2 client, unrelated to fs.readdir).
  • darwin-14-aarch64-test-bun — SIGTERM (exit 3) at test 1749/2027 (test/js/web/fetch/body-stream.test.ts), i.e. the shard hit its time budget; fs.test.ts ran and passed earlier in the same shard.

Every build phase passed on every platform (Linux/macOS/FreeBSD/Android/Windows, all arches). The other Windows test lanes passed too. I already spent my one re-roll so I am handing off — needs a maintainer to kick CI again or merge past the flake.

robobun added a commit that referenced this pull request May 15, 2026
…oding

Reviewers on #30484 caught that the storage representation for dirent_path
was chosen on args.encoding == .buffer while emission is driven by
args.path_is_buffer. Those flags are independent, so the mismatch produces
corrupt parentPath whenever one is set and the other isn't:

- string path + encoding='buffer' (name=Buffer, parentPath=string):
  path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café'
  came back as '/tmp/café'.
- Buffer path + default encoding (name=string, parentPath=Buffer): path
  went through toBunString(.utf8) which replaces invalid UTF-8 bytes with
  U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF
  got lost.

Storage now tracks path_is_buffer at all three Dirent sites
(readdirWithEntries, readdirWithEntriesRecursiveAsync,
readdirWithEntriesRecursiveSync). name storage stays gated on
args.encoding == .buffer, which matches name_as_buffer.

Two new regression tests in fs.test.ts cover both sides:
- non-ASCII string path + encoding='buffer'
- Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
@robobun
robobun force-pushed the farm/19bbeef8/fix-readdir-buffer-dirent branch from f961b56 to 87690f3 Compare May 15, 2026 01:18
Comment thread src/jsc/bindings/NodeDirent.cpp Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — b7758f4 ports the fix to types.rs/node_fs.rs, which resolves the Rust↔C++ ABI mismatch I flagged on Bun__Dirent__toJS (and should clear the readdir/opendir/glob CI failures from 87690f3). I didn't find further issues, but given the FFI signature change plus byte-preservation logic threaded through four readdir paths across three platforms, this warrants a human pass once CI reports on the new commit.

Extended reasoning...

Overview

This PR fixes #30482: fs.readdir({ withFileTypes: true, encoding: 'buffer' }) was returning a flat Uint8Array[] instead of Dirent[] with Buffer-typed name/parentPath. The fix spans six files: the C++ binding Bun__Dirent__toJS gains two bool params and a bunStringToBuffer helper; the Rust Dirent struct and extern declaration are updated to match; args::Readdir gains path_is_buffer and tag() is reordered so withFileTypes wins; and all four readdir code paths (non-recursive, recursive sync, recursive async, Windows UTF-16) thread name_as_buffer/path_as_buffer through with Latin-1 storage for raw-byte preservation. The .zig siblings are updated in lockstep as the porting reference. ~230 lines of test coverage are added.

Security risks

None identified. No auth, crypto, permissions, or untrusted-input parsing. The change is confined to how directory entry names/paths are encoded into JS values; the only external-data surface is filesystem entry names, which were already being read.

Level of scrutiny

High. This is not a mechanical change:

  • It alters an extern "C" signature on a hot path (Bun__Dirent__toJS is called once per directory entry for every readdir({withFileTypes:true})). Any caller/callee arity or ABI mismatch is UB — the 87690f3 CI run demonstrated exactly that before the Rust side was updated.
  • The byte-preservation logic (Latin-1 storage gated on path_is_buffer for parentPath vs. encoding == .buffer for name) is subtle and was already iterated on twice in this PR's review history.
  • Platform-specific branches (Windows UTF-16→UTF-8→Latin-1 transcode reusing re_encoding_buffer) need a reviewer familiar with the Windows path-encoding conventions in this codebase.

Other factors

  • All five of my and CodeRabbit's earlier inline comments have been addressed in subsequent commits (path_is_buffer gating, recursive-async alignment, re_encoding_buffer reuse, macOS test gating, async parentPath assertions).
  • The most recent commit (b7758f4) directly addresses my last 🔴 — the Rust port was missing entirely and the C++ signature change would have caused UB on every withFileTypes call. The diff now shows the Rust extern, struct fields, tag(), and all append_entry* impls updated consistently.
  • robobun's CI status is stale at 87690f3 (pre-Rust-port); the widespread fs/readdir/opendir/glob failures there are explained by the now-fixed ABI mismatch, but there is no green CI on b7758f4 yet.
  • Test coverage is thorough (sync/callback/promises × string/Buffer path × recursive, plus non-ASCII and raw-0xFF regression tests with correct platform gating).

@robobun

robobun commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

CI on b7758f4 currently shows two darwin aarch64 shards marked Expired — buildkite runner queue timeout, both agents at once, not caused by this diff. The rest of the matrix is still running or passing. I've already spent my one re-roll earlier in this PR's life, so handing off here: needs a maintainer to re-kick CI on the darwin aarch64 lane or merge past it.

@robobun

robobun commented May 15, 2026

Copy link
Copy Markdown
Collaborator Author

Update on CI: every build lane passes, every test lane passes except darwin-14-aarch64-test-bun which is marked Expired (runner queue timeout — the darwin-26-aarch64 shard also expired initially and then passed on retry). Diff is green everywhere it ran. Still out of re-rolls; a maintainer kick on the darwin-14 lane should land this.

robobun added a commit that referenced this pull request Jun 5, 2026
…oding

Reviewers on #30484 caught that the storage representation for dirent_path
was chosen on args.encoding == .buffer while emission is driven by
args.path_is_buffer. Those flags are independent, so the mismatch produces
corrupt parentPath whenever one is set and the other isn't:

- string path + encoding='buffer' (name=Buffer, parentPath=string):
  path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café'
  came back as '/tmp/café'.
- Buffer path + default encoding (name=string, parentPath=Buffer): path
  went through toBunString(.utf8) which replaces invalid UTF-8 bytes with
  U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF
  got lost.

Storage now tracks path_is_buffer at all three Dirent sites
(readdirWithEntries, readdirWithEntriesRecursiveAsync,
readdirWithEntriesRecursiveSync). name storage stays gated on
args.encoding == .buffer, which matches name_as_buffer.

Two new regression tests in fs.test.ts cover both sides:
- non-ASCII string path + encoding='buffer'
- Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
@robobun
robobun force-pushed the farm/19bbeef8/fix-readdir-buffer-dirent branch from b7758f4 to 3012e8e Compare June 5, 2026 20:34
Comment thread src/runtime/node/node_fs.rs Resolved
Comment thread src/runtime/node/node_fs.rs Resolved
robobun added a commit that referenced this pull request Jun 16, 2026
…oding

Reviewers on #30484 caught that the storage representation for dirent_path
was chosen on args.encoding == .buffer while emission is driven by
args.path_is_buffer. Those flags are independent, so the mismatch produces
corrupt parentPath whenever one is set and the other isn't:

- string path + encoding='buffer' (name=Buffer, parentPath=string):
  path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café'
  came back as '/tmp/café'.
- Buffer path + default encoding (name=string, parentPath=Buffer): path
  went through toBunString(.utf8) which replaces invalid UTF-8 bytes with
  U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF
  got lost.

Storage now tracks path_is_buffer at all three Dirent sites
(readdirWithEntries, readdirWithEntriesRecursiveAsync,
readdirWithEntriesRecursiveSync). name storage stays gated on
args.encoding == .buffer, which matches name_as_buffer.

Two new regression tests in fs.test.ts cover both sides:
- non-ASCII string path + encoding='buffer'
- Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
@robobun
robobun force-pushed the farm/19bbeef8/fix-readdir-buffer-dirent branch from 578bf6c to 1056ad9 Compare June 16, 2026 19:00
robobun added a commit that referenced this pull request Jun 26, 2026
…oding

Reviewers on #30484 caught that the storage representation for dirent_path
was chosen on args.encoding == .buffer while emission is driven by
args.path_is_buffer. Those flags are independent, so the mismatch produces
corrupt parentPath whenever one is set and the other isn't:

- string path + encoding='buffer' (name=Buffer, parentPath=string):
  path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café'
  came back as '/tmp/café'.
- Buffer path + default encoding (name=string, parentPath=Buffer): path
  went through toBunString(.utf8) which replaces invalid UTF-8 bytes with
  U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF
  got lost.

Storage now tracks path_is_buffer at all three Dirent sites
(readdirWithEntries, readdirWithEntriesRecursiveAsync,
readdirWithEntriesRecursiveSync). name storage stays gated on
args.encoding == .buffer, which matches name_as_buffer.

Two new regression tests in fs.test.ts cover both sides:
- non-ASCII string path + encoding='buffer'
- Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
@robobun
robobun force-pushed the farm/19bbeef8/fix-readdir-buffer-dirent branch from 1056ad9 to 5c0a672 Compare June 26, 2026 17:17
@robobun

robobun commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased on main. The conflict was the Zig→Rust port completing on main: src/runtime/node/node_fs.zig and src/runtime/node/types.zig were deleted upstream, while this PR's early commits still edited them as porting references. Dropped both deleted Zig files; the live fix already lives in node_fs.rs / types.rs / NodeDirent.cpp. Net diff is now 4 files (Rust + C++ + test), no Zig. All 14 readdir-buffer tests pass locally with the debug build.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing all the earlier feedback — I have nothing further on the latest revision, but since this threads new flags across the Rust↔C++ FFI boundary and reworks encoding/byte-preservation across several readdir paths, it's worth a human pass before merge.

Extended reasoning...

Overview

This PR fixes fs.readdir({ withFileTypes: true, encoding: 'buffer' }) to return Dirent objects with Buffer names (matching Node.js) instead of a flat Uint8Array[]. It touches four files: src/jsc/bindings/NodeDirent.cpp (C++ binding — adds a bunStringToBuffer helper and extends Bun__Dirent__toJS from 5→7 params), src/runtime/node/types.rs (adds name_as_buffer/path_as_buffer to Dirent and updates the extern signature), src/runtime/node/node_fs.rs (~250 lines: tag() priority fix, path_is_buffer plumbing, Latin-1 storage for raw bytes, ReaddirEntry trait signature changes including replacing apply_encoding with path_is_buffer), and test/js/node/fs/fs.test.ts (+14 tests).

Security risks

None identified. The change is read-only filesystem enumeration; no auth, crypto, permissions, or untrusted-input parsing is involved. The new bunStringToBuffer helper copies bytes into a fresh GC-managed Buffer and handles null/empty strings explicitly.

Level of scrutiny

Moderate-to-high. This is production-critical core node:fs code on a hot path (readdir with withFileTypes), and it changes an extern "C" signature across the Rust↔C++ boundary — exactly the class of change where a missed call site is an ABI bug rather than a compile error. The encoding semantics (Latin-1 storage vs. UTF-8 decode, gated on path_is_buffer vs. encoding) are subtle and went through three rounds of correction during review. There are platform-specific branches (Windows UTF-16→UTF-8→Latin-1) that CI exercises but a reviewer should sanity-check.

Other factors

I left six inline comments over this PR's lifetime — one critical ABI mismatch (Rust extern not updated), one storage/emission flag mismatch, one macOS test gating issue, and three cleanup nits — all now resolved, with the most recent two addressed in commits 2890bd7 and 5c0a672. The bug-hunting pass on the current revision found nothing. Test coverage is thorough (sync/async/promises/recursive × string/Buffer path × non-ASCII + raw-byte regression cases). No CODEOWNERS match the touched paths. CI has been green modulo unrelated infra flakes per the robobun comments.

That said, the change is neither small nor mechanical: it modifies a trait signature used by three impls, removes a parameter (apply_encoding) whose semantics were folded into the new flow, and adds two struct fields that every Dirent construction site must now populate. This exceeds my bar for bot-only approval, so deferring to a human reviewer.

Jarred-Sumner pushed a commit that referenced this pull request Jun 28, 2026
…compat fixes (#32813)

Four `node:fs` divergences from Node, found by a differential run
against Node v26.3.0. The headline is silent data corruption: the
`encoding` argument of `fs.writeSync(fd, string, position, encoding)`
was parsed and then ignored.

### 1. `fs.write` / `fs.writeSync` / `filehandle.write` ignore the
string encoding

```js
const fd = fs.openSync(p, "w");
fs.writeSync(fd, "abc", 0, "utf16le");
// Node: writes 6 bytes  61 00 62 00 63 00
// Bun:  writes 3 bytes  61 62 63          (raw UTF-8, encoding ignored)
```

Same for `hex`, `base64`, `latin1`, `ascii`, and for the async
`fs.write(fd, string, position, encoding, cb)` and
`filehandle.write(string, position, encoding)`, which share the argument
parser.

**Cause:** `args::Write::from_js` in `src/runtime/node/node_fs.rs`
parses the encoding into `args.encoding`, but the string was already
converted to UTF-8 by `StringOrBuffer::from_js` a few lines earlier, and
`write_inner` / `pwrite_inner` only ever look at `args.buffer.slice()`.
A second bug hid the first for some spellings: the parser only advanced
past the position slot when it held a *number*, so `fs.writeSync(fd,
"abc", null, "utf16le")` never even reached the encoding argument.
Node's `WriteString` binding takes the position from `args[2]`
regardless of its type and the encoding strictly from `args[3]`.

**Fix:** consume the position slot whatever its type (using it only when
it is a number, and only when non-negative, matching the sibling Buffer
arm), then, once the encoding is known, re-encode the string with
`StringOrBuffer::from_js_with_encoding`. That helper is the same path
`writeFileSync(p, s, "utf16le")` (which already worked) and
`Buffer.from(s, enc)` use, and it already matches Node byte-for-byte for
every encoding. The one encoding name the re-encode must not apply to is
`"buffer"`: Node's `StringBytes::Write` treats `BUFFER` the same as
`UTF8`, while the in-tree string encoder's `Buffer` arm is a verbatim
latin1 / low-byte copy, so `writeSync(fd, "\u00e9", 0, "buffer")` would
have written 1 byte instead of Node's 2. The re-encode therefore skips
both `Encoding::Utf8` and `Encoding::Buffer`, keeping the original UTF-8
conversion for both, and the test table covers the `"buffer"` name for
both 8-bit and 16-bit source strings. The one exception is the
`"buffer"` encoding name: Node writes UTF-8 for it (`writeSync(fd,
"\u00e9", 0, "buffer")` writes `c3 a9`), while the encoder's `Buffer`
arm is a verbatim latin1 / low-byte copy, so it is excluded from the
re-encode and keeps the original UTF-8 conversion.

### 2. `fs.readdir` with `{ encoding: "buffer" }` returns
`Uint8Array[]`, not `Buffer[]`

```js
const r = fs.readdirSync(dir, { encoding: "buffer" });
Buffer.isBuffer(r[0]); // Node: true   Bun: false (plain Uint8Array)
```

**Cause:** `ret::Readdir::to_js` used `MarkedArrayBuffer::to_js`, which
builds a plain `Uint8Array`. The sibling `StringOrBuffer::to_js` (used
by `readFile`, `readlink`, `realpath`) already uses `to_node_buffer`,
which goes through `JSBufferSubclassStructure`.

**Fix:** use `to_node_buffer` there too. Both transfer ownership of the
mimalloc-backed bytes to JSC via the same deallocator, so it is a
drop-in. This covers the sync, async, callback, and recursive variants,
which all funnel through `Readdir::to_js`.

### 3. `fs.Dir` is not disposable

```js
using d = fs.opendirSync(dir); // Bun: TypeError, d[Symbol.dispose] is undefined
```

Node's `Dir` prototype has `Symbol.dispose` and `Symbol.asyncDispose`
(and both are no-ops when the directory is already closed, so they
compose with an explicit `close()`).

**Fix:** add both to the `Dir` class in `src/js/node/fs.ts`, delegating
to `closeSync()` / `close()` behind an already-closed guard, using the
same `this.#handle >= 0` check the class's `Symbol.asyncIterator`
already uses.

### 4. An unrecognized flag string throws `ERR_INVALID_ARG_TYPE`

```js
fs.openSync(p, "bogus");
// Node: TypeError [ERR_INVALID_ARG_VALUE]
// Bun:  TypeError [ERR_INVALID_ARG_TYPE]
```

Node's `stringToFlags` throws `ERR_INVALID_ARG_VALUE` for every
unrecognized flag string (including the empty string).
`FileSystemFlags::from_js` used the `ERR_INVALID_ARG_TYPE` helper. Code
that switches on `err.code` gets the wrong branch.

**Fix:** switch the three string-flag error sites to
`ErrorCode::INVALID_ARG_VALUE`. The messages are unchanged.

### 5. Three more `fs.write` gaps, surfaced by vendoring Node's
`test-fs-write.js`

Node's `test/parallel/test-fs-write.js` was missing from the vendored
Node test suite. It is Node's own regression test for this exact bug
class (its
[nodejs/node#38168](nodejs/node#38168) block
asserts `fs.writeSync(fd, 'abcd', 0, 'hex')` returns 2). Vendoring it
surfaced three more gaps on `fs.write`'s string path, all fixed here:

- **`validateEncoding` was never called.** Node rejects an odd-length
string written with the `hex` encoding (`ERR_INVALID_ARG_VALUE`, `is
invalid for data of length N`) from `write`, `writeSync`, and
`filehandle.write`. The C++ validator already existed in
`NodeValidator.cpp` and was exported from `internal/validators` with no
callers; this wires it into the three string write paths where Node
calls it. Messages are byte-identical to Node v26.3.0.
- **`fs.write` was missing `writeSync`'s `typeof buffer !== "string"`
check**, so `new String('x')` (and every other non-string, non-view
value) reached the native parser through the string signature instead of
throwing `ERR_INVALID_ARG_TYPE`.
- **`fs.write` branched on `$isTypedArrayView`, which excludes
`DataView`**, so `fs.write(fd, dataView, offset, length, cb)` was
silently parsed as the `(position, encoding)` string signature. It now
uses Node's `isArrayBufferView` predicate, matching the `writeSync`
sibling one function below.

### Tests

- `test/js/node/fs/fs.test.ts`: a table of `writeSync(fd, string, ...)`
argument shapes whose expected bytes were each produced by Node v26.3.0;
the async `fs.write` + `filehandle.write` variants; `readdir` `{
encoding: "buffer" }` across the sync / recursive / promise / callback
entry points; the flag error code for `openSync` and `open`.
- `test/js/node/fs/dir.test.ts`: `using`, `await using`, and disposing
an already-closed `Dir`.
- `test/js/node/test/parallel/test-fs-write.js`: Node's upstream test,
vendored via `bun node:test:cp`. On the unfixed build it fails on its
first assertion (`'ümlaut sechzig'` written with `latin1` reads back as
`'ümlaut sechzig'`); with the fix it exits 0. The file is byte-for-byte
identical to `nodejs/node@main` with no local patch. Upstream runs it
under V8's `--expose_externalize_string` flag; the
`createExternalizableString` / `externalizeString` / `isOneByteString`
helpers it needs are V8 test globals with no JavaScriptCore equivalent
(present upstream since at least Node v22), so they are polyfilled in
`test/js/node/test/common/index.js` next to the existing `--expose-gc`
polyfill. The test file stays unmodified, and any other vendored Node
test that uses that flag works without a local patch.

All fail on the current build and pass with the fix. The vendored Node
suites `test-fs-write-sync`, `test-fs-write-sync-optional-params`,
`test-fs-write-optional-params`, `test-fs-write-buffer`, `test-fs-open`,
`test-fs-open-numeric-flags`, `test-fs-readdir*`, `test-fs-opendir`, and
`test-fs-promises-file-handle-{write,append-file,dispose}` still pass.

### Out of scope, noted for completeness

- `readdirSync(dir, { withFileTypes: true, encoding: "buffer" })`: Node
returns `Dirent[]` with `Buffer` names; Bun ignores `withFileTypes` when
the encoding is `buffer` and returns a flat array (now of `Buffer`s
instead of `Uint8Array`s). Matching Node needs a `Dirent` variant whose
`name` is a `Buffer`, a larger change than the one-line swap here. That
half is #30482, and #30484 already has a fix open for it (the `Dirent`
name representation, the `args::Readdir::tag` precedence, and
`NodeDirent.cpp`). The two PRs touch disjoint code: #30484 does not
change `ret::Readdir::Buffers` conversion, and this PR does not touch
`Dirent`, so neither replaces the other.
- Flag strings are case-insensitive in Bun (`"A+"` opens; Node throws
`ERR_INVALID_ARG_VALUE`). The uppercase aliases are hand-enumerated in
`FILE_SYSTEM_FLAGS_MAP` and date back to the original implementation, so
I left that deliberate leniency for a maintainer to decide on rather
than silently tightening it. #32966 now does exactly that: it rewrites
`FileSystemFlags::from_js` to match Node's `stringToFlags`
(case-sensitive, exhaustive, always `ERR_INVALID_ARG_VALUE`) and
subsumes the small `types.rs` error-code change in this PR, so whichever
of the two lands second rebases that hunk away. The rest of this PR is
independent of #32966.
robobun and others added 10 commits July 6, 2026 06:07
`fs.readdir`/`readdirSync` with `{ withFileTypes: true, encoding: "buffer" }`
previously returned a flat `Uint8Array[]` instead of `Dirent[]`.

`Arguments.Readdir.tag()` matched the `.buffer` encoding before checking
`with_file_types`, so every withFileTypes+buffer call fell through to the
raw-buffer code path and never constructed Dirents.

Fix:
- withFileTypes takes priority in tag() regardless of encoding.
- When encoding == .buffer, store the name/path bytes as Latin-1 so raw
  filesystem bytes survive round-trip without the UTF-8-via-UTF-16
  replacement that toBunString(.buffer) does.
- Thread two flags (name_as_buffer, path_as_buffer) through Dirent to the
  C++ Bun__Dirent__toJS, which emits the corresponding fields as Buffer
  instead of JSString.
- Track path_is_buffer on Arguments.Readdir so Dirent.parentPath mirrors
  Node: Buffer iff the path arg was a Buffer, independent of encoding.

Covers sync, async callback, promises, and recursive paths. Tests in
test/js/node/fs/fs.test.ts.

Closes #30482. Same root cause as #27914.
…oding

Reviewers on #30484 caught that the storage representation for dirent_path
was chosen on args.encoding == .buffer while emission is driven by
args.path_is_buffer. Those flags are independent, so the mismatch produces
corrupt parentPath whenever one is set and the other isn't:

- string path + encoding='buffer' (name=Buffer, parentPath=string):
  path was stored as cloneLatin1 but emitted via jsString, so '/tmp/café'
  came back as '/tmp/café'.
- Buffer path + default encoding (name=string, parentPath=Buffer): path
  went through toBunString(.utf8) which replaces invalid UTF-8 bytes with
  U+FFFD, then emitted as a Buffer — raw filesystem bytes like 0xFF
  got lost.

Storage now tracks path_is_buffer at all three Dirent sites
(readdirWithEntries, readdirWithEntriesRecursiveAsync,
readdirWithEntriesRecursiveSync). name storage stays gated on
args.encoding == .buffer, which matches name_as_buffer.

Two new regression tests in fs.test.ts cover both sides:
- non-ASCII string path + encoding='buffer'
- Buffer path with raw non-UTF-8 (0xFF) bytes + default encoding
1. test/fs.test.ts: gate the raw 0xFF-byte test with `isWindows || isMacOS`.
   macOS APFS/HFS+ reject non-UTF-8 filenames with EILSEQ, so `mkdirSync`
   of a 0xFF-named dir fails before any assertion. Same gate pattern as
   `test/js/bun/util/filesystem_router.test.ts:475`.

2. src/runtime/node/node_fs.zig: reuse the `re_encoding_buffer` that's
   already allocated once outside the loop (for `is_u16 and encoding != .utf8`)
   in the Windows Dirent + encoding=buffer branch, matching the pattern
   the neighboring `bun.String` branch already uses. Saves one pool
   acquisition per directory entry on Windows.
Cover the async codepaths with Buffer-path parentPath assertions so
regressions in the path_is_buffer threading can be caught independently
from the sync path.
windows-11-aarch64-test-bun shard failed on an unrelated flaky test
(test/cli/install/bun-install-registry.test.ts peer-dep resolution
expecting '1.0.10'). Diff is green on every other shard across Linux,
macOS, FreeBSD, Android, and Windows-2019.
The Rust rewrite (#30412) replaced the Zig node_fs implementation with
src/runtime/node/node_fs.rs, bypassing this PR's original Zig changes
and reintroducing the original bug: tag() matched .buffer encoding
before with_file_types.

Port the whole fix:
- args::Readdir gains path_is_buffer (set in from_js when the path arg
  was a PathLike::Buffer).
- tag() now prioritizes with_file_types over encoding.
- Dirent struct gains name_as_buffer / path_as_buffer flags and the
  Bun__Dirent__toJS FFI signature grows two bool args to match
  NodeDirent.cpp — without this, the bools were reading garbage from
  the stack and could emit Buffers for plain string Dirents
  (test-fs-readdir-recursive.js, pnpm-migration saw parentPath as an
  object).
- ReaddirEntry trait's three append_entry* methods gain path_is_buffer;
  Dirent impl stores name/path as Latin-1 for the buffer emission paths
  (preserves raw filesystem bytes 1:1 instead of UTF-8-via-UTF-16
  round-tripping).
- All three readdir paths (non-recursive, recursive async, recursive
  sync, plus the Windows UTF-16 arm) gate dirent_path storage on
  path_is_buffer and thread it to append_entry*.
- Recursive-async was using apply_encoding=false (clone_utf8); align
  with sync/non-recursive by passing args.encoding through.
After aligning the recursive-async path with sync (both pass
apply_encoding=true), the parameter was invariantly true: the
clone_utf8 else-arms in the BunString and Dirent impls were dead and
the trait doc described a sync/async distinction that no longer
exists. Drop the parameter and the dead branches.
Both call sites are identical since apply_encoding was removed; the
labels no longer contrast anything and the sync-side comment described
an argument that is no longer passed.
@robobun
robobun force-pushed the farm/19bbeef8/fix-readdir-buffer-dirent branch from 5c0a672 to 56da557 Compare July 6, 2026 06:17
@robobun

robobun commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main (af362c9b34); no code changes, the 10 commits applied cleanly. The previous run was from 2026-06-26 and the branch had since fallen behind main, so CI was both stale and red on unrelated lanes.

Re-verified that the bug still reproduces on today's main and that this branch fixes it:

const fs = require("fs");
const dir = fs.mkdtempSync("/tmp/readdir-");
fs.mkdirSync(dir + "/subdir");
console.log(fs.readdirSync(dir, { withFileTypes: true, encoding: "buffer" }));
// main: [ <Buffer 73 75 62 64 69 72> ]
// this branch / node: [ Dirent { name: <Buffer 73 75 62 64 69 72>, ... } ]

Checked against node v26 on three shapes, all now match:

call node this branch
readdirSync(strPath, { withFileTypes, encoding: "buffer" }) Buffer name, string parentPath same
readdirSync(bufPath, { withFileTypes, encoding: "buffer" }) Buffer name, Buffer parentPath same
readdirSync(bufPath, { withFileTypes }) string name, Buffer parentPath same

Non-utf8 filenames round-trip byte-exact ([0x66, 0xff, 0x6f] in, same bytes out), and opendirSync(dir, { encoding: "buffer" }) now yields Dirents too, since Dir#readSync is built on readdirSync({ withFileTypes: true }).

Locally on the debug (ASAN) build: test/js/node/fs/fs.test.ts + test/js/node/fs/dir.test.ts → 416 pass / 0 fail, plus test-fs-readdir-types.js, test-fs-readdir-buffer.js, test-fs-readdir-recursive.js and test-fs-opendir.js all green.

For the record, the failures on the previous build (65063) were all flake on lanes this diff doesn't touch:

  • darwin-14-aarch64 / darwin-14-x64 / darwin-26-aarch64: tls.connect > should have peer certificate and RedisClient TLS hostname verification.
  • windows-2019-x64 / windows-11-aarch64: bun-install > should edit package json correctly with git dependencies, a codeload.github.com ... 400 from the registry fixture.

One divergence from node this PR deliberately leaves alone: on Windows, Dirent.name still ignores the non-buffer encodings (hex, latin1, base64) because append_entry_w clones the raw UTF-16 name for everything except buffer. That is a pre-existing gap (fs.readdirSync without withFileTypes has the same one, and there is a skipped test for it in dir.test.ts), not something this change introduces.

@robobun

robobun commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

CI status (build #68767): neither red lane is caused by this diff, which touches only node_fs.rs, types.rs, NodeDirent.cpp, and fs.test.ts (no cookie or darwin code).

  1. darwin-26-aarch64-test-bun: buildkite-agent artifact download timed out after 120s for step 'darwin-aarch64-build-bun'. The build step passed; the test shard could not fetch its artifact. Infrastructure, not a test failure.

  2. windows-* test-bun: test/js/bun/cookie/cookie-map.test.ts fails because the runtime emits Expires=Thu, 01 Jan 1970 00:00:00 GMT (IMF-fixdate) while the test still asserts Expires=Fri, 1 Jan 1970 00:00:00 -0000. This is a pre-existing break on main: Fix Bun.Cookie Expires to emit an IMF-fixdate #32926 (Jul 5) changed the Expires format, one day after Hardening round 11: input validation, bounds checks, lifetimes #33072 (Jul 4) added that assertion, and the assertion was never updated. Current main HEAD (bdc097dc48) still carries the stale expectation around lines 451-453, so any shard that runs this file fails regardless of this PR.

The readdir/Dirent tests pass on every lane that has run them and under the local fail-before/pass-after gate. The re-trigger re-roll is spent, and re-running will not turn CI green since the cookie failure is deterministic. Flagging for a maintainer: the cookie-map.test.ts expectation needs updating on main (separate from this PR), after which this branch goes green.

@robobun

robobun commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #36064, which combines this with the cp walker fix so cpSync/promises.cp can copy directories containing non-UTF-8 entry names.

@robobun

robobun commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Re-checked on a debug build of main that includes #32813 (which touched readdir + encoding: "buffer"): readdirSync(dir, { withFileTypes: true, encoding: "buffer" }) still returns bare Buffers with no .name (same for the callback, promise and recursive forms), while node v26.3.0 returns Dirent objects with Buffer names. #32813 only changed the flat Buffer[] case and its description lists this case as left for this PR, so #30482 is still open and still needs one of these branches.

Current state of the candidates, for whoever picks this up: this PR (readdir only) and #36064 (the same readdir change plus the cp/cpSync walker fix, which lists this PR as superseded) are both open and both currently conflict with main in src/runtime/node/node_fs.rs. Whichever shape is preferred needs a rebase; the other should then be closed.

@robobun

robobun commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Correction to the note above: #36064 only returns Dirents with a Buffer name (string parentPath, as bun already does for plain withFileTypes). It does not implement node's rule that parentPath follows the type of the path argument, which this PR does and which the repro in #30482 depends on, so this PR is not superseded on that point. #36064's description now says so and no longer claims to fix #30482.

@robobun

robobun commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #42812, which fixes the same bug (#27914) on current main with a smaller change. This branch is about 2,000 commits behind main and conflicts with it.

I built #42812 and compared it with Node v26.3.0 on Linux and Windows. All readdir and opendir forms return Dirent objects with a Buffer name, and non-UTF-8 names keep their bytes. 8 of the 14 tests from this PR pass on it, including the callback and recursive cases.

The 6 tests that fail there cover the one thing this PR has and #42812 does not: a Buffer parentPath when path is a Buffer. Main has that difference from Node for plain withFileTypes: true too, so it is a separate fix, and it is noted on #42812. Node sets parentPath to the path argument itself (the same object, also for a Uint8Array). This PR creates a new Buffer for each entry, so a follow-up should not copy that part.

#36064 carries another copy of the readdir change next to its cp change. Its cp walker and tests pass unchanged on top of #42812.

@robobun robobun closed this Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fs.readdir({ withFileTypes: true, encoding: 'buffer' }) returns Uint8Array[] instead of Dirent[]

1 participant