Skip to content

resolver: keep cached DirEntry when a stale-generation refresh fails - #36874

Closed
robobun wants to merge 5 commits into
mainfrom
farm/7d88e242/resolver-stale-refresh-err-orphan
Closed

robobun wants to merge 5 commits into
mainfrom
farm/7d88e242/resolver-stale-refresh-err-orphan

Conversation

@robobun

@robobun robobun commented Aug 4, 2026 •

Copy link
Copy Markdown
Collaborator

Repro

// build #1 caches ./sub at generation 0
await Bun.build({ entrypoints: ["./entry.js"] });  // entry.js: import "./sub/mod.js"

// ./sub is replaced with a regular file
fs.rmSync("./sub", { recursive: true });
fs.writeFileSync("./sub", "");
await Bun.build({ entrypoints: ["./entry.js"] });  // fails (ENOTDIR), caches Err

// ./sub is a directory again
fs.rmSync("./sub");
fs.mkdirSync("./sub");
fs.writeFileSync("./sub/mod.js", "export const value = 2;");
await Bun.build({ entrypoints: ["./entry.js"] });  // still fails: cached Err is never retried

Cause

read_directory_with_iterator and entries_at_locked re-read a cached directory in place when its stored generation is older than the current bundle generation (BundleThread bumps the generation between Bun.build() calls). When that re-read fails, both routed through read_directory_error, which either writes EntriesOption::Err into the BSSMap slot or re-points the hash index at NOT_FOUND.

The slot previously held Entries(&'static mut DirEntry) where the reference points at a Box<DirEntry> leaked via heap::into_raw. Overwriting the slot drops the reference and orphans the Box together with its data map. The directory's cached .fd cannot safely be closed here either: copies of the descriptor live in Watcher.watchlist, ParseTask.contents_or_fd, and the bake DirectoryWatchStore, so closing it would turn into a use-after-close once the fd value is reused.

Separately, the cached Err carries no generation, so the Entries(e) if e.generation < generation arm never matches it and the directory stays unresolvable for the rest of the process even after it reappears.

Fix

read_directory_error now takes a refreshing_in_place flag. When set, it returns a threadlocal Err and leaves the slot alone. The existing Entries value (and the Box and fd behind it) stay reachable. The next lookup at the same or a later generation sees e.generation < generation, retries the open, and either succeeds (directory is back) or returns another transient error.

The fresh-slot path (refreshing_in_place == false) is unchanged: a directory that has never been listed still caches its error.

Verification

New test rebuilding after a cached directory is replaced with a file in test/bundler/bun-build-api.test.ts drives three in-process Bun.build() calls as above and asserts the third succeeds. It fails on main with Could not resolve: "./sub/mod.js" and passes with this change.


no test proof · iteration 1 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/bundler/bun-build-api.test.ts

read_directory_with_iterator and entries_at_locked re-read a cached
directory in place when its generation is older than the current bundle
generation. When that re-read fails (open_dir or readdir returns an
error) the old code routed through read_directory_error, which overwrote
the BSSMap slot with an Err variant or re-pointed the index at
NOT_FOUND. The slot previously held Entries(&'static mut DirEntry) and
the reference pointed at a leaked Box<DirEntry>; overwriting the slot
orphaned that Box along with its entry map. The cached .fd cannot be
closed on this path either (copies live in Watcher.watchlist,
ParseTask.contents_or_fd, and DirectoryWatchStore), so once the Err was
cached the fd was unreachable as well. The cached Err also never retries
across generations, so a directory that was recreated after being
replaced stayed unresolvable for the rest of the process.

Pass a refreshing_in_place flag through read_directory_error and, when
set, hand back a threadlocal Err without touching the slot. The existing
Entries stays reachable, the Box and fd are not orphaned, and the next
lookup re-tries the open so a recreated directory resolves again.
@github-actions github-actions Bot added the claude label Aug 4, 2026
@robobun

robobun commented Aug 4, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 4:01 AM PT - Aug 4th, 2026

✅ @robobun, your commit a777681f71cc99320273f5894453e77ab885335a passed in Build #88731! 🎉


🧪   To try this PR locally:

bunx bun-pr 36874

That installs a local version of the PR into your bun-36874 executable, so you can run:

bun-36874 --bun

@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The resolver preserves existing directory-cache entries when in-place refreshes fail. A regression test covers repeated Bun.build() calls while a cached directory is replaced by a file and then restored.

Changes

Directory cache refresh handling

Layer / File(s) Summary
Preserve cached entries on refresh failure
src/resolver/lib.rs, test/bundler/bun-build-api.test.ts
Directory-open and directory-read failures pass the in-place refresh state to read_directory_error. Failed refreshes preserve the existing cache entry and return a thread-local error. The regression test validates failure and recovery across repeated builds.

Possibly related issues

  • oven-sh/bun#36732 — Covers stale directory-cache failures across repeated in-process Bun.build() calls.

Possibly related PRs

  • oven-sh/bun#36251 — Directly relates to directory-cache refresh logic in src/resolver/lib.rs.
  • oven-sh/bun#36302 — Relates to directory-cache handling and virtual filesystem directory operations.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main resolver change: preserving cached DirEntry values when stale refreshes fail.
Description check ✅ Passed The description explains the problem, cause, fix, and verification, although it uses different headings from the template.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/resolver/lib.rs`:
- Line 1278: Update the in-place refresh flow around read_directory_error and
its corresponding branches at the other reported call sites so readdir results
are staged in temporary state rather than clearing (*existing).data or
(*e_ptr).data beforehand; only replace the live DirEntry after readdir succeeds,
preserving the cached map on failure. Add coverage verifying a failed refresh
retains the existing map and a subsequent retry still works.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f9279f57-7733-4ab3-b4ff-95b37a9356cc

📥 Commits

Reviewing files that changed from the base of the PR and between 0a14f47 and bd26ef3.

📒 Files selected for processing (2)
  • src/resolver/lib.rs
  • test/bundler/bun-build-api.test.ts

Comment thread src/resolver/lib.rs Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks correct and well-reasoned, but this is process-lifetime resolver cache state with unsafe raw-pointer/leaked-Box management, so it's worth a maintainer glance.

What was checked:

  • All four read_directory_error call sites: the two in entries_at_locked are always inside the stale-generation refresh arm (literal true is correct); the two in read_directory_with_iterator correctly key on in_place.is_some().
  • Callers that now receive a threadlocal Err instead of a BSSMap-backed one: dir_info.rs get_entries* immediately map Err → None, and resolver.rs:5660 reads original_err then bails — none stash the pointer past the call. Generation-0 callers (PackageManager, glob accessor) never hit in_place, so unchanged.
  • After a failed refresh the slot keeps its old generation, so the next lookup at ≥ current generation retries — no permanent stale-empty state.
Extended reasoning...

Overview

Adds a refreshing_in_place: bool parameter to read_directory_error in src/resolver/lib.rs and threads it through four call sites. When set, the function skips the branch that overwrites the BSSMap slot with EntriesOption::Err (or re-points the index at NOT_FOUND) and instead returns a threadlocal Err, leaving the existing Entries(&'static mut DirEntry) slot — and the leaked Box<DirEntry> plus its cached fd behind it — untouched. A new subprocess-isolated regression test in test/bundler/bun-build-api.test.ts drives three in-process Bun.build() calls (dir → file → dir again) and asserts the third build resolves.

Security risks

None. This is internal resolver cache bookkeeping; no user-controlled input reaches new parsing or allocation logic.

Level of scrutiny

High. This is the resolver's process-lifetime directory cache — shared by the bundler, runtime module loader, dev server, and package manager — and the touched code is dense with unsafe raw-pointer dereferences, heap::into_raw-leaked boxes, and fd-sharing invariants (the doc comment names Watcher.watchlist, ParseTask.contents_or_fd, DirectoryWatchStore). REVIEW.md flags memory safety as the most-blocked category, and "fix at the layer that owns the invariant" applies here: the fix intentionally leaves a stale-generation DirEntry (with cleared .data) in the slot rather than reclaiming it, which is the right call given the shared fd, but a maintainer who owns this cache should confirm that's the intended long-term shape (vs. e.g. giving Err a generation so it can be retried without keeping the old DirEntry alive).

Other factors

I traced every caller of entries_at/entries_at_locked/read_directory for whether the newly-threadlocal Err return could outlive the call — none stash it; they all match-and-discard synchronously. The in_place.is_some() gating in read_directory_with_iterator is correct: in_place is only Some when a cached Entries at an older generation was found, exactly the refresh case. Generation-0 callers can never trip in_place (u16 < 0 is always false). The test follows harness conventions (tempDir, bunEnv, concurrent pipe drain, exit-code asserted last). The one open finding is a nit about documenting the Windows skip. Deferring rather than approving because the change, while small, alters ownership/lifetime semantics of a shared unsafe cache slot.

Comment thread test/bundler/bun-build-api.test.ts Resolved
The slot now survives a failed refresh, so clearing .data on the error
path would drop the prev_map the next retry uses to reuse Entry cells
(and orphan them in the process-lifetime EntryStore). Leave the map in
place; readdir only reads from prev_map, so a partial iteration leaves
it consistent.

Also drop the Windows skip: Bun.build runs with store_fd=false, so the
resolver does not hold an open handle to the directory and rmSync
succeeds. Verified against the Windows canary build.
Comment thread src/resolver/lib.rs Resolved
Comment thread src/resolver/lib.rs Resolved
@robobun
robobun force-pushed the farm/7d88e242/resolver-stale-refresh-err-orphan branch from bb7d54a to 3d7eab4 Compare August 4, 2026 05:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/resolver/lib.rs`:
- Line 1604: Update the stale-generation documentation near the directory lookup
logic to state that the existing DirEntry and data map are preserved when
refresh fails, and replaced only after a successful refresh. Align the comment
with the behavior of the error branches around read_directory_error.
- Line 1269: Update the readdir error branch in the directory-reading flow to
close the newly opened handle when store_fd is enabled and need_to_close_files()
is false, before calling read_directory_error. Preserve the existing error
propagation and avoid closing handles that are already managed by the normal
cleanup path.

In `@test/bundler/bun-build-api.test.ts`:
- Line 504: Strengthen the child-driver assertions for the second build in the
“rebuilding after a cached directory is replaced with a file” test: after
checking second.success is false, verify the reported logs identify sub/mod.js
and contain the expected directory-resolution error. Keep the existing
success/failure flow unchanged while ensuring unrelated failures cannot pass the
regression test.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e12b19b8-4121-4ae5-b83f-2a1b78382d24

📥 Commits

Reviewing files that changed from the base of the PR and between bd26ef3 and bb7d54a.

📒 Files selected for processing (2)
  • src/resolver/lib.rs
  • test/bundler/bun-build-api.test.ts

Comment thread src/resolver/lib.rs Resolved
Comment thread src/resolver/lib.rs Resolved
Comment thread test/bundler/bun-build-api.test.ts Resolved
Comment thread src/resolver/lib.rs Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/resolver/lib.rs (2)

1265-1265: ⚠️ Potential issue | 🟠 Major

Close the newly opened handle before returning the refresh error.

When store_fd is true and need_to_close_files() is false, should_close_handle is false. The readdir error at Line [1265] returns before entries.fd = handle at Line [1281]. No owner closes the newly opened descriptor. Repeated failed refreshes can exhaust process descriptors. Keep the guard armed until successful publication transfers ownership to entries.fd.

As per coding guidelines, every resource acquisition must have a release on all error paths.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/resolver/lib.rs` at line 1265, Ensure the readdir error path in the
surrounding refresh logic closes the newly opened handle when ownership has not
yet been transferred to entries.fd. Keep should_close_handle armed through the
read_directory_error return and only disarm it after successful publication of
the handle, preventing descriptor leaks on failed refreshes.

Source: Coding guidelines


1115-1122: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Document the lifetime of in-place refresh errors.

An in-place refresh failure returns the thread-local TEMP_ENTRIES_OPTION even when entry caching is enabled. Update the public contract and ensure callers consume this result before another lookup, or return a correctly scoped value.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/resolver/lib.rs` around lines 1115 - 1122, Update the contract documented
for read_directory_error to state that in-place refresh failures may return the
thread-local TEMP_ENTRIES_OPTION even when entry caching is enabled. Audit its
callers and ensure they consume the returned result before performing another
lookup, or change the return handling to use a correctly scoped value.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/resolver/lib.rs`:
- Line 1265: Ensure the readdir error path in the surrounding refresh logic
closes the newly opened handle when ownership has not yet been transferred to
entries.fd. Keep should_close_handle armed through the read_directory_error
return and only disarm it after successful publication of the handle, preventing
descriptor leaks on failed refreshes.
- Around line 1115-1122: Update the contract documented for read_directory_error
to state that in-place refresh failures may return the thread-local
TEMP_ENTRIES_OPTION even when entry caching is enabled. Audit its callers and
ensure they consume the returned result before performing another lookup, or
change the return handling to use a correctly scoped value.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e4c480d0-1443-4c41-b3a5-e931319a2ebe

📥 Commits

Reviewing files that changed from the base of the PR and between bb7d54a and 3d7eab4.

📒 Files selected for processing (1)
  • src/resolver/lib.rs

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My earlier inline note about the doc-comment length is stale — it was already trimmed to one line in 3d7eab4, so that concern is resolved. I didn't find any bugs in the current revision, but this touches the resolver's process-static DirEntry cache (leaked Boxes, raw-pointer slot writes, fd aliasing across watchers), and the bool-flag-vs-generation-on-Err design choice is one a maintainer should sign off on.

What was reviewed:

  • read_directory_error(..., refreshing_in_place) — verified the new flag only skips the cache write; the fresh-slot path is byte-identical.
  • Confirmed readdir builds into a local DirEntry and only reads prev_map, so removing the three error-path .data.clear() calls cannot leave the live map partially mutated.
  • All four call sites pass the flag consistently (in_place.is_some() in read_directory_with_iterator, literal true in entries_at_locked).
  • Test drains stdout/stderr concurrently, asserts the second build fails specifically on ./sub/mod.js, and asserts the third recovers.
Extended reasoning...

Overview

The PR changes ~30 net lines in src/resolver/lib.rs and adds one regression test in test/bundler/bun-build-api.test.ts. It adds a refreshing_in_place: bool parameter to read_directory_error; when set, the function returns a threadlocal Err instead of overwriting the BSSMap slot, so the existing EntriesOption::Entries(&'static mut DirEntry) (a leaked Box) stays reachable and its stale generation triggers a retry on the next lookup. Three error-path .data.clear() calls are removed so a failed refresh leaves the cached entry map intact.

Security risks

None identified. This is internal resolver caching state; no untrusted input parsing, auth, or network surface is touched.

Level of scrutiny

High. The diff is small but sits inside a block of unsafe raw-pointer manipulation of a process-static singleton (entries_option_map()). Correctness depends on non-local invariants: that readdir never mutates prev_map (verified — it builds into a fresh local DirEntry and only reads from prev_map via add_entry_with_store), that the leaked Box<DirEntry> behind the slot must never be orphaned, and that the cached .fd is aliased elsewhere and cannot be closed here. The PR description and prior review threads walk through all of these, but they are exactly the kind of invariants a maintainer familiar with the resolver's ownership model should confirm.

Other factors

  • All prior review threads (CodeRabbit's data-map-clearing concern, comment-cop's doc-length flags, my Windows-skip note, the second-build assertion) are addressed and resolved. My own inline comment on line 1123 was posted against a stale revision — the doc comment had already been trimmed to one line before I posted, so that thread should be considered resolved.
  • CodeRabbit raised a pre-existing fd-leak on the store_fd && !need_to_close_files() path when iteration fails after a successful open; the author correctly scoped that out as independent of this change.
  • There is a legitimate structural alternative (give EntriesOption::Err a generation so it participates in the same stale-retry logic, eliminating the boolean side-channel). The author chose the minimal fix. That's reasonable, but it's the kind of design call REVIEW.md asks a maintainer to weigh ("Use the simplest mechanism the invariants allow" vs. "when a heuristic keeps sprouting counterexamples, redesign structurally").
  • The test follows harness conventions (tempDir, bunEnv, concurrent pipe drain, combined stdout/stderr assertion before exit code) and now runs on Windows.

BundleThread bumps its generation only after the work queue drains,
which can be after the previous Bun.build() promise has already
resolved. When the race hits, the second build reuses the stale sub/
listing and fails reading the file (ENOTDIR) instead of refreshing the
cache. Re-build until a resolve-level error is observed so the test
always exercises the refresh path.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No bugs found, but this touches resolver directory-cache slot ownership (leaked Box<DirEntry> behind &'static mut, threadlocal-vs-BSSMap error stashing) — worth a human look given how central this path is.

What was reviewed:

  • Verified readdir only reads from prev_map (get_hashed at fs.rs:467), so a mid-iteration failure leaves the in-place map and its EntryStore-backed entries valid after the .data.clear() removals.
  • Confirmed all four read_directory_error call sites pass refreshing_in_place correctly: in_place.is_some() in read_directory_with_iterator, hard-coded true inside the generation < generation arm of entries_at_locked.
  • Checked that skipping the cache write also skips mark_not_found, so an ENOENT during refresh no longer re-points the index at NOT_FOUND either.
  • Test driver's bounded poll loop (100×5ms) waits on the observable "Could not resolve" signal rather than a raw sleep.
Extended reasoning...

Overview

The PR changes read_directory_error in src/resolver/lib.rs to take a refreshing_in_place flag. When set, it returns a threadlocal EntriesOption::Err instead of writing into the BSSMap slot (or re-pointing the hash index at NOT_FOUND). Four call sites are updated: two in read_directory_with_iterator (open failure and readdir failure, gated on in_place.is_some()) and two in entries_at_locked (both hard-coded true since they only run inside the stale-generation arm). Three error-path .data.clear() calls are removed so the existing entry map survives a failed refresh. A subprocess-driven regression test in test/bundler/bun-build-api.test.ts exercises the dir→file→dir sequence across three in-process Bun.build() calls.

Security risks

None. This is internal resolver cache-lifetime management with no user-controlled input reaching the changed branches beyond filesystem state. No auth, crypto, or permission surface.

Level of scrutiny

High. The resolver's directory cache sits under every module resolution, and the code being changed manipulates raw *mut DirEntry pointers, &'static mut reborrows into a process-static BSSMap, and a leaked-Box ownership model under entries_mutex. The fix itself is small and I traced it as correct — add_entry_with_store only reads prev_map via get_hashed, so dropping the local DirEntry on readdir error does not invalidate anything the preserved map points at (entries are EntryStore-owned). But the surrounding invariants (fd aliasing across Watcher/ParseTask/DirectoryWatchStore, the deferred structural alternative of giving EntriesOption::Err a generation, the pre-existing fd-leak the author scoped out) are the kind a maintainer who owns this module should sign off on.

Other factors

All prior review threads (CodeRabbit's data.clear concern, comment-cop's doc-length flags, my own skipIf/doc-comment nits) are resolved in the current head. The entries_at doc comment at lib.rs:1545-1548 still says the stale branch "drops the existing DirEntry"; that's accurate for the success path but no longer describes the error path — minor. CI build #88731 was still running at review time. The test's bounded poll for "Could not resolve" is the right shape per REVIEW.md but couples to a specific error-message prefix; if that wording ever changes the loop falls through to the i === 100 throw with a diagnosable message, so it fails loud rather than silently.

@robobun

robobun commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

CI build #88731 is green for this change: test/bundler/bun-build-api.test.ts passed on every lane after the de-flake in a777681.

The six tests that tripped retries are all unrelated flakes (install migration, bun-install-patch, spawn-maxbuf on Windows, napi GC on Windows, zlib/web-crypto parallel-batch only) and each passed on its retry or when run alone. None touch the resolver.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Closing this PR because it has been inactive for more than 90 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant