Skip to content

watcher(windows): watch workspace packages outside the project root - #35596

Closed
robobun wants to merge 4 commits into
mainfrom
farm/7811a6af/windows-watch-workspace-packages
Closed

robobun wants to merge 4 commits into
mainfrom
farm/7811a6af/windows-watch-workspace-packages

Conversation

@robobun

@robobun robobun commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • On Windows the watcher is one recursive ReadDirectoryChangesW rooted at the cwd, so a file that resolves outside it (a workspace package reached through a node_modules junction, bun link, a file: dependency) is appended to the watch list but its events never fire, and every reload prints File ... is not in the project directory and will not be watched once per file. Fixes Bun doesnt pick changes in monorepos packages #9547.

Fix

  • Each watch root is a Box<DirWatcher> (its own ReadDirectoryChangesW) sharing one IOCP. A completion's lpOverlapped is the address of the DirWatcher that fired, so the watch thread identifies the source without touching the roots Vec and without the mutex.
  • Roots open at the package directory, not the imported file's own directory: the prefix up to the first node_modules component if there is one, else the nearest enclosing directory with a package.json, else the file's directory. A dist/ rebuild no longer deletes its own root, and bun build --watch with an out-of-cwd hoisted node_modules opens one root total for it.
  • A root whose directory disappears is retired in place (flag flip plus handle close, logged to the debug scope); other roots keep running, and add_root reuses dead slots so kill/re-add cycles don't grow the Vec. With every root dead the thread parks on the IOCP until the next out-of-root import arms a new root, like inotify at watch_count zero.
  • stop() runs on both watch-thread exits under the mutex; it marks roots dead, cancels pending I/O and waits for the cancellation (CancelIoEx + GetOverlappedResult(bWait), which needs SYNCHRONIZE in the NtCreateFile mask) before the buffers are freed, then invalidates the IOCP so a later add_root from a still-running --hot process returns early instead of using a closed handle.
  • The initial root is armed by the watch thread (seeded through needs_rearm), not in Watcher::init, so a shutdown before start() has no pending kernel I/O against a freed buffer.
  • The per-cycle path scan snapshots raw slices of the watchlist path column under the mutex (the column is Cow::Owned on Windows, so owned copies would malloc per path per completion) and re-snapshots after a mid-batch dispatch, whose evictions swap_remove-reorder the live list.
  • add_root failures map the NTSTATUS / Win32 code to an errno, so the once-per-root warning names the cause (for example ENOENT) instead of an internal enum variant.
  • Verified: bun bd test test/cli/hot/watch.test.ts (4/4), hot.test.ts (12/12), watch-many-dirs.test.ts on Windows and Linux debug builds; the two new tests fail on main on Windows (warning printed, reload stalls) and pass with this build.

Background

  • ReadDirectoryChangesW watches one directory tree per call and reports paths relative to it; completions are delivered through an I/O completion port (IOCP) the handle is associated with. The OVERLAPPED passed when arming comes back as lpOverlapped, which is why it sits at offset 0 of DirWatcher and doubles as the root's identity.
  • --watch reloads on Windows replace the process, so each generation re-registers roots from scratch; only --hot and the dev server exercise the in-process lifecycle (re-arm rotation, root death and reuse). The new --hot test deletes and recreates a linked package mid-session to cover that path.
  • Closing a directory handle only initiates cancellation of its pending I/O; the kernel finishes writing the OVERLAPPED asynchronously, which is why stop() must wait before the Box<Watcher> is dropped.

no test proof · iteration 4 · Platform-specific test(s) that do not run on this machine. Deferring to CI, which covers all platforms: test/cli/hot/watch.test.ts

@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 10 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e5c5bb8a-6f93-47a4-bb3d-5d0327aea625

📥 Commits

Reviewing files that changed from the base of the PR and between 05dd45e and 6653b89.

📒 Files selected for processing (8)
  • src/runtime/bake/DevServer.rs
  • src/runtime/node/path_watcher.rs
  • src/sys/windows/mod.rs
  • src/watcher/Watcher.rs
  • src/watcher/WindowsWatcher.rs
  • src/watcher/error.rs
  • test/bake/bake-harness.ts
  • test/cli/hot/watch.test.ts

Comment @coderabbitai help to get the list of available commands.

Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
@github-actions

Copy link
Copy Markdown
Contributor

Found 2 issues this PR may fix:

  1. Setting loglevel="error" does not hide warnings about workspace/monorepo files not being watched #22302 - PR removes the per-file "is not in the project directory" warning entirely and fixes the underlying issue (files now actually get watched), making the loglevel suppression request moot
  2. Allow watching other folders with --watch #14183 - PR adds multi-root watching via multiple DirWatcher instances sharing one IOCP, which is exactly what this enhancement requests — watching folders outside the project root on Windows

If this is helpful, copy the block below into the PR description to auto-close these issues on merge.

Fixes #22302
Fixes #14183

🤖 Generated with Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional findings (outside current diff — PR may have been updated during review):

  • 🟡 src/watcher/WindowsWatcher.rs:503-517 — The platform_scratch snapshot is taken once per completion, but when event_id >= 128 triggers a mid-loop process_watch_event_batch → on_file_update → flush_evictions(), swap_remove reorders the live watchlist while the inner for item_idx in 0..n_items loop keeps emitting indices from the frozen snapshot. Subsequent WatchEvent.index values then point at whatever entry swap_remove moved into that slot in the live watchlist, so the wrong file can be dispatched (or a real change dropped by the bounds check). Re-snapshotting platform_scratch under the mutex right after the mid-batch process_watch_event_batch call would keep indices consistent.

    Extended reasoning...

    What the bug is

    watch_loop_cycle now snapshots watchlist.file_path into this.platform_scratch once per outer-loop iteration (once per ReadDirectoryChangesW completion), and n_items is captured from that snapshot. The inner for item_idx in 0..n_items loop then writes item_idx into WatchEvent.index via create_watch_event(&event, item_idx as WatchItemIndex).

    However, when the 128-slot watch_events buffer fills (event_id >= this.watch_events.len()), process_watch_event_batch is called mid inner loop. That reaches dispatch_file_updates → hot_reloader::on_file_update, whose _flush scopeguard (hot_reloader.rs:885-888) unconditionally calls ctx.flush_evictions(). On Windows a FILE_ACTION_REMOVED maps to Op::DELETE, which triggers ctx.remove_at_index(...) (hot_reloader.rs:919), so flush_evictions() executes self.watchlist.swap_remove(item) — reordering and shrinking the live watchlist.

    Control then returns to the inner loop, which keeps matching against the unchanged platform_scratch snapshot and emitting WatchEvent { index: item_idx }. The next process_watch_event_batch hands those events plus &self.watchlist (post-swap_remove) to on_file_update, which reads file_paths[event.index], hashes[event.index], kinds[event.index] from the live watchlist. event.index now resolves to whatever entry swap_remove moved into that slot.

    Step-by-step example

    1. Watchlist has 200 entries; platform_scratch snapshots paths [P0, P1, …, P199], n_items = 200.
    2. A large completion (e.g. git checkout) delivers many events. The first event is FILE_ACTION_REMOVED for P50; the inner loop matches item_idx = 50 (Equal) plus its parent dir, and continues accumulating matches until event_id hits 128.
    3. Mid-loop process_watch_event_batch runs → on_file_update sees Op::DELETE at index 50 → remove_at_index → scopeguard drop → flush_evictions() → watchlist.swap_remove(50). Now live slot 50 holds what was P199; live length is 199.
    4. Loop resumes with event_id = 0, still using the frozen snapshot. A later event in the same completion is a Modified on P50 (common: many editors emit remove+create/write on save; git checkout deletes then recreates). platform_scratch[50] still equals P50, so is_parent_or_equal returns Equal and WatchEvent { index: 50, op: WRITE } is emitted.
    5. In the second process_watch_event_batch, on_file_update reads file_paths[50] from the live watchlist — that's now P199. The reloader enqueues a reload for the wrong module (or, if index 50 now holds a directory entry, walks the directory branch instead). Meanwhile a change to old P199 matched at snapshot index 199 is silently dropped by the event.index >= file_paths.len() guard at hot_reloader.rs:901.
    6. Additionally, if a later event in the same completion is a DELETE that matches snapshot index 50, remove_at_index(File, 50, …) evicts the wrong live entry permanently.

    Why existing code doesn't prevent it

    The bounds check at hot_reloader.rs:901 (event.index as usize >= file_paths.len()) only prevents an OOB panic; it can't detect that a still-in-bounds index now names a different entry after swap_remove. Nothing between the mid-batch flush and the resumed loop refreshes platform_scratch or n_items.

    Regression vs. pre-PR

    Pre-PR, the path (this.watchlist.items_file_path()[item_idx]) and n_items were re-read live on every inner iteration. After a mid-batch swap_remove, the next event's item scan therefore matched against the post-swap layout — the same layout on_file_update reads — so indices stayed consistent (at the cost of the unlocked-realloc race this PR fixes, and a possible OOB panic if item_idx ran past the shrunk length within the same event's item scan). The PR trades that for a wider window of silent wrong-index dispatch: the staleness now spans the entire remaining completion, not just the tail of one event's item scan.

    Impact & severity

    Windows-only, dev-tool watcher, and requires ≥128 matched (event × watched-item) pairs from a single 64 KiB ReadDirectoryChangesW buffer with at least one FILE_ACTION_REMOVED in the first 128 — realistic mainly during git checkout / branch switch in a large watched tree, but still an edge case. Consequence is a spurious or missed hot reload (self-correcting on the next save), or in the worst case a wrong watchlist entry evicted so one file stops hot-reloading until restart. No crash/UB (bounds-checked). Given pre-PR was also broken on the same trigger (potential panic), this is a lateral trade rather than a clean regression, so nit — not worth blocking a PR that fixes the far more common workspace-not-watched bug.

    Suggested fix

    After the mid-batch process_watch_event_batch(this, event_id)?; call (inside the if event_id >= this.watch_events.len() block), re-take the snapshot under the mutex and update n_items:

    process_watch_event_batch(this, event_id)?;
    {
        let _guard = this.mutex.lock_guard();
        this.platform_scratch.clear();
        for p in this.watchlist.items_file_path() {
            this.platform_scratch.push(p.as_ref().to_vec().into_boxed_slice());
        }
    }
    // n_items must be mutable at its declaration
    n_items = this.platform_scratch.len();

    (and break out of the current for item_idx loop so it restarts against the fresh n_items). Alternatively, hoist the snapshot to a small helper and call it both at the top of the outer loop and after each mid-batch flush.

  • 🔴 src/watcher/WindowsWatcher.rs:428-436 — stop() now iterates &self.watchers without Watcher.mutex, but this PR made self.watchers a Vec that transpiler threads push to (via add_file → ensure_watch_root_covers → add_root) under that mutex. On the watch_loop Err path running is still set, so a concurrent Vec::push realloc while stop() is iterating is a UAF on the freed backing buffer. Take me.mutex.lock_guard() in thread_main around me.platform.stop() (or take it inside stop() itself).

    Extended reasoning...

    What the bug is

    WindowsWatcher::stop() reads self.watchers without holding Watcher.mutex:

    pub(crate) fn stop(&mut self) {
        unsafe {
            for dw in &self.watchers {
                w::CloseHandle(dw.dir_handle);
            }
            w::CloseHandle(self.iocp);
        }
    }

    The field's own doc comment says "Appended under Watcher.mutex; never shrunk", and every other accessor of the Vec (covers(), add_root()) is documented and called with that mutex held. stop() is the one accessor that skips it.

    Code path that triggers the race

    Writer (transpiler / JS thread, mutex held):
    RuntimeTranspilerStore::run_from_worker_thread → Watcher::add_file (takes self.mutex.lock()) → append_file_maybe_lock::<_, false> → append_file_assume_capacity → ensure_watch_root_covers(parent_dir) → self.platform.add_root(dir) → self.watchers.push(dw). Vec::push may reallocate the backing storage.

    Reader (watch thread, mutex NOT held):
    Watcher::thread_main → me.watch_loop() returns Err(err) (e.g. prepare() / GetQueuedCompletionStatus / null-overlapped failure) → me.platform.stop() → for dw in &self.watchers. thread_main does not hold me.mutex around this call:

    Err(err) => {
        me.watchloop_handle.store(false);
        me.platform.stop();          // <-- no lock
        if me.running.load() { ... }
    }

    Why nothing prevents it

    On the Err path running has not been cleared — only shutdown() clears it — so a transpiler thread inside add_file proceeds normally. add_file does not check running or watchloop_handle before pushing. And unlike next(), which was carefully designed to identify the firing DirWatcher from the boxed lpOverlapped address so it never touches the Vec, stop() iterates the Vec directly.

    Before this PR the Windows append_file_assume_capacity / append_directory_assume_capacity branches only called is_parent_or_equal(self.top_level_dir(), file_path) and never touched self.platform, and stop() closed a single fixed inline self.watcher.dir_handle. So self.platform was watch-thread-only and stop() was safe. This PR is what made platform.watchers a cross-thread Vec, so stop() becomes the first (and only) unlocked accessor — the race is newly introduced here.

    Step-by-step proof

    1. The watch thread is inside watch_loop_cycle → this.platform.next(Timeout::Infinite). GetQueuedCompletionStatus returns rc == 0 with a non-timeout error (or rc != 0 with overlapped == null), so next() returns Err, which propagates through watch_loop_cycle → watch_loop → thread_main.
    2. thread_main enters the Err(err) arm, stores watchloop_handle = false, and calls me.platform.stop(). me.mutex is not held; me.running is still true.
    3. Concurrently, a transpiler thread resolves an import to a workspace package outside every existing root (the exact scenario this PR enables) and calls add_file. It takes self.mutex — which the watch thread is not contending — and reaches self.platform.add_root(dir).
    4. add_root calls self.watchers.push(dw). The Vec is at capacity, so push allocates a new buffer, memcpys the Box<DirWatcher> pointers over, and frees the old buffer.
    5. The watch thread's for dw in &self.watchers slice iterator is still holding the old (ptr, len) pair. It dereferences the freed backing storage to read dw.dir_handle → use-after-free. Even without a realloc, the concurrent length/pointer write is a data race and UB.

    Impact

    Memory unsafety on Windows: reading freed heap memory and passing whatever is there to CloseHandle. The window is narrow — a hard ReadDirectoryChangesW / IOCP error concurrent with an out-of-root import triggering Vec growth — but per REVIEW.md (Native code memory safety, the most-blocked category): "a mutex only counts if EVERY accessor takes it" and "benign same-value races are still UB".

    Fix

    Trivial: take the same lock the writers take. Either wrap the call site in thread_main:

    Err(err) => {
        me.watchloop_handle.store(false);
        {
            let _guard = me.mutex.lock_guard();
            me.platform.stop();
        }
        if me.running.load() { ... }
    }

    or snapshot the handles under the lock inside stop() before closing them. Either restores the invariant the field's doc comment already claims.

  • 🟡 src/watcher/WindowsWatcher.rs:346-350 — Now that there are N watch roots, any single root's I/O failure still propagates Err through next() → watch_loop → thread_main and calls platform.stop(), terminating watching for all roots including the cwd. Concretely: if a user rm -rf packages/db (opened with FILE_SHARE_DELETE, so it can go delete-pending), the next re-arm on that root fails and the cwd watcher silently stops too. Not blocking — pre-PR these dirs weren't watched at all so this is a strict improvement, and on_error does fire — but per-root isolation (drop the failing extra root, keep the rest alive) would be a good follow-up.

    Extended reasoning...

    What the bug is

    Before this PR the Windows watcher had exactly one root — the cwd — so an error from ReadDirectoryChangesW / GetQueuedCompletionStatus meant the project directory itself was gone, and treating that as fatal was reasonable. This PR generalizes to N roots (one per external directory reached via a workspace symlink / bun link / file: dep) but keeps the single-root error propagation: every error path in next() still returns Err, which bubbles through watch_loop_cycle → watch_loop → thread_main, calls platform.stop() (closing all dir handles), and exits the watcher thread.

    The specific code paths

    There are three per-root error paths, all newly reachable for extra roots:

    1. needs_rearm re-arm at the top of next() (WindowsWatcher.rs:346-350):
      if let Some(dw) = self.needs_rearm.take() {
          unsafe { dw.as_ptr().as_mut().unwrap_unchecked() }.prepare()?;
      }
      If the extra root's directory was deleted/renamed since the last drain, ReadDirectoryChangesW fails (typically ERROR_ACCESS_DENIED on a delete-pending dir) and the ? kills the whole watcher.
    2. Buffer-overflow re-arm in the nbytes == 0 branch — same prepare()? short-circuit.
    3. GetQueuedCompletionStatus returning rc == 0 with a non-timeout error for one root's completion (e.g. after its directory is removed) — returns Err for the whole watcher.

    Why existing code doesn't prevent it

    The pre-PR code had the identical ?/return Err propagation, but only one root existed. The PR adds roots on directories users are far more likely to delete mid-session than the cwd (workspace packages, linked deps) without adding per-root isolation. The dir handle is opened with FILE_SHARE_DELETE, so deletion is permitted while the handle is open — the directory enters delete-pending, and the next ReadDirectoryChangesW on it fails.

    Step-by-step proof

    1. Run bun --watch index.js from apps/myapp, which imports @test/db (junction → packages/db). PR opens root[0] = apps/myapp\, root[1] = packages/db\.
    2. User runs rm -rf packages/db (or git checkout to a branch without it, or renames it) to regenerate the package. The handle on root[1] was opened with FILE_SHARE_DELETE, so the delete succeeds and the directory enters delete-pending.
    3. The delete produces a completion on root[1]; next() returns its EventIterator and sets needs_rearm = Some(root[1]).
    4. watch_loop_cycle drains the iterator, then calls next() again. The needs_rearm block calls root[1].prepare() → ReadDirectoryChangesW on a delete-pending directory → fails (ERROR_ACCESS_DENIED or similar).
    5. prepare()? propagates Err out of next() → watch_loop_cycle returns Err → watch_loop returns Err → thread_main calls platform.stop() (closes root[0]'s handle too) and (on_error)(ctx, err), then the watcher thread exits.
    6. Edits to apps/myapp/index.js — under root[0], which was still perfectly valid — no longer trigger a reload. For DevServer the user sees "hot-reloading is disabled until a restart"; for --watch/--hot the file-watcher thread is simply gone.

    Impact and why this is a nit

    This is a real limitation but I'd flag it as a nit / follow-up, not blocking:

    • Strict net improvement: pre-PR, packages/db wasn't watched at all — deleting it couldn't affect the watcher, but edits to it were never picked up either. The PR fixes the common case (#9547) and only introduces this edge on an operation that would break the running app's next reload anyway.
    • Not silent: on_error is invoked, so DevServer prints a diagnostic; it's "watcher stops, user restarts bun", not data loss or a crash.
    • Edge trigger: deleting an entire directory the running process currently imports from is plausible but not the everyday flow.

    How to fix

    Isolate errors per root: when prepare() on an extra root (index > 0) fails, or GQCS returns a non-timeout error for its completion, log once, close that root's dir_handle, mark the entry dead (e.g. set dir_handle = INVALID_HANDLE_VALUE and skip it in stop()/covers()), and continue — while the remaining roots, especially root[0] (the cwd), keep watching. Only a failure on root[0] should remain fatal. Matches REVIEW.md § Error handling: "Every error/abort/timeout path actively completes the operation" — one root's expected, user-reachable failure shouldn't cascade to unrelated roots.

  • 🟡 test/cli/hot/watch.test.ts:107 — The await using watcher = spawn(...) on line 80 already registers disposal (Subprocess's [Symbol.asyncDispose] kills and awaits exit), so this trailing watcher.kill() is redundant — REVIEW.md's hermetic-tests section says "no manual close alongside using". Neighboring await using runner = spawn(...) tests in hot.test.ts don't add a manual .kill(). Just delete this line.

    Extended reasoning...

    What the issue is

    REVIEW.md's "Tests reviewers reject → hermetic" section states: "Release every resource via using/await using or try/finally registered BEFORE the assertions …; no manual close alongside using". The new test declares await using watcher = spawn({...}) at line 80, which registers automatic disposal via Subprocess[Symbol.asyncDispose] (kills the process and awaits exited). Then at line 107 — the last statement before the function scope closes — it also calls watcher.kill() explicitly. That is precisely the "manual close alongside using" pattern the guideline prohibits.

    Step-by-step

    1. Line 80: await using watcher = spawn({...}) binds watcher and pushes an async disposer onto the scope's dispose stack.
    2. Lines 85–105: the test drives five reload generations and runs its assertions.
    3. Line 107: watcher.kill() sends the kill signal.
    4. Immediately after line 107 the function body ends, the await using disposer runs, and Subprocess[Symbol.asyncDispose] calls this.kill() + await this.exited — which would have killed the process anyway. Step 3 therefore does nothing that step 4 doesn't do one statement later.

    Why nothing else prevents it

    The pre-existing test in this same file (line 47) does call process.kill(), but that test does not use await using — it uses a plain const process = spawn(...) — so the manual kill there is the only cleanup and is not the same pattern. It doesn't establish precedent for pairing await using with a manual kill.

    Neighboring convention

    Sibling tests in test/cli/hot/hot.test.ts (lines 547, 601, 647, 726) and test/cli/hot/watch-many-dirs.test.ts:48 all use await using runner = spawn(...) and none of them add a trailing .kill() — they let the disposer handle it. So the repo convention is consistently enforced in neighboring code.

    Impact

    None at runtime. kill() on a subprocess that's about to be disposed is idempotent; the test behaves identically with or without this line. This is purely a style/convention violation, hence nit severity — not blocking.

    Fix

    Delete line 107 (watcher.kill();). The await using on line 80 already handles teardown on both the pass and fail paths.

Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
Comment thread src/watcher/WindowsWatcher.rs Resolved
…tiring a root

mark_dead now reads every box field before publishing the dead flag,
since a transpiler thread may reuse the slot once the store lands.
Comment thread src/watcher/WindowsWatcher.rs Resolved
@robobun

robobun commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main and reworked per the review, force-pushed as one commit (now 1eb7048 after two follow-up nits):

  • stop() now marks every root dead, cancels and drains pending I/O, closes the handles, and invalidates the IOCP; add_root returns early (ESHUTDOWN) once the IOCP is invalid. It runs on both thread_body exits under the mutex, so it cannot iterate the Vec mid-push after the rebase.
  • The all-roots-dead machinery (live_roots, retire_root, the posted wake packet, the PostQueuedCompletionStatus extern) is gone; the three failure sites mark the root dead and continue, and the thread parks on the IOCP until the next add_root arms a new root.
  • Roots open at the package directory: the prefix up to the first node_modules component, else the nearest enclosing package.json, else the file directory. Dead slots are reused by add_root, and the mark_dead warning is now a debug-scope log.
  • The path snapshot stores raw slices instead of owned copies (the column is Cow::Owned on Windows, so cloned() was a malloc per path per packet); it is retaken after each mid-batch dispatch, which is also what makes the raw slices safe.
  • add_root failures map NTSTATUS / GetLastError through the errno table, so the once-per-root warning prints ENOENT and friends instead of CreateFileFailed or IocpFailed. The unsafe impl Send and the windows_watcher::Error enum are deleted.
  • Tests: the junction test stays on --watch (fresh process per generation covers root registration); the deletion test is rewritten as a --hot lifecycle test that edits across both roots, deletes and recreates the linked package, and asserts the recreated package reloads through a fresh root. Both fail on main on Windows (warning printed, reload stalls to timeout) and pass with the build. The stale comments in path_watcher.rs and DevServer.rs and the dead bake-harness filter are updated/removed.
  • Verified on Windows and Linux debug builds: watch.test.ts, hot.test.ts, watch-many-dirs.test.ts all green.

@alii alii left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earlier items are addressed; two things left, inline.

Comment thread test/cli/hot/watch.test.ts Resolved
Comment thread src/watcher/Watcher.rs Resolved
…dOwnership docs

The test now only passes after the recreated package's edit reloads;
EOF during the last phase fails the assertion. The two comments that
still described the old cwd pin now describe the root-open failure.
Comment thread src/watcher/Watcher.rs Resolved
Comment thread src/watcher/Watcher.rs Resolved
@robobun

robobun commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Both addressed in 6653b89: the hot lifecycle test now requires the final phase (EOF during it fails), and the two stale cwd-pin comments in append_file_maybe_lock and the FdOwnership::Caller doc are updated. Re-verified on Windows: watch.test.ts 4/4.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this revision and didn't find any new bugs. alii's two latest comments (phase-6 assertion in the --hot test, stale FdOwnership/append_file_maybe_lock comment text) look addressed in 6653b89. Given the scope — multi-root IOCP lifecycle, cross-thread unsafe around pending kernel I/O, dead-slot reuse — human sign-off is still the right call.

What was reviewed:

  • mark_dead now reads/logs everything before dead.store(true), closing the reuse-slot race flagged last round.
  • stop() cancels + waits per live root under the mutex and invalidates iocp; add_root early-returns on INVALID_HANDLE_VALUE.
  • pick_watch_root uses PKG.len() for the probe bounds; snapshot_watchlist_paths re-snapshots after mid-batch dispatch so raw slices stay valid across evictions.
Extended reasoning...

Overview

This PR reworks the Windows --watch/--hot filesystem watcher from a single recursive ReadDirectoryChangesW rooted at the cwd to a set of per-root Box<DirWatcher>s sharing one IOCP, so files resolved outside the project root (workspace packages via junctions, bun link, file: deps) are actually watched. It touches WindowsWatcher.rs (substantial rewrite), Watcher.rs (cross-platform stop() ordering, new ensure_watch_root_covers / unwatchable_roots / platform_scratch fields), sys/windows/mod.rs (CancelIoEx/GetOverlappedResult externs), plus comment updates in DevServer.rs/path_watcher.rs, a dead filter removal in bake-harness.ts, and two new tests in watch.test.ts.

Security risks

None in the traditional sense — no untrusted-input parsing or auth surface. The real risk category is memory safety: pending overlapped kernel I/O against heap buffers that get freed, cross-thread access to Vec<Box<DirWatcher>> where the watch thread reaches boxes via raw lpOverlapped pointers while transpiler threads mutate the Vec under Watcher.mutex, and dead-slot reuse racing mark_dead. Prior review rounds surfaced and fixed several concrete UAFs here (unlocked stop() iteration, Ok-exit not calling stop(), CloseHandle not waiting for cancellation, missing SYNCHRONIZE in the access mask, post-publish field reads in mark_dead). This run's bug-hunting pass found nothing new.

Level of scrutiny

High. This is native Windows async-I/O lifecycle code with extensive unsafe, raw-pointer identity tricks (OVERLAPPED at offset 0 of a boxed struct), and a threading model where the mutex covers only some accessors by design. It has already gone through ~8 review rounds with multiple UAF/race findings; alii (human maintainer) has done two substantive review passes and the second one's feedback was just addressed in the head commit. This is exactly the class of change that should not be auto-approved.

Other factors

  • alii's two most recent inline comments (2026-08-13T18:10) are not yet marked resolved, though the head commit 6653b89 appears to address both: the --hot test now advances to phase = 6 on [app] 3 4 and asserts toBe(6), and the FdOwnership::Caller / append_file_maybe_lock comments now describe the new "watch root could not be opened" semantics.
  • The two new tests are stated to fail on main on Windows and pass with the build; the --hot lifecycle test now covers the dead-slot-reuse path that alii specifically called out.
  • The many unresolved github-actions "comment-cop" inline notes are automated lint noise on doc-comment length, not substantive review feedback.

@alii alii left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both remaining items are fixed in 6653b89 (the --hot test now fails on EOF during its last phase, and the two FdOwnership comments describe the new root-open failure); the interdiff contains nothing else. Merges cleanly onto current main. Windows behaviour is unverified from this side; looks ready to merge pending CI.

@robobun

robobun commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Two findings from #39832. That PR fixes the same warning for files inside the project when the cwd is spelled differently from the real path: an 8.3 short name such as C:\Users\AZUREU~1\... (a common %TEMP%), or a junction. On Windows GetCurrentDirectoryW returns the spelling the process was started with, while the entry point and everything imported from it get the GetFinalPathNameByHandle spelling.

  1. covers() compares spellings only. With such a cwd, every module resolved from the entry point is not covered, so ensure_watch_root_covers opens a second ReadDirectoryChangesW root on the project directory itself (or on each directory above the file until a package.json is found). The files do reload, but every change is then read and matched twice. A realpath of each root stored in RootState, with covers() accepting either spelling, removes the duplicate. watcher: accept the real path spelling of the project root on Windows #39832 has the helper (real_root_prefix) and two Windows-only tests in test/cli/hot/hot.test.ts (8.3 cwd, junction cwd) that carry over unchanged.

  2. Watcher::is_eligible_directory still compares against the cwd spelling. The parent directories of those modules are not watched, so their directory events (the directory cache bust on Windows) are lost. On Windows it can use the same covers() check.

If this PR lands first, #39832 becomes those two changes plus its tests on top of it.

Jarred-Sumner pushed a commit that referenced this pull request Aug 28, 2026
…austed (#39183)

### Problem

- The `Comment Cop` check is red on every claude-labeled PR whenever the
repo's GraphQL quota is used up. The step dies before it scans anything:
  ```
  GraphqlResponseError: Request failed due to following response errors:
   - API rate limit already exceeded for site ID installation.
  ##[error]Unhandled error: GraphqlResponseError: ...
  ```
with `x-ratelimit-resource: graphql`, `x-ratelimit-limit: 10000`,
`x-ratelimit-remaining: 0` (run
[31898901053](https://github.com/oven-sh/bun/actions/runs/31898901053)
on #39139). All 33 comment-cop failures on Aug 15 are this error (197
runs succeeded, 171 were skipped); they hit unrelated PRs at the same
time because the quota is shared by every workflow run in the repo, and
they come back whenever PR volume is high.
- Cause, `.github/workflows/comment-cop.yml:124` on main: the step reads
the PR's existing review threads with `github.graphql()`, outside any
`try`, and dedup and posting both sit behind that call. The REST call
just before it (`pulls.listFiles`) had succeeded in each failing run, so
the step had the diff; it only lacked the dedup data, which it was
reading from the exhausted quota.
- The same thread data also fed an auto-resolve of stale threads. That
part does not work: under the Actions `GITHUB_TOKEN` every
`resolveReviewThread` mutation fails with `Resource not accessible by
integration`, after which the step still logs `Resolved N stale
comment-cop thread(s).` Two successful runs from today:
[31900717034](https://github.com/oven-sh/bun/actions/runs/31900717034)
(20 attempted, 20 failed, "Resolved 20") and
[31900354924](https://github.com/oven-sh/bun/actions/runs/31900354924)
(4 attempted, 4 failed, "Resolved 4"). #36959 documents the same thing
and is the PR that moves resolution to a token that can do it. So the
GraphQL query was paying for one thing REST can provide and one thing
that does not happen.

### Fix

- Dedup reads the PR's review comments with `pulls.listReviewComments`
(REST) and collects the `<!-- comment-cop:KEY -->` markers from them.
Every comment the step posts starts with that marker, so the review
comments carry the same keys the thread roots did; REST is the quota the
step already needs for `listFiles` and `createReviewComment`, and it was
available in every failing run.
- The GraphQL query and the resolve loop are removed, so the step makes
no GraphQL request at all; the failure in the Problem section cannot
happen, rather than being caught. Nothing observable is lost: the
mutations the loop issued all fail today, and the only other thing it
did was log `Resolved N stale comment-cop thread(s).` after they had
failed. Resolving stale threads stays with #36959, which will also need
to move its thread lookup into its token step, since this PR removes the
`GITHUB_TOKEN` lookup it currently reuses (noted there).
- What the step posts is unchanged: the script on main and the REST
dedup were dry-run (real reads, writes recorded) against 9 PRs carrying
existing comment-cop threads (#35988, #36956, #36713, #35635, #33632,
#35596, #39139, #30609, #36959) and chose the same comments to post on
every one of them; the canned scenarios below show the same thing with
GraphQL working and with it exhausted.
- Test: `test/internal/source-lints/comment-cop.test.ts` extracts the
script from the workflow and runs it the way `actions/github-script`
does, against a fake `github` whose GraphQL requests all fail with the
rate limit error above. It checks that groups not yet flagged are posted
with the right line ranges, that a group already flagged (and a stale
marker) are left alone, that a second run recognizes the comments the
first run posted and posts nothing, and that no GraphQL request is made.
Both tests fail against the workflow on main (the script throws the
error above) and pass with this change. `source-lints.yml` now also
triggers on changes to `comment-cop.yml`, so the test runs whenever the
script is edited; it is excluded from the Buildkite shards like the rest
of that directory.
- Also ran: `bun bd test
test/internal/source-lints/comment-cop.test.ts`, `bun test
test/internal/source-lints/` (whole directory green), prettier on the
three files. The comment-cop run on this PR itself still executes the
script from main (`pull_request_target`), so the `Source lints` job is
the one that exercises the change here.
- Does not overlap with #37948 (which groups are flagged) or #38127
(where the file list comes from); both touch other parts of the script.
#36959 rewrites the block this PR deletes and will need a rebase either
way.

### Background

- Comment Cop (`.github/workflows/comment-cop.yml`): on each push to a
claude-labeled PR it reads the PR diff, finds multi-line comments added
under `src/`, and posts one review comment per comment block. Each bot
comment starts with `<!-- comment-cop:KEY -->`, KEY being the file path
plus a hash of the block's text; a block whose KEY is already on the PR
is not posted again. The check is advisory (not required for merge).
- GitHub API quotas: REST and GraphQL requests count against separate
hourly quotas (`x-ratelimit-resource` is `core` for REST and `graphql`
for GraphQL). For the `GITHUB_TOKEN` Actions hands out, each quota is
per repository, so every workflow run in oven-sh/bun draws on the same
two pools, and GraphQL-heavy automation (the `gh pr` / `gh issue` / `gh
search` commands used by other workflows go through GraphQL) empties the
GraphQL pool for everything else when PR volume is high.
- Review threads vs review comments: a review thread is GraphQL's
grouping of a line comment with its replies, and is the only place a
thread's id (what `resolveReviewThread` takes) and its resolved flag
exist. `GET /repos/{owner}/{repo}/pulls/{n}/comments` returns every
review comment on the PR, including each thread's root comment, so the
markers are reachable from REST; only resolving needs GraphQL, and under
`GITHUB_TOKEN` GitHub refuses that mutation regardless of the
`pull-requests: write` permission.

<details>
<summary>Script on main vs this branch against a fake github (the fake's
resolve mutation fails the way GITHUB_TOKEN's does)</summary>

```
main   | quota ok, stale threads present                        | exit 0                                   | posts ["src/foo.ts:5-6"] | resolve attempts ["T_STALE_OPEN"] | graphql ["query","query","mutation"] | warnings 1
main   | graphql quota exhausted, stale threads present         | step fails (unhandled rate limit error)  | posts [] | resolve attempts [] | graphql ["query"] | warnings 0
main   | graphql quota exhausted, nothing stale                 | step fails (unhandled rate limit error)  | posts [] | resolve attempts [] | graphql ["query"] | warnings 0
main   | graphql quota exhausted, PR has no review comments yet | step fails (unhandled rate limit error)  | posts [] | resolve attempts [] | graphql ["query"] | warnings 0
fixed  | quota ok, stale threads present                        | exit 0                                   | posts ["src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0
fixed  | graphql quota exhausted, stale threads present         | exit 0                                   | posts ["src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0
fixed  | graphql quota exhausted, nothing stale                 | exit 0                                   | posts ["src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0
fixed  | graphql quota exhausted, PR has no review comments yet | exit 0                                   | posts ["src/foo.ts:2-3","src/foo.ts:5-6"] | resolve attempts [] | graphql [] | warnings 0
```

</details>

<details>
<summary>Dry-run against live PRs: comments the script on main would
post vs the REST dedup (real reads through a user token, writes
recorded)</summary>

```
PR #35988: same posts (0 vs 0)        245 threads on the PR are stale; main attempts to resolve them, this branch does not
PR #36956: same posts (0 vs 0)
PR #36713: same posts (1832 vs 1832)  the stale cached file list that #38127 fixes; identical on both
PR #35635: same posts (0 vs 0)
PR #33632: same posts (0 vs 0)
PR #35596: same posts (0 vs 0)
PR #39139: same posts (1 vs 1)        the comment the failing run above did not get to post
PR #30609: same posts (0 vs 0)
PR #36959: same posts (0 vs 0)
```

</details>

<details>
<summary>Headers from the failing run</summary>

```
errors: [ { type: 'RATE_LIMIT', code: 'graphql_rate_limit',
            message: 'API rate limit already exceeded for site ID installation.' } ]
variables: { owner: 'oven-sh', repo: 'bun', pr: 39139, after: null }
'x-ratelimit-limit': '10000'
'x-ratelimit-remaining': '0'
'x-ratelimit-resource': 'graphql'
'x-ratelimit-used': '10000'
```

</details>
@robobun

robobun commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as stale: this PR conflicts with main, and its last commit is from 2026-08-13.

The branch stays, so the work is not lost. To pick it up again, rebase the branch on main and reopen this PR.

@robobun robobun closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bun doesnt pick changes in monorepos packages

3 participants