fs: allow correct handling of burst in fs-events with AsyncIterator

[pipobscure]

This addresses a bug in fs.watch when used as an AsyncIterator.

The issue is that when consuming the AsyncIteractor returned by fs.watch it yields a value. When using that value and in turn awaiting an asynchrounous action any events happening in the meantime will go missing. The reason is that between exiting the watch function by yielding and reentering it through the next round, the promise inside watch is already resolved. So any events generated will be duplicate resolutions of that promise and therefore ignored.

for await (let found of fs.watch('my dir')) {
  console.log(found);
  await sleep(10000); // any events happening during these 10seconds will  be missed;
}

More reaslistically than this minimal example is when the found files are actually read via await fs.readFile. Then there will be a lag. If the file events happening are happening close together, then the second one will be missed.

To fix this issue I added a queue to the watch function that new file events get pushed onto. The promise is no longer resolved with a value, but is simply the gate gating whether or not there are any events in the queue. The iterator awaits the promise and then yields the items from the queue so long as there are any. When the queue is empty and the watch is still running, then a new promise is created and awaited upon. This whould eliminate the problem entirely and one can now go asynchronous in the loop as long as one wants without missing events.

Verified that the added test fails with v24.0.0 and passes after the fix.

Based on feedback the queuing was made configurable with the maxQueue (default 2048) option determining the maximum size of the queue and the overflow option deciding to either ignore the issue or throw an Error (default: 'ignore'). To effectively get back the previous behavior one would have to pass { maxQueue: 1, overflow: 'ignore' }.

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[codecov]

Codecov Report

Attention: Patch coverage is 72.50000% with 11 lines in your changes missing coverage. Please review.

Project coverage is 90.15%. Comparing base (40d8983) to head (218d461).
Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/fs/watchers.js	71.79%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #58490      +/-   ##
==========================================
- Coverage   90.16%   90.15%   -0.01%     
==========================================
  Files         636      636              
  Lines      187891   187923      +32     
  Branches    36884    36882       -2     
==========================================
+ Hits       169408   169429      +21     
- Misses      11246    11247       +1     
- Partials     7237     7247      +10     

Files with missing lines	Coverage Δ
lib/internal/errors.js	97.48% <100.00%> (+<0.01%)	⬆️
lib/internal/fs/watchers.js	85.16% <71.79%> (-0.63%)	⬇️

... and 30 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[joyeecheung]

This almost LGTM % the suggestions. Though I'd appreciate if others from @nodejs/fs or maybe @jasnell @benjamingr can take a look and see if the new options look good.

[Ethan-Arrowood]

Other than the option values, this LGTM

[joyeecheung]

LGTM % missing updates to the docs

[pipobscure]

Thanks for all the help, guidance & input @joyeecheung & @Ethan-Arrowood ❤️

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/58490
✔  Done loading data for nodejs/node/pull/58490
----------------------------------- PR info ------------------------------------
Title      fs: fix race-condition in fs.watch async iterator (#58490)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     pipobscure:fsbug -> nodejs:main
Labels     fs, semver-minor, needs-ci, commit-queue-squash
Commits    25
 - fs: fix race-condition in fs.watch async iterator
 - add test to validate multiple events not missing
 - switch to using FixedQueue
 - switch to using parallel.status
 - add maxQueue and overflow options
 - add documentation for new options
 - Update test/parallel/test-fs-promises-watch-iterator.js
 - add type tests for new options
 - Update lib/internal/fs/watchers.js
 - Update lib/internal/fs/watchers.js
 - Merge branch 'fsbug' of https://github.com/pipobscure/node into fsbug
 - add standard node error
 - fixed linting
 - hopefully fix linting
 - fixing options tests
 - hopefully fix md formatting
 - Update lib/internal/errors.js
 - Update lib/internal/fs/watchers.js
 - Update doc/api/errors.md
 - Update test/parallel/test-fs-promises-watch.js
 - Update doc/api/fs.md
 - Reverting to .rejects as fs.watch is an async function
 - change overflow option to ignore | error
 - revert tests to reject and use async executor because of async generator
 - Update doc/api/fs.md
Committers 2
 - Philipp Dunkel <[email protected]>
 - GitHub <[email protected]>
PR-URL: https://github.com/nodejs/node/pull/58490
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Ethan Arrowood <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/58490
Reviewed-By: Joyee Cheung <[email protected]>
Reviewed-By: Ethan Arrowood <[email protected]>
Reviewed-By: Benjamin Gruenbaum <[email protected]>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Wed, 28 May 2025 10:03:11 GMT
   ✔  Approvals: 3
   ✔  - Joyee Cheung (@joyeecheung) (TSC): https://github.com/nodejs/node/pull/58490#pullrequestreview-2905470218
   ✔  - Ethan Arrowood (@Ethan-Arrowood): https://github.com/nodejs/node/pull/58490#pullrequestreview-2905320739
   ✔  - Benjamin Gruenbaum (@benjamingr) (TSC): https://github.com/nodejs/node/pull/58490#pullrequestreview-2905504134
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2025-06-07T08:20:31Z: https://ci.nodejs.org/job/node-test-pull-request/67306/
- Querying data for job/node-test-pull-request/67306/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  No git cherry-pick in progress
   ✔  No git am in progress
   ✔  No git rebase in progress
--------------------------------------------------------------------------------
- Bringing origin/main up to date...
From https://github.com/nodejs/node
 * branch                  main       -> FETCH_HEAD
✔  origin/main is now up-to-date
- Downloading patch for 58490
From https://github.com/nodejs/node
 * branch                  refs/pull/58490/merge -> FETCH_HEAD
✔  Fetched commits as 3b111eb3f592..908f3c18c6f3
--------------------------------------------------------------------------------
[main eb729931fd] fs: fix race-condition in fs.watch async iterator
 Author: Philipp Dunkel <[email protected]>
 Date: Wed May 28 10:59:54 2025 +0100
 1 file changed, 14 insertions(+), 7 deletions(-)
[main 6c4b747674] add test to validate multiple events not missing
 Author: Philipp Dunkel <[email protected]>
 Date: Thu May 29 12:22:54 2025 +0100
 1 file changed, 60 insertions(+)
 create mode 100644 test/parallel/test-fs-promises-watch-iterator.js
[main 75466ed2c1] switch to using FixedQueue
 Author: Philipp Dunkel <[email protected]>
 Date: Fri May 30 16:48:43 2025 +0100
 1 file changed, 8 insertions(+), 3 deletions(-)
Auto-merging test/parallel/parallel.status
[main 516d024de1] switch to using parallel.status
 Author: Philipp Dunkel <[email protected]>
 Date: Fri May 30 16:49:04 2025 +0100
 2 files changed, 4 insertions(+), 8 deletions(-)
[main 3558e13c3f] add maxQueue and overflow options
 Author: Philipp Dunkel <[email protected]>
 Date: Wed Jun 4 11:56:15 2025 +0100
 1 file changed, 34 insertions(+), 11 deletions(-)
Auto-merging doc/api/fs.md
[main 8ea11e6519] add documentation for new options
 Author: Philipp Dunkel <[email protected]>
 Date: Thu Jun 5 12:19:33 2025 +0100
 1 file changed, 5 insertions(+)
[main e5c1d5c84b] Update test/parallel/test-fs-promises-watch-iterator.js
 Author: Philipp Dunkel <[email protected]>
 Date: Thu Jun 5 12:19:48 2025 +0100
 1 file changed, 4 insertions(+)
[main 044fee0bb7] add type tests for new options
 Author: Philipp Dunkel <[email protected]>
 Date: Thu Jun 5 12:23:18 2025 +0100
 1 file changed, 14 insertions(+)
[main 71d008fc9f] Update lib/internal/fs/watchers.js
 Author: Philipp Dunkel <[email protected]>
 Date: Thu Jun 5 12:23:52 2025 +0100
 1 file changed, 1 insertion(+), 1 deletion(-)
error: commit 3e71547d8728b8f629946c120022b2151dc64459 is a merge but no -m option was given.
fatal: cherry-pick failed
[main b72a342083] Update lib/internal/fs/watchers.js
 Author: Philipp Dunkel <[email protected]>
 Date: Thu Jun 5 12:24:06 2025 +0100
 1 file changed, 3 insertions(+), 6 deletions(-)
   ✘  Failed to apply patches

https://github.com/nodejs/node/actions/runs/15511732335

[joyeecheung]

The cherry picking failed. Can you rebase on top of the main branch and squash the commits? I think the commit message probably should be rewritten again too at this point

[pipobscure]

Just did a rebase, squash and rename.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/67350/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/67365/

[nodejs-github-bot]

Landed in 5f7dbf4