Rescan events from the notify crate are not being handled, causing cache inconsistencies

[bvinc]

When using buck2 in the default configuration (not using watchman), buck2 uses the notify crate to monitor file system events to keep its view of the file system consistent. While watchman and the notify both sit on top of inotify on Linux, they have different behavior and APIs. Specifically, inotify can miss events when its internal queue overflows. inotify tells you when this happens with a flag.

https://man7.org/linux/man-pages/man7/inotify.7.html

Note that the event queue can overflow. In this case, events are
lost. Robust applications should handle the possibility of lost
events gracefully. For example, it may be necessary to rebuild
part or all of the application cache. (One simple, but possibly
expensive, approach is to close the inotify file descriptor, empty
the cache, create a new inotify file descriptor, and then re-
create watches and cache entries for the objects to be monitored.)

Watchman has logic to handle this:

https://facebook.github.io/watchman/docs/install#linux-inotify-limits

When the kernel reports an overflow, watchman will assume that all the files have been modified and will re-crawl the directory tree as though it had just started watching the dir.

The notify crate, like the inotify API, simply reports to the caller when this happens.

https://docs.rs/notify/8.2.0/notify/event/enum.Flag.html#variant.Rescan

Buck2 does not handle this case, which is a bug, and it can cause buck2's caches to become out of sync with the file system when a large number of changes occur.

One possible fix is for buck2 to have a way to invalidate all of its file system caches when a rescan event is seen.