macOS slowdown due to temporary files outside of Library/Caches

[kornelski]

I've noticed that when I'm working on Rust projects, macOS processes mds and mds_worker are consuming high amount of CPU and disk I/O. These processes are indexing the disk for macOS Spotlight search functionality. I suspect churn of large numbers of files in target/ subdirectories like .fingerprint and incremental causes Spotlight to continually rescan and reindex these directories.

macOS expects well-behaved applications to use ~/Library/Caches or /tmp for all temporary files, so it doesn't have any official APIs for reducing negative impact of file churn elsewhere. Officially the only way to exclude folders from Spotlight is to add them as "private" in GUI settings, which is cumbersome.

I see two ways of fixing this problem:

1. Move build, incremental to a subdirectory in ~/Library/Caches/Cargo.

• The subdirectory could be derived from a hash of dest dir, so that these directories remain separate.
• Optionally the could be moved to the correct temp dir only if CARGO_TARGET_DIR is not set, so that users who have customized CARGO_TARGET_DIR are not surprised by files being put elsewhere.
• Optionally there could be symlinks in target/ to the new temp location, so that 3rd party utilities that peek into Cargo's internals continue to work.

2. or rename them to build.noindex, incremental.noindex. This naming convention is a poorly documented way of excluding folders from Spotlight indexing. It isn't as good as using ~/Library/Caches, but would solve the Spotlight issue with a smaller change to Cargo's behavior.

[steven-joruk]

This page hints that dot prefixed files and directories shouldn't be indexed, unless you explicitly import them.

This web page suggest these items aren't indexed:

• Hidden files: Files whose names begin with a period (.).
• Invisible files: Files whose invisible file-system attribute has been enabled.
• Files within hidden or invisible folders.
• Under Mac OS X 10.4, System-related files or folders. These are indexed and can be searched under Mac OS X 10.5 and later.

If the highlighted line is right then perhaps just using chattr to hide the target directory would be sufficient?

[kornelski]

Thanks for the tip. The hidden attribute may be useful for excluding temp files. However, it can't be just applied wholesale, because it would prevent access to build products. For example, I sometimes open built executables and benches in Instruments.app using its file browser, which doesn't show hidden files.