Add a remote repo contents cache

[fmeum]

RELNOTES[NEW]: The results of reproducible repository rules without dependencies added at runtime (e.g., via repository_ctx.watch or .getenv) can now be cached in a regular HTTP or gRPC remote cache if the new --experimental_remote_repo_contents_cache startup option is provided.

The output of du -h $(bazel info output_base) after bazel build //src:bazel-dev and a fully up-to-date remote cache

• without the flag:

$ bazel info peak-heap-size used-heap-size-after-gc
peak-heap-size: 271MB
used-heap-size-after-gc: 123MB
$ du -h $(bazel info output_base) | tail -1
1.3G	/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722

• with the flag:

$ bazel info peak-heap-size used-heap-size-after-gc
peak-heap-size: 266MB
used-heap-size-after-gc: 120MB
$ du -h $(bazel info output_base) | tail -1
380M	/private/var/tmp/_bazel_fmeum/507738cfc7e6cde00e4a0230e9aa0722

Some repos are still materialized eagerly, which may not be necessary. Patched http_archive also can't be cached yet, so these numbers are likely to improve with further work on this feature.

TODO:

• Recovering from lost BUILD/.bzl files relies upon a solution to Starting with Bazel 8.3.0, archive_override removes all files from the archive root #26450
• Should the debug events be removed?

Fixes #6359

[sluongng]

Hmm, this feels very "heavy-handed", especially after reading the fsForPath below.

What if we first refactor BlazeRuntime to add a dedicated "external repo" entry to ServerDirectories, then make it configurable 🤔

[fmeum]

Been there, but I couldn't get it to work. The problem is that lots of SkyValues have Paths in their corresponding SkyKeys, which means that there needs to be a singleton file system used everywhere for the lifetime of the server. Otherwise you would need to keep some other global object around that knows how to map a logical path to the correct file system for the respective key and add special handling for each file operation that may cross file system boundaries. Pretty much like fsForPath, but not just in a single location.