Skip to content

Enable BAZEL_TRACK_SOURCE_DIRECTORIES by default #25834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pauldraper opened this issue Apr 13, 2025 · 1 comment
Open

Enable BAZEL_TRACK_SOURCE_DIRECTORIES by default #25834

pauldraper opened this issue Apr 13, 2025 · 1 comment
Labels
team-Core Skyframe, bazel query, BEP, options parsing, bazelrc type: feature request untriaged

Comments

@pauldraper
Copy link
Contributor

pauldraper commented Apr 13, 2025

Description of the feature request:

The roll out of BAZEL_TRACK_SOURCE_DIRECTORIES has been taking a while (7+ years).

Any reason it shouldn't be enabled by default?

Which category does this issue belong to?

Core

What underlying problem are you trying to solve with this feature?

One particular example: npm external dependencies.

Starlark globbing every file, Bazel creating runfile symlinks for every file, etc. can be quite expensive (many tens, or even hundreds of thousands of files). Instead, I use one directory for the entire external dependencies. (Even though it's external, it's still a "source" directory.)

Have you found anything relevant by searching the web?

There is no fundamental reason why source artifacts cannot be tree artifacts or unresolved symlink artifacts, it's just that we haven't implemented it yet (we should, though -- referencing a source directory in a BUILD file is one of the few known long-standing incorrectness issues with Bazel; we have an implementation that kind of works which is enabled by the BAZEL_TRACK_SOURCE_DIRECTORIES=1 JVM property)

https://bazel.build/versions/8.1.0/contribute/codebase

But no indication what lacks about this option.

Any other information, logs, or outputs that you want to share?

/**
* A flag to enable / disable tracking of source directories. Uses a system property which can be
* set via a startup flag. The intention is for this code to be temporary, so I didn't want to add
* a permanent flag to startup options (and there's already --host_jvm_args, which we can use to
* roll this out). The flag affects Skyframe dependencies, so it needs to clear the Skyframe graph -
* the easiest way to do that is to restart the server, which is done when --host_jvm_args changes.
*/

@pauldraper
Copy link
Contributor Author

Hm, maybe it's this? #12954

@satyanandak satyanandak added the team-Core Skyframe, bazel query, BEP, options parsing, bazelrc label Apr 14, 2025
copybara-service bot pushed a commit that referenced this issue Apr 30, 2025
The new test subsumes an existing test that focused on empty directories as well as a shell test for package boundary handling.

Work towards #25834

Closes #25859.

PiperOrigin-RevId: 753094192
Change-Id: Ia341aeb56fcdad74c8806ff0e2994580e829d735
copybara-service bot pushed a commit that referenced this issue May 7, 2025
Also add a test verifying that source directories and globbed directory contents have the exact same representation with remote execution.

Fixes #11495
Work towards #25834

Closes #25868.

PiperOrigin-RevId: 755896046
Change-Id: I19d919f7b37e5eb173c0dfb9046f22f9ae7da56c
bazel-io pushed a commit to bazel-io/bazel that referenced this issue May 7, 2025
Also add a test verifying that source directories and globbed directory contents have the exact same representation with remote execution.

Fixes bazelbuild#11495
Work towards bazelbuild#25834

Closes bazelbuild#25868.

PiperOrigin-RevId: 755896046
Change-Id: I19d919f7b37e5eb173c0dfb9046f22f9ae7da56c
github-merge-queue bot pushed a commit that referenced this issue May 9, 2025
…ecution (#26023)

Also add a test verifying that source directories and globbed directory
contents have the exact same representation with remote execution.

Fixes #11495
Work towards #25834

Closes #25868.

PiperOrigin-RevId: 755896046
Change-Id: I19d919f7b37e5eb173c0dfb9046f22f9ae7da56c

Commit
154dae5

Co-authored-by: Fabian Meumertzheim <[email protected]>
copybara-service bot pushed a commit that referenced this issue May 13, 2025
Also add the target for the test that was missed in the merge of ad381cb.

Work towards #25834

Closes #25863.

PiperOrigin-RevId: 758308509
Change-Id: Iafb5854c8986eb3f3b34d96e4f94990f6914f47a
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue May 19, 2025
The new test subsumes an existing test that focused on empty directories as well as a shell test for package boundary handling.

Work towards bazelbuild#25834

Closes bazelbuild#25859.

PiperOrigin-RevId: 753094192
Change-Id: Ia341aeb56fcdad74c8806ff0e2994580e829d735
copybara-service bot pushed a commit that referenced this issue May 20, 2025
This makes it so that changes to the targets of unresolved symlinks invalidate actions that depend on the source directory containing them.

Work towards #25834

Closes #25864.

PiperOrigin-RevId: 760968335
Change-Id: I09c250eb8e7d147006a4e6fdc33a372ce7277bc0
fmeum added a commit to fmeum/bazel that referenced this issue May 20, 2025
This makes it so that changes to the targets of unresolved symlinks invalidate actions that depend on the source directory containing them.

Work towards bazelbuild#25834

Closes bazelbuild#25864.

PiperOrigin-RevId: 760968335
Change-Id: I09c250eb8e7d147006a4e6fdc33a372ce7277bc0
github-merge-queue bot pushed a commit that referenced this issue May 21, 2025
The new test subsumes an existing test that focused on empty directories
as well as a shell test for package boundary handling.

Work towards #25834

Closes #25859.

PiperOrigin-RevId: 753094192
Change-Id: Ia341aeb56fcdad74c8806ff0e2994580e829d735

Commit
ad381cb

Co-authored-by: Fabian Meumertzheim <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-Core Skyframe, bazel query, BEP, options parsing, bazelrc type: feature request untriaged
Projects
None yet
Development

No branches or pull requests

4 participants