go: support "all:" prefix for "go:embed" directives

[malt3]

What type of PR is this?

Feature

What does this PR do? Why is it needed?

Implements the "all:" prefix for //go:embed directives. From Go docs:

If a pattern begins with the prefix ‘all:’, then the rule for walking directories is changed to include those files beginning with ‘.’ or ‘_’. For example, ‘all:image’ embeds both ‘image/.tempfile’ and ‘image/dir/.tempfile’.

Which issues(s) does this PR fix?

Fixes #3441

Other notes for review

[fmeum]

Thanks, looks good!

On last thing: I just noticed that the Go docs only ever mention files starting with . and _ - do you know whether some/_hidden/file would be included also without all:... with the standard Go toolchain?

[malt3]

On last thing: I just noticed that the Go docs only ever mention files starting with . and _ - do you know whether some/_hidden/file would be included also without all:... with the standard Go toolchain?

The Go docs seem to be correct! Any directories are always included (including those starting with . or _). Should I fix the behavior of rules_go to follow this more closely?

[fmeum]

That would be great, yes. Could you keep that change in a separate commit?

@linzhp @achew22 Do you have any backwards compatibility concerns?

[malt3]

A correct implementation should closely follow https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;l=2045;drc=2f2c5e41e7728cae99abd14e8bd5ffcfebf9df8e

I do want to get there but I feel like we would need to change the implementation more drastically to get it to be correct.

Additional edge cases I found:

• Hidden files can be conditionally in-/excluded based on if the folder name is given with a glob or not
  given the folder a and the file a/.hidden, the directive //go:embed a would exclude the file, while //go:embed a/* would include the file.
• There are some exclude patterns for VCS: https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;drc=261fe25c83a94fc3defe064baed3944cd3d16959;bpv=0;bpt=1;l=2203
• Folders named go.mod are ignored https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;l=2158;drc=261fe25c83a94fc3defe064baed3944cd3d16959;bpv=0;bpt=1

I don't have the time available to implement this behavior correctly.

[fmeum]

Thanks for collecting the pointers, that's already very helpful.

Given that you uncovered a number of places in which we (unknowingly) deviate from the native toolchain, let's just keep the scope of this PR limited to all: support. I certainly won't ask you to fix any of these existing issues and adding all: won't make them worse (with one exception, see below).

How do you feel about doing the following as part of this PR?

1. Add excludes for ".bzr", ".hg", ".git", and ".svn", so that users of all: don't accidentally pick up their version control metadata when they wouldn't with the native Go toolchain.
2. Add the list in your last comment as a TODO to the code.

Let me know if you think that you won't have the time for it and we can consider merging as is.

[malt3]

1. Add excludes for ".bzr", ".hg", ".git", and ".svn", so that users of all: don't accidentally pick up their version control metadata when they wouldn't with the native Go toolchain.
   Add the list in your last comment as a TODO to the code
2. Add the list in your last comment as a TODO to the code.

I'll do that. Thanks for the quick responses!

[linzhp]

Do you have any backwards compatibility concerns?
No. I don't see a backward-compatibility issue here.

[malt3]

I implemented excludes for ".bzr", ".hg", ".git", and ".svn" and added TODOs.
I also noticed that the current implementation is already correct when it comes to the inclusion of hidden leaf nodes that I thought was not correct at first:

Hidden files can be conditionally in-/excluded based on if the folder name is given with a glob or not
given the folder a and the file a/.hidden, the directive //go:embed a would exclude the file, while //go:embed a/* would include the file.

This works in the same way that it does for the native toolchain.

[fmeum]

Thanks! I added the links to the comments. This is good to go now.