Improve handling of compile_data with mixed sources

[martingms]

I'm not very well versed in starlark nor the rules_rust codebase, so feel free to ignore this and address the issue in a more fitting way, but this fixes #3171 and a related issue for me.

[sam-mccall]

I was looking at generated sources recently, this change makes sense to me.

Needs an owner, @krasimirgg we were talking about this very issue a couple of days ago :-)

[sam-mccall]

do we need to worry about compile_data being None?

[martingms]

afaict this shouldn't be possible, attr.label_list defaults to [], and we set compile_data explicitly to a depset based on that in all codepaths, but I can add this check back if you think it's better to be defensive. It was a bit confusing to me, which is why I removed it!

[krasimirgg]

Looks good! Please re-format the file that the Buildifier workflow is complaining about.

[krasimirgg]

Please reformat according to https://buildkite.com/bazel/rules-rust-rustlang/builds/13739#01945f02-1578-4657-8c29-aeaef49853d9/127-128

[martingms]

Sorry, missed buildifier and one test affected by the last commit, pushed an updated patch now 👍

[UebelAndre]

This has lead to a regression for targets which have compile data from completely other packages. See repro at (#3193)

ERROR: /Users/user/Code/rules_rust/test/unit/compile_data/src/BUILD.bazel:11:13: in rust_library rule //test/unit/compile_data/src:lib:
Traceback (most recent call last):
	File "/Users/user/Code/rules_rust/rust/private/rust.bzl", line 87, column 32, in _rust_library_impl
		return _rust_library_common(ctx, "rlib")
	File "/Users/user/Code/rules_rust/rust/private/rust.bzl", line 152, column 55, in _rust_library_common
		srcs, compile_data, crate_root = transform_sources(ctx, ctx.files.srcs, ctx.files.compile_data, crate_root)
	File "/Users/user/Code/rules_rust/rust/private/utils.bzl", line 825, column 64, in transform_sources
		generated_compile_data = [_symlink_for_non_generated_source(ctx, src, package_root) for src in compile_data]
	File "/Users/user/Code/rules_rust/rust/private/utils.bzl", line 870, column 64, in _symlink_for_non_generated_source
		src_symlink = ctx.actions.declare_file(paths.relativize(src_short_path, package_root))
	File "/private/var/tmp/_bazel_user/76282c66b0dfe3c5cb9a230bdc913a52/external/bazel_skylib~/lib/paths.bzl", line 247, column 17, in _relativize
		fail("Path '%s' is not beneath '%s'" % (path, start))
Error in fail: Path 'test/unit/compile_data/data/data.txt' is not beneath 'test/unit/compile_data/src'
ERROR: /Users/user/Code/rules_rust/test/unit/compile_data/src/BUILD.bazel:11:13: Analysis of target '//test/unit/compile_data/src:lib' failed

I don't quite know how to solve for this as I'm not sure what the symlink path should be for this foreign path.

[krasimirgg]

Happy if we rolled this back until we have a full fix for this.

[martingms]

@krasimirgg The crash should be fixed by #3196 or similar, but it's unclear to me how the repro in #3193 could have worked (as in the data.txt file is actually available during compile) even before this (#3176) PR was merged, as generated sources were built in the out tree even before. I'm probably missing something! Do you have any pointers to what the situation is in the real world failures?

Fine with me to roll it back too obviously.

[jgsogo]

This PR broke something in my project.

I'm using the diesel_migrations::embed_migrations! macro to collect the migrations for my database. This macro takes a relative path from the Cargo.toml file to the directory that contains the migrations. This works in the Cargo world, however, with Bazel, it fails after this PR.

For a rust library inside a <repo-root>/libraries/diesel_utils directory, the commit that works (5e426fa) and the one that fails (bb74a65) populate the same CARGO_MANIFEST_DIR='${pwd}/libraries/diesel_utils' envvar, but the one that works contains an additional directory with the files from the compile_data in the working directory: libraries/diesel_utils/<value-provided-to-embed_migrations>

I wanted to report my findings here before having a closer look to the PR and maybe open a more detailed issue, or even a PR with a fix.

I have migrations in a rust_library and also in a rust_test. This patch fixes my issue (of course, it requires further investigation):

diff --git a/rust/private/rust.bzl b/rust/private/rust.bzl
index 74d458ee..0dc8b4e6 100644
--- a/rust/private/rust.bzl
+++ b/rust/private/rust.bzl
@@ -203,7 +203,7 @@ def _rust_library_common(ctx, crate_type):
             rustc_env_files = ctx.files.rustc_env_files,
             is_test = False,
             data = depset(ctx.files.data),
-            compile_data = depset(compile_data),
+            compile_data = depset(ctx.files.compile_data),
             compile_data_targets = depset(ctx.attr.compile_data),
             owner = ctx.label,
         ),
@@ -419,7 +419,7 @@ def _rust_test_impl(ctx):
             rustc_env = rustc_env,
             rustc_env_files = ctx.files.rustc_env_files,
             is_test = True,
-            compile_data = depset(compile_data),
+            compile_data = depset(ctx.files.compile_data),
             compile_data_targets = depset(ctx.attr.compile_data),
             owner = ctx.label,
         )

[martingms]

Hi @jgsogo, do you have a minimal reproduction of your issue? It's been a while since I've looked at this, but your patch breaks the stuff that this PR was meant to fix, so I hope we can find a different solution!

[jgsogo]

Here it is: https://github.com/jgsogo/rules_rust-3176

Reproducing the error

git clone https://github.com/jgsogo/rules_rust-3176
cd rules_rust-3176

bazel build //libraries/diesel_utils:integration_tests --verbose_failures --sandbox_debug

Error log (uses rules_rust v0.59.2)

error: proc macro panicked
 --> bazel-out/darwin_x86_64-fastbuild/bin/libraries/diesel_utils/tests/common/mod.rs:3:5
  |
3 |     diesel_migrations::embed_migrations!("tests/common/migrations");
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: message: Failed to receive migrations dir from Some("tests/common/migrations")

error: aborting due to 1 previous error

indeed, if you check the working directory, the libraries/diesel_tuils/tests/common/migrations folder is not there:

➜  rules_rust-3176 git:(main) ls -la /private/var/tmp/_bazel_jgsogo/6a6eb0d5365aa7efc463fc4e16e1d4d9/sandbox/darwin-sandbox/1056/execroot/_main
total 0
drwxr-xr-x   4 jgsogo  wheel  128 Mar 31 13:22 .
drwxr-xr-x   3 jgsogo  wheel   96 Mar 31 11:27 ..
drwxr-xr-x   4 jgsogo  wheel  128 Mar 31 13:30 bazel-out
drwxr-xr-x  13 jgsogo  wheel  416 Mar 31 13:38 external

The issue

In the MODULE.bazel you can modify rules_rust to use different commits, right before and after this PR: https://github.com/jgsogo/rules_rust-3176/blob/c42eb235cde50cf1933666743b543eb9f4a53c1f/MODULE.bazel#L13-L25

Using the commits that works, the migrations are available in the working directory:

➜  rules_rust-3176 git:(main) ✗ ls -la /private/var/tmp/_bazel_jgsogo/6a6eb0d5365aa7efc463fc4e16e1d4d9/sandbox/darwin-sandbox/1251/execroot/_main
total 0
drwxr-xr-x   5 jgsogo  wheel  160 Mar 31 13:49 .
drwxr-xr-x   3 jgsogo  wheel   96 Mar 31 13:49 ..
drwxr-xr-x   4 jgsogo  wheel  128 Mar 31 13:49 bazel-out
drwxr-xr-x  13 jgsogo  wheel  416 Mar 31 13:49 external
drwxr-xr-x   3 jgsogo  wheel   96 Mar 31 13:49 libraries               <-- Here it is!

The issue comes from the target //libraries/diesel_utils:schema that is copying a file into the build directory. If, instead of using this generated file, I use a file in the workspace, it works. See https://github.com/jgsogo/rules_rust-3176/blob/c42eb235cde50cf1933666743b543eb9f4a53c1f/libraries/diesel_utils/BUILD.bazel#L24-L32, and switch from one implementation to the other.

---

Don't judge the way I'm generating this schema.rs file and how I'm building diesel_cli 😅 . This is a side-project and I know I need to refactor this part. Anyway, it shouldn't matter the way I'm generating schema.rs, I think it's just the issue/conflict with two rules contributing files to the working directory.

[jgsogo]

btw, in my project I will decouple the schema.rs generation from the library build, and I think this is the right thing to do. My workflow now looks as follows:

1. A rule that generates the schema.rs file
2. A test rule that validates that the generated schema.rs is equal to the one in the repo
3. A manual rule to copy the generated file into the workspace

With this approach, this is no longer an issue for me. However I can imagine other scenarios where several rules contribute files to a rust library.

[martingms]

I see, the way this library looks for the migrations does mesh well with bazel in general really, as it seems it's hard-coded to search from CARGO_MANIFEST_DIR as the root. You could try a hack by setting CARGO_MANIFEST_DIR with rustc_env somehow, but that might break other stuff if this lib is that tied to cargo..