Continual recompliation of blake3

[justinabrahms]

Blake3 is in my transitive dependency chain. When I edit my code, it recompiles every time. It seems to believe that the environment variable is changing.. but I'm just editing a file, nothing more.

Most seemingly relevant line below.

[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: EnvVarChanged { name: "CC", old_value: None, new_value: Some("gcc") }

Thoughts?

$ CARGO_LOG=cargo::core::compiler::fingerprint=info cargo build
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] dependency on `build_script_build` is newer than we are 1689140428.554280381s > 1689140320.333216314s "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/blake3-1.4.0"
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for WORK_CODE v0.1.0 (/home/abrahms/src/github.com/subconsciousnetwork/WORK_CODE)/Build/TargetInner { name: "WORK_PACKAGE", doc: true, ..: with_path("/home/abrahms/src/github.com/subconsciousnetwork/WORK_CODE/src/main.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for cid v0.10.1/Build/TargetInner { ..: lib_target("cid", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cid-0.10.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "multihash", old_fingerprint: 5789376448465686996, new_name: "multihash", new_fingerprint: 15717749769461962212 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for multihash v0.18.1/Build/TargetInner { ..: lib_target("multihash", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/multihash-0.18.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "blake3", old_fingerprint: 11447239697517040600, new_name: "blake3", new_fingerprint: 9816954190456901003 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for blake3 v1.4.0/Build/TargetInner { ..: lib_target("blake3", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/blake3-1.4.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "build_script_build", old_fingerprint: 13742670213903239432, new_name: "build_script_build", new_fingerprint: 991342934240953990 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for blake3 v1.4.0/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/blake3-1.4.0/build.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: EnvVarChanged { name: "CC", old_value: None, new_value: Some("gcc") }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for libipld v0.16.0/Build/TargetInner { ..: lib_target("libipld", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libipld-0.16.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "libipld_json", old_fingerprint: 17084127054727942497, new_name: "libipld_json", new_fingerprint: 5921226181653514756 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for libipld-cbor v0.16.0/Build/TargetInner { ..: lib_target("libipld-cbor", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libipld-cbor-0.16.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "libipld_core", old_fingerprint: 13071416919958757996, new_name: "libipld_core", new_fingerprint: 17725040881794123426 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for libipld-core v0.16.0/Build/TargetInner { ..: lib_target("libipld-core", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libipld-core-0.16.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for libipld-json v0.16.0/Build/TargetInner { ..: lib_target("libipld-json", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libipld-json-0.16.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "libipld_core", old_fingerprint: 13071416919958757996, new_name: "libipld_core", new_fingerprint: 17725040881794123426 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for libipld-macro v0.16.0/Build/TargetInner { ..: lib_target("libipld-macro", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libipld-macro-0.16.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "libipld_core", old_fingerprint: 13071416919958757996, new_name: "libipld_core", new_fingerprint: 17725040881794123426 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for libipld-pb v0.16.0/Build/TargetInner { ..: lib_target("libipld-pb", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libipld-pb-0.16.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "libipld_core", old_fingerprint: 13071416919958757996, new_name: "libipld_core", new_fingerprint: 17725040881794123426 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-api v0.9.1/Build/TargetInner { ..: lib_target("noosphere-api", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-api-0.9.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-car v0.2.0/Build/TargetInner { ..: lib_target("noosphere-car", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-car-0.2.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-core v0.13.1/Build/TargetInner { ..: lib_target("noosphere-core", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-core-0.13.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-collections v0.6.1/Build/TargetInner { ..: lib_target("noosphere-collections", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-collections-0.6.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-storage v0.7.1/Build/TargetInner { ..: lib_target("noosphere-storage", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-storage-0.7.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for ucan v0.4.0/Build/TargetInner { ..: lib_target("ucan", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ucan-0.4.0/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for ucan-key-support v0.1.7/Build/TargetInner { ..: lib_target("ucan-key-support", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ucan-key-support-0.1.7/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "ucan", old_fingerprint: 14090303343170715210, new_name: "ucan", new_fingerprint: 9002932740954988695 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for dummy-ns v0.1.0 (/home/abrahms/src/github.com/subconsciousnetwork/WORK_CODE2)/Build/TargetInner { name: "dummy-ns", doc: true, ..: with_path("/home/abrahms/src/github.com/subconsciousnetwork/WORK_CODE2/src/main.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-ns v0.9.1/Build/TargetInner { ..: lib_target("noosphere-ns", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-ns-0.9.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere v0.12.1/Build/TargetInner { ..: lib_target("noosphere", ["rlib", "staticlib", "cdylib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-0.12.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-sphere v0.7.1/Build/TargetInner { ..: lib_target("noosphere-sphere", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-sphere-0.7.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint] fingerprint dirty for noosphere-ipfs v0.6.1/Build/TargetInner { ..: lib_target("noosphere-ipfs", ["lib"], "/home/abrahms/.cargo/registry/src/index.crates.io-6f17d22bba15001f/noosphere-ipfs-0.6.1/src/lib.rs", Edition2021) }
[2023-07-12T05:41:37Z INFO  cargo::core::compiler::fingerprint]     dirty: UnitDependencyInfoChanged { old_name: "cid", old_fingerprint: 2196727044934558216, new_name: "cid", new_fingerprint: 7776841732667268909 }

[max-niederman]

I'm also having this issue.

[oconnor663]

build.rs contains these lines, which might be related:

    // The `cc` crate doesn't automatically emit rerun-if directives for the
    // environment variables it supports, in particular for $CC. We expect to
    // do a lot of benchmarking across different compilers, so we explicitly
    // add the variables that we're likely to need.
    println!("cargo:rerun-if-env-changed=CC");
    println!("cargo:rerun-if-env-changed=CFLAGS");

I'd be curious if removing those lines fixes the issue for you. That wouldn't tell us the whole story, but it would be a big part. Also, can you reproduce this issue in any open source project that I could also build for myself. It would help to see it happen on my own machine.

[max-niederman]

Yeah, it's almost certainly to do with those rerun directives.

I was messing around with it earlier and I did notice that it only ever happens in my VS Code terminal, which is quite weird. There are a lot of moving parts w.r.t. the terminal in my particular project because I'm using a Nix shell w/ Direnv, so it's going to be a bit difficult to track down the issue, but I'll set aside some time tomorrow to work on a minimal reproduction.

[pyranota]

I have similar problem on Voidlinux. But with small difference: dirty: EnvVarChanged { name: "CC", old_value: Some("gcc"), new_value: None }. I found a workaround for this. In my case worked exporting CC and AR manually:

1. export CC=gcc
2. export AR=ar

[Eragra3]

I have the same problem on Windows when using Bevy game engine. Any change triggers blake3 recompilation - is this expected?

Edit: it seems like clippy may be triggering that. It happens on code change when I use RustRover, because I configured it to run clippy in the background. I'm still not sure if this is expected or not.

[oconnor663]

Folks who can repro this, can you help me clarify: Are the values of these env vars actually changing every time? If so what sort of values are they taking? (If not, could this be a cc bug?)

[justinabrahms]

Just a note that I no longer work for this company and don't have access to that laptop or code base. Hopefully Max will be of more help to you.

[AviiNL]

On windows with bevy this is happening for me because between clippy and my manual cargo run, the PATH variable is different, probably because vscode runs clippy in either a cmd or powershell environment, while i manually trigger it in a nushell terminal. Wondering why it's even triggering a rebuild when PATH of all things change.

[oconnor663]

@AviiNL do you have time to bisect the rerun lines and figure out which one(s) we'd need to delete to fix your setup?

I'm open to ideas about whether these reruns are causing more harm than good. On the one hand I don't do as much benchmarking as I used to, and it probably wouldn't matter to me if I deleted these. On the other hand, if your compilation environment really is changing, then using stale artifacts might cause weird build issues, and it can be awfully confusing to debug that stuff.

[Nahor]

(Disclaimer: while I was affected by this issue, I've fixed my environment since so I'm not anymore. What follows is my 2cts on this problem)

@oconnor663, compilation is much slower because recompiling Blake3 has a cascading effect and lots of other crates get recompiled as well.
On my beefy machine (64 threads, 64GB RAM), an incremental compilation that normally takes 2s was now taking 1min20s. And that's not counting the time the environment was locked waiting for the IDE rust-analyzer/clippy to finish running.

And to add insult to injuries, in my case, the "path changed" was only duplicating an existing path. The path to cargo/bin was being added a second time by the IDE, on top of one already there. So even though the PATH string was changing, the "canonical environment" did not.
So for me at least, it was more harm than good, a significant but pointless slow down of compilation.

As for the PATH changing being a possible problem, I would think it's no different from a new version of the tools in the same path. Or changes of other stuff in the environment not related to paths.
My own bikeshedding opinion is that detecting path changes is overkill. Personally, I know that if I change my build environment, I ought to do a clean build, that I can't rely an automatic detection of all and every things that may or may not have changed.

[AviiNL]

Yeah pretty much what @Nahor wrote is my experience as well, entries in PATH begin duplicated by the IDE. I've put some ducttape on it in the form of extra environment arguments in settings.json for rust analyzers usage of cargo. rust-analyzer.server.extraEnv with a copy of my own PATH, it now no longer recompiles blake3.

I did a quick check in build.rs, but there doesn't appear to be anything explicitly checking for the PATH variable, so no idea why it's triggering on it.

[oconnor663]

I did a quick check in build.rs, but there doesn't appear to be anything explicitly checking for the PATH variable, so no idea why it's triggering on it.

Yeah exactly. I agree with @Nahor that rebuilding on any PATH change would be overkill, but I don't understand why we're seeing that behavior. Is it possible that the changing PATH is a "red herring", and there's some other variable(s) changing at the same time? The variables we (tell Cargo to) check are:

• CC
• CFLAGS
• CARGO_FEATURE_PURE
• CARGO_FEATURE_PREFER_INTRINSICS
• CARGO_FEATURE_NEON
• CARGO_FEATURE_NO_NEON
• BLAKE3_CI

Those last five are custom settings that are specific to this project, so I don't expect any other tools to ever set them. However, if you do happen to set them yourself in your regular build environment, and your clippy/miri/etc environment doesn't set them, then something like that could be what's causing the build thrashing that you guys are seeing. Do you set any of those variables anywhere in your scripts?

[Nahor]

It was definitely the PATH for me. That's what cargo was complaining about, removing it from the IDE settings fixed my issue, and when I manually update the PATH on the command line it comes back, even between runs from the command line:

$ cargo build
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.35s
$ export PATH=".:$PATH"
$ cargo build
   Compiling blake3 v1.5.1
   [...]
   Finished `dev` profile [optimized + debuginfo] target(s) in 1m 20s
$ cargo build
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.29s   
$ export PATH=".:$PATH"
$ cargo build
   Compiling blake3 v1.5.1
   [...]
   Finished `dev` profile [optimized + debuginfo] target(s) in 1m 19s

It might be related to build.rs because the trace log has this target="build-script-build":

INFO prepare_target{force=false package_id=blake3 v1.5.1 target="build-script-build"}: cargo::core::compiler::fingerprint:     dirty: EnvVarChanged { name: "PATH", old_value: ...

[AviiNL]

Yeah I don't know what to tell you, when running with CARGO_LOG=cargo::core::compiler::fingerprint=info the only line that stands out is

0.371086600s  INFO prepare_target{force=false package_id=blake3 v1.5.1 target="build-script-build"}: cargo::core::compiler::fingerprint:     dirty: EnvVarChanged { name: "PATH", old_value: Some("..."), new_value: Some("...") }

and after forcing PATH in settings.json so rust analyzer uses the same value as my terminal environment, it works as it should

{
    "rust-analyzer.server.extraEnv": {
        "PATH": "..."
    }
}