Disabled optional weak dependencies end up in Cargo.lock
#10801
Comments
ehuss
added
A-lockfile
|
Note that this bug also affects the output of the metadata subcommand, but not the tree subcommand. Weak dependencies are resolved in the |
cargo metadata resolves all dependencies even ones that are disabled (known as weak dependencies) which cargo build doesn't do (see issue rust-lang/cargo#10801). These unresolved dependencies require network access to load their package metadata. As the `--frozen` flag itself blocks network access (in addition to blocking changes to the Cargo.lock) too, we need to remove both the `--frozen` and `--offline` flags. Doing this allows cargo the change the `Cargo.lock` though, which is undesirable (as we're in the process of building the same dependencies that rely on the `Cargo.lock` in the first place).
|
I believe I am running into this with target-specific dependencies: I noticed this when there was a version conflict (since I already had the dependency in the lock file from another dependency). |
cargo metadata resolves all dependencies even ones that are disabled (known as weak dependencies) which cargo build doesn't do (see issue rust-lang/cargo#10801). These unresolved dependencies require network access to load their package metadata. As the `--frozen` flag itself blocks network access (in addition to blocking changes to the Cargo.lock) too, we need to remove both the `--frozen` and `--offline` flags. Doing this allows cargo the change the `Cargo.lock` though, which is undesirable (as we're in the process of building the same dependencies that rely on the `Cargo.lock` in the first place).
|
I'm not sure but I think I ran into this issue in: I found it unexpected that |
|
I've noticed this as well when |
|
@ljedrz that behavior is expected. If you're talking about It's the same reason |
|
@lopopolo indeed, it's that dependency; however, since I'm not building on an applicable target, I'd expect it to not show up in my lockfile just as much as if it were an optional (and disabled) dependency. |
Most of the new dependencies are false positive and are not actually required, but are false dependencies due to a Cargo bug. rust-lang/cargo#10801
A note: this adds serde to the lockfile, but it is not actually used. This is caused by rust-lang/cargo#10801.
|
This also happens with sea-orm which pulls chrono and time when you use the |
I believe this is a known limitation of the lock file generation that is intended to be improved eventually. From #5655 (comment),
While #5133 mentions From #7058 (comment),
|
This only applies to the root crate (the one you are working on) but not the dependencies. When you depend on Cargo.toml[package]
name = "foobar"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
serde = "1.0.147"Cargo.lock# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "foobar"
version = "0.1.0"
dependencies = [
"serde",
]
[[package]]
name = "serde"
version = "1.0.147"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d193d69bae983fc11a79df82342761dfbf28a99fc8d203dca4c3c1b590948965"
This issue is not comparable to platform dependent In my original posts in the |
|
I understand what you mean with weak features now. I'll see if I can find anything mentioned in the past.
Apologies for any confusion, I forgot to add an @ mention to the platform specific part of my comment, which was meant to be in response to @ljedrz #10801 (comment)
|
|
The issue with platforms should be be a separate issue, this is really for the issue where a dependency is truly never used but still included. Let's keep it on topic 👍 |
Indeed, I linked to those earlier in #10801 (comment) so that related discussion can move there. |
epage
changed the title
Cargo.lockCargo.lock
epage
changed the title
Cargo.lockCargo.lock
epage
changed the title
Cargo.lockCargo.lock
Using optional dependency features (`package?/feature`) in Cargo is not too well supported, see: rust-lang/cargo#10801 So instead we now always enable needed features of optional dependencies. This fixes #656. I thought this would be a harder trade-off (e.g. I thought that we'd have to enable a bunch of sub-dependencies for each dependency, and that wouldn't be nice since I'd prefer to have each crate as explicitly imported as possible), but it actually turned out to not be too bad, only a few crates actually enable sub-crates of their dependencies, and then it's usually `objc2-core-foundation`. Additionally, I've cleaned up a lot of our feature handling, which: - Fixes dependent features. - Removes unnecessary imports. - Fixes the last part of #640. There are still a few errors found by `check_framework_features` (esp. regarding the `objc2` feature), but those can be fixed later.
`sqlx-mysql` pulls in the vulnerable `rsa` crate. Janus only works with PostgreSQL, so we don't need that driver at all. `janus_aggregator_core` already enables the slim feature set we need, so we add `default-features = false` to the workspace level dependency to turn off the rest (`cargo` is unhappy if we put `default-features = false` in `aggregator_core/Cargo.toml`). Note that due to an outstanding `cargo` issue ([1], [2]), `sqlx-mysql` and `sqlx-sqlite` still appear in `Cargo.lock`, but are never used or even compiled. [1]: launchbadge/sqlx#2579 [2]: rust-lang/cargo#10801
`sqlx-mysql` pulls in the vulnerable `rsa` crate. Janus only works with PostgreSQL, so we don't need that driver at all. `janus_aggregator_core` already enables the slim feature set we need, so we add `default-features = false` to the workspace level dependency to turn off the rest (`cargo` is unhappy if we put `default-features = false` in `aggregator_core/Cargo.toml`). Note that due to an outstanding `cargo` issue ([1], [2]), `sqlx-mysql` and `sqlx-sqlite` still appear in `Cargo.lock`, but are never used or even compiled. [1]: launchbadge/sqlx#2579 [2]: rust-lang/cargo#10801
* disable unused sqlx drivers `sqlx-mysql` pulls in the vulnerable `rsa` crate. Janus only works with PostgreSQL, so we don't need that driver at all. `janus_aggregator_core` already enables the slim feature set we need, so we add `default-features = false` to the workspace level dependency to turn off the rest (`cargo` is unhappy if we put `default-features = false` in `aggregator_core/Cargo.toml`). Note that due to an outstanding `cargo` issue ([1], [2]), `sqlx-mysql` and `sqlx-sqlite` still appear in `Cargo.lock`, but are never used or even compiled. [1]: launchbadge/sqlx#2579 [2]: rust-lang/cargo#10801 * add `cargo deny advisories` exemption for protobuf We can't update the dependency until `opentelemetry-prometheus` either moves to a fixed `protobuf` or adds a feature letting us opt out of protobuf support. Either way, this advisory doesn't apply to Janus per the reasoning in deny.toml.
Summary: I found this useful for expressing `hashbrown`'s conditional dependency on `allocator-api2`. `actual = None` is a way to disappear a dependency at the incoming edge (like how `incoming_transition` applies), as opposed to the outgoing edge (like `transition_dep`). ```lang=toml # fixups/hashbrown/fixups.toml # Hashbrown does not really depend on allocator-api2 but `cargo metadata` is # claiming that it is, likely because of rust-lang/cargo#10801 omit_deps = ["allocator_api2"] extra_deps = ["//allocator:allocator-api2"] [rustc_flags_select] "//constraints:compiler" = [] "//constraints:library" = ["-Zforce-unstable-if-unmarked"] ``` ```lang=starlark # allocator/BUCK alias( name = "allocator-api2", actual = select({ "//constraints:library": None, # <--- "//constraints:compiler": "//:allocator-api2", }), visibility = ["//:"], ) ``` ```lang=starlark # BUCK, generated by Reindeer rust_bootstrap_library( name = "hashbrown-0.15.2", srcs = [":hashbrown-0.15.2.crate"], crate = "hashbrown", crate_root = "hashbrown-0.15.2.crate/src/lib.rs", edition = "2021", features = select({ "//constraints:compiler": ["default-hasher"], "//constraints:library": [ "alloc", "compiler_builtins", "core", "nightly", "raw-entry", "rustc-dep-of-std", "rustc-internal-api", ], }), named_deps = select({ "//constraints:compiler": {}, "//constraints:library": { "alloc": ":rustc-std-workspace-alloc-1.99.0", "core": ":rustc-std-workspace-core-1.99.0", }, }), rustc_flags = select({ "//constraints:compiler": [], "//constraints:library": ["-Zforce-unstable-if-unmarked"], }), visibility = [], deps = select({ "//constraints:compiler": [ ":foldhash-0.1.5", "//allocator:allocator-api2", ], "//constraints:library": [ ":compiler_builtins-0.1.146", "//allocator:allocator-api2", ], }), ) ``` The correct dependency would be something like `deps = ... + select({"//constraints:library": [], "//constraints:compiler": ["//:allocator-api2"]})` but this is not something that Reindeer's `extra_deps` can express. Maybe when we drop TOML and use Starlark fixups. Note that this diff does not use `attrs.option(..., default = None)` so the new meaning of `None` only kicks in if `actual = None` is written explicitly, as opposed to by default when `actual` is omitted from the `alias` call. Reviewed By: JakobDegen Differential Revision: D73325154 fbshipit-source-id: 9618446cc1762933fbb476ad20e3c9f5d1877d4c
Summary: I found this useful for expressing `hashbrown`'s conditional dependency on `allocator-api2`. `actual = None` is a way to disappear a dependency at the incoming edge (like how `incoming_transition` applies), as opposed to the outgoing edge (like `transition_dep`). ```lang=toml # fixups/hashbrown/fixups.toml # Hashbrown does not really depend on allocator-api2 but `cargo metadata` is # claiming that it is, likely because of rust-lang/cargo#10801 omit_deps = ["allocator_api2"] extra_deps = ["//allocator:allocator-api2"] [rustc_flags_select] "//constraints:compiler" = [] "//constraints:library" = ["-Zforce-unstable-if-unmarked"] ``` ```lang=starlark # allocator/BUCK alias( name = "allocator-api2", actual = select({ "//constraints:library": None, # <--- "//constraints:compiler": "//:allocator-api2", }), visibility = ["//:"], ) ``` ```lang=starlark # BUCK, generated by Reindeer rust_bootstrap_library( name = "hashbrown-0.15.2", srcs = [":hashbrown-0.15.2.crate"], crate = "hashbrown", crate_root = "hashbrown-0.15.2.crate/src/lib.rs", edition = "2021", features = select({ "//constraints:compiler": ["default-hasher"], "//constraints:library": [ "alloc", "compiler_builtins", "core", "nightly", "raw-entry", "rustc-dep-of-std", "rustc-internal-api", ], }), named_deps = select({ "//constraints:compiler": {}, "//constraints:library": { "alloc": ":rustc-std-workspace-alloc-1.99.0", "core": ":rustc-std-workspace-core-1.99.0", }, }), rustc_flags = select({ "//constraints:compiler": [], "//constraints:library": ["-Zforce-unstable-if-unmarked"], }), visibility = [], deps = select({ "//constraints:compiler": [ ":foldhash-0.1.5", "//allocator:allocator-api2", ], "//constraints:library": [ ":compiler_builtins-0.1.146", "//allocator:allocator-api2", ], }), ) ``` The correct dependency would be something like `deps = ... + select({"//constraints:library": [], "//constraints:compiler": ["//:allocator-api2"]})` but this is not something that Reindeer's `extra_deps` can express. Maybe when we drop TOML and use Starlark fixups. Note that this diff does not use `attrs.option(..., default = None)` so the new meaning of `None` only kicks in if `actual = None` is written explicitly, as opposed to by default when `actual` is omitted from the `alias` call. Reviewed By: JakobDegen Differential Revision: D73325154 fbshipit-source-id: 9618446cc1762933fbb476ad20e3c9f5d1877d4c
|
So there is no workaround or fix to this bug after 3 years of cargo development? 😞 |
* disable unused sqlx drivers `sqlx-mysql` pulls in the vulnerable `rsa` crate. Janus only works with PostgreSQL, so we don't need that driver at all. `janus_aggregator_core` already enables the slim feature set we need, so we add `default-features = false` to the workspace level dependency to turn off the rest (`cargo` is unhappy if we put `default-features = false` in `aggregator_core/Cargo.toml`). Note that due to an outstanding `cargo` issue ([1], [2]), `sqlx-mysql` and `sqlx-sqlite` still appear in `Cargo.lock`, but are never used or even compiled. [1]: launchbadge/sqlx#2579 [2]: rust-lang/cargo#10801 * add `cargo deny advisories` exemption for protobuf We can't update the dependency until `opentelemetry-prometheus` either moves to a fixed `protobuf` or adds a feature letting us opt out of protobuf support. Either way, this advisory doesn't apply to Janus per the reasoning in deny.toml.
* disable unused sqlx drivers `sqlx-mysql` pulls in the vulnerable `rsa` crate. Janus only works with PostgreSQL, so we don't need that driver at all. `janus_aggregator_core` already enables the slim feature set we need, so we add `default-features = false` to the workspace level dependency to turn off the rest (`cargo` is unhappy if we put `default-features = false` in `aggregator_core/Cargo.toml`). Note that due to an outstanding `cargo` issue ([1], [2]), `sqlx-mysql` and `sqlx-sqlite` still appear in `Cargo.lock`, but are never used or even compiled. [1]: launchbadge/sqlx#2579 [2]: rust-lang/cargo#10801 * add `cargo deny advisories` exemption for protobuf We can't update the dependency until `opentelemetry-prometheus` either moves to a fixed `protobuf` or adds a feature letting us opt out of protobuf support. Either way, this advisory doesn't apply to Janus per the reasoning in deny.toml. Co-authored-by: Tim Geoghegan <[email protected]>
…5de1de-5 : 006a26c0b546abc0fbef59a773639582b641e500-1 https://chromium.googlesource.com/external/github.com/rust-lang/rust/+log/a2b1646c5973..006a26c0b546 bootstrap/dist.rs changes: https://chromium.googlesource.com/external/github.com/rust-lang/rust/+log/a2b1646c5973..006a26c0b546/src/bootstrap/dist.rs Changes to the `cargo vendor` step need to be reflected in `CargoVendor() in //tools/rust/build_rust.py and in `gnrt gen --for-std` in //tools/crates/gnrt/gen.rs. Ran: tools/clang/scripts/upload_revision.py --skip-clang Contains changes to gnrt. Due to a Cargo bug in an unstable feature now used by hashbrown (which std uses), an extra dependency on allocator-api2 is added incorrectly. gnrt and the gnrt_config.toml are patched to remove this dependency. See rust-lang/cargo#10801 Bug: TODO. Remove the Tricium: line below when filling this in. Change-Id: I10d5c3d20c48ced4ab9a31731ec478b90efb2bee Tricium: skip Disable-Rts: True Cq-Include-Trybots: chromium/try:chromeos-amd64-generic-cfi-thin-lto-rel Cq-Include-Trybots: chromium/try:dawn-win10-x86-deps-rel Cq-Include-Trybots: chromium/try:linux-chromeos-dbg Cq-Include-Trybots: chromium/try:linux_chromium_cfi_rel_ng Cq-Include-Trybots: chromium/try:linux_chromium_chromeos_msan_rel_ng Cq-Include-Trybots: chromium/try:linux_chromium_msan_rel_ng Cq-Include-Trybots: chromium/try:mac11-arm64-rel,mac_chromium_asan_rel_ng Cq-Include-Trybots: chromium/try:ios-catalyst Cq-Include-Trybots: chromium/try:win-asan Cq-Include-Trybots: chromium/try:android-official,fuchsia-official Cq-Include-Trybots: chromium/try:mac-official,linux-official Cq-Include-Trybots: chromium/try:win-official,win32-official Cq-Include-Trybots: chromium/try:linux-swangle-try-x64,win-swangle-try-x86 Cq-Include-Trybots: chrome/try:iphone-device,ipad-device Cq-Include-Trybots: chrome/try:linux-chromeos-chrome Cq-Include-Trybots: chrome/try:win-chrome,win64-chrome,linux-chrome,mac-chrome Cq-Include-Trybots: chrome/try:linux-pgo,mac-pgo,win32-pgo,win64-pgo Cq-Include-Trybots: chromium/try:android-rust-arm32-rel Cq-Include-Trybots: chromium/try:android-rust-arm64-dbg Cq-Include-Trybots: chromium/try:android-rust-arm64-rel Cq-Include-Trybots: chromium/try:linux-rust-x64-dbg Cq-Include-Trybots: chromium/try:linux-rust-x64-rel Cq-Include-Trybots: chromium/try:mac-rust-x64-dbg Cq-Include-Trybots: chromium/try:win-rust-x64-dbg Cq-Include-Trybots: chromium/try:win-rust-x64-rel Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4633240 Reviewed-by: danakj <[email protected]> Commit-Queue: Collin Baker <[email protected]> Cr-Commit-Position: refs/heads/main@{#1162014} NOKEYCHECK=True GitOrigin-RevId: 6b334a89c5758e05e90e990cf2e02ad0c6243bb5
|
That bug would be a good candidate for Google Summer of Code, maybe next year let's think about putting it through the process. It's a big enough task that doing that unpaid is clearly not going to happen. |
and others
Uh oh!
There was an error while loading. Please reload this page.
Problem
Lets say I have two crates, with these contents of
Cargo.toml. The Rust code does not matter here.As you can see,
bardepends onfooand enables theserializationfeature, but it does not enable thetimenor thechronofeature.This results in the following
bar/Cargo.lockfile:bar/Cargo.lockThe relevant snippet is at the end:
Here the dependency
timeis declared, even though it is an optional dependency and nothing enabled the corresponding feature onfoo. Thetimedependency even hasserdelisted as one of its dependencies too.I think this is problematic in a couple of ways.
It bloats the
Cargo.lockand makes it imply that changes are larger than they actually are. In the above example addingfoowill also addtimeeven if that is not necessary.Dependency management tools will get confused by this. I mean tools like dependabot or
cargo audit, which use theCargo.lockas source of truth. Since these tools will see an entry fortime, even if it is unused, they will create pull requests or warn about security vulnerabilities which do not exists.This prevents using two crates with conflicting version requirements. You can see what happens if you uncomment the lines in
bar/Cargo.toml.Uncommenting
timewill fail, since now two conflicting version constraints (=0.3.10!==0.3.11) exist.However, uncommenting
chronoworks with the conflicting versions (=0.4.18!==0.4.19).Steps
foowith the above content forCargo.toml.barwith the above content forCargo.toml.Cargo.lockofbarcontains an entry fortimeand even a transitiveserdedependency intime.Possible Solution(s)
Disabled optional dependencies should not be tracked in
Cargo.lock, even if there is a weak dependency linking to it. This brings it in line with "normal" optional dependencies.Notes
While the
timeentry exists in theCargo.lock, it appears cargo correctly ignores it when building the code. There is neither aCompiling time ...log line nor are there any artifacts in thetargetfolder associated withtime.I checked various issues and documentation entries, but I couldn't find if this is a bug or intended.
The tracking issue (#8832) and documentation (https://doc.rust-lang.org/nightly/cargo/reference/features.html#dependency-features) do not mention any changes in to
Cargo.lock. Neither does the implementation PR (#8818), which does not even seem to have any lock files.Version
The text was updated successfully, but these errors were encountered: