Project Layout Advice

[npmccallum]

We currently have a situation we don't know how to solve using cargo and we could use some advice. I'm posting here because I suspect there isn't a good answer and we'd like to patch cargo to make things work once we have an agreed upon solution.

We have two binaries: foo and bar. They are built for specific hardware and are not portable. The foo binary is built for the platform default (i.e. x86_64-unknown-linux-gnu) but the bar binary is built for x86_64-unknown-linux-musl and is no_std. Then the bar binary is embedded into the foo binary via include_bytes!().

The interface between these two binaries is private. So we don't want to package them as separate published crates. They should always live in the same repo and be published together.

Currently, bar is a subdirectory in the foo crate and we build it by manually shelling out to cargo during build.rs. With a little hackery, this seems to work fine. Our major problem is that we can't seem to publish this configuration. If we attempt to include = ["bar"] in the foo crate, it is ignored because it contains a Cargo.toml.

So we're basically looking for a solution:

• foo depends on the bar binary, which is compiled for a different target.
• foo and bar have their own, separate dependency trees.
• foo and bar should always be distributed in the same package.
• How can we publish this to crates.io?

cc @haraldh

[npmccallum]

We seem to be hitting this check:
https://github.com/rust-lang/cargo/blob/master/src/cargo/sources/path.rs#L403-L406

Just to note what this implies: it is trivial to embed a library from another language, but the above check makes it impossible to embed a Rust library. It is not clear to me why this check exits.

It was first added in 8d5acdf by @alexcrichton. Is it still relevant for some reason? Can we remove it?

[npmccallum]

It looks to me like the original walk() function was internal to the fingerprint() function but over time morphed into something more general. It is not at all clear to me that this shortcut to exclude all subcrates is still relevant.

[npmccallum]

Looks like @seth wrote a test for similar behavior: d33da54

[joshtriplett]

@npmccallum We talked about this in today's @rust-lang/cargo meeting. There are two separate issues here:

• There should be a way to declare a build-dependency on a binary crate, and then have access to all of the resulting binaries. Such a build-dependency should include an optional list of targets to build the binary for (which could be any of "host", "target", or specific target names). The build should unify dependencies with the rest of the crate graph, like any other dependency (so that you can know the binary and your crate share the same version of some library). You should get environment variables similar to the DEP_ environment variables that point to each of the individual binaries built by the binary crate. We'd be very happy to have an implementation of this; last I saw, I think you're working on such an implementation.

• You would like to match the versions of foo and bar. We've talked in the past about potential solutions for that (e.g. private dependencies that aren't separately published), but we don't currently have a path to a solution. In the meantime, you could declare an exact-version dependency from foo to bar, requiring the exact semver you want.

[npmccallum]

@joshtriplett We'd also need:

1. dev-dependency for generating test dependencies
2. a way to not coalesce dependencies across targets
3. a way to get different .cargo/config files for the different targets

I'm wondering if a different approach might be more helpful.

What if we had separate categories:

• [build-artifacts]
• [dev-artifacts]

These would only build bin and maybe cdynlib. They would invoke a separate instance of cargo which would respect .cargo/config for the dependency. Each artifact build would be executed in a separate invocation of cargo with a dependency-specific output directory in target/.

@haraldh Has been doing some work on rebasing the old attempt at an "internal crates" patch. This probably needs an RFC to go with it. But we are very hesitant to publish our internal interfaces on crates.io.

[haraldh]

@haraldh Has been doing some work on rebasing the old attempt at an "internal crates" patch. This probably needs an RFC to go with it. But we are very hesitant to publish our internal interfaces on crates.io.

Which currently is: master...haraldh:cargo_internal

[epage]

We have unstable support for artifact dependencies, see https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#artifact-dependencies

For vendoring packages into your .crate, see #2203 and rust-lang/rfcs#3452

I'm going to close in favor of those issues. If there is a reason for us to re-evaluate, let us know!