Apparent race condition with build pipelining and/or target dir locking

[olson-dan]

Problem

I'll apologize in advance that I don't have a way to easily package a repro for this, but I hope that it's valuable to report this in any case.

We recently upgraded from 1.87.0 to 1.94.0. Our CI/CD started hitting rmeta errors. We thought this might be due to stale rlib/rmeta files from the older version sitting around on our workers. In order to try to deal with this, we hashed the rustc -vV output and appended this to the CARGO_BUILD_TARGET_DIR and CARGO_BUILD_BUILD_DIR folders. We observed that this did not resolve the issue, and moreover that it was occurring on workers that had previously performed successful builds on 1.94.0.

The errors are of this type:

   Compiling once_cell v1.18.0
error: only metadata stub found for `rlib` dependency `cfg_if`
       please provide path to the corresponding .rmeta file with full metadata
error: cannot resolve a prelude import

After some more research I learned about build pipelining, which apparently attempts to use the .rmeta files before the .rlib has been generated to unblock later compilation steps.

I'm seeing this in two cases.

The one I can sort of explain is our test job, where we run cargo test --package <p> on 2-3 crates in the same workspace simultaneously. This could imply a target dir locking issue. Maybe this isn't a recommended way to do this, but it was working without issue on 1.87.0.

The one I can't really explain doesn't feel like it should be an esoteric setup at all. We're invoking cargo from an MSBUILD custom step as part of a larger C++ project. Only one crate builds in this project, with the namespace target and build dirs. So I don't have a good idea of why that one might be occurring, but it's failing with the exact same type of error.

I'm turning pipelining off in my root config.toml and I'm expecting this to solve it for now. Everything else I've tried has been unsuccessful.

Steps

No response

Possible Solution(s)

I'm turning pipelining off in my root config.toml and I'm expecting this to solve it for now. Everything else I've tried has been unsuccessful.

Notes

No response

Version

cargo 1.94.0 (85eff7c80 2026-01-15)
release: 1.94.0
commit-hash: 85eff7c80277b57f78b11e28d14154ab12fcf643
commit-date: 2026-01-15
host: x86_64-pc-windows-msvc
libgit2: 1.9.2 (sys:0.20.3 vendored)
libcurl: 8.17.0-DEV (sys:0.4.84+curl-8.17.0 vendored ssl:Schannel)
os: Windows 10.0.26100 (Windows 11 Enterprise) [64-bit]

[olson-dan]

Ok so disabling pipelining obviously won't work as I just noticed:

[build.pipelining](https://doc.rust-lang.org/cargo/reference/config.html#buildpipelining)

This option is deprecated and unused. Cargo always has pipelining enabled.

I'm pretty close to the point where I might have to revert back to 1.87.0 or another earlier version.

[weihanglo]

It might be related to #16230.

Could you check whether 1.92 is good and 1.93 is bad?

[weihanglo]

Also

Only one crate builds in this project, with the namespace target and build dirs.

Did you set build-dir and target-dir in different locations?

[olson-dan]

Also

Only one crate builds in this project, with the namespace target and build dirs.

Did you set build-dir and target-dir in different locations?

Yes, correct they are different locations eg. <target>_<version_hash> and <target>_build_<version_hash>.

Could you check whether 1.92 is good and 1.93 is bad?

It's unfortunately very challenging for me to get good results here, but I'll work to install 1.92 today and see what happens.

[olson-dan]

I did some napkin math on our deployment just to get an idea of the scale of the issue. We have around 60k cargo invocations per 24h period across approx 500 workers. I saw about 100 "only metadata stub found for rlib dependency" failures yesterday so let's assume a failure rate of around 0.2%.

[olson-dan]

This appears to have gone away by downgrading to 1.92.0. I will test an upgrade to 1.93.0 to see if it occurs there, but I will not be able to do this until around a week from now.

[weihanglo]

It might be related to #16230.

Could you check whether 1.92 is good and 1.93 is bad?

Thanks! So it is likely it.

cc @ranger-ross

Notably environment setup:

• cargo test --package <p> on 2-3 crates in the same workspace simultaneously
• build-dir and target-dis are overridden with separate locations

[ranger-ross]

cargo test --package <p> on 2-3 crates in the same workspace simultaneously

If I recall correctly, we release the locks as soon as compilation is done but before we run the actual test.
However in the error posted above its during compilation, so I don't think this is the cause.

I looked over the changes in #16230 and I didn't spot anything obvious.
Even after that change Cargo should alway taken an exclusive lock on build-dir which is where rustc reads the rlib/rmeta files.

I guess lets wait and see what happens with the 1.93 test?

[ranger-ross]

I wonder if it could be related to #16177