Move env! to env::var_os for buck determinism#4578
Closed
Move env! to env::var_os for buck determinism#4578
Conversation
Author
|
Looks like I need to update a changelog https://github.com/PyO3/pyo3/actions/runs/11042056229/job/30673632736?pr=4578 |
|
Related: #4555 |
Member
|
Thanks very much for the PR! I think you're right that there's a problem here, but I worry that this isn't the complete solution. I'll write up my observations in an issue now... |
Member
|
See #4579 |
facebook-github-bot
pushed a commit
to facebookexperimental/rust-shed
that referenced
this pull request
Sep 27, 2024
Summary: PyO3/pyo3#4578 After investigation with JakobDegen, we found that `OUT_DIR` was being embedded in pyo3-build-config. P1610233931 This introduces non-determinism in our builds when I tried to land D63335477 Reviewed By: dtolnay Differential Revision: D63425591 fbshipit-source-id: ef1ab359e967a8723fddb938d1f5fd6226268f12
facebook-github-bot
pushed a commit
to facebook/sapling
that referenced
this pull request
Sep 27, 2024
Summary: PyO3/pyo3#4578 After investigation with JakobDegen, we found that `OUT_DIR` was being embedded in pyo3-build-config. P1610233931 This introduces non-determinism in our builds when I tried to land D63335477 Reviewed By: dtolnay Differential Revision: D63425591 fbshipit-source-id: ef1ab359e967a8723fddb938d1f5fd6226268f12
Member
|
Given that there's a more complex problem here, which I don't think this solves, I will close this PR. Let's please continue figuring out what to do on #4579. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When building pyo3 with buck2 and remote execution, we're especially sensitive to deterministic builds and being able to compile different rust libraries on different hosts in parallel.
The issue with
resolve_cross_compile_config_pathwas that we were embedding the full path ofOUT_DIRinto the compiled rlib and we are unable to guarantee the path is identical across all invocations ofrustc. The other instances ofenv!("OUT_DIR")are no issue as they are used as part of the larger expressioninclude_str!(concat!(env!("OUT_DIR"), "/pyo3-build-config-file.txt"));and we are able to guarantee that the contents of those files are deterministic.By switching to
env::var_os("OUT_DIR"), we are able to avoid embedding absolute paths into the compiled rust library and maintain backwards compatibility with the rest of the cargo ecosystem.