Repository navigation
feat: remove input-hash for PyPI dependencies - #5174
nichtmacho wants to merge 26 commits into
Conversation
baszalmstra
left a comment
There was a problem hiding this comment.
Nice! This is a good start but its not enough, we need to also compare the requirements, required python version optional dependencies, etc. Furthermore, some fields can be dynamic which means we need to invoke python, and we might also have to deal with setup.py only sources.
Indeed! It might be good to let Claude explore the uv codebase as well for inspiration, although if I recall their satisfiability check is less extensive, they regenerate more often. |
76d5042 to
2b7191e
Compare
2b7191e to
08b5efe
Compare
|
Attaching benchmarks here: Dynamic Metadata (minimal-project with hatch_build.py, setup-project with setup.py)Main branch:
This branch:
Static Metadata (minimal-project without hatch_build.py)Main branch:
This branch:
|
| pub struct EnvironmentBuildCache { | ||
| /// Lazily initialized build dispatch dependencies (interpreter, env, etc.) | ||
| pub lazy_build_dispatch_deps: LazyBuildDispatchDependencies, | ||
| /// Optional conda prefix updater (created during satisfiability checking) | ||
| pub conda_prefix_updater: OnceCell<CondaPrefixUpdater>, | ||
| /// Cached registry client (with Online connectivity) | ||
| pub registry_client: OnceCell<Arc<RegistryClient>>, | ||
| /// Cached index locations | ||
| pub index_locations: OnceCell<IndexLocations>, | ||
| /// Cached build options | ||
| pub build_options: OnceCell<BuildOptions>, | ||
| /// Cached flat index (populated lazily when needed, async initialization) | ||
| pub flat_index: AsyncOnceCell<FlatIndex>, | ||
| /// Cached dependency metadata | ||
| pub dependency_metadata: OnceCell<DependencyMetadata>, | ||
| } |
There was a problem hiding this comment.
I'm not sure we need to cache most of these structs, as the most interesting things are chached to disk. I think there are non in-memory caches, only the InFlight which is not here and maybe in the RegistryClient. I'd rather not introduce another struct containing too many OnceCell's. Maybe we can figure out what we need to keep? I think the LazyBuildDispatch is something we should cache?
There was a problem hiding this comment.
removed all other fields except conda_prefix_updater. I think it's ok to keep it, because it contains an in-memory result ( OnceCell field) of installing a prefix environment.
| "**/*.ipynb", | ||
| "**/docs_hooks.py", | ||
| "scripts/test_native_certs.py", | ||
| "examples/dynamic-source-deps/minimal-project/hatch_build.py", |
There was a problem hiding this comment.
this will be removed when we remove dynamic-source-deps
| @@ -0,0 +1,19 @@ | |||
| [workspace] | |||
There was a problem hiding this comment.
It's added just to show how to test it
|
Test plan:
If all of these cases succeed next to the integration and unit tests I would say it is ready to merge, I did a similar thing for the conda input-hash removal. |
|
added some stuff to cache reading static metadata, these are the new results ( with cache clean on every run ) Dynamic Metadata (minimal-project with hatch_build.py, setup-project with setup.py)Main branch:
This branch:
Static Metadata (minimal-project without hatch_build.py)Main branch:
This branch:
|
|
Might be good to user-test if |
Tested! |
…i-input-hash-9zwnN' into claude/remove-pypi-input-hash-9zwnN
|
I've modified that now we lockfiles that have any I think we should align this with the lockfile bump - otherwise |
|
Is not possible to silently ignore the hashes instead? |
|
I didnt test the code myself yet but before merging, what is now recorded in the lockfile? And how do we deal with dynamic fields? |
|
after discussion with @baszalmstra we will just ignore sha256 field until #5248 is merged |
I think we always re-generate in that case right @nichmor? |
for dynamic-fields ( if they are declared in the |
|
I think we can close this now: It will go in via the feature/lockfile-v7 branch. |
Replace hash-based validation for path-based PyPI packages with metadata-based validation. Instead of computing and storing a hash of pyproject.toml, setup.py, and setup.cfg, we now:
This approach is similar to how PR #5011 removed input_hash for conda packages. The metadata comparison is more explicit about what changed (name vs version) rather than just indicating "something changed via hash mismatch".
Changes:
Description
Fixes #{issue}
How Has This Been Tested?
AI Disclosure
Tools: Claude
Checklist:
schema/model.py.