Skip to content

Respect scoped dependencies when validating metadata-free locks - #20803

Closed
charliermarsh wants to merge 21 commits into
mainfrom
charlie/metadata-free-dynamic-scoped-dependencies
Closed

charliermarsh wants to merge 21 commits into
mainfrom
charlie/metadata-free-dynamic-scoped-dependencies

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Jul 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Metadata-free lock validation must reconstruct dependencies using the same package-scoped override and exclusion semantics as the resolver.

Dynamic source trees intentionally omit their version from uv.lock, so use the freshly built distribution version for scoped rules and bypass static metadata only when relevant scoped rules are present. This accepts unchanged locks and invalidates stale locks when a dynamic version changes which rule applies.

Apply package-scoped overrides and exclusions before flattening recursive extras, matching resolver order. Otherwise an overridden or excluded self-reference can reactivate an extra, invent a dependency edge, and cause an unchanged metadata-free lock to fail offline.

Add deterministic offline integration coverage for version-specific overrides, versionless fallback precedence, scoped exclusions, dynamic-version changes, and overridden or excluded recursive extras.

@charliermarsh charliermarsh changed the title Respect scoped dependencies on dynamic-version packages Respect scoped dependencies when validating metadata-free locks Jul 29, 2026
@astral-sh-bot

astral-sh-bot Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

uv test inventory changes

This PR changes the tests when compared with the main base revision.

  • Added tests: 48
  • Removed tests: 0
  • Changed suites: 3
uv-distribution: +1 / -0

Added:

  • uv-distribution::metadata::requires_dist::test::test_flat_requires_dist_with_production_and_extra_marker

Removed: none

uv-resolver: +1 / -0

Added:

  • uv-resolver::universal_marker::tests::relevant_conflicts

Removed: none

uv::lock: +46 / -0

Added:

  • uv::lock::lock::lock_metadata_free_conflicting_direct_dependencies
  • uv::lock::lock::lock_metadata_free_conflicting_forked_source_dependencies
  • uv::lock::lock::lock_metadata_free_conflicting_forked_source_extras
  • uv::lock::lock::lock_metadata_free_conflicting_transitive_extras
  • uv::lock::lock::lock_metadata_free_conflicting_transitive_groups
  • uv::lock::lock::lock_metadata_free_forked_source_declared_empty_extra
  • uv::lock::lock::lock_metadata_free_forked_source_with_parent_extra_conflict
  • uv::lock::lock::lock_metadata_free_frozen_empty_workspace_selections
  • uv::lock::lock::lock_metadata_free_group_project_conflicting_extras
  • uv::lock::lock::lock_metadata_free_group_scoped_source_registry_fallback
  • uv::lock::lock::lock_metadata_free_group_source_fork_project_conflict_production
  • uv::lock::lock::lock_metadata_free_ignores_non_workspace_self_referential_groups
  • uv::lock::lock::lock_metadata_free_nested_conflicting_forked_sources
  • uv::lock::lock::lock_metadata_free_recursive_extra_preserves_override_markers
  • uv::lock::lock::lock_metadata_free_root_extra_and_group_project_conflict
  • uv::lock::lock::lock_metadata_free_root_extra_project_conflicting_workspace_extra
  • uv::lock::lock::lock_metadata_free_root_extra_scoped_sources
  • uv::lock::lock::lock_metadata_free_workspace_extra_activates_base
  • uv::lock::lock::lock_metadata_free_workspace_extra_conflicting_target_extras
  • uv::lock::lock::lock_metadata_free_workspace_extra_project_conflicting_target
  • uv::lock::lock::lock_regenerates_conflicting_dependency_extra_and_base_package
  • uv::lock::lock::lock_regenerates_dependency_extra_from_multiple_conflicting_python_forks
  • uv::lock::lock::lock_regenerates_dependency_extra_with_project_conflict
  • uv::lock::lock::lock_regenerates_disjoint_python_forked_dependency_extras_with_selected_conflicts
  • uv::lock::lock::lock_regenerates_disjoint_python_forked_dependency_groups_with_selected_conflicts
  • uv::lock::lock::lock_regenerates_dynamic_version_scoped_exclusion
  • uv::lock::lock::lock_regenerates_dynamic_version_scoped_override
  • uv::lock::lock::lock_regenerates_independently_conflicting_dependency_extras
  • uv::lock::lock::lock_regenerates_independently_selected_workspace_member_extras
  • uv::lock::lock::lock_regenerates_marker_scoped_dependency_extra_with_selected_conflict
  • uv::lock::lock::lock_regenerates_marker_scoped_dependency_extra_with_unrelated_conflicts
  • uv::lock::lock::lock_regenerates_marker_scoped_dependency_extras
  • uv::lock::lock::lock_regenerates_overlapping_conflicting_dependency_extras
  • uv::lock::lock::lock_regenerates_project_conflicting_dependency_extra_and_base_package
  • uv::lock::lock::lock_regenerates_project_conflicting_dependency_extra_with_unrequested_alternative
  • uv::lock::lock::lock_regenerates_project_conflicting_incompatible_platform_dependency_extras
  • uv::lock::lock::lock_regenerates_project_conflicting_marker_inactive_dependency_extras
  • uv::lock::lock::lock_regenerates_python_forked_dependency_extra_with_selected_conflict
  • uv::lock::lock::lock_regenerates_python_forked_dependency_extras_in_supported_environments
  • uv::lock::lock::lock_regenerates_python_forked_production_dependency_with_selected_extra
  • uv::lock::lock::lock_regenerates_python_forked_production_dependency_with_selected_group
  • uv::lock::lock::lock_regenerates_recursive_extra_production_marker
  • uv::lock::lock::lock_regenerates_scoped_excluded_recursive_extra
  • uv::lock::lock::lock_regenerates_scoped_overridden_recursive_extra
  • uv::lock::lock::lock_regenerates_source_agnostic_transitive_dependency
  • uv::lock::lock::lock_regenerates_source_constrained_transitive_dependency

Removed: none

@codspeed

codspeed Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 23 untouched benchmarks
⏩ 6 skipped benchmarks1


Comparing charlie/metadata-free-dynamic-scoped-dependencies (4dfbd70) with main (586d850)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

charliermarsh added a commit that referenced this pull request Sep 15, 2026
The previous iterations of `package.metadata`-free lockfiles had missed
a number of cases, and made it hard to ensure that all cases were
actually covered. This PR takes a new approach: Remove the previous
implementation, and then re-add it feature-by-feature in reviewable
commit units. Each commit adds one feature that uv supports.

The new approach is not entirely `package.metadata`-free, instead remote
URL, which can activate other URL source, now have `package.metadata`
which they previously didn't. That's required for offline validation
(which is crucial for `uv lock --check` performance, offline or not) and
IMO the more correct tradeoff: We shouldn't repeat information that's
already available on disk, but we should state what we expect from
remote resources.

`uv.lock` revision 1.4 additionally records empty extras so `--frozen
--extra foo` if the lock wasn't generated with extra `foo` present, but
it was added to `pyproject.toml` afterwards.

This is a lockfile size regression if you're using a lot of remote URLs,
otherwise it's still a much smaller lockfile for large workspaces.
Remote source are usually used sparingly as they are worse for
performance, reliability and auditability than registry sources.

## Validation

This change reflects basically every resolver assumption and form of
dependency specification and also hits a number of preexisting bugs (see
upstack PRs and skipped changes list). Instead of trying to reason
through correctness from the features we're representing, this PR was
tested (and fixed) against large, diverse validation set, checking that
uv's invariants are upheld with lockfile revision 1.4. The PR is
additionally split into one commit per feature the new freshness check
support, to ease reviewing and ensure that all features are covered. The
most import check that needs to always pass is `uv lock --check
--offline --no-cache` on a fresh lockfile. We don't need to check
features such as `uv sync`, as `package.metadata` is only used for fast
path freshness checks, and `uv lock --check --offline --no-cache`
exercises the offline fast path while blocking the slow path if it would
use the internet (which is unacceptable both for performance and for
checks that run in isolated environments). The goal is to cover the
available dependency specification that people use, while not fixing
unrelated bugs in this already large change. There are three of these
larger validations.

The first one is a script finds public GitHub projects with editable
packages, workspaces, Git dependencies or dynamically generated
dependencies and then verifies that both the original lockfile and a
version with package metadata removed pass `uv lock --check --offline
--no-cache` inside an isolated docker container. This dataset contained
254 projects with 14k locked packages.

The second validation uses the check-mode in
https://github.com/astral-sh/uv-ecosystem-testing with
`UV_PREVIEW_FEATURES=lock-without-metadata` .

The third one is running the test suite migrated to lockfile revision
1.4 with
4b98802,
passing 4.8k tests:

```
features=test-python-patch,test-universal,native-auth
cargo insta test --accept --test-runner nextest --workspace --features "$features"
cargo nextest run --workspace --features "$features"
```

The PR handles the test cases and fixes from
#20803 too.

The area with the least coverage wrt to its complexity is conflicts
supports. While the support is implemented, I'd consider its current
support much weaker than the rest of the features.

## Skipped changes

The following independent bugfixes are deliberately excluded from this
stack:

- #21210.
- #21225. The cache assertion remains a TODO here.

The 4.8k-test run above included the add-cache fix and predates this
extraction; it is not a fresh full-suite run of the reduced stack.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>

This branch was previously deployed

1 inactive deployment
automations — 4dfbd703 Deployed Jul 30, 2026 by charliermarsh via review / security review #43846
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant