Support `-c pylock.toml`

[paveldikov]

I'd be really keen on -c pylock.toml, and I think a lot of other users are too. Currently -c doesnt seem to engage the lockfile-aware parser.

-c pylock.toml would provide an excellent compatibility bridge for the bootstrap of future (more feature-complete) versions of pylock-aware tools, including future versions of pip itself.

Especially important considering that ensurepip vendors a (static; generally not up-to-date) version of pip, and getting it to safely upgrade itself to a known-good version is always a bit of a hassle.

Originally posted by @paveldikov in #13952

[pfmoore]

Please clarify what -c pylock.toml should do in practice. It's not at all clear to me how it should behave. I'd like to see some real-world use cases, as well (including the process of generating the lockfile, and other ways in which it would be used, as I'm not understanding why a pinned requirements file wouldn't be a better fit here - what does the lockfile format specifically add?).

[notatallshaw]

If this feature existed I would use it, though given the complexity and tradeoffs I think it would need to be carefully added to pip.

But here is my set up and use case:

I have 3 environment types, each is supposed to be a superset of the other: Production, Development, User Extended. The Production and Development environments are checked into the repo, the User Extended environment is not.

I generate the Development environment using all requirements (production, test, development, optional, etc.), and then I generate the Production environment using the production environment with the Development generated environment as constraints. Finally the User Extended environment is generated ad-hoc when a user or developer wants to add Python packages into their environment to either experiment adding something new or for temporary purposes, these is a predefined location for user requirements that is git ignored, and when they run the command to build their environment that gets included with the Development environment.

So I use the Development environment as constraints in both building the subset Production environment, and the superset User Extended environment. Currently the production environment gets generated as a pylock.toml but the Development environment doesn't, it gets generated as a fully pinning requirements.txt.

If I could use the a pylock.toml as a constraints file I would generate that for the Development environment and use it in generating the Production pylock.toml and the User Extended environment, this would guarantee me the exact distributions, not just that I had pinned to the same version.

[webknjaz]

Copying my use-case posted earlier @ #13876 (comment):

My pip-tools driven workflows usually include a pip install -r deps.in -c deps.txt. So I'd expect things like pip install some-dep -c pylock.transitive-deps.toml to be a replacement.

I feel like this is complimentary to what @notatallshaw describes. And now that #9243's been fixed (by #13887, finally!), reading constraints from a pylock.toml format shouldn't have technical blockers, AFAICS.

@pfmoore as an example, I currently have a bunch of txt "lock" files that are per-env++python+platform constraints that are injected into tox envs automatically and generated with pip-tools + GHA + own tooling: https://github.com/ansible/awx-plugins/tree/3731d98511f555b5a92fb0d042bdad7bb8cac154/dependencies / ansible/awx-plugins#170. Perhaps, eventually, it'll be possible to stuff multiple resolutions into the same lock file but for now, it'd be nice to have the same structure, but converted into TOML.

[pfmoore]

Thanks. But I'd still like to see the full workflow that you imagine here. This scenario isn't at all how I imagined lockfiles being used when we standardised them, and I'm struggling to understand the use case. Specifically, in pip install some-dep -c pylock.transitive-deps.toml, why isn't some-dep in the lockfile? As I said above, I'd like to understand how you expect to generate these lockfiles.

[notatallshaw]

why isn't some-dep in the lockfile?

In my examples:

• In generating the production environment some-dep is in the lock file, I want to generate an exact subset lock file of the larger dev lock file, down to the the distributions, which regular pinning constrains can not guarantee.

• In generating the user extended environment some-dep might not be in the lock file because I don't have control over how the user extends their environment, but I do want to ensure the shared dependencies use the exact same distributions.

[pfmoore]

OK. I guess I see the logic here. Although I'd be inclined to handle your production case by having a lockfile for production, and a second lockfile for the extra stuff that is added in dev. And for the user extended environment, I'm not entirely clear why you wouldn't install the lockfile rather than using it as a constraint (that's what "extended" implies to me, at least).

I don't have any real objections to the feature, but it does feel somewhat odd to me. FWIW, I note that uv doesn't allow lockfiles in their --constraint option.

[paveldikov]

Although I'd be inclined to handle your production case by having a lockfile for production, and a second lockfile for the extra stuff that is added in dev.

Interestingly, for me it's the opposite -- the appeal of having a single lockfile cover both dev and production, and potentially multiple platform/OS combinations, is huge. By being unified, this lockfile elegantly ensures that a project's dev environment and production deployments are as consistent as humanly possible¹ with each other.

Conversely, if you had to have multiple use-case-aligned lockfiles (pylock.*.toml) for a given project (pyproject.toml), you'd realistically then have to validate them to avoid drift/inconsistency -- and it's not trivial to do this validation well. In the case of multi-project ecosystems (the one I manage numbers in the 4 digit range), you'd also have to come up with robust conventions for the naming and usage of these lockfiles, which can be a headache considering the Cartesian-product problem created by optional-dependencies and dependency-groups.

Why pylock.toml and not the other (more traditional) formats? I believe that it's the right strategic construct for ensuring hermetic dev, build and production environments -- especially for fully pyproject.toml-driven projects. It is a more rigorous² format than the old-style stringly-typed .txt format, and conveys much more unambiguous intent³. And since it's one file, all of the aforementioned management problems are solved at the root. In fact, the pylock.toml schema natively allows for the multi-use-case scenario to be expressed even more explicitly through its dependency groups feature -- so one could indeed do a partial install of a lockfile. (Sadly, this capability is not yet widely supported in tooling)

Currently, in my org, we do use -c constraints.txt (where constraints.txt is compiled from pyproject.toml) as the next best thing, and it's alright... with the caveats above. But we always intended this usage as a mere stepping stone to first-class lockfiles.

So, what has support for -c pylock.toml got anything to do with this? It would be tactically incredibly useful, in a way that reinforces the strategic target state instead of deferring it. Since group-wise install of lockfiles is not yet widely supported -- and is a pretty non-trivial piece of work for any installer or lockfile generator to support, so it can't be rushed -- we need a simple and low-risk way to bridge the gap.

To this end, the ability to hot-swap the stringly-typed txt-file format with a structured, standardised pylock.toml file whilst preserving constraints-file semantics would immediately give us the benefits of fully-formed lockfiles without forcing us (or the community as a whole) to wait for all of the tools to support group-wise sync semantics in a uniform way. And once support does land across the board, flipping install -c pylock.toml into a sync pylock.toml --group ... operation is a simple and low-risk change -- not a flag-day build tooling overhaul.

(Additionally, at the point tooling starts supporting this more elaborate usage -- I could pip install pip -c pylock.toml as a much more hermetic version of the traditional pip install -U pip dance, which is far from reproducible. And then my shiny new pip sync operations would be hermetic all the way down.)

FWIW, I note that uv doesn't allow lockfiles in their --constraint option.

From what I gather, they are open to doing so, but are perhaps mildly reluctant to stray from pip's feature set as far as their uv pip commands are concerned. So it's a sort of chicken-and-egg?

Footnotes

1. With the notable, though unfortunate, exception of build-system dependencies. ↩

2. By explicitly expecting that the whole dependency graph, and not some arbitrary subset thereof is present. And mandatory checksums + URLs helps too. It's also nice that it solves all of my SBOM problems too. 🙂 ↩

3. Traditional requirements/constraints files, always suffered from ambiguous intent. Is it a: 1) human-editable high-level declarative spec of stuff that the developer actively cares about 2) autogenerated low-level fully-pinned graph expansion 3) anything in-between? The latter scenario being particularly fraught... This is a practical problem as inexpert users will see these files and naively edit them, and often brick their builds and require (costly!) expert intervention. ↩

[manfred-kaiser]

I work professionally on Reproducible Builds for Python; that work turned up an unrelated hash-verification bug in pypa/build along the way (fixed in #1140), and it's how I ran into the --build-constraint gap here specifically — a case nobody's touched in this thread yet. Here's what I found, with actual commands rather than just the shape of it.

This is the core mechanism: a requirements file expresses what you want — loosely, if you like (psutil>=6,<8, hatchling>=1,<2) — while a constraint narrows whatever actually gets resolved to one exact, hashed version, without ever triggering installation on its own. -r states intent, -c/--build-constraint pins the resolved reality; dependency resolution keeps working automatically in between. That resolved reality — name, version, URL, and hash per package in one structured file — also serves directly as an SBOM/provenance record, not just as a pin.

For runtime dependencies: pip lock "psutil==7.2.2" -o pylock.psutil.toml generates the constraint, with exactly one hash for psutil (its own C extension, multiple wheels per platform) — pip-compile --generate-hashes (from pip-tools) produces 21 hashes for the same package instead, because it resolves across platforms by default, which adds up once a project constrains many such packages, not just one. With psutil>=6,<8 in the requirements file, pip install -r requirements.txt -c pylock.psutil.toml then installs exactly 7.2.2 — the correct platform-specific wheel too, not just any hash.

For build dependencies — the --build-constraint case that hasn't come up in this thread — the same workflow applies to a backend with a real dependency tree like hatchling (packaging, pathspec, pluggy, trove-classifiers): pip lock "hatchling==1.31.0" -o pylock.build.toml generates the constraint, and with hatchling>=1,<2 in build-system.requires, pip wheel --build-constraint pylock.build.toml -w dist/ . builds successfully, resolving exactly to 1.31.0.

None of the three constraint paths (-c/--constraint, PIP_CONSTRAINT, --build-constraint) currently understand pylock.toml, though — all three fail with the same parse error, since they go through the same internal function that doesn't recognize pylock yet. Only -r/--requirement (as an install source, a different purpose) already supports it.

I put together an implementation that enables pylock.toml for all three paths at once: PR #14285 (currently a draft — pypa/pip limits non-collaborators to one open PR at a time, and it's stacked on #14284 below; will mark it ready once that one merges). Along the way I found a pre-existing resolver bug unrelated to pylock itself: when a package is only needed transitively and its constraint is given in URL form, its hash never reaches the resolved candidate (_iter_candidates_from_constraints() takes hash_options from the package that pulls it in rather than from the constraint), which breaks the install with a false "hash missing" error under --require-hashes mode — structurally always the case for pylock.toml, practically never for classic constraints, since those rarely use URL form. Filed and fixed separately, since #14285 depends on it: #14283 / #14284.

[notatallshaw]

I've been thinking a lot about this problem as I develop nab (a new and experimental dependency resolver and lock generator), and in general I've come to the conclusion the better solution is to generate lock files with selectable groups and extras rather than using a constraints file.

This has a number of advantages over constraints files:

1. Regardless what scenario you're selecting for (production, dev, build, etc.) each transitive resolve is individually and entirely derivable from the lock file.
2. A constraints file is inherently a subset resolve, nothing stops you from accidentally specifying additional requirements in your requirements file that now aren't part of the lock, and aren't reproducible.

However, while hopefully pip will soon be able to install from pylock.toml by explicitly selecting extras and groups, it is unlikely it will be able to generate these lock files any time soon, as to do the bare minimum of generating selectable groups and extras (ignoring multi-platform resolves and conflicts) you need to be able to do marker set algebra (or some very hacky regexes and string manipulation), and there's no independent library that supports this right now (I hope to offer an experimental one soon, independently installable from the rest of nab, and then eventually upstream it to packaging).

So in the mean time, if we expect users to be able to a pip lock ... and then distribute that file and do pip install ..., it might make sense to support -c pylock.toml, but I do think that eventually it will be bad practice.

[manfred-kaiser]

Thanks for the framing — the approach of generating lock files with
selectable groups/extras is a reasonable direction for the runtime case.

Build requirements:

For --build-constraint specifically, though, this unfortunately doesn't
work, because build requirements are categorically different from what
extras/groups cover: extras/dependency-groups in PEP 751 only cover
[project.optional-dependencies] and PEP 735 dependency groups, both
runtime concepts. Build requirements (PEP 518 [build-system].requires)
are independent of that: they're resolved in an isolated environment
that's discarded after the build, never part of the installed environment,
and aren't something you "select" like an extra at install time.

PEP 751 itself already considered and deliberately dropped a mechanism for
this ("Rejected Ideas" → "Locking build requirements for sdists"): an
earlier draft had a packages.build-requires key, removed again as too
confusing with too many edge cases, deferring to a future, separate PEP.
Build requirements still need a classic constraint mechanism either way —
"select a group" isn't an architectural fit for that, regardless of
tooling maturity. So I'd see --build-constraint pylock.toml as a
separate use case alongside the groups question for runtime dependencies,
rather than something selectable groups would replace.

Runtime dependencies:

I also tested the transitive case from your second point, with a loosely
referenced package and a constraints file that doesn't cover its
transitive dependencies. Result: pip has automatically switched into
hash-checking mode whenever any hash appears in the requirement set since
26.2 — --require-hashes doesn't need to be set explicitly for that —
and it also applies to purely transitively discovered packages. A missing
package therefore causes a hard error instead of a silent, unpinned
resolution. Since pylock.toml carries hashes in practice essentially
always, that applies automatically to the -c pylock.toml workflow too.

---

I do like the underlying idea of selectable groups and would welcome
seeing it implemented for the runtime case — build requirements will just
need to keep going through a constraint mechanism regardless.

[notatallshaw]

@manfred-kaiser Your posts are very verbose and difficult to read, if you're editing or generating them with LLMs please stop.

Build requirements

I cover build requirements in nab by being able to generate synthetic groups, as I linked in my prior post: notatallshaw/nab#790 (comment)

Runtime dependencies

Users can opt out of implicitly accepting require hashes with --no-require-hashes: #14186