feat(export): pypi pip requirements export by synapticarbors · Pull Request #2049 · prefix-dev/pixi

synapticarbors · 2024-09-13T19:29:01Z

This is a follow-up to #1873 that adds a new pixi project export pypi-requirements to export a pip requirements.txt from a pixi lock file. For pixi environments that specify both conda and pypi dependencies, this would allow users to pixi project export conda-explicit-spec output --ignore-pypi-errors together with the new command to create a series of files that could then be installed into a conda env using a combination of conda/mamba and pip/uv.

Since the pip requirements file has many valid ways of representing a package as well as flags, I think we side step those complexities by specifying the full url path from the pixi.lock. I'd definitely like feedback from people who work with pip more since typically live fully in conda-land.

One known complexity is that a requirements.txt that contains any package with a hash specified is treated as if the --require-hashes flag has been used for the whole file. This means that you can't mix packages with hashes and without in the same pip install -r .... It does appear that uv will handle mixed files. To deal with this, I followed the currently recommended workaround and split requirements into two files. There is fix PR in pypa/pip#11968, at least for packages specified from VCS urls, but that's been sitting for a while. As seen for the output of running this on the pypi-source-deps example, this produces:

$ eza --tree test_output
test_output
├── default_linux-64_requirements.txt
├── default_linux-64_requirements_nohash.txt
├── default_osx-64_requirements.txt
├── default_osx-64_requirements_nohash.txt
├── default_osx-arm64_requirements.txt
├── default_osx-arm64_requirements_nohash.txt
├── default_win-64_requirements.txt
└── default_win-64_requirements_nohash.txt

and the outputs look like:

$cat default_linux-64_requirements.txt
https://files.pythonhosted.org/packages/bb/2a/10164ed1f31196a2f7f3799368a821765c62851ead0e630ab52b8e14b4d0/blinker-1.8.2-py3-none-any.whl --hash=sha256:1779309f71bf239144b9399d06ae925637cf6634cf6bd131104184531bf67c01
https://files.pythonhosted.org/packages/1c/d5/c84e1a17bf61d4df64ca866a1c9a913874b4e9bdc131ec689a0ad013fb36/certifi-2024.7.4-py3-none-any.whl --hash=sha256:c198e21b1289c2ab85ee4e67bb4b4ef3ead0892059901a8d5b622f24a1101e90
https://files.pythonhosted.org/packages/ee/fb/14d30eb4956408ee3ae09ad34299131fb383c47df355ddb428a7331cfa1e/charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl --hash=sha256:90d558489962fd4918143277a773316e56c72da56ec7aa3dc3dbbe20fdfed15b
https://files.pythonhosted.org/packages/22/7e/d71db821f177828df9dea8c42ac46473366f191be53080e552e628aad991/idna-3.8-py3-none-any.whl --hash=sha256:050b4e5baadcd44d760cedbd2b8e639f2ff89bbc7a5730fcc662954303377aac
https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl --hash=sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374
https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl --hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef
https://files.pythonhosted.org/packages/31/80/3a54838c3fb461f6fec263ebf3a3a41771bd05190238de3486aae8540c36/jinja2-3.1.4-py3-none-any.whl --hash=sha256:bc5dd2abb727a5319567b7a813e6a2e7318c39f4f487cfe6c89c6f9c7d25197d
https://files.pythonhosted.org/packages/42/d7/1ec15b46af6af88f19b8e5ffea08fa375d433c998b8a7639e76935c14f1f/markdown_it_py-3.0.0-py3-none-any.whl --hash=sha256:355216845c60bd96232cd8d8c40e8f9765cc86f46880e43a8fd22dc1a1a8cab1
https://files.pythonhosted.org/packages/0a/0d/2454f072fae3b5a137c119abf15465d1771319dfe9e4acbb31722a0fff91/MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl --hash=sha256:f5dfb42c4604dddc8e4305050aa6deb084540643ed5804d7455b5df8fe16f5e5
https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl --hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8
https://files.pythonhosted.org/packages/08/aa/cc0199a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl --hash=sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124
https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl --hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669
https://files.pythonhosted.org/packages/f7/3f/01c8b82017c199075f8f788d0d906b9ffbbc5a47dc9918a945e13d5a2bda/pygments-2.18.0-py3-none-any.whl --hash=sha256:b8e6aca0523f3ab76fee51799c488e38782ac06eafcf95e7ba832985c8e7b13a
https://files.pythonhosted.org/packages/c7/d9/c2a126eeae791e90ea099d05cb0515feea3688474b978343f3cdcfe04523/rich-13.8.0-py3-none-any.whl --hash=sha256:2e85306a063b9492dffc86278197a60cbece75bcb766022f3436f567cae11bdc
https://files.pythonhosted.org/packages/ca/1c/89ffc63a9605b583d5df2be791a27bc1a42b7c32bab68d3c8f2f73a98cd4/urllib3-2.2.2-py3-none-any.whl --hash=sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472
https://files.pythonhosted.org/packages/4b/84/997bbf7c2bf2dc3f09565c6d0b4959fefe5355c18c4096cfd26d83e0785b/werkzeug-3.0.4-py3-none-any.whl --hash=sha256:02c9eb92b7d6c06f31a782811505d2157837cea66aaede3e217c7c27c039476c
-e ./minimal-project

and:

$ cat default_linux-64_requirements_nohash.txt
https://github.com/pallets/click/releases/download/8.1.7/click-8.1.7-py3-none-any.whl
git+https://github.com/pallets/flask@f93dd6e826a9bf00bf9e08d9bb3a03abcb1e974c
git+https://github.com/pytest-dev/pytest.git@c947145fbb4aeec810a259b19f70fcb52fd53ad4
git+https://github.com/psf/requests.git@0106aced5faa299e6ede89d1230bd6784f2c3660

I'll add tests and docs once I get some feedback on the design.

…ments

ruben-arts

Thanks @synapticarbors This looks good and simple enough. For the tests could you add a test to uv/pip install the file into an environment @abkfenris added one in #2003

We need these tests to be fast.

ruben-arts · 2024-09-18T06:28:52Z

+    /// Conda dependencies are not supported in pip requirements files.
+    /// This flag allows creating the requirements file even if Conda dependencies are present.
+    #[arg(long, default_value = "false")]
+    pub ignore_conda_errors: bool,


You cannot have a pixi project without conda-dependencies if their are pypi dependencies. So this is a mandatory flag. This logic should be removed.

Sounds good. I'll try to make the changes you requested shortly and will hopefully have everything ready today

…ments

synapticarbors · 2024-09-18T16:42:37Z

@ruben-arts -- I just pushed changes that add the in-module insta tests, updated the docs and removed the flag having to do with conda packages. I also added a flag --split-reqs-no-hashes that controls whether to split the requirements if there are mix of requirements with and without hashes. I defaulted it to true since that's what needed for pip compatibility, but maybe it's better to opt-in. Also I'm not sure if I love the name.

As far as the tests that @abkfenris added in tests/test_export.sh, the one complication that I'm not sure about how best to deal with is that the export files for this (and the conda explicit spec) are platform dependent. Is it safe to assume that all tests will be run on linux-64? It's slightly problematic if you're running locally on osx-64 or osx-arm64. Or is there a robust way to get the platform from within a bash script?

ruben-arts

I like your style on the amount of tracing! This is often forgotten!

For the bash test, it should not be to hard to just do an if else on the matrix of the GitHub machines giving a variable that sets the platform name. I guess that would be the quickest way to test them platform specific in CI.

ruben-arts · 2024-09-19T06:20:05Z

+- `--environment <ENVIRONMENT> (-e)`: Environment to render. Can be repeated for multiple envs. Defaults to all environments.
+- `--platform <PLATFORM> (-p)`: The platform to render. Can be repeated for multiple platforms. Defaults to all platforms available for selected environments.
+- `--ignore-pypi-errors`: PyPI dependencies are not supported in the conda explicit spec file. This flag allows creating the spec file even if PyPI dependencies are present.
+- `--split-reqs-no-hashes`: Create a separate requirements.txt for dependencies that do not have an associated hash


I'm not sure I understand this flag properly. It's default true, but that means setting it will create a false? which would not split the reqs right? If this thought is correct I would rather name it --no-split or something similar.

Thinking about it couldn't the users also want to just ignore the hashes? e.g. --no-hash? This would then also not require the split.

ruben-arts · 2024-09-19T06:26:25Z

+    render_pypi_requirements(target, &base)?;
+
+    if !nohash.is_empty() {
+        tracing::info!(


Info is only shown on --verbose is that what you want? If it should show on normal level you can either print it with eprintln or with tracing::warn!

ruben-arts · 2024-09-19T06:27:31Z

+        match package {
+            Package::Pypi(p) => pypi_packages.push(PypiPackageReqData::from_pypi_package(&p)),
+            Package::Conda(cp) => {
+                tracing::warn!(


I don't think this warning is required as it is kind of intended. We should make it clear in the help message and documentation.

ruben-arts · 2024-09-19T06:29:24Z

+        // Split package list based on presence of hash since pip currently treats requirements files
+        // containing any hashes as if `--require-hashes` has been supplied. The only known workaround
+        // is to split the dependencies, which are typically from vcs sources into a separate
+        // requirements.txt and to install it separately.


I had to read this part to understand the flag. Could you incorporate this information in the documentation and help of the flag?

ruben-arts · 2024-09-19T06:30:01Z

+                false,
+            ),
+        };
+        //


Suggested change

//

matthewfeickert · 2024-10-24T07:23:04Z

(Just dropping in to say thank you in advance @synapticarbors for working on this, as this will be super helpful for me getting pixi used more in my field!)

synapticarbors · 2024-10-25T14:21:13Z

@matthewfeickert I hope to pick up this PR again at some point, but I'm still not 100% happy with the design. The requirements.txt is sort of complicated in terms of all of the different specifications it supports (including integrating cli flags), issues with how hashes interact with pip, etc. Since I don't tend to use pypi packages, I'm also not sure what most users might want, and how to support most or typical use cases.

benglewis · 2025-02-05T09:59:53Z

@synapticarbors I understand that you aren't 100% happy with this, but something is better than nothing, and the export of the full URL path would be perfect for the vulnerability scanning use-case, say using pip-audit, so I would really appreciate this being merged 🙏

baszalmstra · 2025-11-21T11:18:59Z

@synapticarbors Do you think you'll continue with this PR? If not, I will close it for now.

synapticarbors added 2 commits September 13, 2024 14:57

add pypi pip requirements export to cli

d314f85

Merge remote-tracking branch 'upstream/main' into export-pypi-require…

063a59a

…ments

ruben-arts reviewed Sep 18, 2024

View reviewed changes

synapticarbors added 2 commits September 18, 2024 10:59

Merge remote-tracking branch 'upstream/main' into export-pypi-require…

d215893

…ments

continued work on export pypi dependencies

4a340a8

ruben-arts reviewed Sep 19, 2024

View reviewed changes

synapticarbors mentioned this pull request Oct 1, 2024

Having a pip installable file(like requirements.txt) that can be used directly(without installing pixi or conda) inside docker environment. #2172

Closed

fecet mentioned this pull request Nov 4, 2024

Question: Support for PyPi Packages planned? Quantco/pixi-pack#38

Closed

lucascolley added pypi enhancement New features area:export labels Jul 21, 2025

lucascolley added the needs-design Needs a design so it can be implemented label Jul 29, 2025

lucascolley marked this pull request as draft July 29, 2025 10:50

lucascolley changed the title ~~feat: Add pypi pip requirements export~~ feat(export): pypi pip requirements export Sep 8, 2025

Uh oh!

Conversation

synapticarbors commented Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruben-arts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

synapticarbors commented Sep 18, 2024

Uh oh!

ruben-arts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthewfeickert commented Oct 24, 2024

Uh oh!

synapticarbors commented Oct 25, 2024

Uh oh!

benglewis commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

baszalmstra commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

synapticarbors commented Sep 13, 2024 •

edited

Loading

benglewis commented Feb 5, 2025 •

edited

Loading