init fetchPipMetadata / pip #504
Conversation
...and report.json for now
...to simplify for now, we'll re-introduce it later
...remove the venv and wait for the proxy via http, because the cert is only generated a bit after mitmproxy is started
...and the venv. Also make hash gathering more solid
with fetchPipMetadata
buildInputs. There's probably a cleaner way
for proper support of platform-specific and optional features. Turns out it, it isn't that hard after all with a bit of recursion.
...to prevent inifite recursion in nix later on
...but don't activate it by default
| }; | ||
| mkDerivation = { | ||
| src = l.mkDefault (l.fetchurl {inherit (metadata.${config.name}) url sha256;}); | ||
| doCheck = l.mkDefault false; |
There was a problem hiding this comment.
Tried to do so, but that causes lots of failures because most packages don't include their requirements to run the tests in the default set.
I'd like to leave it like that for now as I am going to add an interface to request "extras" soon. Together with a method to set default for all .drvs in a package, this would allow us to add {"test", "tests"} to extras by default if doCheckis true, which should solve that problem for most packages in a cleaner way.
...using a map() instead of list-comprehension resulted in an iterator which was consumed in the first branch of our dependency tree where it was encountered, resulting in an empty set of requirements in other branches.
phaer
force-pushed
the
fetch-pip-metadata
branch
from
c067046 to
7416e44
Compare
don't hard-core x86_64-linux
It's broken and we don't want to invest into fixing it yet.
...in order to ensure non-flake nix compat
phaer
force-pushed
the
fetch-pip-metadata
branch
3 times, most recently
from
ae4d21f to
3472d8b
Compare
nix command missing after refactoring
instead of makeWrapper
phaer
force-pushed
the
fetch-pip-metadata
branch
from
3472d8b to
2a27fce
Compare
|
Exciting work! Would this enable editable installs for multiple packages (as described here: NixOS/nixpkgs#11957 (comment))? |
|
@raj-magesh Thanks! While this can buiild many python packages, we haven't added devShells yet. There's a prototype which iiuc should be able to support editable installs for multiple packages, but we still need to finish a few things - such as switching dev dependencies on and off from the same lock file, you'd currently need to re-lock. I'd expect a pull request in the next month or so :) |
This PR forks
fetchPipandmach-nix-xsinto newfetchPipMetadataandpipmodules. This happens in order to allow us to experiment with two quite different approaches on howto handle python packaging.In short:
fetchPipMetadatais now a normal, non-sandboxed script which uses pips install report only to acquire dependency metadata to write into a lock file. This lock file is then used bypipto build a granular tree of python package derivations with normal nixpkgs fetchers and without a FOD or IFD, which positively affects performance.Writing all package metadata, including urls and hashes of packages, directly into the lock file also allows easier inspection of changes using diffs of said lock file.
Drv substitutions are out of scope for this PR, and nixpkgs-overrides can be enabled on a per-package basis. There's also preliminary support for pythons environment markers and editable installs (for the toplevel package).