ENH: Compatibility with upcoming lief 0.17

[h-vetinari]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

While looking at the release notes for the just-released 0.16.5 (which conda-build 25.3 finally became compatible with after a chain of fixes culminating in #5627), I noticed the following: https://lief.re/doc/latest/changelog/pe-0-17-0.html

In short, there's been a big refactor in how windows DLLs are being treated, including again some renamed structures (see "Resources" section), though from a quick code search here, conda-build doesn't seem to be using any of them.

It's probably mostly about testing conda-build together with lief 0.17 as comprehensively as possible.

We already have a defensive upper bound on the conda-forge feedstock that will ensure we can publish lief 0.17 without breaking the world (this was done without knowing about the PE changes, just caution based on recent breakages - good call from @beckermr! 😉)

The recipe in this repo is still lacking such a bound though:

conda-build/recipe/meta.yaml

Line 51 in f3315ff

- py-lief

Sidenote: py-lief should probably be listed under the dependencies in pyproject.toml? 🤔

conda-build/pyproject.toml

Line 24 in f3315ff

dependencies = [

What we can do once 0.17 is out is to publish an uncapped conda-build to a separate label, and test it on as many feedstocks as we feel necessary.

CC @tkelman

Why is this needed?

Staying up to date on lief is essential for getting support for new python versions, performance improvements, fixes etc.

What should happen?

• Add a <0.17.0a0 bound to recipe/meta.yaml
• Add py-lief <0.17 to pyprojects.toml
• Once lief 0.17 is out
  • do any adaptations that are necessary for conda-build; test them in CI here
  • once there's a compatible conda-build release, publish it to a label and test it in conda-forge

Additional Context

No response

[tkelman]

Has anyone gone to the lengths of building and testing against a pre-release dev version of LIEF? That would be one way of getting a head-start in seeing for sure whether the upper bound will be necessary and what kind of adaptations are needed to handle the new version, without having to wait until it's done and tagged upstream (then packaged in conda-forge or defaults).

A difficulty I had in working through the prior related PR's to #5627 was in test coverage for the parts of conda-build that use LIEF. Part of that may have been the testing pulling from defaults, but it also seemed like we weren't seeing some code paths get called until after the latest versions of everything were fully rolled out and the broader ecosystem was disrupted by my mistakes or incomplete fixes here.

[beckermr]

Yes test coverage is definitely an issue in conda-build for sure.

[h-vetinari]

Has anyone gone to the lengths of building and testing against a pre-release dev version of LIEF?

We'd have to create a dev branches on the feedstocks for lief and conda-build, then we could try publishing builds from lief's main to a special label, e.g. conda-forge/label/conda_build_dev and start testing that together with builds of conda-build (also pushed under that label) where the lief cap is removed. I've done that sort of testing from main in the LLVM stack (e.g. recently in the context of conda-forge/flang-feedstock#80), but it needs the buy-in of the respective feedstock maintainers. Luckily, the extra branch and some builds under a non-default label usually don't bother anyone, but it still takes extra work/attention.

[h-vetinari]

but it also seemed like we weren't seeing some code paths get called until after the latest versions of everything were fully rolled out and the broader ecosystem was disrupted by my mistakes or incomplete fixes here.

Sidenote: Please don't blame yourself here, these things are hard to shake out, and bugs were missed despite everyone's best effort (and several people looking at it). On the contrary, your fixes were very much appreciated! I'm just glad to finally be rid of an extra 20-30min on several feedstocks that were due to conda-forge/lief-feedstock#45 🥳

[h-vetinari]

Looks like conda-forge/lief-feedstock#63 was merged already. Now we mainly need testing that this doesn't break the world. I think the best way would be to publish a build of conda-build without the lief <0.17 constraint to a special label, which can then be easily pulled into various feedstock

[tkelman]

conda-forge/conda-build-feedstock#263 is a really good start to this. Chime in if anyone disagrees but I think these are the next areas of work, now that there's a lief_dev label for the purposes of testing and experimenting here:

1. Immediate specific testing of lief v0.17.x to see what needs adjusting so we can raise the defensive upper bound on the main conda-forge channel to py-lief <0.18.0a0 # [cap_lief] and get 0.17.x rolled out broadly. I think the state of the lief_dev channel label right now is usable to get this testing going. Do folks envision this testing happening on individual maintainers' local environments, or can we start opting in certain downstream feedstocks helping out with this testing in their CI / an automated bot process somehow? It's not quite the same workflow as a migration but it looks a little similar. Should we collect a written list of specific feedstocks that are known to exercise the parts of conda-build that use py-lief in ways that have broken before? We have past lief-related breakage issue reports we can look through, and core maintainers who know offhand which feedstocks are complicated and therefore good test cases.

2. Longer term, should we also think about publishing pre-release dev preview versions for upcoming versions of lief and/or conda-build to the same lief_dev channel label? Or separate new channels labels of their own? This could help lower the turnaround time and getting the relevant fixes upstream earlier, without having to wait for final release tags of each codebase.

[h-vetinari]

conda-build 26.1 removed the lief caps, so this issue can be closed. This might also make it possible to solve #4881 by switching to the new lief-based patchelf implementation