Add: conda-forge publication tutorials #177

lwasser · 2024-01-29T19:27:30Z

This is a pr that overviews the conda publication process.

This is open to review starting now.... ✨

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1206499479429130
- https://app.asana.com/0/0/1206499479429135

lwasser · 2024-02-01T20:41:24Z

@jezdez @ocefpaf this is our conda-forge tutorial. can you both have a look and let me know what you think? it think it would be wonderful to have someone from the conda team review this to ensure i'm representing things accurately. This lesson should be beginner friendly and pure python focused. i also tried to emphasize that conda-forge is not a place to practice recipes - it's a final stop for those who have a maintained package. many thanks 👐

jaimergp

Thanks for putting this tutorial together! I'm sure it will help a lot of people! I added some comments to clarify some aspects, but it's mostly minor edits, nothing drastic!

package-structure-code/python-package-distribution-files-sdist-wheel.md

package-structure-code/publish-python-package-pypi-conda.md

jaimergp · 2024-02-05T16:30:10Z

tutorials/publish-conda-forge.md

+:::{todo}
+From leah - Can someone validate that this description is accurate or suggest alternative language please? i don't fully understand static library links
+:::


For practical purposes, a static library can be considered a "copy" of the library within another library. Many software licenses require redistribution of the license itself when the software is distributed with other pieces of software. Hence, when a library is built statically, it includes bits of the libraries it depends on, and their licenses must be added to the metadata.

For pure Python packages, this is never an issue, since there are no compiled libraries involved. If the Python package contains compiled extensions, then I'd say it's safe to assume the author knows about this distinction and will be able to ask in the review process.

The license aspect is one issue, though there are quite a few others that make static libraries unsuitable for conda-forge (except in some very rare circumstances). You might be interested in this summary about shared-vs-static libraries, which should be relatively accessible.

thank you both! this is helpful!

tutorials/publish-conda-forge.md

jaimergp · 2024-02-05T16:31:30Z

tutorials/publish-conda-forge.md

+Once you have completed all of the above, you are ready to open up a pull request in the `conda-forge/staged-recipes repository`.
+
+1. Submit a pull request from your fork/branch of the staged-recipes repository.
+1. Remember that the conda-forge maintainers are volunteers. Be patient for someone to respond and supportive in your communication with them.


jaimergp · 2024-02-05T16:32:41Z

tutorials/publish-conda-forge.md

+    - pip
+```
+
+If you have more advanced tests that you wish to run, you can add them here. However, you can also simple leave the tests section as it is.


It's not needed or even recommended, I'd even say. We don't want folks to run their full CI here because we presume they have done it before publication. These should just be minimal tests to ensure that things would work when installed.

It depends on the kind of package. A pure python package does not need as extensive testing as something that is - potentially - deeply affected by the changes necessary to get it into conda-forge.

Obviously people should not abuse conda-forge as their CI, but for the general case that releases are rare, I much prefer testing things. I regularly find stuff that doesn't work as intended in conda-forge packaging (though I'm probably biased towards looking at more complicated recipes) - often that's just because various test suites make assumptions that only hold locally, but it can also be related to using different BLAS implementations, etc.

ok thank you both. this makes sense to me. This is a tutorial for beginners and we are assuming it is pure python. in that case, should we recommend that they don't need anything extra in terms of testing than what we have in this tutorial?

but i can have a small breakout that includes caveats around how if your package is more complex / has compilation steps / non pure python, you may need additional tests in this step. just as a note?

ALternatively for this tutorial i can keep things super simple. You don't need to worry about tests. Then I can add this as a breakout to our broader conda-forge page that is not an opinionated tutorial but rather a resource so folks know how conda-forge works.

thoughts?

Fine for me to keep it simple in the tutorial. For a more general recommendation about testing "depth", perhaps Jaime and I should have a separate chat 😁

jaimergp · 2024-02-05T16:33:17Z

tutorials/publish-conda-forge.md

+```yaml
+test:
+  imports:
+    - pyospackage # Test importing the package into a python environment


This assumes that the __init__ module is populated. In some cases it's not. I'd add a note here that this list should try to target the "main" module(s) of the package; aka where the action happens.

ahhhh so if a Python package doesn't have an init file (in our tutorial we definitely have them add that file but it's empty). So Python should recognize the directory as one that is a package if they are following the tutorial.

but i think it's worth a breakout around this too - however i don't fully understand how it would work - yet. if a directory structure was missing an init file what would this section look like?

so say the directory looked like this:

src/ pyospackage/ module1.py module2.py

no init file is in that above example structure .
many thanks!

jaimergp · 2024-02-05T16:35:02Z

tutorials/publish-conda-forge.md

+
+<img src="../images/conda-forge-staged-recipes-ci.png" alt="Image showing the 5 CI tasks that will run against your package in the GitHub interface after you'ce created a pull request. " width="700px">
+
+Wait until you have green checks on all of the CI steps in your pull request. At that point your pull request is ready for review.


In some cases, when packaging noarch: python projects, OSX and Windows builds might fail due to missing dependencies. And that's ok (those dependencies might become available in the future, and then things will work). We only require Linux CI to pass for noarch: python.

h-vetinari

Cool that you're doing this! Didn't read everything in detail, but some comments from a quick read-through.

h-vetinari · 2024-02-05T16:31:19Z

images/python-pypi-conda-channels.png

"Anyone can publish to conda-forge" is probably not the best formulation. Only the CI of conda-forge can publish packages, but every package can be packaged in principle, provided it goes through review in staged-recipes.

h-vetinari · 2024-02-05T16:34:21Z

package-structure-code/publish-python-package-pypi-conda.md

- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data
- [**bioconda**](https://anaconda.org/bioconda): this channel focuses on biomedical
+- **defaults:** this is a channel managed by the Anaconda, inc. It is the version of the Python packages that you will install if you install the Anaconda Distribution. Anaconda decides what packages live on the default channel.
+- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data. Anyone can publish a package to this channel.


Same here. The focus on geospatial data seems out of place for me, conda-forge has a much, much wider focus. It's rather the opposite that certain ecosystems (biology, robotics, etc.) have their own channels, but also building on conda-forge.

Ahhhh i hear you @jaimergp . that is my personal experience as a geospatial person. so maybe we can use geospatial as a breakout example. What other examples of non PyPI support dependencies would help users understand why we need conda? Especially with wheels now making things a bit easier, this is just my go to - if you use gdal conda-forge might make your life easier.

h-vetinari · 2024-02-05T16:38:36Z

package-structure-code/publish-python-package-pypi-conda.md


-```{tip}
+Many of the core scientific Python projects depend upon or wrap around tools and extensions that are written in other languages such as C++. In the early stages of the scientific ecosystem's development, these non python extensions and tools were not well supported on PyPI making publication difficult. In recent years there is more support for complex builds that allow developers to bundle non Python code into a python distribution using the [wheel distribution format](python-wheel).


I don't know how deeply you want to go into detail here; for historical context around conda, the best reference I'm aware of is https://technicaldiscovery.blogspot.com/2013/12/why-i-promote-conda.html

While the situation has much improved outside of conda, many very tricky problems remain, and most of these are solved by conda.

h-vetinari · 2024-02-05T16:41:22Z

package-structure-code/python-package-distribution-files-sdist-wheel.md

+
+<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px">
+
+Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. Conda forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge.


This is the default, but not required (obviously this is the case for all non-Python packages in conda-forge). A git tag on a public repo is enough.

PS. spelling consistency "Conda forge" -> "conda-forge"

fixing this throughout now! thank you!

h-vetinari · 2024-02-05T16:41:45Z

package-structure-code/python-package-distribution-files-sdist-wheel.md

+<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px">
+
+Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. Conda forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge.
+:::

 You need to build your Python package in order to publish it to PyPI (or Conda). The build process organizes your code and metadata into a distribution format that can be uploaded to PyPI and subsequently downloaded and installed by users. NOTE: you need to publish a sdist to PyPI in order for conda-forge to properly build your package automatically.


There's some duplication between the addition above and the existing "NOTE:" here.

h-vetinari · 2024-02-05T16:43:34Z

tutorials/publish-conda-forge.md

+Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. Conda forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge.
+:::
+
+## What is conda forge?


I think it would be good to consistently spell it with a hyphen.

ok this is fixed. thanks @h-vetinari ! does conda-forge care about capitalization at the beginning of a sentence? i'm using lower case everywhere else.

I cannot speak for everybody, but I think either would be fine (capitalized at the beginning of a sentence or not). I've never seen it written in upper case, but at the same time, people understand capitalization rules.

tutorials/publish-conda-forge.md

h-vetinari · 2024-02-05T16:56:52Z

tutorials/publish-conda-forge.md

+:::{todo}
+From leah - Can someone validate that this description is accurate or suggest alternative language please? i don't fully understand static library links
+:::


The license aspect is one issue, though there are quite a few others that make static libraries unsuitable for conda-forge (except in some very rare circumstances). You might be interested in this summary about shared-vs-static libraries, which should be relatively accessible.

h-vetinari · 2024-02-05T16:56:52Z

tutorials/publish-conda-forge.md

+    - pip
+```
+
+If you have more advanced tests that you wish to run, you can add them here. However, you can also simple leave the tests section as it is.


It depends on the kind of package. A pure python package does not need as extensive testing as something that is - potentially - deeply affected by the changes necessary to get it into conda-forge.

Obviously people should not abuse conda-forge as their CI, but for the general case that releases are rare, I much prefer testing things. I regularly find stuff that doesn't work as intended in conda-forge packaging (though I'm probably biased towards looking at more complicated recipes) - often that's just because various test suites make assumptions that only hold locally, but it can also be related to using different BLAS implementations, etc.

kierisi

made grammatical and spelling suggestions. one thing that jumped out to me was inconsistency in capitalization of conda - I flagged it in the review, and whatever the decision is, it should be consistent throughout the doc!

package-structure-code/publish-python-package-pypi-conda.md

kierisi · 2024-02-07T16:22:20Z

package-structure-code/publish-python-package-pypi-conda.md

- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data
- [**bioconda**](https://anaconda.org/bioconda): this channel focuses on biomedical
+- **defaults:** this is a channel managed by the Anaconda, inc. It is the version of the Python packages that you will install if you install the Anaconda Distribution. Anaconda decides what packages live on the default channel.
+- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data. Anyone can publish a package to this channel.


package-structure-code/publish-python-package-pypi-conda.md

Co-authored-by: Jesse Mostipak <[email protected]>

lwasser · 2024-02-15T17:05:10Z

images/pyopensci-python-package-pypi-to-conda-forge.png

@kierisi this graphic now that i'm looking at it has really nice hierarchy and typography and flow. whereas the one in canva was totally different. did i look at the wrong graphic in canva? this graphic is just missing a bit of color pop but it's really clean and easy to digest.

lwasser · 2024-02-15T17:13:49Z

@all-contributors please add @jaimergp for code, review, tutorials

allcontributors · 2024-02-15T17:13:59Z

@lwasser

I've put up a pull request to add @jaimergp! 🎉

Co-authored-by: Jesse Mostipak <[email protected]>

Co-authored-by: jaimergp <[email protected]>

lwasser · 2024-02-15T17:29:59Z

@all-contributors please add @h-vetinari for code, review, tutorials

allcontributors · 2024-02-15T17:30:08Z

@lwasser

I've put up a pull request to add @h-vetinari! 🎉

Co-authored-by: jaimergp <[email protected]>

lwasser · 2024-02-27T17:02:22Z

ok everyone! thank you for all of the feedback. this PR has been open for i think over a month at this point. i'm going to merge it. if anyone catches issues, problems typos feel free to open an issue or PR to address! 👐

ocefpaf · 2024-02-27T17:26:09Z

After reading some AI generate "stuff?" on packaging it is really refreshing to read this instead. Great documentation, written by those who experienced the points written here, and very useful to newcomers and experienced folks alike.

Thank you @lwasser for leading this effort!

lwasser added 4 commits January 29, 2024 12:26

Fix: cross link across pages

0d65afb

Fix: finished draft 1 of this lesson - phew

3ca9010

Fix: move image up

948729c

Fix: Final tutorial edits

9e7d1f6

lwasser marked this pull request as ready for review February 1, 2024 19:46

fix: new images

5e8060c

lwasser changed the title ~~[DRAFT] Add: conda-forge publication tutorials~~ [Bug bash] Add: conda-forge publication tutorials Feb 1, 2024

lwasser changed the title ~~[Bug bash] Add: conda-forge publication tutorials~~ 🐛 bug bash 🐞 Add: conda-forge publication tutorials Feb 1, 2024

lwasser added the 🚀 ready-for-review label Feb 1, 2024

jaimergp reviewed Feb 5, 2024

View reviewed changes

h-vetinari reviewed Feb 5, 2024

View reviewed changes

lwasser added the bug-bash label Feb 6, 2024

kierisi reviewed Feb 7, 2024

View reviewed changes

lwasser and others added 2 commits February 15, 2024 09:55

Update package-structure-code/publish-python-package-pypi-conda.md

48f45bc

Co-authored-by: Jesse Mostipak <[email protected]>

Fix: changes from @kierski

08f35e4

Co-authored-by: Jesse Mostipak <[email protected]>

lwasser commented Feb 15, 2024

View reviewed changes

allcontributors bot mentioned this pull request Feb 15, 2024

docs: add jaimergp as a contributor for code, review, and tutorial #182

Merged

lwasser and others added 2 commits February 15, 2024 10:16

Fix: edits from @jaimergp

e2508d7

Co-authored-by: Jesse Mostipak <[email protected]>

Apply suggestions from code review

5a3a2ac

Co-authored-by: jaimergp <[email protected]>

allcontributors bot mentioned this pull request Feb 15, 2024

docs: add h-vetinari as a contributor for code, review, and tutorial #183

Merged

lwasser and others added 5 commits February 15, 2024 10:34

Merge branch 'main' into conda-force

a1f4812

Apply suggestions from code review

b86354e

Co-authored-by: jaimergp <[email protected]>

Fix: edits from review

9aeefbd

Fix: conda-forge fixes throughout

150a5b5

Fix: conda forge

7c742e8

lwasser added 4 commits February 21, 2024 13:02

Fix: edits from review

368f9c0

Fix: conda forge final edits

a6274c5

Fix: cleanup todos

b46abde

Merge branch 'main' into conda-force

a5122e9

lwasser changed the title ~~🐛 bug bash 🐞 Add: conda-forge publication tutorials~~ [march 1 merge] Add: conda-forge publication tutorials Feb 27, 2024

lwasser changed the title ~~[march 1 merge] Add: conda-forge publication tutorials~~ Add: conda-forge publication tutorials Feb 27, 2024

lwasser merged commit 7f08998 into pyOpenSci:main Feb 27, 2024

lwasser deleted the conda-force branch February 27, 2024 17:03

eriknw mentioned this pull request Feb 27, 2024

Fix typo of duplicate lines #185

Merged


		<img src="../images/conda-forge-staged-recipes-ci.png" alt="Image showing the 5 CI tasks that will run against your package in the GitHub interface after you'ce created a pull request. " width="700px">

		Wait until you have green checks on all of the CI steps in your pull request. At that point your pull request is ready for review.


		```{tip}
		Many of the core scientific Python projects depend upon or wrap around tools and extensions that are written in other languages such as C++. In the early stages of the scientific ecosystem's development, these non python extensions and tools were not well supported on PyPI making publication difficult. In recent years there is more support for complex builds that allow developers to bundle non Python code into a python distribution using the [wheel distribution format](python-wheel).


		<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px">

		Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. Conda forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge.

Add: conda-forge publication tutorials #177

Add: conda-forge publication tutorials #177

Conversation

lwasser commented Jan 29, 2024 • edited Loading

lwasser commented Feb 1, 2024

jaimergp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kierisi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lwasser commented Feb 15, 2024

allcontributors bot commented Feb 15, 2024

lwasser commented Feb 15, 2024

allcontributors bot commented Feb 15, 2024

lwasser commented Feb 27, 2024

ocefpaf commented Feb 27, 2024

lwasser commented Jan 29, 2024 •

edited

Loading