-
Notifications
You must be signed in to change notification settings - Fork 72
Add: conda-forge publication tutorials #177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
0d65afb
3ca9010
948729c
9e7d1f6
5e8060c
48f45bc
08f35e4
e2508d7
5a3a2ac
a1f4812
b86354e
9aeefbd
150a5b5
7c742e8
368f9c0
a6274c5
b46abde
a5122e9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Anyone can publish to conda-forge" is probably not the best formulation. Only the CI of conda-forge can publish packages, but every package can be packaged in principle, provided it goes through review in staged-recipes. |
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,14 +9,22 @@ from a public community repository such as PyPI or a conda channel such as | |
Below you will learn more about the various publishing options for your Python | ||
package. | ||
|
||
```{admonition} Take Aways | ||
:::{admonition} Take Aways | ||
|
||
* Installing packages in the same environment using both pip and conda can | ||
lead to package conflicts. | ||
* To minimize conflicts for users who may be using conda (or pip) to manage local environments, consider publishing your package to both PyPI and the conda-forge channel of the Anaconda Cloud. | ||
|
||
Below you will learn more specifics about the differences between PyPI and conda publishing of your Python package. | ||
``` | ||
::: | ||
|
||
|
||
:::{figure-md} pypi-conda-channels | ||
|
||
<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px"> | ||
|
||
Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. Conda forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
::: | ||
|
||
## What is PyPI | ||
|
||
|
@@ -32,6 +40,14 @@ a package is that conda can install any package regardless | |
of the language(s) that it is written in. Whereas `pip` can | ||
only install Python packages. | ||
|
||
:::{button-link} ../tutorials/publish-pypi.html | ||
:color: primary | ||
:class: sd-rounded-pill float-left | ||
|
||
Click here for a tutorial on publishing your package to PyPI. | ||
::: | ||
|
||
|
||
```{tip} | ||
On the package build page, we discussed the [two package distribution | ||
types that you will create when making a Python package](python-package-distribution-files-sdist-wheel): SDist (packaged as a .tar.gz or .zip) and | ||
|
@@ -40,6 +56,7 @@ be published on PyPI when you use [a standard build tool](python-package-build-t | |
your package. | ||
``` | ||
|
||
(about-conda)= | ||
## What is Anaconda Cloud and conda? | ||
|
||
conda is an open source package and environment management tool. | ||
|
@@ -49,20 +66,26 @@ repository](https://repo.anaconda.com/). | |
Anaconda cloud (anaconda.org) contains public and private repositories for | ||
packages. These repositories are known as channels (discussed below). | ||
|
||
The most common public channels are: | ||
:::{admonition} A brief history of conda's evolution | ||
:class: note | ||
|
||
- defaults | ||
- conda-forge and | ||
- bioconda | ||
The conda ecosystem evolved years ago to provide support for and | ||
simplify the process of managing software dependencies in scientific | ||
Python projects. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```{tip} | ||
Many of the core scientific Python projects depend upon or wrap around tools and extensions that are written in other languages such as C++. In the early stages of the scientific ecosystem's development, these non python extensions and tools were not well supported on PyPI making publication difficult. In recent years there is more support for complex builds that allow developers to bundle non Python code into a python distribution using the [wheel distribution format](python-wheel). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't know how deeply you want to go into detail here; for historical context around conda, the best reference I'm aware of is https://technicaldiscovery.blogspot.com/2013/12/why-i-promote-conda.html While the situation has much improved outside of conda, many very tricky problems remain, and most of these are solved by conda.
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Conda provides a mechanism to manage these dependencies and ensure that the required packages are installed correctly. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
::: | ||
|
||
:::{tip} | ||
While conda was originally created to support Python packages, it | ||
is now used across all languages. This cross-language support | ||
makes it easier for some packages to include and have access to | ||
tools written in other languages such as c/c++ (gdal), Julia, or R. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Creating environment that mixes all those packages are usually easier and more | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
consistent with full fledged package managers like conda. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
``` | ||
::: | ||
|
||
### conda channels | ||
|
||
|
@@ -72,20 +95,21 @@ channels. The conda package manager can install packages from different channels | |
There are several core public channels that most people use to install | ||
packages using conda including. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- **defaults:** this is a channel managed by the Anaconda, inc. It is the version of the Python packages that you will install if you install the Anaconda Distribution. | ||
- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data | ||
- [**bioconda**](https://anaconda.org/bioconda): this channel focuses on biomedical | ||
- **defaults:** this is a channel managed by the Anaconda, inc. It is the version of the Python packages that you will install if you install the Anaconda Distribution. Anaconda decides what packages live on the default channel. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data. Anyone can publish a package to this channel. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same here. The focus on geospatial data seems out of place for me, conda-forge has a much, much wider focus. It's rather the opposite that certain ecosystems (biology, robotics, etc.) have their own channels, but also building on conda-forge. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ahhhh i hear you @jaimergp . that is my personal experience as a geospatial person. so maybe we can use geospatial as a breakout example. What other examples of non PyPI support dependencies would help users understand why we need conda? Especially with wheels now making things a bit easier, this is just my go to - if you use gdal conda-forge might make your life easier.
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- [**bioconda**](https://anaconda.org/bioconda): this channel focuses on biomedical tools. | ||
|
||
**conda-forge** emerged as many of the scientific packages did not | ||
exist in the default Anaconda cloud channel. | ||
|
||
:::{figure-md} conda-channels | ||
:::{figure-md} pypi-conda-channels | ||
|
||
<img src="../images/conda-channels-geohackweek.jpeg" alt="ADD." width="700px"> | ||
<img src="../images/python-pypi-conda-channels.png" alt="Graphic with the title Python package repositories. Below it says Anything hosted on PyPI can be installed using pip install. Packaging hosted on a conda channel can be installed using conda install. Below that there are two rows. the top row says conda channels. next to it are three boxes one with conda-forge, community maintained; bioconda and then default - managed by the anaconda team. Below that there is a row that says PyPI servers. PyPI - anyone can publish to pypi. and test pypi. a testbed server for you to practice. " width="700px"> | ||
|
||
ADD source: GeoHackWeek | ||
Conda channels represent various repositories that you can install packages from. Because conda-forge is community maintained, anyone can submit a recipe there. PiPY is also a community maintained repository. Anyone can submit a package to PyPI and test PyPI. Unlike conda-forge there are no manual checks of packages submitted to PyPI. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||
::: | ||
|
||
|
||
## conda channels, PyPI, conda, pip - Where to publish your package | ||
|
||
You might be wondering why there are different package repositories | ||
|
@@ -147,16 +171,16 @@ Once your package is on PyPI, the process to add your package to conda-forge | |
is straight forward to do. [You can follow the detailed steps provided | ||
by the conda-forge maintainer team.](https://conda-forge.org/docs/maintainer/adding_pkgs.html#generating-the-recipe). | ||
|
||
Generally those steps are: | ||
|
||
1. Fork the staged recipes conda-forge GitHub repository | ||
1. Create a new recipe using the `grayskull` tool. You can [learn more about grayskull here](https://conda.github.io/grayskull/). | ||
1. Add the recipe file created by `grayskull` in a folder within the recipes/ directory of the [**conda-forge/staged-recipes**](https://github.com/conda-forge/staged-recipes) repository. | ||
1. Open a pull request with your change. Your package will be tested on Windows, Mac and Linux using the repository CI. | ||
:::{button-link} ../tutorials/publish-conda-forge.html | ||
:color: primary | ||
:class: sd-rounded-pill float-left | ||
|
||
Click here for a tutorial on adding your package to conda-forge. | ||
::: | ||
|
||
If you want a step by step tutorial, click here. | ||
|
||
Once you have submitted a recipe to conda-forge, a maintainer will work | ||
with you to iron out any bugs in your recipe. They will then add your package | ||
to the conda-forge channel. | ||
|
||
Once your package is added, you will have a feedstock repository on GitHub with your packages name | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,19 +1,24 @@ | ||
# Learn about Building a Python Package | ||
|
||
:::{figure-md} build_workflow | ||
<img src="../images/python-package-development-process.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. those items all go into your package. Documentation and data are below that box because they aren't normally published in your packaging wheel distribution. an arrow to the right takes you to a build distribution files box. that box leads you to either publishing to testpypi or the real pypi. from pypi you can then connect to conda forge for an automated build that sends distributions from pypi to conda-forge. " width="700px"> | ||
|
||
:::{figure-md} pypi-conda-channels | ||
|
||
<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px"> | ||
|
||
Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. Conda forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the default, but not required (obviously this is the case for all non-Python packages in conda-forge). A git tag on a public repo is enough. PS. spelling consistency "Conda forge" -> "conda-forge" There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. fixing this throughout now! thank you! |
||
::: | ||
|
||
You need to build your Python package in order to publish it to PyPI (or Conda). The build process organizes your code and metadata into a distribution format that can be uploaded to PyPI and subsequently downloaded and installed by users. NOTE: you need to publish a sdist to PyPI in order for conda-forge to properly build your package automatically. | ||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's some duplication between the addition above and the existing "NOTE:" here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thank you! |
||
::: | ||
|
||
(build-package)= | ||
## What is building a Python package? | ||
|
||
To [publish your Python package](build_workflow) and make it easy for anyone to install, you first need to build it. | ||
To [publish your Python package](publish-python-package-pypi-conda) and make it easy for anyone to install, you first need to build it. | ||
|
||
But, what does it mean to build a Python package? | ||
|
||
[As shown in the figure above](build_workflow), when you build your Python package, you convert the source files into something called a distribution package. A distribution package contains your source code and metadata about the package, in the format required by the Python Package Index, so that it can be installed by tools like pip. | ||
[As shown in the figure above](#pypi-conda-channels), when you build your Python package, you convert the source files into something called a distribution package. A distribution package contains your source code and metadata about the package, in the format required by the Python Package Index, so that it can be installed by tools like pip. | ||
|
||
:::{note} | ||
The term package used to mean many different things in Python and other languages. On this page, we adapt the convention of the [Python Packaging Authority](https://www.pypa.io/en/latest/) and refer to the product of the | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kierisi this graphic now that i'm looking at it has really nice hierarchy and typography and flow. whereas the one in canva was totally different. did i look at the wrong graphic in canva? this graphic is just missing a bit of color pop but it's really clean and easy to digest.