Skip to content

Add: conda-forge publication tutorials #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added images/conda-forge-staged-recipes-ci.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/publish-python-package-pypi-conda.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/pypi-project-landing-page-no-meta.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/python-pypi-conda-channels.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
86 changes: 58 additions & 28 deletions package-structure-code/publish-python-package-pypi-conda.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,22 @@ from a public community repository such as PyPI or a conda channel such as
Below you will learn more about the various publishing options for your Python
package.

```{admonition} Take Aways
:::{admonition} Take Aways

* Installing packages in the same environment using both pip and conda can
lead to package conflicts.
* To minimize conflicts for users who may be using conda (or pip) to manage local environments, consider publishing your package to both PyPI and the conda-forge channel of the Anaconda Cloud.

Below you will learn more specifics about the differences between PyPI and conda publishing of your Python package.
```
:::


:::{figure-md} pypi-conda-channels

<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px">

Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. conda-forge requires a source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge.
:::

## What is PyPI

Expand All @@ -32,6 +40,14 @@ a package is that conda can install any package regardless
of the language(s) that it is written in. Whereas `pip` can
only install Python packages.

:::{button-link} ../tutorials/publish-pypi.html
:color: primary
:class: sd-rounded-pill float-left

Click here for a tutorial on publishing your package to PyPI.
:::


```{tip}
On the package build page, we discussed the [two package distribution
types that you will create when making a Python package](python-package-distribution-files-sdist-wheel): SDist (packaged as a .tar.gz or .zip) and
Expand All @@ -40,6 +56,7 @@ be published on PyPI when you use [a standard build tool](python-package-build-t
your package.
```

(about-conda)=
## What is Anaconda Cloud and conda?

conda is an open source package and environment management tool.
Expand All @@ -49,43 +66,52 @@ repository](https://repo.anaconda.com/).
Anaconda cloud (anaconda.org) contains public and private repositories for
packages. These repositories are known as channels (discussed below).

The most common public channels are:
:::{admonition} A brief history of conda's evolution
:class: note

- defaults
- conda-forge and
- bioconda
The conda ecosystem evolved years ago to provide support for and
simplify the process of managing software dependencies in scientific
The conda ecosystem evolved years ago to provide support for, and
simplify the process of, managing software dependencies in scientific
Python projects.

```{tip}
Many of the core scientific Python projects depend upon or wrap around tools and extensions that are written in other languages, such as C++. In the early stages of the scientific ecosystem's development, these non-Python extensions and tools were not well supported on PyPI, making publication difficult. In recent years there is more support for complex builds that allow developers to bundle non-Python code into a Python distribution using the [wheel distribution format](python-wheel).

Conda provides a mechanism to manage these dependencies and ensure that the required packages are installed correctly.
:::

:::{tip}
While conda was originally created to support Python packages, it
is now used across all languages. This cross-language support
makes it easier for some packages to include and have access to
tools written in other languages such as c/c++ (gdal), Julia, or R.
Creating environment that mixes all those packages are usually easier and more
consistent with full fledged package managers like conda.
```
tools written in other languages, such as C/C++ (gdal), Julia, or R.
Creating an environment that mixes all of these packages is usually easier and more
consistent with full-fledged package managers like conda.
:::

### conda channels

conda built packages are housed within repositories that are called
channels. The conda package manager can install packages from different channels.

There are several core public channels that most people use to install
packages using conda including.
packages using conda, including:

- **defaults:** this is a channel managed by the Anaconda, inc. It is the version of the Python packages that you will install if you install the Anaconda Distribution.
- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data
- [**bioconda**](https://anaconda.org/bioconda): this channel focuses on biomedical
- **defaults:** this is a channel managed by Anaconda. It is the version of the Python packages that you will install if you install the Anaconda Distribution. Anaconda decides what packages live on the default channel.
- [**conda-forge:**](https://anaconda.org/conda-forge) this is a community-driven channel that focuses on scientific packages. This channel is ideal for tools that support geospatial data. Anyone can publish a package to this channel.
- [**bioconda**](https://anaconda.org/bioconda): this channel focuses on biomedical tools.

**conda-forge** emerged as many of the scientific packages did not
exist in the default Anaconda cloud channel.

:::{figure-md} conda-channels
:::{figure-md} pypi-conda-channels

<img src="../images/conda-channels-geohackweek.jpeg" alt="ADD." width="700px">
<img src="../images/python-pypi-conda-channels.png" alt="Graphic with the title Python package repositories. Below it says Anything hosted on PyPI can be installed using pip install. Packaging hosted on a conda channel can be installed using conda install. Below that there are two rows. the top row says conda channels. next to it are three boxes one with conda-forge, community maintained; bioconda and then default - managed by the anaconda team. Below that there is a row that says PyPI servers. PyPI - anyone can publish to pypi. and test pypi. a testbed server for you to practice. " width="700px">

ADD source: GeoHackWeek
Conda channels represent various repositories that you can install packages from. Because conda-forge is community maintained, anyone can submit a recipe there. PyPI is also a community maintained repository. Anyone can submit a package to PyPI and test PyPI. Unlike conda-forge there are no manual checks of packages submitted to PyPI.
:::


## conda channels, PyPI, conda, pip - Where to publish your package

You might be wondering why there are different package repositories
Expand Down Expand Up @@ -147,16 +173,16 @@ Once your package is on PyPI, the process to add your package to conda-forge
is straight forward to do. [You can follow the detailed steps provided
by the conda-forge maintainer team.](https://conda-forge.org/docs/maintainer/adding_pkgs.html#generating-the-recipe).

Generally those steps are:

1. Fork the staged recipes conda-forge GitHub repository
1. Create a new recipe using the `grayskull` tool. You can [learn more about grayskull here](https://conda.github.io/grayskull/).
1. Add the recipe file created by `grayskull` in a folder within the recipes/ directory of the [**conda-forge/staged-recipes**](https://github.com/conda-forge/staged-recipes) repository.
1. Open a pull request with your change. Your package will be tested on Windows, Mac and Linux using the repository CI.
:::{button-link} ../tutorials/publish-conda-forge.html
:color: primary
:class: sd-rounded-pill float-left

Click here for a tutorial on adding your package to conda-forge.
:::

If you want a step by step tutorial, click here.

Once you have submitted a recipe to conda-forge, a maintainer will work
with you to iron out any bugs in your recipe. They will then add your package
to the conda-forge channel.

Once your package is added, you will have a feedstock repository on GitHub with your packages name

Expand All @@ -170,5 +196,9 @@ Once your package is on the conda-forge channel, maintaining it is simple.
Every time that you push a new version of your package to PyPI, it will
kick off a continuous integration build that updates your package in the
conda-forge repository. Once that build is complete, you will get a
notification to review the update. You can merge the pull request for
that update once you are happy with it.
notification to review the update.

You can merge the pull request for that update once you are happy with it.
A ready-to-merge PR usually means ensuring that your project's dependencies
(known as runtime requirements) listed in the updated YAML file found in the
pull request match the PyPI metadata of the new release.
Original file line number Diff line number Diff line change
@@ -1,19 +1,24 @@
# Learn about Building a Python Package

:::{figure-md} build_workflow
<img src="../images/python-package-development-process.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. those items all go into your package. Documentation and data are below that box because they aren't normally published in your packaging wheel distribution. an arrow to the right takes you to a build distribution files box. that box leads you to either publishing to testpypi or the real pypi. from pypi you can then connect to conda forge for an automated build that sends distributions from pypi to conda-forge. " width="700px">

You need to build your Python package in order to publish it to PyPI (or Conda). The build process organizes your code and metadata into a distribution format that can be uploaded to PyPI and subsequently downloaded and installed by users. NOTE: you need to publish a sdist to PyPI in order for conda-forge to properly build your package automatically.
:::{figure-md} pypi-conda-channels

<img src="../images/publish-python-package-pypi-conda.png" alt="Image showing the progression of creating a Python package, building it and then publishing to PyPI and conda-forge. You take your code and turn it into distribution files (sdist and wheel) that PyPI accepts. then there is an arrow towards the PyPI repository where ou publish both distributions. From PyPI if you create a conda-forge recipe you can then publish to conda-forge. " width="700px">

Once you have published both package distributions (the source distribution and the wheel) to PyPI, you can then publish to conda-forge. conda-forge requires an source distribution on PyPI in order to build your package on conda-forge. You do not need to rebuild your package to publish to conda-forge.
:::

You need to build your Python package in order to publish it to PyPI (or a conda channel). The build process organizes your code and metadata into a distribution format that can be uploaded to PyPI and subsequently downloaded and installed by users. NOTE: you need to publish a sdist to PyPI in order for conda-forge to properly build your package automatically.
:::

(build-package)=
## What is building a Python package?

To [publish your Python package](build_workflow) and make it easy for anyone to install, you first need to build it.
To [publish your Python package](publish-python-package-pypi-conda) and make it easy for anyone to install, you first need to build it.

But, what does it mean to build a Python package?

[As shown in the figure above](build_workflow), when you build your Python package, you convert the source files into something called a distribution package. A distribution package contains your source code and metadata about the package, in the format required by the Python Package Index, so that it can be installed by tools like pip.
[As shown in the figure above](#pypi-conda-channels), when you build your Python package, you convert the source files into something called a distribution package. A distribution package contains your source code and metadata about the package, in the format required by the Python Package Index, so that it can be installed by tools like pip.

:::{note}
The term package used to mean many different things in Python and other languages. On this page, we adapt the convention of the [Python Packaging Authority](https://www.pypa.io/en/latest/) and refer to the product of the
Expand Down Expand Up @@ -87,7 +92,7 @@ represent on your PyPI landing page. These classifiers also allow users to sort
```

:::{figure-md} build-workflow
<img src="../images/python-package-development-process.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. those items all go into your package. Documentation and data are below that box because they aren't normally published in your packaging wheel distribution. an arrow to the right takes you to a build distribution files box. that box leads you to either publishing to testpypi or the real pypi. from pypi you can then connect to conda forge for an automated build that sends distributions from pypi to conda-forge. " width="700px">
<img src="../images/python-package-development-process.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. those items all go into your package. Documentation and data are below that box because they aren't normally published in your packaging wheel distribution. an arrow to the right takes you to a build distribution files box. that box leads you to either publishing to testpypi or the real pypi. from pypi you can then connect to conda-forge for an automated build that sends distributions from pypi to conda-forge. " width="700px">

You need to build your Python package in order to publish it to PyPI (or Conda). The build process organizes your code and metadata into a distribution format that can be uploaded to PyPI and subsequently downloaded and installed by users. NOTE: you need to publish a sdist to PyPI in order for conda-forge to properly build your package automatically.
:::
Expand Down
26 changes: 16 additions & 10 deletions tutorials/intro.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
# Python packaging 101
*A start to finish beginner-friendly tutorial*

:::{toctree}
:hidden:
:caption: Python Packaging 101

What is a Python package? <self>
Make your code installable <1-installable-code>
Publish to PyPI <publish-pypi>
Publish to conda-forge <publish-conda-forge>
:::

Welcome to the pyOpenSci Python packaging tutorial series. The lessons
on the upcoming pages walk you through the core steps needed to
create a Python package.
Expand Down Expand Up @@ -38,6 +48,7 @@ Get to know Hatch <get-to-know-hatch>
What is a Python package? <self>
Make your code installable <1-installable-code>
Publish to PyPI <publish-pypi>
Publish to conda-forge <publish-conda-forge>
:::


Expand Down Expand Up @@ -306,25 +317,20 @@ If you want to make your package directly installable without having
to download the code to your computer locally then you need to
publish it in a repository such as **PyPI** or **conda-forge**.

Learn [how to publish your package to PyPI in this tutorial.](publish-pypi.md)

You will learn [how to publish your package to PyPI in this tutorial.](publish-pypi)

:::{todo}
This lesson is also not published - yet but will be reviewed soon.
:::

Then you can create a conda-forge recipe using the [Grayskull](https://github.com/conda/grayskull) tool. You can then submit this recipe to conda-forge.

:::{todo}
remove this todo when this page is published
[You will learn more about the conda-forge publication process here.](7-publish-conda-forge.md)
:::
[You will learn more about the conda-forge publication process here.](publish-conda-forge.md)


:::{figure-md} build-workflow-tutorial
<img src="../images/tutorials/publish-package-pypi-conda.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. Those items all go into your package. Documentation and data are below that box because they aren't normally published in your packaging wheel distribution. an arrow to the right takes you to a build distribution files box. that box leads you to either publishing to testPyPI or the real PyPI. From PyPI you can then connect to conda forge for an automated build that sends distributions from PyPI to conda-forge." width="700px">
<img src="../images/tutorials/publish-package-pypi-conda.png" alt="Graphic showing the high level packaging workflow. On the left you see a graphic with code, metadata and tests in it. Those items all go into your package. Documentation and data are below that box because they aren't normally published in your packaging wheel distribution. an arrow to the right takes you to a build distribution files box. that box leads you to either publishing to testPyPI or the real PyPI. From PyPI you can then connect to conda-forge for an automated build that sends distributions from PyPI to conda-forge." width="700px">

In the image above, you can see the steps associated with publishing
your package on PyPI and conda forge. Note that the distribution files that PyPI requires are the [sdist](#python-source-distribution) and [wheel](#python-wheel) files. Once you are ready to make your code publicly installable, you can publish it on PyPI. Once your code is on PyPI it is straight forward to then publish to conda forge. You create a recipe using the Grayskull package and then you open a pr in the conda-forge recipe repo. You will learn more about this process in the [conda-forge lesson](#).
your package on PyPI and conda-forge. Note that the distribution files that PyPI requires are the [sdist](#python-source-distribution) and [wheel](#python-wheel) files. Once you are ready to make your code publicly installable, you can publish it on PyPI. Once your code is on PyPI it is straight forward to then publish to conda-forge. You create a recipe using the Grayskull package and then you open a pr in the conda-forge recipe repo. You will learn more about this process in the [conda-forge lesson](#).
:::

## Yay, your package has users! Now what?
Expand Down
Loading