wheels on PyPI

[carlwgeorge]

I'm using the createrepo_c Python bindings in a project of mine. For the CI test run I'm using the python:3.7 container image. I'd like to be able to pip install createrepo_c, but this fails due to multiple missing system libraries. I can work around this by installing them with apt first (at a minimum cmake, libmagic-dev, and librpm-dev), but I believe if a wheel package of createrepo_c was uploaded to PyPI it would just work.

Can a createrepo_c wheel be uploaded to PyPI?

[hroncok]

Note that the createrepo-c PyPI package appears to be maintained by pulp maintainers.

[carlwgeorge]

/me checks pulp/pulp_rpm commit history for recently active maintainers...

@goosemania @ipanova @daviddavis @dralley Would y'all be able to help here?

[dralley]

Hello @carlwgeorge

Currently uploading a binary wheel of createrepo_c to PyPI is blocked by some changes in the Python build infrastructure pypa/manylinux#279

Since compiled packages link against system libraries, and the system libraries are different on different systems, Python created a build environment called "manylinux" and then built tooling that wraps up all the dynamically-linked libraries into the wheel so that it doesn't depend on any system-provided libraries. In order to upload a binary wheel to PyPI, it has to have been created in this build environment because otherwise a package built on Ubuntu and uploaded would not work on CentOS.

Unfortunately, that common build environment is based on CentOS 5 and createrepo_c won't build on CentOS 5. The new environment discussed in the PR above is based on CentOS 6 and that will be usable. But it's taking a really, really long time for them to get it ready.

That's why, for now, all we've been able to do is upload a source package and have our CI install all of the build dependencies just like you're doing.

https://github.com/pulp/pulp_rpm/blob/master/.travis/install.sh#L44-L46

[carlwgeorge]

Thanks for the clarification @dralley. I am vaguely familiar with manylinux, but I didn't know that it was still building against EL5. That's understandable that a createrepo_c wheel is blocked on the build environment getting refreshed to EL6.

[dralley]

Yup, sorry I couldn't be more helpful! It was disappointing to us too when we realized that.

[carlwgeorge]

Hey, having a clear understanding of the blocker is absolutely helpful. Thanks!

[hroncok]

but I didn't know that it was still building against EL5

manylinux1 will always build against EL5.

manylinux2010 (imagine "manylinux2", but the naming schema has changed) will always build against EL6.

[dralley]

Also, once that environment does come out, there may be added complications since CentOS 6 doesn't have zchunk or libmodulemd packages available. Hopefully they can at least be compiled on CentOS 6. If not, I'm not really sure what the solution is...

https://pkgs.org/download/zchunk-libs
https://pkgs.org/download/libmodulemd

As it stands, these new deps make it (virtually?) impossible to build a fully-featured createrepo_c Python package from source on any Debian-based distro.

[carlwgeorge]

Since those exist in Fedora (and aren't in RHEL) it's just a matter of requesting the respective Fedora maintainers add an el6 branch and build it for EPEL 6. Of course the dependencies will need to be worked out too. Right off the bat I see that meson and gobject-introspection will be needed, and probably a few others.

[sgallagher]

@dralley just pointed me at this ticket.

So, libmodulemd cannot realistically be built on RHEL/EPEL 6. It needs a far newer glib2 and gobject-introspection than will ever be available on that platform. (And we can't ship newer ones in EPEL for a variety of reasons, not least of which being that they would conflict with or replace the official RHEL packages and in so doing cause lots of chaos).

If this is a significant-enough blocker to justify it, we could look into producing a version of libmodulemd that doesn't provide GObject Introspection bindings and statically-links the libyaml and glib2 dependencies, but that's a big undertaking.

[hroncok]

If this was actually desired and some effort put into this, conflicts with RHEL or EPEL wouldn't really matter. A copr repo (for example) would do.

[hroncok]

As an example, with admesh, I just build my own EL5 RPM:

https://github.com/admesh/python-admesh/tree/v0.98.9/travis
https://github.com/admesh/python-admesh/blob/027f9fe1922d7aada4e032632fcfcb1a4bde5f76/.travis.yml#L48

This is of course much easier, because it is not a full stack of dependencies but just one tiny library.

[carlwgeorge]

@sgallagher

• gobject-introspection isn't in RHEL6, so why couldn't it be added to EPEL6?
• What is the minimum version of glib2 needed to build libmodulemd? The libmodulemd spec file build requires pkgconfig(gobject-2.0), which is provided in RHEL6 by glib2-devel-2.28.8-10.el6. If a newer version is necessary it would be preferable to build require at least that version.
• For libyaml, the build requirement is pkgconfig(yaml-0.1). That isn't provided by RHEL6's libyaml-devel, but that could just be a packaging oversight as the difference between RHEL6 and RHEL7 here is just 0.1.3 to 0.1.4.

[carlwgeorge]

I don't need libmodulemd or zchunk support for my use case. I just want pip to be able to install a createrepo_c wheel with basic functionality.

[carlwgeorge]

Also related, I tried to just use the system python3-createrepo_c package on Fedora, but it's missing metadata so pip can't see it.

https://bugzilla.redhat.com/show_bug.cgi?id=1695677