Dive into fixing rpm-md

[cgwalters]

This is and has for a long time been a big problem:

$ du -shc /var/cache/rpm-ostree/repomd/fedora/repodata/48986ce4583cd09825c6d437150314446f0f49fa1a1bd62dcfa1085295030fe9-primary.xml.gz
15M	/var/cache/rpm-ostree/repomd/fedora/repodata/48986ce4583cd09825c6d437150314446f0f49fa1a1bd62dcfa1085295030fe9-primary.xml.gz
15M	total
$ zcat /var/cache/rpm-ostree/repomd/fedora/repodata/48986ce4583cd09825c6d437150314446f0f49fa1a1bd62dcfa1085295030fe9-primary.xml.gz > /tmp/primary.xml
$ du -shc /tmp/primary.xml 
130M	/tmp/primary.xml
130M	total
$ 

It's insane. And the problem is that with rojig ♲📦 even for Atomic Host/Silverblue users who don't use package layering, we'll go back to "download 30+ MB of repodata, uncompress to 100+MB" just to check whether there are any updates at all.

The plus side is us working on this will also benefit containers and traditional, if we can do it right.

One idea I had is to "presolve" - a lot of this data is completely redundant dependencies. Take this chunk from the very first package I looked at, 0ad:

    <rpm:requires>
      <rpm:entry name="libstdc++.so.6()(64bit)"/>
      <rpm:entry name="libstdc++.so.6(CXXABI_1.3)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(CXXABI_1.3.5)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(CXXABI_1.3.8)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(CXXABI_1.3.9)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.11)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.14)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.15)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.18)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.19)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.20)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.21)(64bit)"/>
      <rpm:entry name="libstdc++.so.6(GLIBCXX_3.4.9)(64bit)"/>
   ...

But those are all provides of the libstdc++ package - and I don't think we're ever going to have different symbol versions provided by separate packages.

So doing a pass where we just drop redundant requires would probably make a notable difference.

Taking that idea slightly farther, do "presolve" - 0ad has a requires on /bin/sh, but that's already going to be a dependency of one of its library dependencies I bet.

A user-visible result of this would be that yum info would no longer show all of the dependencies, but...eh.

Of course another thing to do is simply to split up the repos. For Atomic Host in particular we can omit all of the desktop apps, all of the -devel packages etc.

[dustymabe]

I still don't see why we don't use something like git (or even ostree) to just download just the changes to the metadata (i.e. the diff) rather than downloading the entire thing every time.

[cgwalters]

Changing the format is a larger topic; I wouldn't say it's out of scope here, but it feels like we need to do both? Part of the inefficiency here is feeding 100MB of data to libsolv, complete with redundant dependencies.

AIUI @james-antill did some analysis versus Debian and he concluded that the "file dependencies" were a major part of the wire size. And yes holy cow, I just looked at a filelists.xml. I think my vote there would be to only do file entries for "entrypoints" like /usr/bin - there's really no sane scenario where an RPM package should Require: /usr/share/doc/GeographicLib-doc/html/C/annotated.html or whatever.

Briefly though, AIUI Rust's cargo actually does use git for metadata in the "crate index". I can't find any documentation on the formats in a quick search though.

There's also a project from the dnf team on this somewhere, I forget the name.

Using libostree for this isn't crazy; we could do something like split each XML entry into its own file. But it's quite a new nontrivial format to impose on the rpm side, and would also be quite ironic to do after jigdo...

[Conan-Kudo]

@cgwalters You're referring to the deltarepo project? https://github.com/rh-lab-q/deltametadata-prototype

There are some realistic cases where people do file requires on more than /usr/bin, but my vote would be that rpm-md should be extended to cover /usr/libexec in addition to /usr/bin and /usr/lib(64) and /etc. Maybe also /usr/include, which would more or less basically mean that only /usr/share wouldn't be included.

Debian does have a file list metadata now (that's the Contents file), but it's downloaded on-demand rather than always like we do.

The main trouble, as I understand it, is that it's not easy to "reset" the sack for on-demand fetching of filelists like Yum did. Once you're over that hurdle, it's really simple, as we can control what files are fetched for the solver cache in libdnf.

The other thing is that people do want a way to access all of the rpm-md repodata, including the changelogs, but we don't yet offer that capability. It's one of the pain points mentioned in the devel mailing list a while back when @ignatenkobrain solicited feedback about dropping Yum.

[igor-raits]

See also: https://pagure.io/packaging-committee/issue/714

[cgwalters]

I quickly glanced and of course cargo prints out the relevant URL; browsing https://github.com/rust-lang/crates.io-index makes it fairly obvious how things work there.

[cgwalters]

One thing we can do in jigdo mode is only download the primary metadata. Or at least avoid the filelists. There's also the (currently desktop specific I believe) appstream data we could skip.

[cgwalters]

However, doing this quickly blocks on #1114 ...

[cgwalters]

Let me briefly talk more about ostree vs git here. Where ostree is most valuable is when one uses its hardlink checkouts. Which is really most useful for OS/container filesystem trees.

One can use libostree to store arbitrary data and read it via the API, but...it's just not what its primarily designed to do. For example, libostree is also explicitly not intended to be a backup system.

Now obviously as I mentioned above Rust's cargo is using git to store metadata. I think that's generally a great idea, however, there are some details such as the fact that OSTree uses SHA-256 and not SHA-1. That's obviously going to be addressed in git...someday.

The other major thing about git is that when using it as we'd want to do in libdnf, I think we'd really want a shared library. And libgit2 is great but it's also not ABI stable, which is somewhat problematic in the context of a base operating system updater. libostree is fully API/ABI stable.

But my overall feeling here is that for metadata, git is probably a better choice.

[Conan-Kudo]

Is there a particular reason you'd want to have repodata checkouts via git? Wouldn't something like casync or zsync help in alleviating the "download too much every time" thing?

Generally speaking, git checkouts of metadata doesn't make as much sense because the point in time part isn't very helpful when it isn't part of the rest of the tree.

[cgwalters]

I just explicitly said above that I didn't see a use case for repodata hardlink checkouts, right? Which was an argument against libostree.

As far as improving network efficiency; yes, those would help, but so would ostree and git.

I think a bifurcation here is - are we defining a new on-disk format? I would argue to do so because...XML is not great. The way cargo uses a lot of split-up JSON seems pretty sane to me.

Anyways this isn't quite the right place to debate this, rpm-ecosystem list is.

[james-antill]

This started when I was on holiday, but I'm not sure how much it's worth me stating the obvious things ... mainly: everything is too big, something needs to change. Seven years ago we discussed and Seth wrote down: http://yum.baseurl.org/wiki/dev/NewRepoDataIdeas ... that doesn't take into account weak deps. but it's the same general idea: Only way to win is to ship less data.

For older data there is:

• http://yum.baseurl.org/wiki/YumBenchmarks

• http://yum.baseurl.org/wiki/apt2yum

...note that since those were written Fedora has gone from ~24k packages to ~55k, with the obvious increases in metadata size.

[dustymabe]

Only way to win is to ship less data.

ehh. I think only downloading data the client doesn't have is a win too. how much metadata really changes in two weeks?