set up containers-storage in supermin cache #4286
Conversation
In the container native build flow where we are building from quay.io/fedora/fedora-bootc:xx it's better for the local developer use case if we don't need to pull the container from the registry on each iteration. Let's use containers-storage from the cache and also use `buildah build --layers=true` in all cases. This also bumps the default runvm cache size to account for the extra usage associated with the containers storage.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors build-with-buildah to set up and use a containers-storage cache within the supermin VM. This is a significant improvement as it unifies the build logic for both direct and supermin paths, where buildah now consistently builds to containers-storage, and skopeo handles the export to an OCI archive. The changes in src/supermin-init-prelude.sh to set up the cache via a symlink are well-commented and seem correct. The related adjustments to cache size and the unmount command in src/cmdlib.sh are also appropriate.
I have two main concerns regarding src/cmd-build-with-buildah: one about argument quoting in the generated script, which could lead to build failures, and another about a logic change for mounting the overrides directory that might break some use cases. Please see the detailed comments.
We now have persistent containers storage in the supermin VM with the cache. Let's unify the direct and non-direct paths in the code. This does mean we need to skopeo copy to the ociarchive inside the supermin VM (because we don't have access to that containers storage outside of it). Alternatively we could have tried to run the `cosa import` inside the supermin VM as well, but the use of a bare-user repository over a virtiofs share made that option not work.
Somehow hardcoding `tmp/repo` here isn't actually working. Here's
what happens if I blow away my `tmp/` and then attempt to do another
operation (like `cosa osbuild qemu`):
```
Wrote: ostree-unverified-image:oci-archive:/srv/tmp/cosa-import-op0cg6_c/out.ociarchive => e202ee3cb8a2a8d814bd2b325cdef5500459fd2381d4fa766cd15ab730ae0e80
2273 metadata, 8746 content objects imported; 1.2 GB content written
1471 metadata, 1534 content objects imported; 133.6 MB content written
Imported OCI image as build 42.20250821.dev.0
Pruning build 44.20250826.dev.0
Deleting 5 blob refs
[coreos-assembler]$
[coreos-assembler]$ rm -rf tmp/*
[coreos-assembler]$ cosa osbuild qemu
Config commit: e5f88e72120b3f89cf2c4d81b45bbe791d9bd79d
Using manifest: /srv/src/config/manifest.yaml
Will build qemu
Extracting e202ee3cb8a2a8d814bd2b325cdef5500459fd2381d4fa766cd15ab730ae0e80
layers already present: 0; layers needed: 66 (975.9 MB)
3725 metadata, 21742 content objects imported; 1.7 GB content written 3
2603 metadata, 10508 content objects imported; 250.5 MB content written
error: No such metadata object e202ee3cb8a2a8d814bd2b325cdef5500459fd2381d4fa766cd15ab730ae0e80.commit
Traceback (most recent call last):
File "<string>", line 10, in <module>
cmdlib.import_ostree_commit(workdir, builddir, buildmeta, extract_json=('1' == '1'))
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 362, in import_ostree_commit
extract_image_json(workdir, commit)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/usr/lib/coreos-assembler/cosalib/cmdlib.py", line 272, in extract_image_json
raise Exception("Failed to extract image.json")
Exception: Failed to extract image.json
failed to execute cmd-osbuild: exit status 1
```
Let's just have the caller explicitly tell us the path to the repo
we want to import into is.
If there are no rpms then let's clean up the repo metadata if it exists which should prevent cosmetic warts like: ``` Updating and loading repositories: overrides ???% | 0.0 B/s | -1.0 B | ? >>> Curl error (37): Could not read a file:// file for file:///run/src/overrides >>> Usable URL not found ``` Or fatal ones like: ``` error: Installing packages: importing RPMs: failed to open /run/src/overrides/rpm/systemd-258~rc3-2.fc44.x86_64.rpm ```
dustymabe
force-pushed
the
dusty-cache-containers-storage
branch
from
e949a94 to
b9dbc5c
Compare
| cat <<EOF > "${tempdir}/build-with-buildah-script.sh" | ||
| set -euxo pipefail | ||
| env -C ${tempdir}/src TMPDIR=$(realpath cache) buildah $@ | ||
| skopeo copy --quiet "${final_ref}" "${tmp_oci_archive}" |
| fi | ||
| cat <<EOF > "${tempdir}/build-with-buildah-script.sh" | ||
| set -euxo pipefail | ||
| env -C ${tempdir}/src TMPDIR=$(realpath cache) buildah $@ |
There was a problem hiding this comment.
I usually wouldn't worry about spaces the way Gemini did, but there's an easy way I think to do this which is to use \"\$@\" here and then just pass the arguments when you call $cmd below.
| @@ -161,8 +161,13 @@ build_with_buildah() { | |||
| fi | |||
|
|
|||
| if [ -d overrides ]; then | |||
| if [[ -n $(ls overrides/rpm/*.rpm 2> /dev/null) ]]; then | |||
| (cd overrides/rpm && rm -rf .repodata && createrepo_c .) | |||
There was a problem hiding this comment.
Hmm, I don't understand what exactly you were hitting.
Actually, is the bug here just s/.repodata/repodata? Though this was cargo-culted from
coreos-assembler/src/cmdlib.sh
Line 447 in 6e863df
and use it for
cosa build-with-buildah.See individual commit messages.