Skip to content

USHIFT-5375: Document embedding container images in bootc builds #4635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 11, 2025
Merged

USHIFT-5375: Document embedding container images in bootc builds #4635

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf73580
Document embedding container images in bootc builds
ggiguash Mar 5, 2025
5bacd91
Address code review comments
ggiguash Mar 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/config/Containerfile.bootc-embedded-rhel9
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
ARG USHIFT_BASE_IMAGE_NAME
ARG USHIFT_BASE_IMAGE_TAG
FROM $USHIFT_BASE_IMAGE_NAME:$USHIFT_BASE_IMAGE_TAG

# Pull the container image dependencies into /var/lib/containers/storage-preloaded
RUN --mount=type=secret,id=pullsecret,dst=/run/secrets/pull-secret.json \
images=$(jq -r '.images | .[]' "/usr/share/microshift/release/release-$(uname -m).json") ; \
for i in ${images} ; do \
podman pull \
--authfile /run/secrets/pull-secret.json \
--root /var/lib/containers/storage-preloaded \
"docker://${i}" ; \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this preserve cosign signatures? I did not verify with a corresponding config to ensure that those local images still can be verified with the Red Hat release signer key. Do you know or want me to verify?

Copy link
Contributor Author

@ggiguash ggiguash Mar 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same code we're using in CI.
Cosign signature preservation / copy requires a few more configuration steps as performed here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that are two different aspects:

  1. verify during image build, during above podman pull that the images are signed correctly. That could probably done with the registry configuration and policy.json to be used during the build.

  2. Once the bootc image is deployed, could a customer verify that the OCI images that are part of the bootc image, do not have been tampered with. I tried to configure this using a policy.json on the edge device, but that seems to be used only during image pull (which does not happen, as the image is obviously already there).

Lets add both aspects at a later stage. For now, this is simple and works.

done

# Edit the container storage configuration file to include the new path
RUN sed -i '/^additionalimagestores.*/a\ "/var/lib/containers/storage-preloaded",' /etc/containers/storage.conf
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means the images are part of /var, which is mutable. Meaning: when switching to a new version, the old images are still present. That has a couple of pros and cons:

  • pro: we avoid the issue with layer sharing and some layers disappearing during an upgrade
  • con: we violate the "immutable" contract: if the images get somehow deleted from this folder, the workload might not be able to starte.
  • con: how is garbage collection working? At some point in time, we need to clean up older images. But only when there is no chance for a rollback. That sounds really complicated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I tried this with an update from 4.17.1 to 4.17.12, both images with embedded offline containers. I was expecting the /var/... folder to be merged, meaning to see both sets of images to be in there.
Before that update, I saw 11 images (crictl images | wc -l) .
Right after the reboot, I still saw 11 images, which feels strange and wrong.
During MicroShift startup, image count increased to 20. It felt like microshift is actually pulling images, despite they should be in the bootc update already. So I guess that did not work?
Happy to share / reproduce this on a call - this is available in the COE MUC Lab via VPN.

Copy link
Contributor Author

@ggiguash ggiguash Mar 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an open issue for fixing this: https://issues.redhat.com/browse/OCPBUGS-52420
I will take care of all the fixes in the same PR. For now, let's keep it consistent with what we have in CI?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, fine for me!


# Apply a workaround to set the SELinux context on the new storage directory and
# also restore 'NET_BIND_SERVICE' capability that is currently lost when including
# images in the container.
#
# Note: This requires setting the additional image stores path to a read-write
# location on the file system. The images will still be treated as read-only by
# the container subsystem.
# See https://github.com/ostreedev/ostree-rs-ext/issues/654
#
# hadolint ignore=DL3059
RUN cat > /usr/bin/microshift-imagestore-config <<'EOF'
#!/bin/bash
set -euxo pipefail
DEF_IMGPATH="$1"
NEW_IMGPATH="$2"
semanage fcontext -a -e "${DEF_IMGPATH}" "${NEW_IMGPATH}"
restorecon -R "${NEW_IMGPATH}"
find "${NEW_IMGPATH}" -type f -path "*/usr/sbin/haproxy" -exec setcap "cap_net_bind_service=+ep" {} \;
EOF

# hadolint ignore=DL3059
RUN cat > /etc/systemd/system/microshift-imagestore-config.service <<'EOF'
[Unit]
Description=Configure the image store directory for MicroShift
Before=microshift.service
[Service]
Type=oneshot
ExecStart=/usr/bin/microshift-imagestore-config /var/lib/containers/storage /var/lib/containers/storage-preloaded
[Install]
WantedBy=multi-user.target
EOF

RUN chmod 755 /usr/bin/microshift-imagestore-config && \
systemctl enable microshift-imagestore-config.service
4 changes: 2 additions & 2 deletions docs/config/Containerfile.bootc-rhel9
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
FROM registry.redhat.io/rhel9-eus/rhel-9.4-bootc:9.4

ARG USHIFT_VER=4.17
ARG USHIFT_VER=4.18
# hadolint ignore=SC1091
RUN . /etc/os-release && dnf upgrade -y --releasever="${VERSION_ID}" && \
dnf config-manager \
--set-enabled "rhocp-${USHIFT_VER}-for-rhel-9-$(uname -m)-rpms" \
--set-enabled "fast-datapath-for-rhel-9-$(uname -m)-rpms" && \
dnf install -y firewalld microshift && \
dnf install -y firewalld jq microshift microshift-release-info && \
systemctl enable microshift && \
dnf clean all

Expand Down
Loading