mislabeled files in /usr/etc when building with newer rpm-ostree-2025.11-1

[dustymabe]

The ext.config.selinux.file-context-policy-match fails. Here's the reason:

[core@cosa-devsh ~]$ sudo restorecon -vnr /var/ /etc/ /usr/ /boot/
Would relabel /usr/etc/systemd/system from system_u:object_r:etc_t:s0 to system_u:object_r:systemd_unit_file_t:s0
...

There are 96 mislabeled files. Doing the build with rpm-ostree-2025.10-3 does not yield this problem.

It was tagged in to the continuous tag last week:

[dustymabe@media ~]$ koji list-tagged f44-coreos-continuous 
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
fedora-release-44-0.1                     f44-coreos-continuous  jnsamyak
rpm-ostree-2025.10-3.fc44                 f44-coreos-continuous  jlebon
rpm-ostree-2025.11-1.fc44                 f44-coreos-continuous  jmarrero

[dustymabe]

Building using the newer rpm-ostree also causes the ext.config.extensions.package test to fail:

Sep 14 15:39:35 qemu0 kola-runext-package[2266]: + rpm-ostree install --apply-live htop strace tcpdump tree crio
Sep 14 15:39:35 qemu0 kola-runext-package[2275]: notice: auto-inferring -y/--assumeyes when not run interactively; this will change in the future
Sep 14 15:39:38 qemu0 kola-runext-package[2275]: Checking out tree 4452920...done
Sep 14 15:39:38 qemu0 kola-runext-package[2275]: Enabled rpm-md repositories: fedora-cisco-openh264 rawhide
Sep 14 15:39:39 qemu0 kola-runext-package[2275]: Updating metadata for 'fedora-cisco-openh264'...done
Sep 14 15:39:41 qemu0 kola-runext-package[2275]: Updating metadata for 'rawhide'...done
Sep 14 15:39:45 qemu0 kola-runext-package[2275]: Importing rpm-md...done
Sep 14 15:39:45 qemu0 kola-runext-package[2275]: rpm-md repo 'fedora-cisco-openh264'; generated: 2025-08-13T16:56:53Z solvables: 0
Sep 14 15:39:45 qemu0 kola-runext-package[2275]: rpm-md repo 'rawhide'; generated: 2025-09-14T05:33:50Z solvables: 77365
Sep 14 15:39:45 qemu0 kola-runext-package[2275]: Resolving dependencies...done
Sep 14 15:39:45 qemu0 kola-runext-package[2275]: Will download: 6 packages (25.2 MB)
Sep 14 15:39:46 qemu0 kola-runext-package[2275]: Downloading from 'rawhide'...done
Sep 14 15:39:47 qemu0 kola-runext-package[2275]: Importing packages...done
Sep 14 15:39:47 qemu0 kola-runext-package[2275]: Checking out packages...done
Sep 14 15:39:47 qemu0 kola-runext-package[2275]: Running systemd-sysusers...done
Sep 14 15:39:47 qemu0 kola-runext-package[2275]: Running pre scripts...done
Sep 14 15:39:47 qemu0 kola-runext-package[2275]: Running post scripts...done
Sep 14 15:39:48 qemu0 kola-runext-package[2275]: Running posttrans scripts...done
Sep 14 15:39:48 qemu0 kola-runext-package[2275]: Writing rpmdb...done
Sep 14 15:39:52 qemu0 kola-runext-package[2275]: Writing OSTree commit...done
Sep 14 15:39:54 qemu0 kola-runext-package[2275]: Staging deployment...done
Sep 14 15:39:56 qemu0 kola-runext-package[2275]: Computing /etc diff to preserve...done
Sep 14 15:39:56 qemu0 kola-runext-package[2275]: Updating /usr...done
Sep 14 15:39:56 qemu0 kola-runext-package[2275]: error: Changed directories are not supported yet

[dustymabe]

I've untagged all of these for now. cc @jmarrero

[dustymabe@media ~]$ koji untag-build f42-coreos-continuous rpm-ostree-2025.11-1.fc42
[dustymabe@media ~]$ koji untag-build f43-coreos-continuous rpm-ostree-2025.11-1.fc43
[dustymabe@media ~]$ koji untag-build f44-coreos-continuous rpm-ostree-2025.11-1.fc44

[dustymabe]

Note that upstream rpm-ostree CI may have caught this, but our overrides/rpm mechanism is currently broken. So that may have contributed to this issue slipping through.

[jlebon]

The labeling change is from coreos/rpm-ostree#5485 and expected. I think we need to adapt our tests.

Note that upstream rpm-ostree CI may have caught this, but our overrides/rpm mechanism is currently broken. So that may have contributed to this issue slipping through.

Ahh yeah, makes sense. (Though I suspect you mean Bodhi testing there, right? CI in upstream rpm-ostree would still be using the legacy path.) This should be fixed by coreos/coreos-assembler#4319.

Building using the newer rpm-ostree also causes the ext.config.extensions.package test to fail:

Hmm, are we sure that one isn't due to an update in one of the packages we try to apply live? Possibly something is trying to change e.g. mode bits on an existing directory.

[dustymabe]

The labeling change is from coreos/rpm-ostree#5485 and expected. I think we need to adapt our tests.

Our test just runs restorecon here on a running system. Are you saying restorecon is wrong?

Note that upstream rpm-ostree CI may have caught this, but our overrides/rpm mechanism is currently broken. So that may have contributed to this issue slipping through.

Ahh yeah, makes sense. (Though I suspect you mean Bodhi testing there, right? CI in upstream rpm-ostree would still be using the legacy path.) This should be fixed by coreos/coreos-assembler#4319.

+1

Building using the newer rpm-ostree also causes the ext.config.extensions.package test to fail:

Hmm, are we sure that one isn't due to an update in one of the packages we try to apply live? Possibly something is trying to change e.g. mode bits on an existing directory.

I'm pretty sure on this. Both branched and rawhide now build ever since I did #2030 (comment)

[jlebon]

The labeling change is from coreos/rpm-ostree#5485 and expected. I think we need to adapt our tests.

Our test just runs restorecon here on a running system. Are you saying restorecon is wrong?

Was the new rpm-ostree in both the build stage and the target system? It might need to be in both.

[dustymabe]

The labeling change is from coreos/rpm-ostree#5485 and expected. I think we need to adapt our tests.

Our test just runs restorecon here on a running system. Are you saying restorecon is wrong?

Was the new rpm-ostree in both the build stage and the target system? It might need to be in both.

yes, for rawhide it was. for branched it wasn't

[jmarrero]

I'll try to replicate here locally and see how we fix this without reverting coreos/rpm-ostree#5485

[jmarrero]

Is there a specific way CI is building the image vs what I have done here? I upgraded my toolbox to rpm-ostree 2025.11, then added the rpms to the cosa overrides/rpm directory and did a cosa fetch && cosa build && cosa buildextend-qemu

Fedora CoreOS 42.20250915.dev.0
[core@cosa-devsh ~]$ sudo restorecon -vnr /var/ /etc/ /usr/ /boot/
[core@cosa-devsh ~]$ 

[jmarrero]

jmarrero@toolbx ~/d/cosa-build> cosa kola run ext.config.extensions.package
kola -p qemu run ext.config.extensions.package --output-dir tmp/kola
⏭️  Skipping kola test pattern "ext.config.multipath.resilient":
  👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
=== RUN   ext.config.extensions.package
--- PASS: ext.config.extensions.package (47.71s)
PASS, output in tmp/kola
jmarrero@toolbx ~/d/cosa-build> 

I wonder if this is a 43/rawhide only thing... testing.

[jmarrero]

I see both failures in rawhide, did a new fresh build (new cosa build dir) of F42 just to double check with the same rpm-ostree and still works as expected. Something else is contributing to this on rawhide.

[alexlarsson]

I can't say I understand all the details related to this, but there is something called RPMOSTREE_SELINUX_MODE_V1, which is where /usr/etc is labeled as /etc. Maybe the changes we did in coreos/rpm-ostree#5485 need to be only done when this is enabled?

[alexlarsson]

[core@cosa-devsh ~]$ sudo restorecon -vnr /var/ /etc/ /usr/ /boot/
Would relabel /usr/etc/systemd/system from system_u:object_r:etc_t:s0 to system_u:object_r:systemd_unit_file_t:s0

This is basically saying that we when building the image we applied the changes (add file aliases) to the policy config, but we didn't relabel after that.