Lab updates at 2025-09-02

[DanielFroehlich]

• inf1 - dnf update
• inf31 - dnf update
• ISAR Cluster - cluster update to latest channel (stop mad-metro/regional DR Cluster VMs)
• ISAR Cluster - Update operators
• Upgrade keycloak operator (check channel)
• Upgrade LVM Storage (check channel)
• Upgrade ODF (check channel)
• Upgrade ACM (check channel)
• Upgrade Quay (check channel)
• Upgrade OCPVirt (check channel)
• NetApp CSI driver (check channel)
• Create follow up issue and calendar blocker for next update on the first tue of following month

Follow up to #283

[DanielFroehlich]

ODF requires special attention - it claims to be at 4.19.3 from the operator, but inside the logs, there are some complains about missing install plans and dependencies - see #283 (comment)

[DanielFroehlich]

Operator upgrades:

• Virt 4.19.1 > 4.19.3
• MTV: 2.9.1 > 2.9.2 >2.9.3
• MetalLB: 4.19.0.202506 > 4.19.0.202508
• NMState: 4.19.0.202506 > .4.19.0-202508120121
• LocalStorage: 4.19.0.202506 > 4.19.0.202508130123
• GitOps: 1.16.2 > 1.17.0
• Pipelines: 19.1 > 1.19.3
• Keycloak: 26.2.6 > 26.2.7

MetalLB, NMState and LocatStorage were suffering from "error finding unpacked bundle configmap for ref " on the Subscription tab.
Solution: https://access.redhat.com/solutions/7108964
Deleted the failed install plans and restarted operator updates.

[DanielFroehlich]

storm6 did not come back from "not ready" state after >30 minutes. Console looked like a regular reboot. Did a manually powercycle via IDRAC, that fixed it.

ceph11 and ceph2 did not manage to cordo, had to manually delete the odf cephs osd pods to continue to update. Something is strange with the pod distruption budgets:

oc get pdb -o=wide -n openshift-storage
NAME                                              MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
rook-ceph-mds-ocs-storagecluster-cephfilesystem   1               N/A               1                     335d
rook-ceph-mgr-pdb                                 N/A             1                 1                     516d
rook-ceph-mon-pdb                                 N/A             1                 0                     654d
rook-ceph-osd-zone-muc-b                          N/A             0                 0                     2m3s
rook-ceph-osd-zone-muc-c                          N/A             0                 0                     2m6s

The last ones for zone-muc-b+c dont allow a disruption - that is strange. And: why is there none for muc-a? I guess this is due to the ODF limbo state we are currently in.

Logged #299 for the ODF issue

[DanielFroehlich]

Update is done!