Skip to content

fix(e2e): move MinIO/mc to the pgsty fork; preload digest refs via crictl - #485

Merged
perfectra1n merged 1 commit into
mainfrom
fix/e2e-minio-images
Sep 25, 2026
Merged

fix(e2e): move MinIO/mc to the pgsty fork; preload digest refs via crictl#485
perfectra1n merged 1 commit into
mainfrom
fix/e2e-minio-images

Conversation

@perfectra1n

@perfectra1n perfectra1n commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Why

quay.io/minio/{minio,mc} started returning 401 to anonymous pulls on 2026-09-24 — the second registry upstream has withdrawn (Docker Hub already 404s, see the old MINIO_IMAGE doc comment). The MinIO pod sits in ImagePullBackOff, every S3-backed e2e test waits out its 420s readiness timeout ×3 nextest tries, and the shard fails.

PR CI on 09-23 was green, so this only showed in the merge queue: both queue runs for #481 failed on bootstrap-deadline + repo-breaker with exactly this (failed to resolve reference "quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z": … 401 UNAUTHORIZED). Every PR entering the queue will hit it until this lands.

What

  • MINIO_IMAGE / MC_IMAGE → docker.io/pgsty/{minio,mc} (community rebuild; same minio entrypoint, /usr/bin/mc + /usr/bin/sh present), pinned by tag and digest.
  • minio-preload: digest-pinned refs are now pulled through the node's CRI (crictl pull) and verified with crictl inspecti. docker pull by digest stores no repo tag, so the old docker save | ctr import path imported them nameless while still printing preloaded — kubelet never matched them. That was already silently the case for the digest-pinned nfs-ganesha image.

Verification

  • cargo test -p kopiur-e2e --lib — 32 passed.
  • repo-breaker shard end-to-end in a fresh kind cluster (mirroring the CI job's steps): 1 passed, twice (before and after the preload change).
  • After the preload change, all three digest-pinned images are on the node by name, and kubelet reports Container image "docker.io/pgsty/minio:…@sha256:b6bf…" already present on machine for the MinIO pod (same for mc-mkbucket) — no in-cluster pull.

Not touched: the node-seed busybox and CSI-stack preload helpers are separate copies; the CSI ones are tag-pinned and unaffected.

@perfectra1n
fix(e2e): move MinIO/mc to the pgsty fork; preload digest refs via cr…
754f0a6 
…ictl

quay.io/minio/{minio,mc} started answering 401 to anonymous pulls on
2026-09-24, the second registry upstream has withdrawn (Docker Hub already
404s). Every S3-backed e2e shard then waited out a 420s readiness timeout per
attempt on an ImagePullBackOff MinIO pod, which is what failed every merge-queue
run since (e.g. #481: bootstrap-deadline, repo-breaker).

- MINIO_IMAGE / MC_IMAGE -> docker.io/pgsty/{minio,mc}, a drop-in community
  rebuild (same entrypoints, /usr/bin/mc and /usr/bin/sh present), pinned by
  tag AND digest so a re-tag on a third-party repo can't change what runs.
- minio-preload: digest-pinned refs now go through the node's CRI with
  `crictl pull` and are confirmed with `crictl inspecti`. `docker pull` by
  digest stores no repo tag, so the old save|ctr-import path imported them
  NAMELESS while still printing "preloaded" - kubelet never matched them and
  pulled in-cluster anyway. This was already silently true for the
  digest-pinned nfs-ganesha image.
@perfectra1n
perfectra1n added this pull request to the merge queue Sep 25, 2026
Merged via the queue into main with commit a4f0233 Sep 25, 2026
40 checks passed
@perfectra1n
perfectra1n deleted the fix/e2e-minio-images branch September 25, 2026 16:59
@sticky-gecko sticky-gecko Bot mentioned this pull request Sep 24, 2026
Merged
doonga pushed a commit to greyrock-labs/home-ops that referenced this pull request Sep 25, 2026
renovate[bot] git.greyrock.io Instance
fix(container): update image ghcr.io/home-operations/charts/kopiur (0…
84e43d5 
….10.9 ➔ 0.10.10) (#280)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.9` → `0.10.10` |

---

### Release Notes

<details>
<summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary>

### [`v0.10.10`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#01010-2026-09-25)

[Compare Source](home-operations/kopiur@0.10.9...0.10.10)

##### Bug Fixes

- batch snapshot deletes and checkpoint-poisoned replication ([#&#8203;477](home-operations/kopiur#477)) ([#&#8203;481](home-operations/kopiur#481)) ([183337c](home-operations/kopiur@183337c))
- **container:** update image busybox ([`cac8f90`](home-operations/kopiur@cac8f90) → [`fd7dc98`](home-operations/kopiur@fd7dc98)) ([#&#8203;479](home-operations/kopiur#479)) ([945a106](home-operations/kopiur@945a106))
- **container:** update image busybox ([`dc2d74b`](home-operations/kopiur@dc2d74b) → [`cac8f90`](home-operations/kopiur@cac8f90)) ([#&#8203;478](home-operations/kopiur#478)) ([8531656](home-operations/kopiur@8531656))
- **e2e:** move MinIO/mc to the pgsty fork; preload digest refs via crictl ([#&#8203;485](home-operations/kopiur#485)) ([a4f0233](home-operations/kopiur@a4f0233))
- **rust:** update crate clap (4.6.6 → 4.6.7) ([#&#8203;465](home-operations/kopiur#465)) ([fcddd45](home-operations/kopiur@fcddd45))
- **rust:** update crate rand (0.10.2 → 0.10.3) ([#&#8203;473](home-operations/kopiur#473)) ([026c0f4](home-operations/kopiur@026c0f4))
- **rust:** update crate thiserror (2.0.20 → 2.0.21) ([#&#8203;482](home-operations/kopiur#482)) ([1457622](home-operations/kopiur@1457622))

##### Continuous Integration

- **github-action:** update github-actions ([#&#8203;468](home-operations/kopiur#468)) ([af27b2e](home-operations/kopiur@af27b2e))

##### Miscellaneous Chores

- **github-action:** update action codecov/codecov-action (v7.1.0 → v7.1.1) ([#&#8203;472](home-operations/kopiur#472)) ([5a3f76b](home-operations/kopiur@5a3f76b))
- **mise:** update mise tools ([#&#8203;463](home-operations/kopiur#463)) ([69698aa](home-operations/kopiur@69698aa))
- **mise:** update tool aqua:astral-sh/uv (0.12.15 → 0.12.16) ([#&#8203;474](home-operations/kopiur#474)) ([8a45270](home-operations/kopiur@8a45270))
- **mise:** update tool aqua:astral-sh/uv (0.12.16 → 0.12.17) ([#&#8203;475](home-operations/kopiur#475)) ([b94926b](home-operations/kopiur@b94926b))
- **mise:** update tool node (24.20.0 → v24.21.0) ([#&#8203;457](home-operations/kopiur#457)) ([fc67c05](home-operations/kopiur@fc67c05))
- **mise:** update tool oxfmt (0.66.0 → 0.68.0) ([#&#8203;459](home-operations/kopiur#459)) ([0614d12](home-operations/kopiur@0614d12))
- **mise:** update tool rust (1.98.0 → 1.98.1) ([#&#8203;447](home-operations/kopiur#447)) ([3b2d33e](home-operations/kopiur@3b2d33e))
- **renovate:** inherit the automerge rules from the shared preset ([#&#8203;483](home-operations/kopiur#483)) ([5b9feb6](home-operations/kopiur@5b9feb6))

</details>

---

### Configuration

📅 **Schedule**: (in timezone America/New_York)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMDUuMiIsInVwZGF0ZWRJblZlciI6IjQ0LjEwNS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19-->

Reviewed-on: https://git.greyrock.io/todd/home-ops/pulls/280
gabrielcosi pushed a commit to gabrielcosi/home-ops that referenced this pull request Sep 25, 2026
bot-owl @gabrielcosi
fix(container): update image ghcr.io/home-operations/charts/kopiur (0…
cbcb763 
….10.9 ➔ 0.10.10) (#931)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.9` → `0.10.10` |

---

### Release Notes

<details>
<summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary>

### [`v0.10.10`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#01010-2026-09-25)

[Compare Source](home-operations/kopiur@0.10.9...0.10.10)

##### Bug Fixes

- batch snapshot deletes and checkpoint-poisoned replication ([#&#8203;477](home-operations/kopiur#477)) ([#&#8203;481](home-operations/kopiur#481)) ([183337c](home-operations/kopiur@183337c))
- **container:** update image busybox ([`cac8f90`](home-operations/kopiur@cac8f90) → [`fd7dc98`](home-operations/kopiur@fd7dc98)) ([#&#8203;479](home-operations/kopiur#479)) ([945a106](home-operations/kopiur@945a106))
- **container:** update image busybox ([`dc2d74b`](home-operations/kopiur@dc2d74b) → [`cac8f90`](home-operations/kopiur@cac8f90)) ([#&#8203;478](home-operations/kopiur#478)) ([8531656](home-operations/kopiur@8531656))
- **e2e:** move MinIO/mc to the pgsty fork; preload digest refs via crictl ([#&#8203;485](home-operations/kopiur#485)) ([a4f0233](home-operations/kopiur@a4f0233))
- **rust:** update crate clap (4.6.6 → 4.6.7) ([#&#8203;465](home-operations/kopiur#465)) ([fcddd45](home-operations/kopiur@fcddd45))
- **rust:** update crate rand (0.10.2 → 0.10.3) ([#&#8203;473](home-operations/kopiur#473)) ([026c0f4](home-operations/kopiur@026c0f4))
- **rust:** update crate thiserror (2.0.20 → 2.0.21) ([#&#8203;482](home-operations/kopiur#482)) ([1457622](home-operations/kopiur@1457622))

##### Continuous Integration

- **github-action:** update github-actions ([#&#8203;468](home-operations/kopiur#468)) ([af27b2e](home-operations/kopiur@af27b2e))

##### Miscellaneous Chores

- **github-action:** update action codecov/codecov-action (v7.1.0 → v7.1.1) ([#&#8203;472](home-operations/kopiur#472)) ([5a3f76b](home-operations/kopiur@5a3f76b))
- **mise:** update mise tools ([#&#8203;463](home-operations/kopiur#463)) ([69698aa](home-operations/kopiur@69698aa))
- **mise:** update tool aqua:astral-sh/uv (0.12.15 → 0.12.16) ([#&#8203;474](home-operations/kopiur#474)) ([8a45270](home-operations/kopiur@8a45270))
- **mise:** update tool aqua:astral-sh/uv (0.12.16 → 0.12.17) ([#&#8203;475](home-operations/kopiur#475)) ([b94926b](home-operations/kopiur@b94926b))
- **mise:** update tool node (24.20.0 → v24.21.0) ([#&#8203;457](home-operations/kopiur#457)) ([fc67c05](home-operations/kopiur@fc67c05))
- **mise:** update tool oxfmt (0.66.0 → 0.68.0) ([#&#8203;459](home-operations/kopiur#459)) ([0614d12](home-operations/kopiur@0614d12))
- **mise:** update tool rust (1.98.0 → 1.98.1) ([#&#8203;447](home-operations/kopiur#447)) ([3b2d33e](home-operations/kopiur@3b2d33e))
- **renovate:** inherit the automerge rules from the shared preset ([#&#8203;483](home-operations/kopiur#483)) ([5b9feb6](home-operations/kopiur@5b9feb6))

</details>

---

### Configuration

📅 **Schedule**: (in timezone Europe/Berlin)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMDMuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjEwMy4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19-->

Reviewed-on: https://git.xcd.dev/gabrielcosi/home-ops/pulls/931
eleboucher pushed a commit to eleboucher/homelab that referenced this pull request Sep 26, 2026
bot-owl @eleboucher
fix(container): update image ghcr.io/home-operations/charts/kopiur (0…
30362fe 
….10.9 ➔ 0.10.10) (#2200)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.9` → `0.10.10` |

---

> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information.

---

### Release Notes

<details>
<summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary>

### [`v0.10.10`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#01010-2026-09-25)

[Compare Source](home-operations/kopiur@0.10.9...0.10.10)

##### Bug Fixes

- batch snapshot deletes and checkpoint-poisoned replication ([#&#8203;477](home-operations/kopiur#477)) ([#&#8203;481](home-operations/kopiur#481)) ([183337c](home-operations/kopiur@183337c))
- **container:** update image busybox ([`cac8f90`](home-operations/kopiur@cac8f90) → [`fd7dc98`](home-operations/kopiur@fd7dc98)) ([#&#8203;479](home-operations/kopiur#479)) ([945a106](home-operations/kopiur@945a106))
- **container:** update image busybox ([`dc2d74b`](home-operations/kopiur@dc2d74b) → [`cac8f90`](home-operations/kopiur@cac8f90)) ([#&#8203;478](home-operations/kopiur#478)) ([8531656](home-operations/kopiur@8531656))
- **e2e:** move MinIO/mc to the pgsty fork; preload digest refs via crictl ([#&#8203;485](home-operations/kopiur#485)) ([a4f0233](home-operations/kopiur@a4f0233))
- **rust:** update crate clap (4.6.6 → 4.6.7) ([#&#8203;465](home-operations/kopiur#465)) ([fcddd45](home-operations/kopiur@fcddd45))
- **rust:** update crate rand (0.10.2 → 0.10.3) ([#&#8203;473](home-operations/kopiur#473)) ([026c0f4](home-operations/kopiur@026c0f4))
- **rust:** update crate thiserror (2.0.20 → 2.0.21) ([#&#8203;482](home-operations/kopiur#482)) ([1457622](home-operations/kopiur@1457622))

##### Continuous Integration

- **github-action:** update github-actions ([#&#8203;468](home-operations/kopiur#468)) ([af27b2e](home-operations/kopiur@af27b2e))

##### Miscellaneous Chores

- **github-action:** update action codecov/codecov-action (v7.1.0 → v7.1.1) ([#&#8203;472](home-operations/kopiur#472)) ([5a3f76b](home-operations/kopiur@5a3f76b))
- **mise:** update mise tools ([#&#8203;463](home-operations/kopiur#463)) ([69698aa](home-operations/kopiur@69698aa))
- **mise:** update tool aqua:astral-sh/uv (0.12.15 → 0.12.16) ([#&#8203;474](home-operations/kopiur#474)) ([8a45270](home-operations/kopiur@8a45270))
- **mise:** update tool aqua:astral-sh/uv (0.12.16 → 0.12.17) ([#&#8203;475](home-operations/kopiur#475)) ([b94926b](home-operations/kopiur@b94926b))
- **mise:** update tool node (24.20.0 → v24.21.0) ([#&#8203;457](home-operations/kopiur#457)) ([fc67c05](home-operations/kopiur@fc67c05))
- **mise:** update tool oxfmt (0.66.0 → 0.68.0) ([#&#8203;459](home-operations/kopiur#459)) ([0614d12](home-operations/kopiur@0614d12))
- **mise:** update tool rust (1.98.0 → 1.98.1) ([#&#8203;447](home-operations/kopiur#447)) ([3b2d33e](home-operations/kopiur@3b2d33e))
- **renovate:** inherit the automerge rules from the shared preset ([#&#8203;483](home-operations/kopiur#483)) ([5b9feb6](home-operations/kopiur@5b9feb6))

</details>

---

### Configuration

📅 **Schedule**: (in timezone Europe/Paris)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMDQuMiIsInVwZGF0ZWRJblZlciI6IjQ0LjEwNC4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2200
perfectra1n added a commit that referenced this pull request Sep 26, 2026
@perfectra1n
Merge origin/main into feat/web-ui
5cde0e0 
Brings in 0.10.9/0.10.10 (#467 stream sources + streamExec restores, #481
batched deletes, #485 MinIO fork) and the dependency bumps since the branch
last synced.

Most conflicts were main editing code this branch had moved out of
crates/cli into crates/ops; main's edits are re-applied at the new home:

- restore: `RestoreArgs -> RestoreRequest` is now `TryFrom`, with main's
  total `restore_target_from_args` (no `unreachable!`) refusing by name via
  `CliError::UnresolvedRestoreTarget`, since `streamExec` has no flag.
  Main's two target tests move to cli.rs beside the conversion.
- ops snapshot: `mover: None` on the built spec, plus main's stream-policy
  planned-cells test.
- ops doctor: the privileged-movers annotation value is `true`, not `allow`.
- cli error: keep GoneWhileWaiting/AmbiguousTarget in OpsError (moved), add
  main's UnresolvedRestoreTarget.

Semantic conflicts (merged cleanly, did not compile or silently misbehaved):

- kopia fake_shim: main's new test built an `ObjectId` via `.into()`; this
  branch made it a validated newtype, so use `ObjectId::parse`.
- ui: `SnapshotSpec.mover` in a test fixture; exhaustive `RestoreTarget`
  match in a cli test gains `StreamExec`.
- ui policies view: a `stream` source fell through to "(unrecognized
  source)"; it now renders as `stream/<fileName>`, with a test.

Unions: e2e `ws` feature + ui-model dep; e2e scenario list gains both sides'
buckets; mise tool bumps from main keep this branch's pnpm; expand tests
keep both sides' appended cases. Cargo.lock re-points the new crates at
thiserror 2.0.21.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@perfectra1n