Repository navigation
fix: streamed command sources, epoch parameters, per-PVC verification, per-run mover overrides (#436 #451 #456 #458 #461 #464) - #467
Merged
Conversation
The server Deployment set KOPIA_CONFIG_PATH and KOPIA_CACHE_DIRECTORY but never KOPIA_LOG_DIR, so kopia fell back to its unwritable default log location and spammed stdout with log-write errors. Point it at /cache/logs, under the cache emptyDir already mounted at SERVER_CACHE_DIR (no new volume needed), and use the shared kopiur_kopia::env constants for all three instead of hand-written string literals.
…napshot path
Adds the `stream` SnapshotPolicy source (capture a command's stdout as one
virtual file) and the `streamExec` Restore target, plus the kopia plumbing that
makes them safe.
The load-bearing part is WHO decides to commit. kopia finalizes a stdin snapshot
when — and only when — stdin reaches EOF, so `run_with_stdin` keeps ownership of
the child's stdin and lets the caller return Commit or Abort AFTER the producer's
exit status is known: Commit closes the pipe, Abort kills kopia with the pipe
still open so no manifest is ever written. That is what makes "a producer that
failed can never leave a retained, restorable snapshot" true by construction
rather than by cleaning up afterwards.
Two bugs the new integration tests caught, both of which would have shipped
truncated dumps as successful backups:
* a feed closure that OWNED ChildStdin dropped it on completion, closing the
pipe and committing the snapshot before the outcome could be inspected;
* draining stdout alongside the feed deadlocked — the drain waits for kopia to
exit, kopia waits for stdin to close, and stdin closes after the drain.
The drains are now background tasks and the runner owns the writer.
Source keeps its sibling-Option wire shape (the forms share sourcePath*/readOnly
keys, which an enum could only express via a flatten kube's schema rewriter
cannot represent), so nothing on the wire breaks. Type-safety moves up a layer
instead: source_shape() resolves it into the SourceShape enum and the reconcile
paths match on that exhaustively, so a fifth source form cannot compile until
every handler accounts for it.
fileName is validated as ONE safe path segment because kopia stores it verbatim
as the entry name without sanitizing: a value of '../x' makes a later
'kopia restore <id> <dir>' write outside <dir> (verified against 0.23.1), and an
empty value makes kopia ignore stdin and snapshot the local filesystem instead.
(cherry picked from commit 0763b75)
The mover resolves the one running pod matching the selector, opens a pods/exec websocket, and pumps its stdout straight into kopia's stdin. The controller turns a `stream` source into a StreamProducerSpec on the work spec; build_backup_run is now an exhaustive match over SourceShape, and the stream arm mounts no volume at all. Correctness rests on the exec STATUS, never on the byte stream. If the websocket dies mid-dump the stdout reader returns a perfectly clean EOF — indistinguishable from a finished command — so committing on EOF would silently store a truncated database dump as a successful backup. Only `Success` commits; a non-zero exit, a closed status channel (NoStatus), a timeout, or an IO error all abort, which kills kopia with stdin still open so no manifest is ever written. Buffers are raised to 4 MiB: kube-rs defaults its exec duplex pipes to 1 KiB (remote_command.rs MAX_BUF_SIZE), which would move a multi-gigabyte dump one kilobyte per wakeup. Backpressure still works end to end without extra machinery — a full duplex blocks the message loop, which stops reading the socket, which TCP-backpressures the kubelet, which blocks the producer's write. The dump bytes are only ever copied pipe-to-pipe: never collected into a String, logged, attached to an error, or written to a status. Only stderr is captured, capped at 8 KiB. The restore direction mirrors this and additionally requests no stdout channel at all, so restored data has no path back into the mover. Zero, several, unready, or terminating selector matches are each a distinct named failure rather than an arbitrary pick: a backup that quietly dumped a different replica than intended is worse than one that stops and says why. (cherry picked from commit aedc81b)
…pace opt-in A stream source needs pods/exec in the workload namespace. That verb deliberately does NOT go on the generic kopiur-mover role: every ordinary mover Job in the namespace runs as that ServiceAccount, so granting it there would let any backup Job run arbitrary commands in any pod in the namespace. Instead it lives on a dedicated kopiur-stream-mover ClusterRole/Role bound to its own minted SA — the same separation, for the same reason, as kopiur-snapshot-replication-mover. resourceNames cannot narrow pods/exec further: the pod name is not known until the selector resolves at run time, and RBAC has no label-selector form. The real bound is therefore the namespace, so stream sources additionally require a cluster-admin opt-in — the new kopiur.home-operations.com/stream-exec-movers namespace annotation. Without it, someone who can write a SnapshotPolicy in a namespace would effectively acquire pods/exec there without holding it, which is a separation Kubernetes makes on purpose. The gate reuses the MoverPermitted condition but carries its own reason (StreamExecNotPermitted) and Event action: sharing the privileged-mover reason would send an admin to the wrong annotation. Registered in STRUCTURAL_GATES and pinned in GATE_WRITERS, so the two existing drift tests keep doctor and the reconcilers in agreement — both failed until it was registered, which is exactly what they are for. The gate keys off the WORK SPEC rather than re-deriving from the policy, so the RBAC minted and the permission checked cannot drift from what the Job executes. Committed with --no-verify: the format-yaml hook exits 2 when every staged YAML file is excluded by its ignore rules (only .tpl templates here). fmt-check, clippy, gen-check and the full suite were all run manually and pass. (cherry picked from commit 4d1d61d)
…over A streamExec restore carries a StreamConsumerSpec on the work spec instead of a target path, and runs under the same dedicated exec-capable ServiceAccount and the same namespace opt-in as a stream backup — restoring INTO a pod is at least as sharp an edge as dumping out of one, so it gets the identical gate rather than a weaker one. (cherry picked from commit 7fec5d3)
The picked stream-restore wiring reached for a local `destination` binding that #449 replaced with the `RestoreDispatch` struct, so the streamExec gate and the `stdout` work-spec field now go through `dispatch.destination`.
Explains when a logical dump beats a volume copy (no CSI snapshots, portability, application consistency) and when it does not, why the namespace opt-in exists, what 'success' means when kopia cannot tell a finished dump from a dropped connection, and which PVC-shaped fields are rejected versus ignored and why. (cherry picked from commit b5e6f86)
The fork numbered them 41/42, which collide with our existing 41-repository-seed-from-backend and 42-repository-seed-from-repository. Our ladder ends at 44, so they land at 45/46; the two `--8<--` snippet references in docs/stream-sources.md follow.
A ClusterRepository declaring `advanceOnCount: 5` + `advanceOnSizeMiB: 2`
alongside a legal `minDuration` landed NONE of the three. Nothing in kopiur
drops the fields — `EpochParametersSpec::from_api` copies all six, `epoch_drift`
emits all six, `SetParametersArgs::args()` renders every flag. kopia refuses
them: `repository set-parameters` MERGES the declared flags into the
repository's existing parameters and validates the whole resulting set, so one
out-of-range value refuses the entire call and discards every other parameter
in the same apply.
Move those ranges to admission, in a pure `epoch_floor_errors()` shared by both
repository kinds:
* `advanceOnCount >= 10` — "epoch advance on count too low"
* `advanceOnSizeMiB >= 1` — "epoch advance on size too low"
* `checkpointFrequency >= 1` — "invalid epoch range compaction period"
* `deleteParallelism >= 1` — kopiur's own floor; kopia never validates it
* `refreshFrequency <= 80m` — kopia needs `cleanupSafetyMargin >= 3x` it and
kopiur cannot set the margin, so it stays at kopia's 4h default
* `minDuration >= max(10m, 3 x refreshFrequency)`, defaulting the refresh to
kopia's untouched 20m — which is why `minDuration: 10m` alone ALWAYS fails
on a live repository ("epoch refresh period is too long"), and why the fix
offers both branches: use 60m, or declare a smaller refreshFrequency with it.
Every floor and quoted error text was verified against the pinned kopia 0.23.1
binary, not read off the source. Two findings the messages now carry: kopia's
CLI silently treats `0` on these integer flags as "flag not set" and reports
`no changes` (success, nothing applied), and `--epoch-min-duration=10m
--epoch-refresh-frequency=3m` genuinely succeeds where `10m` alone does not.
Also names the floors on `health::FIX_EPOCH_PARAMETERS_ACTION`, so the Warning
event's remediation label points at the same list.
Covers the acceptance criteria, written to fail for the right reason: byte-identity proved by reading the restored file back out of the target pod (a phase-only assertion would pass on an empty capture), a producer that emits real bytes and THEN exits non-zero which must leave nothing restorable, selector mismatches, the namespace gate parking and then releasing, and a leak check that greps status, Events and mover logs for a distinctive payload sentinel. (cherry picked from commit c0fe4a0)
Rejecting PVC-only fields on a stream source used `is_some()`, which is wrong for any field carrying a CRD schema default: the API server materializes those onto every source before admission runs. `readOnly` defaults to true and `sourcePathStrategy` to PvcName, so `is_some()` was ALWAYS true and every valid stream policy was rejected with 'readOnly does not apply to a stream source' — naming a field the user never wrote. Caught by applying the feature to a real cluster, not by the unit tests, because the tests built Source literals directly and so never saw the defaults. The new tests go through the YAML→JSON→typed bridge with the defaults spelled in, which is what the API server actually sends. Now only values that would MEAN something are refused: readOnly: false (there is no mount to make writable) and a non-default sourcePathStrategy. A materialized default is inert and indistinguishable from absence, so it is accepted. acknowledgeLiveMutation has no schema default, so its presence is still real user intent and still rejected. (cherry picked from commit 7f8ec7f)
The scenarios flip the namespace annotation that gates stream-exec on and off to prove the gate works, so they cannot share a cluster: another binary's Snapshot could be parked or released by an edit that was not its own. (cherry picked from commit 33c28f3)
The operator-authored CEL rule in the Restore CRD still named only pvc/pvcRef/populator, so the API SERVER rejected every streamExec Restore before the webhook or any reconciler saw it — with a message listing only the old variants. The Rust enum being exhaustive does not help here: the rule lives in the generated schema, so a variant can compile, pass every test, and still be unusable. Found by applying a streamExec Restore to a real cluster. Added restore_target_cel_rule_lists_every_variant, which drives its expectation off kind_str() over a value of every variant, so a future variant fails in the test suite with instructions instead of in someone's cluster. (cherry picked from commit 26e5297)
`epoch_drift` divided the OBSERVED byte threshold by MiB and compared the quotient, which truncates. A repository holding 10 MiB + 1 byte reads back as `10`, so a declared `advanceOnSizeMiB: 10` looked converged and kopiur never corrected it — a declared value that silently never applies, which is the same class of bug as #458's headline symptom. Compare in bytes instead: kopia stores the threshold as `declared << 20`, so `desired * MIB` is bit-for-bit what a converged repository reports and the comparison converges exactly in both directions (one byte over AND one byte under are now drift). `saturating_mul` keeps a pathological declared value from overflowing into a wrapped negative. `observed_epoch` now rounds the mirror UP (`bytes_to_mib_ceil`, hand-rolled because `i64::div_ceil` is still unstable). Rounding down would print exactly the value the user declared while the repository held something else — re-introducing the same silent convergence inside `status`. No new fields.
…PathStrategy `policy_own_source_path` gives a non-PVC restore target (today `target.streamExec`) the policy's OWN kopia path, since the per-PVC derivation #449 added answers a question a stream restore never asks. Naming `SourcePathStrategy` in a consumer crate flipped the wiring ratchet (`SnapshotPolicy.spec.sources.sourcePathStrategy` read as wired while its allowlist entry still stood), so the helper mirrors step (3) of `restore_source_path` — `sources.first()` plus `strategy_for`, including its zero-source legacy tolerance — instead.
…icy (#464) An ad-hoc one-shot `Snapshot` could not adjust mover settings without editing the shared, GitOps-managed `SnapshotPolicy` that every scheduled run also uses. `Restore` already had `spec.mover`; `Snapshot` now does too. - `MoverSpec::merge_over(base)` overlays the HIGHER layer field-wise, reusing the existing primitives so a per-run override can never resolve differently from a per-recipe one: `merge_context_pair` for the (container, pod) security-context pair (field-wise + identity promotion, so a policy's container `runAsUser` cannot shadow a snapshot's pod-level one), `merge_resources_opt` for resources, `CacheDefaults::merge` for cache, `.or()` for the whole-value knobs. - `snapshot::effective_backup_mover(snapshot_spec, policy_spec)` is the one pure answer shared by the controller and the webhook. `moverDefaults` still enters underneath it inside `resolve_mover`, giving `hardened ⊂ moverDefaults ⊂ inherited ⊂ policy.mover ⊂ snapshot.mover`; the pair merge is associative, so pre-folding the top two layers is identical to a flat merge. - `validate_backup` runs `validate_mover` + `forbid_snapshot_inherit` on it. `inheritSecurityContextFrom.pvcConsumer` is deliberately ADMITTED: the validator is client-free and cannot see the policy's sources, so the controller resolves it and parks the run with a named reason when it cannot. - Controller computes the merged mover ONCE and threads it through the inherit resolution, the privileged-mover gate, `resolve_mover`, the recorded `kopiur-meta` identity and `report_inherit_outcome` (which must judge THIS run's context, not the policy's). - The policy-owned PERSISTENT cache PVC deliberately does NOT take the per-run layer: the claim is named per-policy and shared by every sibling Snapshot. Extracted as `persistent_cache_spec(repo, policy)`, which takes no `Snapshot` at all — the absent parameter is the guarantee — with a test proving a snapshot-level `mover.cache` cannot move it while the per-run budgets do honor it. - Webhook `handle_snapshot` passes the merged mover to `secctx::backup_warnings`. Documented behaviour change, covered by a test: a Snapshot that ADDS an inherit on top of a policy that pinned a UID now silences the warning the policy alone emitted, because every inherit mode resolves at reconcile time. Also: a field-count + exhaustive-destructuring change detector so an eighth `MoverSpec` field cannot be silently forgotten by the merge.
- `docs/backups.md`: a new `mover` section under Snapshot spelling out the three-layer precedence, with the two things a per-run override deliberately cannot do — resize a policy-owned persistent cache PVC, or bypass the privileged-mover namespace opt-in — plus the backup-side inherit rules. - `deploy/examples/09-mover-permissions.yaml`: a one-shot Snapshot that overrides the recipe's UID, resources, cache budgets and Job TTL, snippet- included by the new section. The header now describes three layers, not two. - `docs/security-context.md`, `docs/permissions.md`, `docs/restores.md`: table rows and cross-links so each page points at the per-run layer from where a reader actually lands.
`epoch_floor_errors` quotes kopia's own error strings back to the user, and an
unchecked quote is a lie waiting to happen. This runs the pinned kopia 0.23.1
binary and asserts every one of them, plus the behaviors the admission messages
assert about kopia:
* each floor's exact text ("epoch advance on count too low", "epoch advance on
size too low", "invalid epoch range compaction period", "minimum epoch
duration too low", "invalid cleanup safety margin, must be at least 3x epoch
refresh frequency");
* MERGE-THEN-VALIDATE: `{advance-on-count: 5, checkpoint-frequency: 9}` is
refused as a whole and the LEGAL checkpoint-frequency is discarded with it,
verified by reading `repository status --json` back — this is exactly how the
reporter lost their legal `minDuration`;
* `minDuration: 10m` alone fails on the 3x rule against the untouched 20m
refresh, and the identical 10m succeeds with a 3m refresh beside it — the two
branches of the webhook's fix text;
* a `0` is silently DROPPED by kopia's CLI (exit 0, `no changes`, value
unchanged), which is why the webhook rejects it instead of reading it as
"leave this alone";
* the reporter's corrected declaration (count 10 + size 2 MiB) applies and is
visible, with the size stored as `2 << 20` — the premise of the byte-exact
drift comparison.
Two findings worth their own asserts. First: a NEGATIVE integer never reaches
kopia's validator through kopiur at all — `SetParametersArgs::args()` emits
`["--flag", "value"]` and kopia's parser reads the leading `-` as another flag
("expected argument for flag '--epoch-advance-on-size-mb'"), so kopiur's argv
rendering, not kopia's validation, decides that failure. The range errors are
therefore also asserted through `=`-joined argv, which is the only way to put a
negative in front of kopia's validator.
Second: `maintenance set --owner` leaves `schedule.runs` byte-identical — held
for both the `--owner me` no-op and a genuine owner change, asserted on kopia's
own emitted JSON because the typed model drops `runs`. A later before/after
run-delta depends on this.
Also reflows the `EpochParameters` floors paragraph into a list: a doc line
beginning with `>=` reads as a markdown blockquote and trips clippy's
`doc_lazy_continuation`. Source-only — schemars takes only the first paragraph
of a struct doc, so the generated schemas are unchanged (the per-field floor
text, which IS user-visible, landed with the validator).
A `SnapshotPolicy` whose source is a `pvcSelector` resolved a PATHLESS kopia identity for its verify run (`user@host:`), because `verify_identity` derived the source path from `sources[0]`'s own `pvc`/`nfs` — and a selector source has neither. Deep verify then failed with `deep verify found no snapshot to restore for source path ""`. The quick tier was broken too, and silently: kopia's `ParseSourceInfo` falls through to treating `user@host:` as a RELATIVE filesystem path, lists zero manifests and exits 0. Every multi-PVC policy's quick verification has been passing falsely and stamping `lastVerified`. Verification is now one cell per (READY repository x verification member): * `verify_members` mirrors `expand_sources` exactly — a policy mixing a plain `pvc:` source with a `pvcSelector` yields ONLY the selector members, built on `EffectiveSource::kopia_source_path` + `strategy_for`. A selector matching nothing yields no members: warn, spawn nothing, stamp nothing (mirroring `SlotMintPlan::NothingMatched`), never a pathless run. * `verify_identity_for` delegates to `config_identity_for_path`, the #443 restore-fan-out kernel, feeding the derived path in as a verbatim `sourcePathOverride`. * `VerifyTarget.member6` is orthogonal to `repo_key`: the repository dimension still drives `repo_tag6`, `VERIFY_REPO_LABEL` and the projected-credentials prefix (members under one (policy, repository) share that Secret), while the member drives the `-m<6>` Job-name segment, the new `VERIFY_MEMBER_LABEL` and the stamp key's member segment. Cron seeds become `<uid>|<path>` so members do not all fire in the same second. * New `VerifyOp.stamp_key` (serde default; `repository_key` is NOT repurposed — it also names the Job and the creds prefix, and in-flight Jobs carry the old shape, so the mover falls back to it). * The stamps prune is keyed by `rsplit_once('#')`: it used to keep only keys equal to a bare repo key, so it deleted every member stamp on the next reconcile (and nulled the whole map each pass for a single-repo policy), which would have kept `lastVerified` frozen and re-fired deep verify every slot. * The fold is the MIN over (repository x member) cells; `status.verification[repo]` is the MIN over that repo's members, so a partially-verified repository never looks fully verified. * Two hoists out of the per-repository `verify_step`: `match_pvcs` and the #168 discovered-snapshot probe, both once per reconcile / per repository. Upgrade safety: the single-flight LISTs WITHOUT the member label and filters client-side, so an in-flight Job minted by an operator predating the label still holds every member's slot — narrowing the LIST would have spawned N Jobs beside it, i.e. N+1 concurrent deep scratch restores. Entering fan-out also clears the flat `lastVerified` once, since the pre-fix value came from the pathless false pass. Cost guard: quick members run concurrently, deep members SEQUENTIALLY — each provisions its own `deep.capacity` of scratch, so a 4-PVC policy at 500Gi would otherwise ask for 2 TiB of ephemeral storage at once. Two visible behaviour changes, both correct: a policy with sources but no derivable path now resolves `/data` (what the backup recorded it under) instead of `""`, and an unresolvable CEL identity PARKS the policy instead of being masked as a `kopiur-verify@<ns>:` sentinel that verified nothing and reported success. A zero-source legacy policy still passes `None` and keeps its pathless identity byte-for-byte. Refs #456
…456) * `crates/e2e/tests/verification_fanout.rs::verification_fans_out_per_matched_pvc` asserts the POSITIVE shape a pre-fix operator cannot produce: two verify Jobs with distinct member labels, each carrying its own derived kopia source path and `#<member6>` stamp key in its embedded work spec, two stamps in `status.verificationStamps`, and a flat `lastVerified` equal to their MIN. Asserting merely "a verify succeeded" would have passed before the fix, since the pathless quick verify exited 0. It rides the `multi-pvc` CI shard, not `reshape-async`: it needs TWO labelled source PVCs and `reshape-async` provisions one. Own repo subpath `vfyfanout` in `REPO_SUBPATHS` AND the mise `node-seed` list (the lockstep test enforces the pair), and a `backup=` label value no other scenario uses. * The #168 verification gate is now PER MEMBER (`member_has_success`): a PVC that joins the selector later stays gated until its OWN first backup succeeds. A per-policy gate would have reopened the #456 false pass for it — its quick verify would filter on a path with no manifests, match zero and exit 0. A path-less (non-selector) member keeps the per-policy/per-repository input byte-identically, and a child from an older recipe shape derives no path so it never false-matches. * `verify_step`'s spawn arm moves into `spawn_if_slot_free` to stay under the cognitive-complexity ratchet (29, unchanged). * docs/backups.md gains "verification with a `pvcSelector`", covering the per-member Jobs/stamps/MIN, the quick-concurrent vs deep-sequential rule, the per-member gate, the upgrade warning that a pre-fix `lastVerified` is not evidence, and the two visible behaviour changes. Refs #456
…assumptions (#458) Review round 1. The headline hole: `refreshFrequency: 0s` was still ADMITTED — the one epoch field that got a ceiling but no floor. `parse_go_duration` returns a real zero for "0s"/"0", and kopia's `setDurationParameter` is `if v == 0 { return }`, so the flag never reaches the format blob, `anyChange` stays false, and the command exits 0 with `no changes`. Verified on 0.23.1: the repository keeps its 20m. Downstream that is worse than a no-op — `EpochParametersSpec::from_api` renders "0s", `dur_drift` compares 0 against the live 1_200_000_000_000 forever, so kopiur re-issues the flag on every bootstrap (invalidating every other client's cached format blob each time) while `status.parameters.epoch.refreshFrequency` never agrees with `spec`. That is verbatim the failure mode this task exists to close. `minDuration: 0s` already fell to the 10m floor and the four integers to their `>= 1` floors, so this was the last field; the new test asserts the minDuration case too so a refactor cannot reopen it. Both DERIVED bounds stated an assumption as fact, and each can reject a call kopia would accept. The 80m ceiling assumed the 4h cleanupSafetyMargin default, but kopia's CLI exposes `--epoch-cleanup-safety-margin` — raise it to 23h and a 90m refresh is genuinely legal (now pinned in the integration test). The minDuration bound assumed the 20m refresh default, which is reachable through kopiur itself: apply `{minDuration: 60m, refreshFrequency: 5m}`, drop refreshFrequency from spec (a spec `None` means don't touch, so the live 5m stays), and `minDuration: 30m` becomes legal. Both bounds stay — they are right for every repository kopiur configured, and a pure validator cannot read live state — but the messages now say kopiur ASSUMES the default BECAUSE of that, and point at `status.parameters.epoch.cleanupSafetyMargin` / `.refreshFrequency` where the live values are already mirrored. The module doc's unknowable-rules paragraph now covers all three. Also: * `suggest_refresh` is capped at the 80m ceiling. `{minDuration: 300m, refreshFrequency: 200m}` suggested "100m or less", which the ceiling then rejects — advice that fails the next admission is worse than none. * the `min < 10m` branch named the derived floor while explaining only the absolute rule. Both violated rules are now reported when both bite, and the fix offers the route that works (raise to 10m WITH a small refreshFrequency); "lowering the refresh cannot help" only held with minDuration pinned, which is exactly what the user is being asked to change. With the 3x rule already satisfied, the message still blames only the absolute rule. * `ZERO_IS_DROPPED` is gated on `v == 0`, so `advanceOnCount: 5` no longer carries ~230 characters about what kopia does with a zero. * doc nits: `validate_repository_parameters` said "every count must be positive", which understated the floors; the C4 header named `epoch_floor_errors` as a public path when it is private. The integration test grows two steps for the behaviors the new messages assert: a zero DURATION is dropped exactly like a zero integer, and a 23h margin really does make 90m legal. The second sends `minDuration: 24h` alongside, because step (6) had left it at 10m and a 90m refresh must also clear the 3x rule — the merge semantics again, this time admitting a call either value alone would fail. Not in scope, per review: the `args()` two-argv rendering, and the epoch-disabled repository (kopia's Validate() returns nil when !Enabled).
`Maintenance.status.{quick,full}.lastContentReclaimedBytes` was hardcoded to
`0`, so a run that freed gigabytes reported "reclaimed nothing" — and the
`kopiur_maintenance_reclaimed_bytes` gauge published that zero.
`kopia maintenance run` prints nothing machine-readable, but it DOES append a
per-task run history to the schedule blob that `kopia maintenance info --json`
reports (`schedule.runs`), which the model previously dropped entirely. Model it
and diff the pre-run `info` the lease decision already reads against a post-run
one.
The figure counts only tasks that issue real blob deletes:
`quick/full-delete-blobs` (`deleteUnreferencedPacksStats.deletedTotalSize`),
`delete-superseded-epoch-indexes` and `cleanup-logs`. It deliberately excludes
`snapshotGCStats.deletedContentSize` — `snapshot-gc` only MARKS contents deleted
in the index and frees no storage, so reporting it would show tens of gigabytes
"reclaimed" while the bucket did not shrink.
Details that are load-bearing, each with a test:
* kopia PREPENDS runs and caps the history at 50 per task, so the delta keys on
`start`, never on index or length (a saturated history does not grow).
* a fresh repository emits `"runs": null`, which `#[serde(default)]` alone
rejects — hence the explicit null-as-default deserializer.
* `RunInfo.Success` is `omitempty`, so a FAILED run has no `success` key, and
`extra` is only populated on success: a failed run contributes nothing.
* `extra[].data` is a nested object whose shape depends on `kind` and is
sometimes `null`, so it is a plain `Value` with typed accessors rather than a
tagged enum — an unknown kind from a newer kopia must still decode.
* `None` (key omitted) means "not measured"; `0` means a measured zero. A quick
run on an epoch-enabled repository reports `None`, because
`runQuickMaintenance` short-circuits into epoch tasks that delete no blobs.
* a FULL run stamps the figure only on `full`; the subsumed `quick` clock gets
an explicit null so a full run's bytes are never misattributed to quick.
The measurement is strictly best-effort: a failed post-run `info` degrades to
"not measured" and never fails a Job whose maintenance already succeeded.
Fixtures are real `kopia maintenance info --json` captures taken before and
after a genuine `maintenance run --full` on kopia 0.23.1, and a source-scan test
pins that only the mover writes these keys.
All four fixes from review round 1 landed together.
IMPORTANT-1 — a per-run `mover.cache` was silently ignored for the EPHEMERAL
cache volume, while six user-facing statements promised it worked.
`resolve_cache_volume` decides BOTH provisioning modes from the single effective
`CacheDefaults` it is handed, and the backup caller handed it the policy-only
spec. So a per-run `capacity`/`storageClassName`/`mode` was dropped for the
ephemeral volume too, and a per-run `capacity` against a policy that set none
left the run on an `emptyDir`; only the content/metadata budgets took the
per-run layer.
Fixed in the code, not the docs — the docs described the behaviour we want, and
an ephemeral cache volume is genuinely Job-scoped (a generic ephemeral volume
bound to this run's pod and GC'd with it), so there is no shared state for a
per-run size or class to disturb. `cache_volume_spec(repo, policy, run_mover)`
now splits the decision by who owns what:
- The MODE comes from `policy_cache_spec` alone (renamed from
`persistent_cache_spec`, which now also names what it guards), defaulting to
`Ephemeral`. Exhaustive `match` on the mode, so a new provisioning mode must
declare whether it is shared.
- `Persistent` → the policy's spec verbatim. The per-run layer still
structurally cannot reach `ensure_cache_pvc`: `policy_cache_spec` takes no
`&Snapshot`, so an ad-hoc Snapshot cannot resize or re-class the policy-named,
sibling-shared PVC. That guarantee is preserved, not weakened.
- `Ephemeral` → the MERGED `capacity`/`storageClassName`, with `mode` pinned
back to `Ephemeral` so a per-run `mode: Persistent` can never mint the shared
PVC through this path.
The old regression test only exercised `Persistent`, which is exactly why it
could not catch this. Added `a_snapshot_mover_cache_does_size_and_class_an_
ephemeral_cache_volume` — (a) per-run size/class winning under Ephemeral, (b) a
per-run capacity upgrading a cache-less policy off `emptyDir`, (c) a per-run
`mode: Persistent` staying Ephemeral, (d) nothing-anywhere staying `None` — plus
`an_ephemeral_per_run_capacity_reaches_the_mover_volume_with_no_api_call`, which
drives the real `resolve_cache_volume` through a `tower::service_fn` client that
fails EVERY request, proving both that the per-run capacity reaches
`CacheVolume::Ephemeral` and that the ephemeral path performs no API call at all.
The `Persistent` case still asserts the hostile override cannot move the PVC
spec. All six statements were rewritten to the corrected, narrower truth.
IMPORTANT-2 — the e2e scenario. `crates/e2e/tests/mover_config.rs` now exercises
all three layers: a third Snapshot on the same policy sets
`spec.mover.securityContext.runAsUser: 3001` over the recipe's 3000 over
`moverDefaults`' 65532. The overlay is bare spec fields (a `{"spec": ...}`
wrapper is dropped by serde and pruned by the apiserver), the CR is read BACK and
the field asserted present — a stale CRD would prune it and every later
assertion would be satisfied by the recipe's own 3000 — and the pod `fsGroup`
must still come from `moverDefaults`, proving the per-run layer merges rather
than replacing the chain.
MINOR-3 — narrowed the webhook warning. `assessable_sources` uses the pinned
source when `spec.source` carries one, the full list only when it does not. A
fanned-out Snapshot over an N-source policy no longer emits N-1 warnings naming
PVCs the run never touches, once per schedule slot. An out-of-range pin assesses
nothing rather than guessing at whichever source now sits elsewhere: the policy
shrank out from under the pin, which is the reconciler's named terminal failure
to report.
MINOR-4 — example 09 said "`generateName` … so you can repeat this apply";
`kubectl apply` rejects a nameless object. Now "`kubectl create` (not apply)",
matching the sibling examples and the field reference.
Regenerated CRDs + field-reference and refreshed the insta snapshot (the
`SnapshotSpec.mover` field doc changed).
…hing (#464) A `mover.inheritSecurityContextFrom.workloadSelector` that matches NO pod — the workload is scaled to zero, which is precisely when a quiesced backup is most useful — used to surface as a generic, condition-less `Error::MissingDependency`: a fast (30s) retry loop forever, with nothing on the object naming the selector that found nothing. Replace it, for the LIVE-pod modes only, with a typed hold: * `kopiur_api::consts::SECURITY_CONTEXT_RESOLVED_CONDITION` + `INHERIT_SOURCE_MISSING_REASON`, registered as `gates::INHERIT_SOURCE_MISSING_GATE` (SnapshotOrRestore / False / Fail) so `kubectl kopiur doctor` reports the park instead of passing a cluster whose backups are quietly wedged. * `Error::InheritSourceMissing`, classified `Structural` (300s), precedent `Error::MissingRecordedIdentity`. Deliberately a new Error rather than a new `InheritOutcome` variant: an outcome means "we resolved contexts and are proceeding", and it would flow into `recorded_meta`, which could stamp a kopia `kopiur-meta` tag for a run that must not happen. * One generic writer, `io::park_on_inherit_source_missing`, called from the `Snapshot` and `Restore` resolution arms: live conditions re-read (a `conditions` patch REPLACES the array), `upsert_gate` from the registry row, `phase: Pending`, and a Warning Event fired only on a real transition. * `io::inherit_source_missing_message` — pure, byte-stable, unit-tested verbatim — states what happened, why the run is HELD rather than run as the wrong UID, and the three levers that clear it, and always NAMES the selector (or the source PVC) that resolved nothing. `SecurityContextInherited` is untouched and stays advisory and unregistered. Registering one of its five reasons would make `doctor`'s `first_gate` report the other four as "the operator is newer than the plugin", and the registry pins one severity per condition+scope so they cannot be registered as `Warn` alongside a `Fail`. A regression test in `gates.rs` asserts it stays out. The existing fallback is unchanged: when an explicit `mover.securityContext` pins a `runAsUser`, that context still stands in and the run still proceeds with `InheritFallback`. The restore-only `snapshot` mode and a `pvcConsumer` with no source PVC keep their current propagation — `live_inherit_source_label` returns `None` for both, so the `MissingRecordedIdentity` hold still catches the former and the latter is not told to "scale the workload up".
) `map(|p| p.clone())` -> `cloned()`, a `let_and_return` in `feed_from_pod`'s transfer future, and a runtime `assert!` over two constants moved into a `const` block. All three are on the adopted stream-source code and fail `clippy --all-targets -- -D warnings`; clearing them first keeps every later commit in this series green.
…ry identity site (#451) Five identity sites re-derived a policy's kopia source path inline from `sources.first()`'s `pvc`/`nfs`. A `stream` source has NEITHER, so each of them resolved a pathless-or-`/data` identity while the backup path recorded `/stream/<fileName>`. Every consequence was silent: * a `Restore` with `source.fromPolicy` looked for `user@host:/data`, found nothing, and reported `SnapshotNotFound`; * quick verification verified the EMPTY path — kopia reads `user@host:` as a RELATIVE filesystem path, matches no manifests, and `snapshot verify` exits 0, so a verification that looked at nothing reported `Verified=True`; * the collision webhook collapsed every stream policy onto one pathless identity, raising collisions that were not real and missing the real one between a stream policy and a PVC policy at the same `username@hostname`; * `adoption::identities_match` compares source paths field-by-field, so a DISCOVERED stream snapshot could never be adopted; * `status.resolved.sources[].sourcePath` was blank for a stream source. Fixed by routing all six identity sites (the five broken ones plus the backup path that was already right) through ONE derivation, `expand::identity_source_path`, which is `EffectiveSource::kopia_source_path` under the governing source's `strategy_for` — byte-identical to the old inline chain for every pvc/nfs/selector/override shape, and additionally correct for `stream`. A seventh site cannot drift, because `IdentityInputs` no longer ACCEPTS the raw pieces to re-derive from: `pvc_name` + `default_source_path` + `source_path_override` collapse into a single `source_path` that the caller must have derived. The kernel records it verbatim. Also fixes `policy_own_source_path` (adopted in the conflict resolution), which mirrored step (3) of `restore_source_path` WITHOUT step (3)'s `!has_selector` precondition. For `target.streamExec` + `source.fromPolicy` naming a `pvcSelector` policy with a shared `sourcePathOverride`, it resolved that override, matched the newest snapshot of an ARBITRARY matched member, and the mover would have piped that volume's bytes into the user's `psql` stdin — the exact configuration `restore_source_path` step (5) refuses, and one nothing upstream prevents (`validate_restore` never constrains the referenced policy, and cross-CR validation is unavailable at admission). It now fails closed on the effective source's SHAPE with what/why/fix text. Tests: 10 that fail without the fix — one per identity site plus the shared derivation, the adoption path's `identities_match` equality, the fail-closed guard, an explicit-`sourcePath` override, the zero-source legacy tolerance, and a negative guard that a PVC target still routes through the per-PVC `restore_source_path` derivation (the whole `from_policy_identity` arm a stream restore takes previously had zero coverage in any tier). Also moves the 15-line #443 rationale back onto `from_policy_identity`, where it belongs: the conflict resolution had reattached it to `policy_own_source_path`, so the canonical explanation of the fail-closed rule documented the one function that did not obey it. Three type-safety repairs ride along, because they are in the same two files and are the same class of defect (an `_ =>` where CLAUDE.md requires an exhaustive match): * `snapshot::build::build_backup_run`'s outer `match` on `SourceShape` had a `_ =>` arm; it now names `Pvc(_) | PvcSelector(_) | Nfs(_)` explicitly, so a new source kind must decide what kopia records and what is mounted before it compiles. The inner `(eff.pvc, source.nfs)` resolution is unchanged. * `work_spec_uses_stream` was `_ => false` over `Operation`. That decision MINTS `pods/exec` RBAC, so a silent arm is an RBAC defect in either direction. Now exhaustive over all eleven operations, with the `Restore` arm reading `RestoreOutput` — the same resolver the mover uses to decide where the bytes go, so the grant and the behaviour cannot diverge. * `drive_direct_restore` resolved its target in two matches, the second with `_` in the target position, so a future non-PVC variant yielding `None` compiled clean and produced a runtime `Invariant` error. The two-step existed for a real borrow reason (`RestoreDestination` borrows a `&str` that must outlive the awaits); a new owned `DirectDestination` intermediate carries the `String`, collapsing it to ONE exhaustive match plus a total, unreachable-free conversion.
Review round 1. CRITICAL: the #456 false pass was still reachable — the fix so far removed its guaranteed trigger, not the class. `run_verify_flow`'s QUICK arm resolved the newest manifest for the run's identity with a `_ =>` catch-all that fell through to `verify_ok_body` and stamped `status.lastVerified`. Since `kopia snapshot verify --sources <spec>` EXITS 0 when its filter matches zero manifests, any quick verify naming a path the repository has no snapshot for still reported success. Two reachable routes with no operator error: flipping a selector's `sourcePathStrategy` or changing a `sourcePathOverride` (the controller gate re-derives from the CURRENT policy, so existing children appear to cover the new path), and snapshots being gone externally while a Succeeded Snapshot CR is still in the reflector window. That made the controller-side gate an INFERENCE about coverage, and the only safety property. The arm is now three EXHAUSTIVE cases with no `_`: * `Ok(Some(entry))` -> the `successExpr` environment, via the new pure `quick_coverage` (counts saturate rather than wrap); * `Ok(None)` -> TERMINAL `MoverError::VerifyNoSnapshot`, exactly as the deep arm already treats it, with a status message naming the identity and pointing at a strategy/override change as the likely cause; * `Err(_)` -> TERMINAL, and a DIFFERENT verdict: the integrity result may be sound but what was covered cannot be proven, so success must not be claimed. The catch-all conflated the last two and reported success for both. That converts the whole class into a loud failure and demotes the controller gate to an optimisation. IMPORTANT: a one-member selector policy is no longer collapsed to the flat shape. Keeping it there left the pre-#456 false-pass `lastVerified` in place, which is not just a displayed lie: via `verify_cell_anchor` it ANCHORS `due_tier`, deferring the first real verification by up to a full period of the coarsest tier — a month for a monthly deep drill. Running it in stamps-map mode makes the existing `(None, prior_flat) => Null` arm clear it on the first reconcile with no new state, and it also means growing a selector 1 -> 2 no longer discards the verified member's real result. Byte-identity was already gone for this shape (a different identity source path IS the fix) and Job-name continuity is moot, since a legacy unlabelled in-flight Job blocks the new name anyway. Non-selector policies are untouched. IMPORTANT: the deep single-flight no longer blocks on ANY in-flight verify Job. It blocked on a sibling member's short QUICK run, and with N members and a tight quick cron the odds of that at deep-due time scale with N — while a held deep also suppressed that member's own quick, since `due_tier` prefers deep. Deep now blocks only on non-terminal DEEP Jobs plus unlabelled legacy ones, using the tier the Job name already carries (`-vfy-q-` / `-vfy-d-`, last occurrence, so a policy whose own name contains the marker still reads correctly; an unreadable name is "cannot tell" and blocks). The durable cross-reconcile sequencing property is preserved. Smaller findings: * The generated `lastVerified` / `verificationStamps` / `RepoVerification` field docs said "keyed by the normalized repository key" and "Never written for the single-repo shape", both false now. Rewritten around the (repository x source) cell model and regenerated — no CRD TYPE changed, which is exactly why `gen-all --check` passed and this slipped. * `sourcePathOverride` + `pvcSelector` is now REFUSED at admission. It is one literal path, so it collapses N matched PVCs onto one kopia source; `expand_sources` already refused it at run time, leaving the shape admissible but inert — and verification then legitimately derived a member for a path nothing was ever written to. The message names `sourcePathStrategy` as the supported knob. * `verify_flight_selector` extracted as a pure fn with a test asserting it can NEVER contain `VERIFY_MEMBER_LABEL` — the one composition whose regression would silently produce the N+1-concurrent-scratch-restores failure the design exists to prevent. * The `#`-free repo-key invariant `parse_stamp_key` relies on is now pinned by a test over real `repo_key` output plus a `debug_assert` in `stamp_key`. Without it a future key format admitting `#` would silently delete a LIVE stamp every reconcile and freeze `lastVerified`. * `verify_members` no longer `continue`s past a pathless member; it mirrors `expand_sources`' own `/data` fallback byte-for-byte. Unreachable today, but a silent drop from a COVERAGE calculation is the wrong default. * docs/backups.md: dropped the "/data instead of empty" behaviour-change claim (unreachable from the verify path — a `Source` is exactly one of pvc/pvcSelector/nfs), and documented that a verification covering nothing is now a failure, the deep-vs-quick blocking rule, and the selector override refusal. Refs #456
A repository's index-blob count was only ever refreshed by a bootstrap Job, so
the `IndexBlobHealth` condition kept quoting a PRE-compaction number — "8000
content-index blobs, maintenance is not compacting them" — for as long as a day
after maintenance had actually compacted it to a few hundred. Compaction happens
during maintenance, so the count right after a run is the freshest truth there
is.
The mover now re-counts (`kopia index list`) after a successful run and publishes
`Maintenance.status.observedIndexBlobs: {count, observedAt}`. The repository
reconcilers adopt it in their steady-state arm when it is newer than their own
observation, folding it through the existing `reconcile_index_blob_health` and
the same guarded status patch.
Comparing the two observations needs a timestamp neither side had, so
`StorageStats.indexBlobCountAt` joins `indexBlobCount` and is stamped by all
three of its existing writers. It is deliberately NOT `lastObservedAt` (the
catalog scan's) or `health.lastProbeAt` (the backend probe's) — the index-blob
count has its own writers and its own cadence.
Correctness details, each with a test:
* the recount is strictly best-effort and bounded by an explicit timeout. The
repository this measurement is FOR is the unhealthy one with thousands of
blobs, i.e. exactly where the LIST is slowest — and a hung LIST must never
turn a maintenance run that already succeeded into a failed Job.
* `None` (no recount) leaves the bootstrap value standing. It is never zero,
which would read as a perfectly healthy index and silently clear a real
warning. That covers both an unsynced informer AND the genuine scope gap: the
Maintenance informer is namespace-scoped under a namespaced install, while a
ClusterRepository's Maintenance sits in `spec.maintenance.namespace`.
* the freshness comparison only ever moves the count FORWARD: a bootstrap that
ran after the last maintenance keeps its own number, ties go to the
repository (so the steady state is a byte-stable no-op, not a hot loop), and
an unparseable timestamp loses rather than winning by accident.
* the bootstrap finalize arms stamp the JOB's completion time, not `now` — they
re-run on every reconcile while the finished Job lingers, and re-stamping the
same observation as "just now" would let a stale count outrank a fresher
recount.
* the in-process filesystem arm re-counts every reconcile, so its stamp only
moves when the count moves; otherwise the status patch would always differ
and re-trigger its own reconcile.
No new watch is needed: both repository controllers already watch Maintenance,
so the mover's status patch wakes the reconcile that reads the recount.
… stem (D1)
`kopiur.moverName` was `trunc63(fullname + "-mover")` and `kopiur.streamMoverName`
was `trunc63(fullname + "-stream-mover")` — the cap applied AFTER the distinguishing
suffix. The controller only ever sees the first of those (`KOPIUR_MOVER_CLUSTERROLE`)
and derived the others by stripping `-mover`, so the two sides diverged, and at the
top of the range the chart collided with itself. Two distinct defects:
* `fullname` 58-61: the env value ends `-move`, `strip_suffix("-mover")` returns
`None`, the fallback yields `base + "-stream"`, and `trunc63` folds that straight
back to `base`. The controller then minted a RoleBinding whose `roleRef` RESOLVES
— to the GENERIC mover ClusterRole. No "role not found" breadcrumb, escalation
prevention passes, the binding applies cleanly, and the stream Job runs on the
SAME ServiceAccount as every ordinary mover Job, which is exactly what
clusterrole-mover.tpl says must never happen.
* `fullname` 62-63: purely chart-side. `kopiur.streamMoverName ==
kopiur.moverName`, so the chart emitted the stream role (WITH `pods/exec`) and
the generic mover role under ONE object name. Whichever template rendered last
won; if the stream role won, every ordinary mover Job in every namespace
inherited `pods/exec`, cluster-wide. `helm template` with a 62-char
`fullnameOverride` rendered THREE ClusterRoles under one name.
A strip-then-cap fix handles 58-61 but cannot handle 62-63 — at those lengths the
distinguishing suffix is truncated away entirely, so no plain suffix scheme can
separate the names.
The scheme chosen: **cap the STEM, not the finished name.** A new
`kopiur.moverBaseName` = `fullname | trunc 36 | trimSuffix "-"`, where 36 is
63 minus the longest suffix any mover helper appends (`-snapshot-replication-mover`,
27 bytes). All three names are then that stem plus a whole suffix, and no suffix is
ever cut. Why this over a shorter suffix or a hash:
* It makes the distinctness STRUCTURAL rather than probabilistic. All three names
share one stem and differ only by suffix, so they are pairwise distinct for any
stem, at any length, with no collision argument to get wrong.
* It makes `kopiur.moverName` exactly `<stem>-mover`, so the controller recovers
the stem LOSSLESSLY from `KOPIUR_MOVER_CLUSTERROLE` and performs no truncation
at all on the chart path. The root cause was a lossy env value; this removes the
loss rather than teaching both sides to lose identically.
* A hash discriminator would have to be computed the same way in sprig and in
Rust. Nothing in the workspace has sha256, and a hermetic test could not then
reproduce what Helm computes — the failure mode that produced the fake-green
test this replaces.
`kopiur.snapshotReplicationMoverName` had no `| trunc 63` at all (a 62-char
`fullname` rendered an 89-char name), so it is folded into the same scheme; that is
where the 27-byte budget comes from. On the Rust side both derivations now go
through one `derived_mover_name`, whose truncating branch is reachable only for a
hand-set `KOPIUR_MOVER_CLUSTERROLE` and keeps the suffix whole, spending the
reclaimed room on a stable FNV-1a hash of the stem.
The pinning test was fake-green: it fed `"r".repeat(80) + "-mover"`, an 86-char base
the chart can never emit, and asserted `trunc 63` was PRESENT in the template rather
than that the two sides AGREED. It is replaced by a sweep over `fullname` 50..=63
plus adversarial shapes (a cut landing on a dash; names already ending in
`-stream`, `-stream-mover`, `-snapshot-replication-mover`) asserting three
properties: the chart's name equals what the Rust derives from the chart's mover
name; the three names are pairwise distinct; every emitted name is a valid ≤63-byte
DNS-1123 subdomain. Sprig `trunc`/`trimSuffix` are reproduced faithfully in the test,
a second test pins that the template still implements the scheme, and a third covers
the off-chart custom-name path. Confirmed against `helm template` at `fullname`
57/58/60/62/63.
Behavior change, documented in docs/upgrade.md: installs whose chart `fullname`
exceeds 36 characters get renamed mover Roles. `fullname` ≤ 36 — the default
`kopiur` and essentially every normal install — renders byte-identical names.
… document the stream-exec grant (D2, D3, D4) D4 — the privileged-mover refusal named the wrong object. The gate is correct: it evaluates the MERGED mover, so a per-run override cannot bypass the namespace opt-in. The MESSAGE hardcoded the kind to `"SnapshotPolicy"` and ended "remove the elevated securityContext/privilegedMode from the SnapshotPolicy `spec.mover`". Since #464 made a per-run `Snapshot.spec.mover` the common source of elevation, that sent the operator to a policy containing nothing elevated — so they took the only other fix on offer, `kubectl annotate namespace <ns> privileged-movers=true`, PERMANENTLY granting every future backup in that namespace the right to run privileged in order to unblock one ad-hoc run. `io::attribute_elevation` now picks the most specific authored layer that is elevated on its own — invocation (`Snapshot.spec.mover`) → recipe (`spec.mover`) → repository (`spec.moverDefaults`) — with a `Composed` arm for an elevation that is inherited from a workload pod or assembled across individually-benign layers, which is never silently attributed to the recipe. An enum with an exhaustive `match` at each call site, so a future layer cannot be added without every refusal deciding what it names. `privileged_mover_message` takes the carrying object's kind, name AND field path, and states the spec-edit fix FIRST, labelling the annotation as the namespace-wide standing grant it is. Applied to the Snapshot gate (three layers) and, for the same `moverDefaults` misattribution, to the Restore and Maintenance gates. Purely a message change — attribution is advisory and cannot widen or narrow what is refused. D3 — `docs/rbac.md` opens "Everything Kopiur is allowed to do in your cluster, in one place. Use it for security review", lists three principals, and omitted `kopiur-stream-mover` (`pods` get/list + `pods/exec` create/get) entirely, while its far less sensitive sibling `kopiur-snapshot-replication-mover` got a full subsection. Adds the fourth principal to the table and a matching subsection: the rules, the three things that bound `create pods/exec`, and the revoke caveat below. Also corrects the `namespaces` row, which still described only its two old purposes and not the `stream-exec-movers` check — the one that fails CLOSED when that read is unavailable. D2 — RULED: document, do not reap. `ensure_mover_rbac` applies the SA and RoleBinding with no owner reference and nothing deletes them. For the generic and snapshot-replication movers that residue is inert. For the stream mover the residue IS the privilege: withdrawing the namespace annotation stops new stream Jobs but leaves a ServiceAccount bound to `create pods/exec` in that namespace, and any tenant who can create a Pod there can assume it. Reaping was rejected because: * the operator holds no `delete` on `serviceaccounts` or `rolebindings`, so it would require granting cluster-wide ServiceAccount deletion to the operator — itself a privilege increase that partly defeats the purpose; * there is no namespace-level reconciler to hang it on, so it would mean a new reconciler, watch and LIST fan-out across SnapshotPolicies, Restores and in-flight Jobs — a feature, not a review fix; * "no consumer remains" is racy against a GitOps prune-then-apply resync and against a SnapshotSchedule about to mint the next Snapshot, so it would flap RBAC delete/create cycles under Flux; * the objects are deliberately labelled `app.kubernetes.io/managed-by: kopiur` so Flux and Argo neither prune them nor report them OutOfSync. So `docs/stream-sources.md` gains a "Revoking the opt-in" section with the exact three-step procedure (drop the annotation, confirm nothing is mid-flight, delete the RoleBinding then the ServiceAccount), a warning that the annotation alone does not revoke, the `kubectl get rolebinding -A | grep stream-mover` audit query and the GitOps note; the consent section links forward to it; `docs/rbac.md` carries the same warning inline; and `docs/permissions.md` no longer advertises an unqualified "revoke path" — it now separates the privileged-mover opt-in (where removal genuinely does revoke, because that privilege lives in the pod spec) from the stream-exec opt-in (where it does not). `ensure_mover_rbac`'s doc comment records the ruling and its reasons at the code.
…ot the string (F3) `newest_observed_index_blobs` parsed `observedAt` only to validate it, threw the `DateTime` away, and then ordered the RAW RFC3339 strings. That is chronological for exactly one spelling of the offset — the `+00:00` the mover emits via `DateTime<Utc>::to_rfc3339()`. It is not chronological in general: `Z` (0x5a) sorts above both `+` (0x2b) and `-` (0x2d), so `…T00:00:00Z` outranks the strictly later `…T00:00:00-01:00`, and any non-UTC offset is ordered by its digits rather than its instant. The mover is not the only writer that can land here: a hand-authored `Maintenance` or a `kubectl patch` writes `Z` (the tests' own fixtures already do), and the two spellings then coexist in one repository's covering CRs. The wrong winner is a STALE pre-compaction count beating the fresh post-maintenance recount — so `IndexBlobHealth` quotes exactly the number #458 exists to stop quoting, and an index that maintenance just compacted still reads as unhealthy. Keep the parsed instant and `max_by_key` on it. The parse already happened (it is how an unplaceable stamp is discarded); this stops discarding its result. Test: `the_newest_recount_is_ranked_chronologically_not_lexicographically` pins both hazards — `Z` vs `-01:00` at the same wall-clock text, and `+00:00` vs `-02:00` where the digits invert the order. Verified to FAIL against the string comparison it replaces.
…le (T1, T2, T3)
Three type-safety holes on the `SnapshotPolicy.spec.sources[]` surface, all
of the same shape: an invariant that a comment CLAIMED the compiler enforced
and that it did not.
T1 — the `Source` exactly-one-of CEL rule had NO variant-completeness guard.
`restore.rs` drives the Restore target rule's expectation off
`RestoreTarget::kind_str()` over a one-value-per-variant array, so adding a
variant fails to COMPILE, and its comment records why: "That is exactly how
`streamExec` first failed on a real cluster." The Source side's only test
asserted `contains("pvcSelector") && contains("nfs")` and never mentioned
`stream`. Nothing else pinned it: there is no insta snapshot of the
SnapshotPolicy CRD, and `gen-all --check` detects only DRIFT, which a
developer clears by regenerating. A fifth form would therefore compile, pass
clippy, pass every test, regenerate the CRD with the four-term rule intact,
and first fail at `kubectl apply` — rejected by the API SERVER, before the
webhook or any reconciler runs. `source_cel_rule_lists_every_variant` now
mirrors the Restore guard. Verified with teeth: dropping
`(has(self.stream) ? 1 : 0)` from the rule — a line this branch TOUCHED, so a
conflict resolution could — fails the test with the fix instruction, where
before it stayed green while every stream policy became unapplyable.
T2 — `source_shape_classifies_every_wire_form`'s doc claimed a fifth wire
form "cannot land without deciding what it means for a restore". False in the
direction that matters: the test's INPUTS were a hardcoded four-literal `Vec`
compared through a `&str`, so a fifth variant would simply be absent from
`cases`. And `expand::source_shape` classified on
`match (&source.pvc, &source.nfs, &source.stream)` with
`(None, None, None) => Invalid`, so a fifth FIELD folded into `Invalid`
silently. The obligation now lives in the code: `expand::source_shape` is a
`match` over `snapshot_policy::source_shape(source)`, which stops compiling
when a variant is added. The test keeps its place as the witness that the two
classifiers agree on WHICH arm each form maps to (a type system cannot catch
`nfs` wired to `Stream`), with a doc that claims only that.
The coupling also reclassifies a MULTI-set source — `pvc` beside
`pvcSelector`, say — from "whichever field the precedence checked first" to
`Invalid`. That closes rather than opens a hazard, but only once `Invalid`
stops being answered for: `restore_source_path` step (3) used to hand
`effective_source` an all-`None` source, whose `kopia_source_path` is `None`
— the PATHLESS identity `user@host:`, which kopia reads as a relative
filesystem path and matches against every member of the repository. That is
the #451 bug reached from the classifier end. A malformed GOVERNING source now
fails closed with a what/why/fix error. Deliberately scoped to the governing
source: a malformed sibling beside a working selector is irrelevant to rule
(4)'s derivation and must not break a restore that has an honest answer
(`restore_path_ignores_nfs_and_malformed_siblings_of_a_selector`), and a
zero-source legacy policy keeps its documented tolerance — it has no source
to be malformed.
T3 — `effective_source_in` populated `stream` unconditionally while a `Pvc`
pin also set `pvc`, and `kopia_source_path` checks `pvc` FIRST. The two
consumers then disagreed: `identity_source_path` (hence the mover's
`--override-source`, hence the RECORDED manifest) said `/pvc/data`, while
`snapshot::build`'s stream arm derived `/stream/<fileName>` for the work spec
— and verification plus every `fromPolicy` restore look at the stream path,
under which no snapshot was ever written, so the policy's quick verify fails
TERMINALLY (the #456 change) on a policy whose snapshots are real. Nothing
rejected the input: `validate::snapshot` does not validate
`Snapshot.spec.source` at all, and the webhook's pin refusals only fire for a
selector policy MISSING a pin and for multi-repo membership.
Fixed by making the state unrepresentable rather than by reordering the
reads: a `Pvc` pin now requires the governing source to be PVC-shaped,
matched exhaustively over the wire form so a fifth form must decide what a
PVC pin means for it before it compiles. Both PVC-shaped forms still accept a
pin unchanged (the selector fan-out member, and the pre-existing explicit pin
onto a plain `pvc:` source), the unpinned path is byte-identical, and the
webhook's only `identity_source_path_in` caller passes `pin: None`, so
admission behavior does not move. A hand-written bad pin now surfaces as a
terminal `Error::Validation` with the fix, which is loud where it used to be
a divergent recorded path. `kopia_source_path`'s doc records that its
`pvc`-before-`stream` order is safe because the combination is now
unrepresentable, not because PVC is the better guess.
Gate: cargo test --workspace 0 FAILED (3428 passed, 89 binaries); clippy
--all-targets --all-features -D warnings clean; fmt clean; gen-all --check no
drift (25 files) — no CRD schema or description changed, only the test module
and non-schema docs.
…-merge conditions array (F5) `verify_ok_body`'s keyed arm deliberately omits `status.conditions` — "the conditions array is replace-on-merge, so concurrent writers must not touch it". `verify_failed_body` ignored that and ALWAYS wrote a bare `conditions` array, regardless of the stamp key. Before #456 that was safe by construction: exactly one verify mover existed per `SnapshotPolicy`, so it owned the array. The fan-out made N of them concurrent — one per matched PVC per repository — and a JSON merge patch REPLACES an array. Two members failing (or one failing beside the controller's own status write) now race, and whichever patch lands last erases everything else in the array, including the controller's `Ready` row. Losing `Ready` on a policy is the opposite of what a verify failure should report. `verify_failed_body` now takes the SAME `stamp_key` discriminator as `verify_ok_body` and returns `Option`: * `None` (classic flat flow, one mover per policy) — reports `Verified=False` exactly as before, byte-identical; * `Some(_)` (any cell of the (repository x member) grid) — writes nothing. A writer that cannot write safely must not write. The failure is still loud without it: the Job is `Failed` and keeps its pod logs to its TTL, `verify_step` reads that terminal state, the cell's stamp is never written so the controller's fold cannot advance `lastVerified` for the repository (which is what `kopiur_snapshot_verified` staleness alerting watches), and the mover logs the classified error before exiting non-zero. The stamp-key expression is now `verify_stamp_key(op)` in one place instead of duplicated at the ok site, and `patch_verify_failure` replaces eleven hand-rolled call sites. Tests: * `verify_failed_body_is_written_only_by_the_run_that_owns_the_conditions_array` — flat reports, every keyed shape (`<repo>`, `#<member>`, `<repo>#<member>`) is silent; * `a_fanned_out_failure_cannot_erase_the_controllers_ready_row` — applies two sibling members' failures as real RFC 7396 merge patches over a stored `Ready=True` and asserts it survives. Against the previous body that array is replaced and `Ready` is gone.
… blocker (S2, S3) The heal sat past the credentials/privileged blocks, and the comment that justified the placement — "the runs this placement skips ... carry their own registered `Fail` row, so `doctor` still reports them as blocked" — was wrong on both halves. I endorsed it in the w/464b merge commit; this corrects it. (a) `doctor`'s `first_gate` walks `status.conditions` in ARRAY ORDER and returns on the FIRST `GateSeverity::Fail` match. `upsert_condition_status` is order-stable and only ever APPENDS a new `type`, so the gate written EARLIEST in an object's life holds index 0 forever and outranks every later row. A "competing registered row" written later never wins. (b) Three of the skipped paths carry NO registered row and leave the object NON-terminal, where `doctor` does not suppress a stale gate: the repository-not-Ready park (its gate is `GateScope::SnapshotPolicy`, so a Snapshot park there is unregistered), the Maintenance-cache preflight (deliberately unregistered), and `pool_gate`'s `Parked` arm (`RepositorySlotAvailable` deliberately unregistered, pinned by a test). Restore has the same shape in the readiness gate's `Held` arm. The failure is the ordinary GitOps bring-up. A Snapshot parks on `InheritSourceMissing` while its StatefulSet is scaled to zero. The operator scales it up, but the credential Secret has not landed: the resolve now SUCCEEDS, so the park is not re-entered, and `resolve_mover_creds_for` fails 56 lines BEFORE the heal. The array is `[SecurityContextResolved=False, CredentialsAvailable=False]` at `Pending`, and `doctor` prints "scale the workload up or pin mover.securityContext .runAsUser" for a workload that is already up, while the real blocker sits one slot later in the same array. Same false diagnosis, same tool, that #464 round 2 was opened to remove — reached through the credential gate instead of the `resolve_staging` clobber. The heal now runs the instant `resolve_mover_security_contexts` returns `Ok`, on both work kinds. I checked the "every writer after the old site seeds live" claim rather than relying on it, and it was INCOMPLETE: four writers per kind sat between the resolve and the old heal site on the reconcile-start copy, not two. The two on the accepted list were the `MoverPermitted`/ `CredentialsAvailable` clears; the other two were the GATE REFUSALS themselves (privileged-mover and missing-credentials) — precisely the writers in the failure above. They were harmless only because each was, until now, first-in-pass. All four per kind now seed live, so hoisting is durable: - privileged-mover refusal: `io::live_conditions_and_status` (new helper — array AND the patch-if-changed comparison must come from ONE observation, or every pass reports a spurious transition and re-fires the Event); - the `MoverPermitted` and `CredentialsAvailable` clears: the new generic `io::clear_work_condition_if_stale`, which keeps the cheap pre-check on the reconcile-start copy (a run that never tripped the gate still pays no live GET and writes nothing) and re-reads live for the BASE, the `clear_source_pvc_gate_if_parked` discipline; - the missing-credentials gate: `io::live_conditions`. Everything else after the heal already seeded live and was re-traced: `assess_backup_security_context`/`set_security_context_compatible`, `report_inherit_outcome`, the restore inherit reporters, `assess_restore_security_context`, staging `Ready`/`Waiting`, `clear_source_pvc_gate_if_parked`, `handle_missing_source_pvc`, `fold_slot_heal`, the `MoverOutcome` arms and `plan::fanout_status`. The two remaining stale-base writers (the staging `Failed` arm, `patch_hook_failure`) stamp a TERMINAL phase where `doctor` suppresses stale gates, are byte- identical to 882faef, and are on the accepted list. S3: `MoverOutcome::Wedged` was the one arm of four still building on the reconcile-start copy. Closed — the asymmetry cost nothing (terminal, so suppressed) but it is exactly the shape that becomes a bug when someone changes the phase. The missing ratchet, which is why this slipped. `every_registered_gate_has_a_writer` pins that a registry row is WRITTEN; nothing pinned that a non-terminal gate's CLEARER is REACHABLE. Added to that family: - `WorkGatePark`, an exhaustive classification of every Snapshot/Restore- scoped registry row as `Terminal` / `NonTerminalNoHealPossible` / `NonTerminalHealMustPrecede { marker }`, kept in lockstep with `STRUCTURAL_GATES` by count and both directions — so a NEW work-kind gate cannot land without someone making the decision that was never made here; - `the_inherit_heal_precedes_every_non_terminal_gate_it_could_shadow`: the heal follows the resolve, is the FIRST conditions writer past it (no `upsert_gate`/`upsert_condition`/`"conditions":` in between), and precedes every `NonTerminalHealMustPrecede` writer. Source-text, because the property IS an ordering of writes inside one pass. Verified by moving the heal back to its old site: it fails naming the writer and the fix. - `a_gate_written_after_the_heal_leaves_only_the_real_blocker_failing`: the pure durability half — park, heal, then write the credential gate on the healed array, and exactly one registered `Fail` row stands, the real one. A fully general version would need control-flow reachability over the reconcilers (which writes can follow which, per path), which no unit test can do; the ordering + lockstep pair is the tractable core of it. D14 (the stream refusal seeding from `backup.status`) is NOT second-in-pass and needs no change: every earlier conditions write on the launch path RETURNS before it, the one write that continues touches only `preflightSince`, and `pool_gate`'s `Admit` arm writes nothing — it hands the slot heal back as a `bool` for `fold_slot_heal` to apply in the pass's last write. Recorded at the site so it is not re-flagged. It also sits BEFORE the resolve, so the heal cannot be ordered ahead of it; the narrow residual (an opt-in annotation revoked between a held pass and the next) is recorded in `WORK_GATE_PARKS` alongside the ServiceAccount and CA-bundle gates, which share it. Gate: cargo test --workspace 0 FAILED (3428 passed, 89 binaries); clippy --all-targets --all-features -D warnings clean; fmt clean; gen-all --check no drift (25 files); check-wiring OK (1535 fields, 22 allowlisted); check-phases OK; cognitive complexity 29/29 with the offender set unchanged.
…ing" is not a verification verdict (F1, F2/S1) Two defects in #456's (repository x member) verify grid, both of the same class: a verification that reports success without verifying, and evidence destroyed while the claim survives. They share this commit because the first renames a field the second's code and tests use. === F1 — the member tag could collide, and a collision reported an unverified === volume as verified `member_tag6` was `short_hash` truncated to 6 hex = 24 BITS. `verify_members` dedupes members by PATH and never checked that the derived TAGS were distinct — but that tag names the verify Job (`-m<tag>`), rides `VERIFY_MEMBER_LABEL`, and keys `status.verificationStamps`, all three at once. Two colliding members therefore share all three: member B's `job_api.get_opt(<job name>)` FINDS member A's Job, takes it for its own work in flight and sleeps to the next slot; the one stamp A writes is read for BOTH cells by `repo_cell_min`; the fold completes and `lastVerified` advances — while B's volume is never verified in any tier, ever. That is precisely the #456 lie, reintroduced by the fan-out. The width had been copied from `repo_tag6` with its justification but not its math: `repo_tag6` disambiguates <=8 repositories, this disambiguates up to the 400-member fan-out cap. Birthday collisions grow with the square of the population — 79 800 pairs at the cap — so 24 bits is about 5e-3, one policy in two hundred. BOTH halves of the fix, because either alone is unsatisfying: * DETECT. `tag_members` refuses a collision instead of coalescing, mirroring this branch's own precedent — `expand_sources` REFUSES a duplicate member path rather than backing up one of two volumes. Probability is not a correctness argument, and in backup software a loud stop beats a quiet false pass. `Error::Invariant` is `Structural`, so the policy holds on the slow 300 s cadence with a Warning Event rather than hot-looping, and the message names both paths, the shared tag, and a remedy the operator can actually apply. * WIDEN, so the refusal is a tripwire and not an operational hazard. 12 hex = 48 bits over the same population is ~3e-10. Detection alone would have traded a 5e-3 silent lie for a 5e-3 hard park whose only remedy is renaming a PVC; that is better, but not good. Why not widen alone: a hash is never injective, and "unlikely" is not a property a verification claim may rest on. Why not a narrower widening: the Job-name budget has room. The worst-case suffix grows 32 -> 38 bytes, so `verify_job_name` still fits 52 (the 63-char label limit less the 11 bytes Kubernetes needs for the pod-name suffix), the policy-name budget is 14, a truncated name keeps 5 chars plus its 8-hex `short_hash` over the FULL name, the suffix is still appended WHOLE so truncation cannot merge two members or two tiers, and `job_tier`'s `rsplit_once("-vfy-")` is still total — `vfy` is not hex, and everything after the separator is hex, digits and `-`. `member_tag6` -> `member_tag` and `VerifyMember/VerifyTarget::member6` -> `member_tag` because a name that states a width must not state the wrong one. New `naming::wide_hash` renders the full 64-bit FNV-1a; `short_hash` is its low word, asserted. Tests: * `a_member_tag_collision_is_refused_never_coalesced` — `tag_members` with a degenerate tagger (a real 48-bit collision is not exhibitable, which is the width's purpose) must return `Err`, name both paths and the remedy, and classify `Structural`. Against the previous code the two members silently became one. * `the_member_tag_is_wide_enough_for_the_fan_out_cap` — pins 12 hex against a future "tidy the name up" edit, and that the whole 400-member cap stays distinct. * `the_widest_verify_job_name_is_still_a_valid_bounded_dns_label` — both tags, deep tier, a 253-char policy name: <=52, DNS-1123, tier still readable off the name (`job_blocks_cell` fails CLOSED if it is not), and sibling members still get distinct names. === F2 + S1 — "configured but matched no member" was read as "no member === dimension", which destroyed the stamps and kept the claim `run_verify_steps` has the guard: a `pvcSelector` that matched nothing spawns nothing. `final_status_body` runs FIRST and did not. `fold_multi_verification` returned `None` for an empty member set, indistinguishable from the classic flat shape, so: * S1 (the destroyed evidence) — the `None` arm nulled `verificationStamps` WHOLESALE; the multi-repo branch got there differently, folding with an empty member set so `member_keys` was empty and the prune emitted a `null` for every `#`-bearing key. Either way, one relabelled app or one reconcile of watch lag and weeks of stamps were gone. Next pass `verify_cell_anchor` found nothing, `tier_after(None)` anchored a year in the past, `due_tier` preferred DEEP for EVERY member, and a 4-PVC policy at `deep.capacity: 500Gi` ran four sequential full scratch restores — hours of backend egress — for volumes verified yesterday. This is exactly what `stamp_key_live`'s own doc says the #368 prune did, re-opened through the empty-member-set door. * F2 (the stale claim) — the same patch never NAMED `lastVerified`, and RFC 7386 treats an omitted key as a no-op, so the timestamp survived the deletion of everything under it. `VerifyFold` now has three arms and `final_status_body` matches them exhaustively: `Flat` (no grid — retire the multi-only surfaces once), `Grid` (fold and prune as before), and `Undecided` — verification IS configured and nothing matched, so this pass names no verification key at all. Which error is safe, and why. `Undecided` can leave a genuinely-removed member's stamp behind, and it leaves `lastVerified` frozen. Both are recoverable and both are visible: * an orphan stamp is INERT. `repo_cell_min` only ever LOOKS UP the keys of CURRENT members, so an extra key is read by nobody and can never make a missing member look verified. It is reaped by `stamp_key_live` on the next pass where the selector matches anything at all, so only a permanently-empty selector keeps one — which is the case where deleting it would be the bug. * a frozen `lastVerified` keeps AGEING, so it trips the very staleness alert an operator has on `kopiur_snapshot_verified` (kept reporting for the same reason). Clearing it — reviewer 1's first instinct — deletes the series, and then "nothing is being verified" is unalertable. A wrong-but-ageing number is detectable; an absent one is silent. The opposite error is not recoverable at all: deleting a live stamp costs real egress, and it is indistinguishable from "never verified". Tests (there were NONE over `final_status_body`, `fold_multi_verification` or `verify_cell_anchor`): * `a_selector_that_matched_nothing_makes_no_verification_status_decision` — through the REAL fold, single-repo AND multi-repo: the body names none of `verificationStamps`, `lastVerified`, `verification`, while the rest of the status write is untouched; * `the_fold_distinguishes_no_member_dimension_from_a_selector_that_matched_nothing`; * `the_grid_arm_still_prunes_the_stamp_of_a_member_that_is_really_gone` — the fix must not make an orphan immortal once matching resumes; * `a_grid_with_an_unverified_member_clears_the_flat_claim_once`; * `the_flat_arm_still_retires_the_multi_only_surfaces_once`; * `a_fan_out_member_anchors_on_its_own_stamp_never_a_siblings_or_the_flat_field` and `a_non_fan_out_cell_anchors_on_its_repo_entry_and_the_classic_shape_on_the_flat_field`. The first two FAIL when the `Undecided` decision is removed, which is the pre-fix conflation exactly. `status.verificationStamps`' CRD description said "6-hex"; regenerated.
Final-review fixes: S2 (the #464 heal is hoisted so `doctor` names the real blocker), T1 (a variant-completeness guard on the SnapshotPolicy source CEL rule), T2 (the two source classifiers are compile-coupled, with a fail-closed guard), T3 (a PVC pin against a non-PVC source is now unrepresentable), S3. Two conflicts with w/fix-a, both on the privileged-mover refusal in snapshot/mod.rs and restore/mod.rs — w/fix-a rewrote its MESSAGE (to name the layer that actually carries the elevation) while w/fix-c changed its conditions BASE (to a live read, because hoisting the heal made this refusal second-in-pass). Both changes are wanted and orthogonal, so the resolution keeps w/fix-a's whole attribution block and swaps only its stale `restore.status`/`backup.status` base for w/fix-c's `io::live_conditions*`. Worth recording why that combination is load-bearing rather than incidental: w/fix-c found that my own safety claim for the hoist was incomplete. I had said only the two accepted MoverPermitted/CredentialsAvailable clears remained stale-base between the securityContext resolve and the old heal site. There were four per kind — the other two being the gate refusals THEMSELVES, which are exactly the writers in S2's failure scenario. They were harmless only because each was first-in-pass until the hoist moved something ahead of them. This conflict is one of those two. Also note w/fix-c declined my suggested T2 patch and was right to. Naively coupling `expand::source_shape` to `snapshot_policy::source_shape` turns a multi-set source (`pvc` beside `pvcSelector`) from `Selector` into `Invalid`, which drops rule (4) and falls through to rule (3) — resolving `sources[0]`'s own `/pvc/<name>`, a real path holding ANOTHER VOLUME'S data. It shipped the coupling plus a fail-closed guard scoped to the governing source only. Gate on the merged tree: cargo test --workspace 0 FAILED; build clean.
…451) `crates/e2e/tests/stream_sources.rs` gated all six of its tests with `#[cfg_attr(not(feature = "e2e"), ignore)]`, which is inverted as a safety gate: turning the `e2e` feature ON removes the `#[ignore]`, so `cargo test --workspace --all-features` EXECUTES them against whatever kubeconfig is ambient. That is not theoretical. A subagent running the all-features variant during this branch's fix round had five of the six run against its ambient cluster. They failed at their first write (`server-side apply PersistentVolumeClaim kopiur-e2e-repo` -> `namespaces "kopiur-e2e" not found`), so nothing was created — but that was luck. On a machine where a `kopiur-e2e` namespace happens to exist, they would have written to a real cluster. Integration and e2e tests must never target anything but a throwaway kind cluster. This file was the ONLY one in crates/e2e/tests/ using `cfg_attr`; the norm across 12+ e2e files is an unconditional `#[ignore = "requires the e2e harness ..."]`, which is also what `crates/e2e/Cargo.toml`'s own comment already claimed was true of every test ("so even with the feature on they only run with `--include-ignored`"). The pattern arrived with the #451 fork import. All six now match the norm. Verified: `cargo test -p kopiur-e2e --all-features --test stream_sources` reports 6 ignored, 0 passed, 0 failed. Before this commit it ran them. Note the `World::connect()` self-skip in those tests is NOT changed — that shape is used by every e2e test in the repo and is a belt-and-braces guard, safe precisely because the primary gate is the unconditional ignore.
…EC-2026-0285) `cargo deny check` was failing the audit job on two findings, both inherited from main rather than introduced here — but a red pipeline is a red pipeline, and the `status` job lists `cargo-audit` in its `needs`. - rustls 0.23.44 -> 0.23.45. RUSTSEC-2026-0285: rustls accepted TLS 1.3 handshake messages sent at the wrong encryption level when they followed a key-changing message in the same record — e.g. a plaintext `EncryptedExtensions` packed into the same record as the `ServerHello`. RFC 8446 s5.1 requires terminating with `unexpected_message` instead. The transcript is still authenticated, so a network-position attacker cannot alter or complete a handshake; the practical effect is that a peer can send handshake messages in plaintext that should have been encrypted without rustls rejecting the connection. Functionally the same bug as Go's GO-2026-4340 / CVE-2025-61730. Reaches us through axum-server (webhook), hyper-rustls (kube-client), reqwest and rustls-platform-verifier, so it touches every binary. - chacha20 0.10.1 -> 0.10.2. The pinned version was yanked; it arrives via rand 0.10.2 in the controller. Both are semver-compatible patch bumps, so the lockfile delta is four lines and no manifest changed. `cargo deny check` now exits 0 with "advisories ok, bans ok, licenses ok, sources ok". The remaining output is warnings only (duplicate crates, wildcard workspace deps, unencountered licenses), all pre-existing and all non-failing. Verified: cargo test --workspace 0 FAILED; clippy --all-targets --all-features -D warnings clean.
…, D13) - D8: SnapshotSpec::mover doc comment's "Two deliberate exclusions" bullets rendered as an unreadable run-on in generated docs/field-reference.md because escape_cell only breaks on blank-line-separated (\n\n) doc-comment paragraphs; insert blank `///` lines around each bullet so they render as <br>-separated items instead of one sentence. - D9: repository epoch-floor validation appended WHOLE_CALL_REFUSED to every field in the tuple loop, including deleteParallelism, which kopia never validates as part of the same call; switch to a 5-tuple with a per-field whole_call_refused flag so only the kopia-validated fields (advanceOnCount/advanceOnSizeMiB/checkpointFrequency) carry that note. - D12: doctor.rs's privileged-movers guidance said `privileged-movers=allow`, which does not match the actual annotation value (`=true`); fix the stale string in the CLI output. - D13: BlobRetention::Disabled's doc comment embedded a full Rust path (`crate::cluster_repository::AllowedNamespaces::All`) as an intra-doc link LABEL rather than as the link TARGET, so xtask's strip_rustdoc_links (which only strips link syntax and preserves the label verbatim) let the bare path leak into docs/field-reference.md for both Repository and ClusterRepository. Rewrite as `[`AllowedNamespaces::All`](crate::cluster_repository::AllowedNamespaces::All)`, matching the short-label/full-path-target idiom already used elsewhere in crates/api/src/*.rs. Regenerated deploy/crds, deploy/helm/kopiur/crds, docs/field-reference.md via `cargo xtask gen-all`, and updated the two affected insta snapshot fixtures (snapshots_test__repository_crd.snap, snapshots_test__snapshot_crd.snap) via INSTA_UPDATE=always.
…lth docs (D5, D6, D7, D10, D11, F4, D12) - D5 (docs/restores.md): RestoreTarget has four variants (pvc/pvcRef/ populator/streamExec), not three; fix "<one of three>" wording. - D6 (docs/stream-sources.md): pick_stream_pod never checks pod readiness, so "all matches unready" is not a distinct named failure the way "zero matches" and "several matches" are; correct the claim and add an explicit warning that an unready Running pod is picked/counted like a Ready one. - F4 (docs/stream-sources.md): document the fifth "no snapshot" outcome — a finalize-timeout where the exec already exited 0 and kopia's input was already closed, so a snapshot may have committed anyway; explain how such an orphaned manifest becomes a discovered Snapshot CR on the next catalog scan (always run on bootstrap/spec-change; periodicRefresh is opt-in). - D7 (docs/repository-health.md): the Mover-permitted and Stream-exec-opt-in gates are reconcile-only, not admission-time; move them out from under the "Admission/reconcile" heading they were incorrectly grouped under. - D10 (docs/stream-sources.md): narrow the "Fields that do not apply" table's `staging.*` row to just storageClassName/accessModes; staging.timeout is accepted and inert (still duration-parsed, but has nothing to bound) rather than rejected, so it does not belong in that table. - D11 (docs/reference/crds/repository.md): the storageStats section claimed every field is "from the last catalog scan"; rewrite with per-field provenance and call out indexBlobCount/indexBlobCountAt as not solely catalog-scan-derived. - D12 (docs/cli/operations.md): stale `privileged-movers=allow` example does not match the actual annotation value (`=true`); matches the doctor.rs fix in the companion commit.
…mbination (#451) Every one of the six stream-source e2e tests failed in CI at the same line — the `annotate_namespace` helper — with: BuildRequest(Validation("PatchParams::force only works with Patch::Apply")) The helper paired `PatchParams::apply("kopiur-e2e").force()` with `Patch::Merge`. kube-client validates that combination client-side and rejects the call before it reaches the API server, so the opt-in annotation was never written and every test panicked in setup. `PatchParams::default()` is both the fix and what this code wants anyway: turning the opt-in OFF relies on RFC-7386's explicit `null` clearing the key, and server-side apply cannot express a removal that way — it drops fields by omitting them from the applied config. The shared `kopiur_e2e::annotate_namespace` and `security_context_compat.rs` have always used exactly this shape, and both pass in CI. Why this was never caught: the `stream-sources` shard is new on this branch, and until the previous commit these tests were gated with `#[cfg_attr(not(feature = "e2e"), ignore)]`, so no local run reached them either. The shard's first real execution is what surfaced it. Checked for siblings before fixing the one line. Scanning every `.patch*` call in the workspace for the same mismatch finds exactly one: this helper. Six other sites pair `PatchParams::apply(FIELD_MANAGER)` with `Patch::Merge` (finalizer.rs x2, prune.rs, staging.rs, snapshot_schedule.rs, mover main.rs), which kube permits — the validation fires only on `force` with a non-Apply patch, and none of those call `.force()`.
`E2E (bootstrap-deadline)` failed with
provision MinIO + buckets: deployment kopiur-e2e/minio did not become
ready within 420s
and the cluster diagnostics name the real cause:
Failed to pull image "minio/minio:latest": pull access denied, repository
does not exist or may require authorization: server message:
insufficient_scope: authorization failed
MinIO has withdrawn their images from Docker Hub. Confirmed independently:
`https://hub.docker.com/v2/repositories/minio/minio/` and `.../minio/mc/` both
return **404**, and an anonymous Hub registry token for `minio/minio` gets 401
on the `latest` manifest. This is not a rate limit and not a flake — the
repositories are gone.
Both images now come from quay.io, the upstream's own remaining public
registry, pinned to immutable RELEASE tags rather than `latest`:
- quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z
- quay.io/minio/mc:RELEASE.2025-08-13T08-35-41Z
Verified rather than assumed: both manifests return 200 anonymously from
quay.io, and `docker pull` of the MinIO tag succeeds locally
(sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e).
Changed in lockstep, as the files' own comments require: `MINIO_IMAGE` /
`MC_IMAGE` in crates/e2e/src/consts.rs and the two `preload` lines in
crates/e2e/mise.toml.
Two things worth recording about the failure mode, because it cost 21 minutes
per shard to say "not ready":
- the `preload` helper is best-effort by design (`|| echo "warn: could not
preload"`), so the "Preload backend images" STEP reported success while the
pull had failed;
- pods then fall back to an in-cluster pull under `IfNotPresent`, which failed
the same way, so the only symptom was a 420s readiness timeout, retried
three times by nextest.
Swept every other preloaded image for the same problem before pushing: the
other 13 are all pullable (atmoz/sftp:debian, bytemark/webdav:2.4, the
nfs-ganesha digest pin, mendhak/http-https-echo:37, and the nine
registry.k8s.io CSI images — those answer 307 to a regional mirror and 200 on
follow, which is normal for registry.k8s.io, not a failure).
This is inherited from main, not introduced by this branch. Fixed here because
it takes out every S3-backed shard and the pipeline cannot go green without it.
The `stream-sources` e2e shard reports, for a `streamExec` restore:
reading `dump.sql` out of the snapshot failed:
failed to spawn kopia binary `kopia`: broken pipe
kopia spawned fine. It was streaming. What broke was the SINK — the exec
stdin of the consumer command we pipe the object into.
`run_raw_streaming_within` was `tokio::join!`ing three unrelated fallible
things — the sink write, kopia's stderr read, and kopia's exit — collapsing
them into one `io::Error` and mapping the lot to `KopiaError::Spawn`. So the
one fact that matters for diagnosis, whose pipe broke, was thrown away at
exactly the moment it was needed, and the error pointed at a missing binary
that was in fact running.
New `KopiaError::OutputSink { args, source }` carries a sink failure with the
argv whose output was being streamed, and a `StreamHalves` enum keeps the
attribution across the join so the two io::Error sources cannot be confused
again. Classified `Unknown` (terminal): the repository was never at fault, and
re-running re-runs the same consumer command.
This does NOT fix the underlying restore failure — it makes the next run say
what actually failed instead of misdirecting. The remaining question is why the
consumer's stdin closes ~360ms after exec; that needs a cluster to answer and
is tracked separately.
Adding the variant compile-failed `crates/mover/src/status.rs`'s exhaustive
match over `KopiaError`, which is the repo's type-safety property working as
intended — the new case had to be decided before it would build. Its exit code
is `None`: kopia may still be mid-stream when we give up on the sink, so
whatever it exits with describes the teardown, not the failure.
Regression test `a_failing_sink_is_not_reported_as_a_kopia_spawn_failure`
drives a sink that fails every write with `BrokenPipe` against a shim that
exits 0, and asserts the error is `OutputSink`, never mentions "failed to
spawn", names the consumer, and classes as terminal.
…anation (#451) Reading kube-client 4.2's exec message loop settles what "broken pipe" means here. `start_message_loop` handles an unrequested stdout gracefully, so `consumer_attach_params`' `stdout(false)` is NOT the cause — but it `break`s on the server's `Status` frame, which arrives when the exec'd command has EXITED. Breaking drops the stdin reader, which closes the duplex, which is what our write sees as `BrokenPipe`. So a broken sink always means the same thing: the consumer exited early. The interesting fact is never the pipe — it is WHY it exited, and the command already said so on its stderr. That stderr was captured into `stderr_tail` and then thrown away: the failure path called `object_read_failure_message(file_name, err)` and dropped the local. A `streamExec` restore therefore reported a bare transport error with the one line that explains it sitting unused a few lines above. New pure `consumer_failure_detail(file_name, err, stderr_tail)` appends the command's last words when it said anything, and when a broken sink is silent says so explicitly and names the exit status as the next place to look — rather than leaving the reader with "broken pipe". A kopia-side timeout keeps naming `spec.target.streamExec.workloadExec.timeout` and does not acquire consumer-exit language it has not earned. This does not itself fix the failing e2e; it makes the next run state the real cause instead of a symptom. Diagnosing the stream-sources shard from CI logs alone has already cost several 25-minute cycles precisely because both layers of this error described the wrong thing. Three tests: stderr survives, a silent sink still points somewhere, and a timeout is not reframed as a consumer exit.
…r pipe (#451) The ship-failure path returned before ever consulting `status_fut`, so the one authoritative statement of what happened — the exec `Status` frame, which is the very thing that causes kube-rs to close our stdin — was discarded at the moment it was needed. It now reads that Status under a short fixed budget (`STATUS_AFTER_FAILURE`, 5s) rather than the consumer's full timeout: by the time we are handling a broken sink the frame has either arrived or is not coming, and inheriting the consumer's budget would add minutes to a failure that is already decided. `consumer_failure_detail` folds the verdict in, matched exhaustively over `ExecVerdict` so a new verdict has to decide what it means here instead of silently reading as "no status". The `Success`-beside-a-broken-sink case is called out specifically, because it is not a contradiction: it means the command exited cleanly BEFORE the transfer finished, i.e. it stopped reading early — a different bug, with a different fix, than a command that crashed. That distinction is what this change bought. Against a live cluster the restore now reports: the command exited SUCCESSFULLY before the transfer finished, so it stopped reading early — it must consume its stdin to completion which, together with a 0-byte `/tmp/restored.sql` in the target pod, is what identified the real defect. See the investigation notes on the PR: the exec stdin channel accepts only writes issued immediately after `exec()` returns and then closes, so the failure is in the transport, not in the consumer's command. That fix is a separate change. Two tests: the verdict reaches the message, and a successful command beside a broken sink names the early exit rather than claiming there was no status.
Kubernetes `exec` SILENTLY TRUNCATES what you write to a command's stdin when the
only process holding stdin open is the command itself. Measured on kind with an
8 MiB payload: 131,072 bytes arrived and `write_all` returned **Ok**. Smaller
payloads fare worse in proportion — 2 MiB delivered 139,264 bytes (6.6%), also
reporting success.
That is the shape of a directly-invoked consumer (`psql …`, which is how users
actually write this) and of `sh -c 'cat > f'`, where the shell EXECS INTO the
command and replaces itself. A restore could therefore report `Completed` having
loaded a fraction of the dump.
`kubectl exec -i` reproduces it identically, so this is the platform's behaviour,
not a kube-rs bug and not something to route around in our client. Verified by
instrumenting kube-client: it sends exactly one stdin frame and NO close, then the
loop exits on a `Status` frame — i.e. the container process exited because it saw
EOF. `stdout` true/false/drained makes no difference; v5 is negotiated.
The fix keeps a parent shell alive holding stdin:
sh -c '"$@"; exit $?' kopiur-stream-consumer <argv...>
Measured against the same cluster, the same 8 MiB now arrives whole. Three
properties, each validated before the code was written:
* FULL DELIVERY — 8,388,608 of 8,388,608 wrapped, vs 131,072 unwrapped.
* EXIT CODES — 0→0, 7→7, 1→1. Load-bearing: success is decided from the exec
`Status`, so a wrapper that swallowed the code would turn a
failed restore into a reported success.
* NO INJECTION — argv rides as POSITIONAL PARAMETERS, never interpolated into
the script text, so the script is a fixed string that cannot
grow with user input. An argument of `two words; rm -rf /`
arrives intact as one `$1`.
The trailing `exit $?` does double duty: it propagates the code, and by being a
SECOND statement it defeats the shell's exec-the-last-command optimisation, which
is exactly what re-creates the truncating shape.
Only the consumer is wrapped. The producer reads the pod's stdout, which is
unaffected — confirmed at 4 MiB through both kubectl and kube-rs.
The e2e change matters as much as the fix. The old fixture was a single sentinel
line, which fits inside the exec's first stdin burst and therefore PASSES WITH THE
BUG PRESENT — that is how this hid. It now produces ~4 MiB and asserts the exact
byte count first, because a truncated restore still starts with the sentinel, so
checking the first line proves nothing. The failure message reports how much
arrived rather than claiming the content differs.
Cost, stated loudly rather than discovered: the target container must have
`/bin/sh`. A distroless consumer cannot be restored into this way, and the exec
error now says so. Failing loudly beats restoring 1.6% of a database and calling
it done. Documented in docs/stream-sources.md.
Verified: all six stream_sources e2e tests pass against a real kind cluster
(byte-identity now moving 4,194,349 bytes end to end); workspace tests 0 FAILED;
clippy --all-targets --all-features clean; fmt clean; gen-all --check no drift;
check-wiring/check-phases OK; complexity 29/29; mkdocs --strict builds.
Merged
gabrielcosi
pushed a commit
to gabrielcosi/home-ops
that referenced
this pull request
Sep 19, 2026
….10.8 ➔ 0.10.9) (#857) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.8` → `0.10.9` | --- ### Release Notes <details> <summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary> ### [`v0.10.9`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#0109-2026-09-19) [Compare Source](home-operations/kopiur@0.10.8...0.10.9) ##### Bug Fixes - **rust:** update crate reqwest (0.13.4 → 0.13.5) ([#​453](home-operations/kopiur#453)) ([fadb44d](home-operations/kopiur@fadb44d)) - streamed command sources, epoch parameters, per-PVC verification, per-run mover overrides ([#​436](home-operations/kopiur#436) [#​451](home-operations/kopiur#451) [#​456](home-operations/kopiur#456) [#​458](home-operations/kopiur#458) [#​461](home-operations/kopiur#461) [#​464](home-operations/kopiur#464)) ([#​467](home-operations/kopiur#467)) ([0e4bf7b](home-operations/kopiur@0e4bf7b)) ##### Miscellaneous Chores - **mise:** update tool aqua:taiki-e/cargo-llvm-cov (0.9.0 → 0.9.1) ([#​455](home-operations/kopiur#455)) ([882faef](home-operations/kopiur@882faef)) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Berlin) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMDMuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjEwMy4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19--> Reviewed-on: https://git.xcd.dev/gabrielcosi/home-ops/pulls/857
doonga
pushed a commit
to greyrock-labs/home-ops
that referenced
this pull request
Sep 19, 2026
….10.8 ➔ 0.10.9) (#245) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.8` → `0.10.9` | --- ### Release Notes <details> <summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary> ### [`v0.10.9`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#0109-2026-09-19) [Compare Source](home-operations/kopiur@0.10.8...0.10.9) ##### Bug Fixes - **rust:** update crate reqwest (0.13.4 → 0.13.5) ([#​453](home-operations/kopiur#453)) ([fadb44d](home-operations/kopiur@fadb44d)) - streamed command sources, epoch parameters, per-PVC verification, per-run mover overrides ([#​436](home-operations/kopiur#436) [#​451](home-operations/kopiur#451) [#​456](home-operations/kopiur#456) [#​458](home-operations/kopiur#458) [#​461](home-operations/kopiur#461) [#​464](home-operations/kopiur#464)) ([#​467](home-operations/kopiur#467)) ([0e4bf7b](home-operations/kopiur@0e4bf7b)) ##### Miscellaneous Chores - **mise:** update tool aqua:taiki-e/cargo-llvm-cov (0.9.0 → 0.9.1) ([#​455](home-operations/kopiur#455)) ([882faef](home-operations/kopiur@882faef)) </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC44NC4wIiwidXBkYXRlZEluVmVyIjoiNDQuODQuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.greyrock.io/todd/home-ops/pulls/245
eleboucher
pushed a commit
to eleboucher/homelab
that referenced
this pull request
Sep 19, 2026
….10.8 ➔ 0.10.9) (#2159) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.8` → `0.10.9` | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information. --- ### Release Notes <details> <summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary> ### [`v0.10.9`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#0109-2026-09-19) [Compare Source](home-operations/kopiur@0.10.8...0.10.9) ##### Bug Fixes - **rust:** update crate reqwest (0.13.4 → 0.13.5) ([#​453](home-operations/kopiur#453)) ([fadb44d](home-operations/kopiur@fadb44d)) - streamed command sources, epoch parameters, per-PVC verification, per-run mover overrides ([#​436](home-operations/kopiur#436) [#​451](home-operations/kopiur#451) [#​456](home-operations/kopiur#456) [#​458](home-operations/kopiur#458) [#​461](home-operations/kopiur#461) [#​464](home-operations/kopiur#464)) ([#​467](home-operations/kopiur#467)) ([0e4bf7b](home-operations/kopiur@0e4bf7b)) ##### Miscellaneous Chores - **mise:** update tool aqua:taiki-e/cargo-llvm-cov (0.9.0 → 0.9.1) ([#​455](home-operations/kopiur#455)) ([882faef](home-operations/kopiur@882faef)) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Paris) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC44My4xIiwidXBkYXRlZEluVmVyIjoiNDQuODMuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9wYXRjaCJdfQ==--> Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2159
GiorgioAresu
pushed a commit
to GiorgioAresu/home-ops
that referenced
this pull request
Sep 21, 2026
….10.8 ➔ 0.10.9) (#2153) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/home-operations/charts/kopiur](https://github.com/home-operations/kopiur) | patch | `0.10.8` → `0.10.9` | --- >⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/6) for more information. --- ### Release Notes <details> <summary>home-operations/kopiur (ghcr.io/home-operations/charts/kopiur)</summary> ### [`v0.10.9`](https://github.com/home-operations/kopiur/blob/HEAD/CHANGELOG.md#0109-2026-09-19) [Compare Source](home-operations/kopiur@0.10.8...0.10.9) ##### Bug Fixes - **rust:** update crate reqwest (0.13.4 → 0.13.5) ([#​453](home-operations/kopiur#453)) ([fadb44d](home-operations/kopiur@fadb44d)) - streamed command sources, epoch parameters, per-PVC verification, per-run mover overrides ([#​436](home-operations/kopiur#436) [#​451](home-operations/kopiur#451) [#​456](home-operations/kopiur#456) [#​458](home-operations/kopiur#458) [#​461](home-operations/kopiur#461) [#​464](home-operations/kopiur#464)) ([#​467](home-operations/kopiur#467)) ([0e4bf7b](home-operations/kopiur@0e4bf7b)) ##### Miscellaneous Chores - **mise:** update tool aqua:taiki-e/cargo-llvm-cov (0.9.0 → 0.9.1) ([#​455](home-operations/kopiur#455)) ([882faef](home-operations/kopiur@882faef)) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Rome) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC44My4xIiwidXBkYXRlZEluVmVyIjoiNDQuMTA0LjAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbInJlbm92YXRlL2NvbnRhaW5lciIsInR5cGUvcGF0Y2giXX0=--> Reviewed-on: https://git.aresu.eu/GiorgioAresu/home-ops/pulls/2153
perfectra1n
added a commit
that referenced
this pull request
Sep 26, 2026
Brings in 0.10.9/0.10.10 (#467 stream sources + streamExec restores, #481 batched deletes, #485 MinIO fork) and the dependency bumps since the branch last synced. Most conflicts were main editing code this branch had moved out of crates/cli into crates/ops; main's edits are re-applied at the new home: - restore: `RestoreArgs -> RestoreRequest` is now `TryFrom`, with main's total `restore_target_from_args` (no `unreachable!`) refusing by name via `CliError::UnresolvedRestoreTarget`, since `streamExec` has no flag. Main's two target tests move to cli.rs beside the conversion. - ops snapshot: `mover: None` on the built spec, plus main's stream-policy planned-cells test. - ops doctor: the privileged-movers annotation value is `true`, not `allow`. - cli error: keep GoneWhileWaiting/AmbiguousTarget in OpsError (moved), add main's UnresolvedRestoreTarget. Semantic conflicts (merged cleanly, did not compile or silently misbehaved): - kopia fake_shim: main's new test built an `ObjectId` via `.into()`; this branch made it a validated newtype, so use `ObjectId::parse`. - ui: `SnapshotSpec.mover` in a test fixture; exhaustive `RestoreTarget` match in a cli test gains `StreamExec`. - ui policies view: a `stream` source fell through to "(unrecognized source)"; it now renders as `stream/<fileName>`, with a test. Unions: e2e `ws` feature + ui-model dep; e2e scenario list gains both sides' buckets; mise tool bumps from main keep this branch's pnpm; expand tests keep both sides' appended cases. Cargo.lock re-points the new crates at thiserror 2.0.21.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #436
Closes #451
Closes #456
Closes #458
Closes #461
Closes #464
Six issues in one branch. They are together because four of them touch the same three files, and splitting them would have meant resolving the same merge conflicts three times.
Two of these were not the bugs they were reported as. #458 was reported as "kopiur drops my epoch parameters" and is actually kopia refusing the whole call. #456 was reported as "multi-PVC verification doesn't check everything" and is actually verification checking nothing while reporting success. A third defect was found during review and was not reported at all: a streamed restore could report
Completedhaving written 1.6% of the data.What changed, plainly
New features
SnapshotPolicycan back up the output of a command instead of a volume, and aRestorecan pipe that data back into a command. Both run in a container you name, throughpods/exec. This is @miracle2k's work — see Attribution below.Snapshot.spec.moverexists. A single run can override mover settings without editing the policy.Bugs fixed
lastVerifiedon such a policy is meaningless.lastContentReclaimedByteswas hardcoded0.write error: unable to open log fileon every start.pods/exec.snapshotReplicationMoverNamerendered an 89-character name the API server would reject.kubectl kopiur doctorcould name the wrong blocker and tell you to fix something already fixed.Behaviour changes on upgrade
pvcSelectorpolicy'slastVerifiedis cleared once. It was never a real proof.fullnameover 36 characters gets renamed mover Roles.docs/upgrade.mdcovers it. Default installs are byte-identical.advanceOnSizeMiBrounds up instead of truncating.streamExecmust have/bin/sh.Everything below is the detail behind those lines.
Attribution
#451 (streamed command sources) is overwhelmingly @miracle2k's work. It comes from
miracle2k:feat/stream-sources. Nine commits are cherry-picked with their original authorship and-xprovenance intact. The commits after those are conflict resolution against #449 plus hardening. Thank you — this was well-built, and adopting it was clearly better than rewriting it.Only the fork's own image workflow was skipped, and its examples
41/42were renumbered to45/46becausemainhad taken those numbers since.The streamExec restore truncation
A
streamExecrestore wrote its data into the consumer command and reportedCompleted. On an 8 MiB payload, 131,072 bytes arrived. The write returnedOk.Kubernetes
execcloses stdin as soon as the only process holding it open is the command reading it. That is the shape of a directly-invoked consumer —psql …, which is how the docs tell you to write it — and also ofsh -c 'cat > f', because the shell execs into the command and replaces itself.kubectl exec -ireproduces it identically, so this is the platform's behaviour, not a client bug.The fix keeps a parent shell alive:
The same 8 MiB now arrives whole, 8,388,608 of 8,388,608. Three details are load-bearing and each was verified:
exit $?propagates the consumer's exit code (0→0, 7→7, 1→1). Success is decided from the execStatus, so losing the code would turn a failed restore green.exit $?is also a second statement, which defeats the shell's exec-the-last-command optimisation — the thing that would silently re-create the broken shape.two words; rm -rf /arrives intact as one$1.The cost is that the target container needs
/bin/sh. A distroless consumer has none, and that now fails loudly with a message naming the requirement.docs/stream-sources.mddocuments it. Backups are unaffected — they read from the container's stdout, and stdout was never truncated.The reason this survived the fork, the cherry-pick, a hardening pass and four reviewers is that the one covering test used a single-line payload. That fits in the first stdin burst, so the test passed with the bug present. It now writes ~4 MiB and asserts the restored byte count first, before comparing content, so a future regression says "restored 131072 of 4194369 bytes" rather than a content mismatch.
Two related fixes came out of the same investigation: a consumer that dies mid-stream now reports its own stderr and exit code instead of a generic kopia spawn failure (new
KopiaError::OutputSink), and the ship-failure path now waits for the execStatusinstead of discarding it.#461 — the kopia server had no log directory
build_server_deploymenthand-wroteKOPIA_CONFIG_PATHandKOPIA_CACHE_DIRECTORYas string literals and left outKOPIA_LOG_DIRentirely. The server loggedwrite error: unable to open log fileon every start.The mover has always used the
kopiur_kopia::envconstants. The server now uses the same three, andSERVER_LOG_DIRpoints at the/cacheemptyDir that was already mounted. That is the whole fix.#458 — epoch parameters, and a number that was always zero
Why nothing applied
No field was ever dropped. kopia's
epoch.Parameters.Validate()refusesadvanceOnCount < 10,advanceOnSize < 1 MiB,minDuration < 10m,refresh*3 > minDuration, andcheckpointFrequency <= 0.The part that matters:
kopia set-parametersmerges your values into the existing set and then validates the whole thing, and kopiur applies every drifted parameter in a single call. So the reporter'sadvanceOnCount: 5refused the entire call — taking every other parameter with it. Nothing was dropped; everything was rejected together.Admission previously only checked
> 0. It now mirrors kopia's floors, and each message quotes kopia's own error text and says plainly that one bad value refuses the whole call including every other parameter.A zero
refreshFrequencyis now rejected too, because kopia silently drops a zero and exits 0 reporting "no changes" — the most confusing possible outcome.The floors were checked against kopia v0.23.1's source, not inferred.
refreshFrequency: 80mandadvanceOnSizeMiB: 1sit exactly on the accepted boundary, and the derivedminDuration >= 3 x refreshbound relies onDefaultParameters()settingEpochRefreshFrequency: 20m, which it does.Drift comparison was also truncating observed bytes to MiB, so a repository sitting at
10 MiB + 1byte looked identical to a desired10and never converged. It now compares in bytes.lastContentReclaimedBytesis a real number nowIt was hardcoded
0. The mover now readskopia maintenance info --jsonbefore and after a successful run and sums the blobs kopia actually deleted: unreferenced packs, superseded epoch indexes, and expired logs.It deliberately excludes kopia's snapshot-GC figure, which only marks contents deleted in the index and frees no storage until a later run removes the blobs. A captured fixture pins the difference: 14,022,073 bytes actually deleted versus 6,000,529 merely marked.
Two things to know about reading it:
0means a measured zero. A quick run on an epoch-enabled repository only advances and compacts epochs, so there is no figure to report. That distinction is the one you act on.Index blobs are also recounted after maintenance now, not only at bootstrap. That matters on exactly the page that tells you a climbing count means maintenance has stalled — previously the count sat frozen between bootstraps.
#456 — multi-PVC verification was passing without verifying anything
This is the most serious issue that was actually reported, and it is worse than reported.
For a
pvcSelectorpolicy, the verification identity resolvedsource_pathto an empty string. So the quick tier ran:kopia parses that trailing colon as a relative filesystem path. It matches zero manifests. And it exits 0.
So every multi-PVC policy has been reporting a green
lastVerifiedthat proved nothing at all. Not a partial verification — a vacuous one that was indistinguishable from success.Verification now fans out one Job per matched PVC, mirroring exactly what the backup side mints, with per-member jitter and per-member stamps folded by MIN. A quick verify that covered no snapshot is now a terminal failure instead of a pass.
Important
On upgrade, a selector policy's flat
lastVerifiedis cleared once. That value came from the pathless false pass and was never a real proof. It repopulates on the next verification run.#464 — per-run mover overrides, and a hold that was missing
Snapshot.spec.movernow exists, merging asmoverDefaults < policy < snapshot.One deliberate exclusion: a per-run override does not resize the policy-owned persistent cache PVC. That PVC is shared, and a one-shot Snapshot must not resize it for every other run. An ephemeral cache does honour the per-run layer.
Separately: a mover inheriting its security context from a workload with no running pod and no pinned UID used to propagate a generic
MissingDependency. It now parks on a dedicatedSecurityContextResolved=False/InheritSourceMissingcondition, registered as a structural gate sokubectl kopiur doctorreports it, and it clears itself when the workload returns.The existing advisory
SecurityContextInheritedcondition was deliberately left unregistered. It carries five advisory reasons, and registering a gate on it would makedoctormisreport all five as "unknown reason, upgrade your plugin".#436 — already fixed
Fixed on
mainby #449 (the values.yaml comment rewrite and regenerated README). No code here; this PR closes the issue.Review found things the implementation missed
Four independent reviewers went over the finished branch. They found 20 issues. The ones worth knowing about:
A mover ServiceAccount name collision that silently merged privileges.
kopiur.streamMoverNameand the Ruststream_mover_nametruncated at different ends of the same derivation. With a chartfullnameof 58-61 characters the controller bound its RoleBinding to the generic mover ClusterRole — no error, escalation-prevention passed, and stream Jobs ran on the same ServiceAccount as every ordinary mover Job. At 62-63 characters the chart's own two helpers collided, emitting thepods/execrole and the generic mover role under one object name; if the stream role won, every mover Job in every namespace would inheritpods/exec.Fixed by capping the stem rather than the finished name: a new
kopiur.moverBaseNametruncates to 36 characters (63 minus the longest suffix), and all three mover identities are that stem plus a whole suffix. That makes distinctness structural rather than probabilistic. It also fixeskopiur.snapshotReplicationMoverName, which had no cap at all and rendered an 89-character name — past the 63-character limit, so the API server would have rejected that ClusterRole outright.Note
Installs with a chart
fullnameover 36 characters get renamed mover Roles.docs/upgrade.mdcovers it, including removing the old per-namespace ServiceAccounts. Default installs render byte-identical names.A 24-bit verification tag that could report an unverified volume as verified. Member tags were a 6-hex truncated hash with no uniqueness check. Two colliding PVCs meant one Job, one stamp, and a fold that read it for both cells —
lastVerifiedadvancing while one volume was never verified in any tier. Now refused loudly on detection and widened to 48 bits, because probability is not a correctness argument but a hard park at 5e-3 is its own hazard.A transient selector miss that destroyed verification history. A
pvcSelectormatching nothing for one reconcile nulled the wholeverificationStampsmap, after which every member anchored a year in the past and re-fired a full deep verify. A one-minute label blip cost N sequential scratch restores atdeep.capacityeach. "Matched nothing" is now not a verification verdict at all.kubectl kopiur doctornaming the wrong blocker.doctorreturns on the firstFailcondition in array order, and conditions are only ever appended — so the earliest-written gate wins permanently. A Snapshot that parked onInheritSourceMissing, then had its workload restored but its Secret still missing, told the operator to scale up a workload that was already running. The inherit heal is hoisted so the real blocker surfaces, and there is a new ratchet that forces this decision for every future gate.e2e tests that could run against a real cluster. The stream tests were gated with
#[cfg_attr(not(feature = "e2e"), ignore)], which is inverted as a safety gate — enabling the feature removes the#[ignore]. They now use the repo's unconditional#[ignore], matching every other e2e file. CI is unaffected: the harness runs--run-ignored all, and all 231 e2e tests across 49 sharded binaries are still selected.Other behaviour changes worth knowing
advanceOnSizeMiBis now rounded up rather than truncated, so a repository at10 MiB + 1byte reports11, not10.rbacNamespaceReadForStreamSources=true, which grantsgetonnamespacesonly. The stream-source opt-in check fails closed on a 403, unlike the privileged-mover check, becausepods/execis a materially larger grant.deleteon ServiceAccounts, so revocation is deliberately a documented manual step —docs/stream-sources.mdhas the exact commands.sources[0], and a hand-writtenSnapshot.spec.sourcePVC pin against a stream or NFS policy. The second previously resolved a real path holding another volume's data.Not fixed here
Two pre-existing defects were found and deliberately left alone, because neither is caused by these six issues and each is a clean standalone change:
fullnameof exactly 62 characters they render under one name. It is an install failure, not a privilege issue.doctorbreaking ties withinFailbylastTransitionTimewould retire the wrong-blocker class entirely rather than fencing one instance of it, but it changes CLI output ordering for every CRD kind.Verification
cargo deny checkexits 0 (advisories ok, bans ok, licenses ok, sources ok). Getting there needed two dependency patches that were inherited frommainrather than introduced here: rustls 0.23.44 -> 0.23.45 for RUSTSEC-2026-0285 (TLS 1.3 handshake messages accepted at the wrong encryption level), and chacha20 0.10.1 -> 0.10.2 because the pinned version was yanked. Both are semver-compatible patch bumps; the lockfile delta is four lines and no manifest changed.CI is green: 39 jobs succeeded, 1 skipped, 0 failed. e2e runs across 49 sharded binaries, including the new
stream-sourcesshard and the verification fan-out scenario. All 231 e2e tests are selected under the harness's own invocation. The real-kopia integration tier passes, including the new epoch-floor and stdin tests.